Live fact-checking: What could go wrong?


The mainstream fact checkers can’t seem to wait to grasp one of the Holy Grails of fact-checking: Live fact-checking.

The problem? Live fact-checking leads directly toward a magnification of one of fact-checking’s standard problems, selection bias.

In standard fact-checking, the fact checkers pick a claim and then take their time in (hopefully!) getting the facts right and then reporting on them. There’s plenty of time to tweak the presentation to make it seem nonpartisan.

Throw that out the window with live fact-checking.

If fact checkers need a new way to unintentionally clue readers in to their biases, this is a fine way to do it. The first few fact checks from NPR’s annotated presidential debate transcript serve well to illustrate.

Live fact-checking Clinton’s opening statement

Near the end of Hillary Clinton’s opening statement, NPR’s fact checkers found something worth annotating. Clinton said that we need to make the economy more fair, and that means instituting a national minimum wage.

NPR’s fact checkers showed no concern over whether instituting a minimum wage would make the economy more fair. Instead, Marilyn Geewax implied that it’s about time we increase the minimum wage again:

The federal minimum wage has not been increased by Congress since 2009.


We probably should not even wonder if Congress was wise to raise the minimum wage while the job market was weak.

Before Clinton had finished her sentence about making the economy more fair, she also called on the United States to “guarantee, finally, equal pay for women’s work.”

Was Clinton calling for equal pay for equal work? Or, more radically, for women to receive equal pay for their work relative to men regardless of the job performed? NPR’s Danielle Kurtzleben split the difference by telling readers about the current raw gender wage gap:

According to the latest Census income report, full-time, year-round working women earned 80 percent of what men earned in 2015. Lots of factors go into that wage gap, including hours and jobs worked, as well as interruptions in women’s careers due to maternity.


It suits the Democratic Party well to conflate the raw gender wage gap with differences in wages caused by gender discrimination. The wage gap caused by discrimination is almost certainly lower than 5 percent, but mentioning the 80 percent figure instead of a more realistic 95 percent figure leads the audience into one of the fallacies Daniel Kahneman pointed out in his bestselling 2011 book “Thinking, Fast and Slow”:

 It is an anchoring effect. It occurs when people consider a particular value for an unknown quantity before estimating that quantity. What happens is one of the most reliable and robust results of experimental psychology: the estimates stay close to the number that people considered—hence the image of an anchor. If you are asked whether Gandhi was more than 114 years old when he died you will end up with a much higher estimate of his age at death than you would if the anchoring question referred to death at 35.


Thanks to Kurtzleben, we now have the gender wage gap for discrimination anchored with the raw wage gap number, helping to ensure that the audience overestimates the wage gap from discrimination. And that’s fact-checking. Normally Democrats provide the anchoring number without help from fact checkers.

Next, Clinton’s mention of supporting debt-free college prompts Anya Kamenetz to point out that debt-free college was Clinton’s primary campaign promise after the primary season. That’s important to know, right? And it’s simply a fact that it was Clinton’s “primary” campaign promise, as though the primacy of a campaign promise isn’t an opinion.

Live fact-checking Trump’s opening statement

On we go to Trump’s opening statement.

Trump said China has been devaluing its currency without an effective response from the United States.

Enter NPR fact checker Anthony Kuhn:

In fact, over the past two years, Beijing has been selling off some of its roughly $4 trillion in foreign exchange reserves to prop up the value of its currency, the renminbi or yuan. This has contributed to a lower U.S. trade deficit with China. Beijing allowed the RMB to appreciate against the dollar for about a decade until 2014, leading the IMF to judge the RMB as fairly valued in May of last year.


Maybe there’s an implicit admission from Kuhn that China has a history of devaluing its currency without an effective response? Perhaps Kuhn is pointing out valuable information, but he leaves it unclear how this information intersects with Trump’s point about China successfully gaming its trade relationship with the United States to its own benefit. Kuhn primarily produces the impression Trump’s statement was wrong, beginning as it does with the corrective “In fact.”

We could not tell from NPR’s transcript whether Trump talked about China’s history of computer hacking. Maybe he did and the transcript simply doesn’t fully capture it. The comment following Kuhn’s came from Aarti Shahani:

Obama has brokered an agreement with Chinese President Xi to reduce hacking, and IP theft — and cybersecurity experts who monitor intrusions from China say that attacks are in fact down.


Does that show Trump was wrong about previous administrations failing to confront China on trade issues? Maybe. But the Washington Post reported that China was already scaling back its cyber attacks before the agreement Shahani mentions. The Post article mentions factors potentially affecting the downturn that Shahani omits. We do not see it as reasonable to take Shahani’s comment as a proof Trump was wrong, yet that is the impression it generates.

Finally, we have Trump’s comments about good jobs moving out of states like Michigan and Ohio. Trump specifically mentions Ford moving its small-car division to Mexico.

Cue the NPR fact checkers. First up, Geewax:

Unemployment in Michigan is 4.5 percent; Ohio rate is 4.7 percent. Both are better than the national average of 4.9 percent.


By talking about overall employment, Geewax shifts the subject from the “good jobs” Trump was mentioning to jobs in general. This approach to fact-checking better fits the definition of Democrat apologetics. Was Trump right about Ford moving its small-car division to Mexico? Who cares, really? Let’s talk about why it’s not a big deal.

Speaking of losing the Ford’s small-car division, next up for NPR is Shahani:

Ford CEO Mark Fields says zero jobs will be lost in Michigan because Ford will build two new vehicles at that plant.


Right. Sending Ford’s small-car manufacturing to Mexico is not a big deal because Ford will employ those workers making other cars. Is NPR sweeping under the rug the fact that sending more auto production to Mexico invariably cuts the share of Ford automobiles made in the United States? It kind of looks like it, doesn’t it?

The part we can’t figure out about all this is how these fact checkers think they can produce material like this and not look like liberals. When we publish a fact check, we try to have in mind the way somebody would attack it if they disagreed with it. How would NPR defend its work from criticism?

What could go wrong with live fact-checking?

The fact checkers may easily let their masks of objectivity and nonpartisanship slip. NPR’s live fact-checking effort looks like a fine example of that.


Correction Oct. 10, 2016: Apologies to Aarti Shahani, for we misspelled his last name on second reference (“Shani”).

Leave a Comment

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.