Brooks Jackson and firefighting

Brooks Jackson, the founder of Annenberg Fact Check, published a review in December of his past nine years at FactCheck.org.  Titled “Firefighters, Fact-Checking and American Journalism,” the review focuses largely on the role of the fact checker.

Jackson deserves credit for steering what is and has been the best fact-checking organization.  But his defense of mainstream fact checking does much to underscore the reasons I’ve offered showing the need for a higher standard in fact checking.

Jackson likens fact checkers to firefighters.  Falsehoods are like fires.  Firefighters put out fires.  Fact checkers expose falsehoods.

Jackson’s analogy serves the critics of fact checking well.

Fighting fires involves a fairly standard approach.  Take away the fuel and the fire goes out.  There are only a few different categories of fires, so the firefighter needs a pretty simple tool kit to effectively fight most fires.

Fighting falsehoods, in contrast, requires the fact checker to recognize the falsehood.  Falsehoods, unfortunately for fact checkers, come in an infinite variety and often wear disguises.  We see the disadvantage of the fact checker immediately when we observe the favored method of checking difficult facts:  Get somebody else to do it.  Journalists and information researchers routinely rely on others to provide the essential information needed to check facts.  Fact checkers are like a firefighter who gets an emergency call and then has to go to another professional to figure out what to do.

The trick, of course, lies in choosing the right professional to offer the solution.  Fact checking, when taken beyond the he-said/she-said model of traditional journalism, falls into non-objective judgments faster than a politician can think of something false to say.  Most journalists probably lack the research skills to judge the best sources or the existence of an expert consensus.  And even the very best researchers quickly run into the problem of finite human knowledge.  We often just don’t know which statements are solid facts and which are falsehoods.

It’s disturbing watching Jackson effectively ignore the gigantic challenges faced by fact checkers, and no less disturbing seeing him give anecdotal evidence in his article that journalists may sometimes struggle with analytic thinking.

The Newhouse quotation

Both the Obama and Romney campaigns, Jackson says, used dishonest tactics without apology.

 

Romney’s pollster, Neil Newhouse, challenged by reporters to defend the campaign’s “most effective” (but false) ad, said, “We’re not going to let our campaign be dictated by fact checkers.”

 

Newhouse put his finger directly on the problem of fact checker bias in the full quotation.

 

“People are always going to get Pinocchios for this stuff,” Newhouse said. “We stand behind those ads and behind the facts in those ads.”

Newhouse suggested the problem was with the fact-checkers, not the facts themselves: “Fact-checkers come to this with their own sets of thoughts and beliefs and you know what? We’re not going let our campaign be dictated by fact-checkers.”

 

Jackson proves Newhouse’s point by taking the statement out of context. Newhouse wasn’t saying he didn’t care whether the ad was accurate or not.  Newhouse was saying the fact checkers made the wrong call and would routinely make such mistakes–so why run the campaign according to their rulings?

Jackson and other fact checkers made light of Newhouse’s comment to advance their own interests.  That’s ironic.

Fact checking works because the readers say it works

Jackson dismisses complaints from politicians about poor quality fact checking.  He knows that fact checking works, in part, because the readers say so.

Of course they’ll say that.  Otherwise they wouldn’t read fact checks.  But it doesn’t mean the readers are in any better position to judge the truth than the journalists, and we’ve already looked at the magnitude of that problem.  Brooks at least hints that he realizes that reader testimony serves as a poor evidence, and he adds another evidence in support.

 

In addition to these testimonials, we now have some solid data as well. People who visited fact-checking websites during the 2012 campaign actually knew significantly more about the issues under discussion than those who didn’t.

(…)

The results? Persons who reported that they had sought out fact-checking information on the Internet answered 55.5 percent of the “political knowledge” questions correctly, more than 10 points higher than those who didn’t visit a fact-checking site.

 

Causation or mere correlation?  Did the Annenberg Public Policy Center study the effects of fact checking or simply the tendency of better-informed voters to visit fact checking sites?  The press release offers few indications that the study achieves a separation between the two.  But it’s good enough for Jackson to call it “solid data.”

Jackson concludes his section defending the efficacy of fact checking thus:

 

So based on this, I can say with confidence that we fact-checkers are making voters harder to fool. And that was our goal all along. We can’t stop the political arsonists, but we can and do limit their damage.

 

Jackson’s confidence is misplaced. His evidence is far weaker than he supposes, and Annenberg Fact Check suffers some of the same problems as other mainstream fact checkers. For example, the three biggest names in the fact check business augmented the Obama campaign’s personal attack on Republican candidate Mitt Romney with a set of preposterous fact checks of his claim that Jeep would open a Jeep production line in China.  The fact checks did more to mislead than the Romney ad.

Jackson tries to fend off criticism right and left

Is there any doubt that fact checkers employ selection bias in choosing stories?

Is there any doubt that the population of American journalists is significantly to the left of the average American?

Jackson brings a steadfast agnosticism to the implications.  When criticized from the right for the measurably harsher treatment of Republicans by fact checkers, Jackson says it “doesn’t necessarily” indicate bias.  That’s true.  It probabilistically indicates bias rather than necessitating its presence.  But Jackson goes on to claim that the notion that Republicans simply make more untrue statements explains the data just as well.

Does it?  We at least have a reliable baseline measurement of the leftward lean of journalists.  Where’s the comparable baseline for the speech of politicians?  What’s the support for assuming that ideological bias plays no significant role in fact checking?

Jackson thinks that the general agreement between mainstream fact checkers supports their accuracy.  That works if a common leftward lean doesn’t influence the outcomes.

Other than noting that it exists, we’ll ignore Jackson’s disgraceful straw-man version of Mark Hemingway’s argument that Glenn Kessler does a better job of fact checking than PolitiFact.

As for the criticism from the left that fact checkers indulge in “false equivalency,” Jackson again does a poor job of justifying his defense.  Jackson says fact checkers “don’t try to keep score.”  Yet some of them do.  As noted at the site PolitiFact Bias, PolitiFact editor Bill Adair has specifically said that PolitiFact tries to pick an approximately equal number of claims from Democrats and Republicans.  Whether or not anyone keeps a running tally, it is human nature to keep track at some level.

Jackson concludes with a candidate for 2012’s unintended irony award:

 

It is just human nature for biased observers to see bias in us when we cite facts that tend to discredit their beliefs, but that’s not evidence of bias on our part. And it’s also human nature for the politicians we criticize to strain for reasons to deflect the criticism, rather than acknowledge responsibility for their own deceptions.

 

Fortunately for all of us, fact checkers rise above the human tendency for biased observers to see what they want to see.  Nor do they rise to the temptation to strain to deflect criticism rather than acknowledging responsibility for their own deceptions.

Just kidding.

Want to fight fires?  Keep a fire extinguisher in your home, know how to use it and know how to contact somebody who knows what to do if your fire extinguisher isn’t up to the job.  In terms of the firefighter analogy, mainstream fact checkers will often fail to adequately fill the role of the fire department.