Review: “The Epistemology of Fact Checking”

A critical review of and reflection on The Epistemology of Fact Checking by Joseph E. Uscinski and Ryden W. Butler

 

epis·te·mol·o·gy:  the study or a theory of the nature and grounds of knowledge especially with reference to its limits and validity

 

In their paper The Epistemology of Fact Checking (hereafter “Epistemology“), published in October 2013, authors Joseph E. Uscinski and Ryden W. Butler identify the  epistemology of journalistic fact checking and offer examples of its failures.

We agree with the bulk of the paper. We’ll review the paper’s main points, express agreement where appropriate, and then grapple with conclusions that hit close to home.

Representative sample?

Epistemology notes that fact checkers’ ratings produce the impression that some politicians lie more than others. The authors produce an eloquent and persuasive argument suggesting that fact checkers end up with a non-representative collection of statements. These fact checkers communicate poorly founded inferences to some unwary readers who notice that some politicians receive more unfavorable ratings than others.

Epistemology also notes that fact checkers often apply one rating to a claim with multiple parts or focus on just one element of a compound claim, creating an inviting avenue for introducing a fact checker’s bias.

We made some of these same observations in our recent critique of Lying in Politics: Evidence from the U.S.

Causal Claims and Predictions

Epistemology takes issue with the way fact checkers handle causal claims and predictions.

The authors illustrate the difficulty of establishing a good case for causation by discussing the methods social scientists use. Epistemology notes that fact checkers do not describe a similarly rigorous method in their work.

Epistemology treats fact checks of predictions similarly: Predictions depend on future outcomes for verification, so present-day statements about future events do not count as facts.

Lying Liars?

Epistemology charges fact checkers with using a deceptively easy distinction between truth and lies. The charge is memorably worded in the section on predictions:

Finally, we may find here a root of the tendency — although one that is not universal among all fact checkers — to classify what is supposedly inaccurate as a “lie.” … A lie is an intentional falsehood; when the falseness of a statement cannot be established, the fact checker settles for checking the intention. But there is almost never any evidence of such an intention except the failure of the politician to agree with predictions that the fact checker finds so self-evidently credible that only a deliberate liar would contradict them.

 

PolitiFact and the Washington Post Fact Checker undoubtedly create the impression that persons making false statements are lying. PolitiFact does it with a “Pants on Fire” rating tied in tradition and rhyme with the word “liar,” along with “report cards” PolitiFact urges on its readers when elections draw near. The Fact Checker does it with a grading system based on the famed fictional character Pinocchio, whose nose grew when he lied. Both fact checkers do plenty of implying with very little accusing. We think Epistemology could have been more clear about the fact checkers’ effort to have it both ways in the name of gimmickry.

Interpreting Political Rhetoric

Epistemology finishes by pointing out, with examples, how fact checkers tend to give their subjects’ statements narrow interpretations. The fact checkers tend to give the impression that accurately interpreting politicians’ statements is easy stuff. The art of interpretation, of course, provides yet another route through which bias may contaminate fact checking.

The Fact Checkers’ Epistemology

Combining the problem areas addressed in Epistemology gives us the epistemology of the fact checkers. The authors use the term “naïve” a number of times in describing the journalistic approach to fact checking. It seems fair to sum up Epistemology as saying applied naivete serves as the foundation for the epistemology of fact checking.

 

Afterword/Introspection

It’s natural to apply these criticisms of fact checking to Zebra Fact Check.  This is, after all, a site that bills itself as a fact checking website.

Many of the criticisms in Epistemology occurred to us during PolitiFact’s early years, and we designed this fact-checking enterprise to avoid PolitiFact’s errors.

We published a FAQ shortly after Zebra went online announcing that selection bias would occur in our selection of stories. It’s not a representative sample, and we don’t intend it that way.  We will never urge readers to draw conclusions about an entity’s overall record of truth-telling based on our sample.  We’ll try to remind ourselves to avoid doing that even implicitly.

We’re aware of the problem with the truth values of causal claims, and we’ll remind readers of the problems with causal claims when we analyze rhetoric involving them.

Though our rating system avoids implying that false claims amount to lies, we do occasionally declare that a claim seems intentionally misleading. We don’t do this for claims that are simply inaccurate, saving the judgment for claims where the speaker was very likely to know better and used rhetoric particularly well-suited for deceit. Those things we count as good evidences of an intent to deceive.

Interpretation

How do we avoid copying from a conservative perspective the interpretation problems at the mainstream fact checkers?

We aggressively use the principle of charitable interpretation. We treat problem statements like a puzzle regardless of the source, trying to envision ways we could count the statement true. Naturally this is easier when one is sympathetic to the speaker.  But we have years of experience applying the principle.  We did a story on a fact check of President Obama in January that shows the results we can achieve.  We went directly against the findings of the mainstream fact checkers that were harsh on the president’s claim.  Conservative bias didn’t pose much of a problem.

Naive?

Sure, we’re naive.

We’ve used our “About” page to touch on the difficulties journalists face in doing fact checking.  Journalists aren’t experts in much. At best we have a reasonably good handle on the limits of our knowledge, an ability to find information that improves that knowledge, and an ability to synthesize the information into a coherent whole our readers can understand.

We think it’s helpful to understand “fact checking” less literally, more along the lines of the way journalist Ezra Klein describes his work:  “explanatory journalism.” We’re not always checking facts, but at the best of times we’re helping people gain a better understanding of things.

And we’ll explain things better if we keep in mind many of the points made in Epistemology.

 

Special thanks to Joseph E. Uscinski for making Epistemology easier to access.

 

Correction Jan 22, 2014: We misspelled Mr. Uscinski’s name throughout this article for an unforgivably long period of time. We even spelled the tag incorrectly. We apologize for the error.