Jeff D and I started the PolitiFact Bias blog back in 2011 because we both knew that PolitiFact’s methodology and results were highly suspect. We thought PolitiFact’s very design suggested a biased approach to fact-checking. “Beyond the Big Lie,” by PolitiFact’s founding editor Bill Adair, confirms we were right. “Beyond the Big Lie” has as its central pillar the notion that Republicans as a party lie more and more egregiously than Democrats. Adair takes an approach to the question that should repel a neutral and competent fact checker.
Adair clothes his central idea with tales surrounding a handful of figures such as Nina Jankowicz, highlighting the way GOP lies led to harm for her and others. That has the effect of an appeal to emotion, perhaps intended to distract from the fact that Adair can’t back up his central claim that Republicans lie more.
Adair’s Evidence
Of course Adair used PolitiFact and Washington Post Fact Checker ratings to confirm his suspicion that Republicans lie more. What else should we expect?
Adair’s methodology made a nod toward fairness by excluding Donald Trump from its dataset. But Adair neglected to explain why his investigation only used ratings between January of 2016 and June of 2021. We’ll address that shortly.
Adair (Ch. 9, Page 146 of our digital copy):
Our hunch about the partisan divide was right. For PolitiFact, the results confirmed our hypothesis regardless of whether my team combined the claims into a “false-ish” category (those rated Mostly False, False, and Pants on Fire) or we looked at those ratings individually. For Democrats, false-ish ratings accounted for 31.3 percent of all the claims; for Republicans, the share was significantly higher—55.9 percent.
In the book’s preceding paragraph Adair admitted to part of the problem with his evidence, while excusing “Truth-O-Meter” ratings as “the best data available”:
It’s important to note that our data has limitations. Although the ClaimReview database provides a new and creative way to analyze political falsehoods, it is a product of journalists, not social scientists. The articles were created for readers based on the daily decisions of editors and reporters about what is newsworthy and relevant to check. It is not a random sample. Still, it has been used by other academic researchers because it’s the best data available for analyzing the accuracy of politicians’ claims.
“It is not a random sample.”
An objective approach to the question of party truthfulness should start with a random sample. A convenience sample, or whatever we want to call fact checker editorial judgment, must count as suspect. But Adair, fact-checking expert, doesn’t see it that way.
Adair (Page 159):
But weighing all the complaints—I’ve read and considered lots of them since I started PolitiFact in 2007—I have never seen sufficient evidence that we missed enough False or Pants on Fire claims by Democrats to make a difference. Selection bias doesn’t cause the difference. Our ratings simply reflect what’s happening in the real world.
Adair treats his skepticism as a methodology sufficient to defend his interpretation of PolitiFact’s rating data. But a researcher’s doubt that selection bias affected the findings counts for nothing, including as an excuse for poor methods. Competent researchers use randomized samples partly to avoid having to make such embarrassing statements.
If Adair’s attempt to extinguish concerns about selection bias failed, his approach to concern over fact checkers issuing biased ratings failed even more miserably.
Biased Fact Checker Ratings?
Adair followed the playbook described by fellow fact-checking expert Lucas Graves (from Graves’ book “Deciding What’s True”):
One strategy is responding [to criticism] only minimally or in carefully chosen venues, and always asserting their balance, often by showing the criticism they receive from the other side of the spectrum. Fact-checkers make this point constantly.
Adair takes that approach in his book (Chapter 9, Page 158):
I believe it’s probably true that fact-checkers, like other journalists, are more likely to be liberal. But all political journalists are expected to put aside their own viewpoints, and it seemed ours always did. We made it clear they were expected to be nonpartisan in their work—and I found they were.
It’s important to note that Democrats and their own allies in the media saw an opposite reality about our work. They said fact-checkers were so afraid of being called biased that they awarded unwarranted False or Four Pinocchio ratings to show they were balanced.
He’s received criticism from the right and from the left. So is he biased against one, or biased against the other? Adair implicitly asks “Who knows?” Jeff D. correctly identifies that as a middle ground fallacy (I expounded on that here).
Maybe Adair should just stick with answering the criticism instead of using whataboutism to avoid it? Of course, that only works when one has a way of plausibly answering the criticism. In our experience, Adair lacks those answers. Instead of telling us how left-leaning fact-checkers avoid things like confirmation bias, Adair simply tells us they’re expected to exhibit non-partisanship and he thinks they succeed at it. Let’s not ask whether Adair himself is biased, eh?
A Handwave for the Critics
Two academic researchers looked at bias in PolitiFact’s ratings. Adair gave both a brief hand wave.
Eric Ostermeier of the University of Minnesota noted PolitiFact’s comparatively harsh treatment of Republicans in 2011, suggesting that selection bias may explain the disparity.
Robert Lichter of George Mason University looked at PolitiFact ratings during a 2013 news cycle that was unkind to Democrats and found PolitiFact remained tougher on Republicans regardless.
Adair dismissed both studies as lacking in detailed analysis (Chapter 9, Page 157): “But like Lichter, Ostermeier did not offer any significant analysis to back his allegations.”
Readers of Adair’s book may well wonder what allegations Adair was talking about. From his description we doubt Adair understood Lichter’s study. And Ostermeier’s study made the gentle criticism that PolitiFact’s ratings serve as a poor measure of party truthfulness without ruling out that selection bias accounts for the difference.
We doubt Ostermeier would find Adair’s personal assurance adequate.
In Part II, we will support the proposition that PolitiFact is biased in a way that may well account for the ratings disparity.