Yes, PolitiFact is Biased and Yes That Bias Could Account for the Ratings Disparity
Tellingly, when Adair talked about conservative accusations of bias he only mentioned one line of evidence (Chapter 9, Page 156):
Many conservatives said PolitiFact was biased. To reach that conclusion, critics could simply add up our Truth-O-Meter ratings.
Adair’s talking about the common approach of noting that PolitiFact treats Republicans more harshly than Democrats on its “Truth-O-Meter.” Republicans tend to get more of the harsh ratings like “False” and “Pants on Fire.”
In context, Adair went directly to the Lichter and Ostermeier studies as examples. Lichter’s arguably serves as a poor example of the approach Adair described. But of greater concern was Adair’s failure to mention other approaches.
PolitiFactBias.com has existed over a decade. It has published many dozens of fact check critiques and a number of studies. PolitiFact Bias has never adopted the approach Adair identified, and Adair could hardly be ignorant of this.
The “Pants on Fire” bias study
One long-running study from PolitiFact Bias led to two convincing sets of evidence showing PolitiFact’s liberal bias.
The Bias Measure
The study tracks PolitiFact’s “False” and “Pants on Fire” ratings. But the study focuses on percentages instead of raw numbers. Specifically, the study looks at the percentage of false (“False” plus “Pants on Fire”) statements rated “Pants on Fire” and looks at party differences.
Republicans, over PolitiFact’s entire history, are 34 percent more likely to have a false claim rated “Pants on Fire.” Why does that count as a bias measure? Because PolitiFact has never defined the two ratings in any objective way. PolitiFact says “False” claims are not true, and “Pants on Fire” claims are not true and also “ridiculous.”
“Ridiculous,” of course, does not count as any type of objective measure. A researcher might define it to make it so, but there’s no sign that has ever happened at PolitiFact. When PolitiFact editor Angie Drobnic Holan was put on the spot to describe the difference between the two rating she offered “(T)he line between ‘False’ and ‘Pants on Fire’ is just, you know, sometimes we decide one way and sometimes decide the other.”
And though we’d like to credit Adair with hyperbole on this, he has said “Lord knows the decision about a Truth-O-Meter rating is entirely subjective.”
Our study concludes that on this subjective decision PolitiFact exhibits a bias against Republicans. Were that 34 percent difference present across the board where bias might affect PolitiFact’s decisions, it might well cancel out the disparity Adair uses as his evidence for Republicans lying more.
The States Vary
The second evidence of PolitiFact’s bias came to our attention incidentally while doing the “Pants on Fire” study. For as long as it made sense, we tracked PolitiFact’s franchises individually. PolitiFact has had state franchises in various states for years, including PolitiFact Texas, PolitiFact Florida and PolitiFact Wisconsin.
PolitiFact National was fairly consistent with its ratios, but the state franchises varied widely. Some even showed an anti-Democrat bias in their “Pants on Fire” ratings.
We count that as a hard evidence of bias in PolitiFact’s ratings.
Notably, PolitiFact ironed out most of those differences over time by making it unclear when a state fact check was done by the staffers dedicated to that state. It is unlikely today that PolitiFact franchises provide their own “Star Chamber” of editors to decide “Truth-O-Meter” ratings.
The charts above, Fig. A and Fig. B, show the contributions of PolitiFact’s state franchises to its pool of false claims. The block representing PolitiFact National counts as a clear minority up through 2014. PolitiFact Wisconsin’s heavy contribution waned and started to mostly favor the Democrats after 2015.
Recall that Adair, without explaining why, limited the data set for use in his book to between January 2016 and June 2021. That window tends to exclude the years where some state operations mediated the marked bias of the rest of the organization.
Of course we’d love to see Adair’s explanation for picking the window he chose.
So, PolitiFact’s pattern of mistakes paints it as biased. Its preference for using the subjective “Pants on Fire” rating on (supposed) Republican falsehoods likewise shows its bias. And the variation from one PolitiFact franchise to the other in treatment of the “Pants on Fire” bias shows different patterns for the franchises.
These lines of evidence all support the hypothesis PolitiFact leans left. Was Adair ignorant of all this, or did he simply choose to ignore it? Neither alternative reflects well on a fact checker.