On Dec. 18, 2020 Zebra Fact Check submitted a new complaint about the mainstream fact checker PolitiFact to the International Fact-Checking Network.
PolitiFact and the IFCN are both owned by the nonprofit Poynter Institute.
In a nutshell, the complaint points toward a long-running study of PolitiFact’s subjective application of its “Pants on Fire” rating as evidence PolitiFact does not exhibit nonpartisanship and fairness in its judgments as the IFCN Code of Principles would require.
Strong Evidence of Partisan Bias at PolitiFact
PolitiFact overwhelmingly skewed its subjective “Pants on Fire” rating away from Democrats in 2020, producing strong evidence of a partisan bias.
This is not a complaint that Republicans received more “Pants on Fire” ratings from PolitiFact. One might explain that statistic away with the possibility that Republicans simply lie more.
The bias in PolitiFact’s use of its “Pants on Fire” rating stems from the apparently purely subjective nature of the rating. In its explanation of its rating system, PolitiFact identifies one distinguishing factor between the “False” rating and the “Pants on Fire” rating. Both are false claims and the latter are “ridiculous” claims.
By 2012, after observing PolitiFact for about three years, it became apparent that PolitiFact had never bothered to offer any objective baseline criterion for identifying a claim as “ridiculous.” Not wishing to assume the line of demarcation between the two ratings was as arbitrary as it appeared, I randomly surveyed about 15 percent (1⁄6) of the “Pants on Fire” ratings and looked for evidence in the text showing the use of some objective measure.
That search came up empty.
Later, PolitiFact principals Bill Adair (founding editor) and Angie Drobnic Holan (now PolitiFact’s editor-in-chief) made statements seeming to confirm the notion that the line of demarcation between “False” and “Pants on Fire” was a subjective one.
During an interview with Michael Schulson, Adair said “Lord knows the decision about a Truth-O-Meter rating is entirely subjective.”
In 2014, interviewer Andrea Seabrook asked Holan [6:21] about the difference between a “False” rating and a “Pants on Fire” rating. Holan replied (bold emphasis added) “You know, that’s an interesting question. We have definitions for all of our ratings. The definition for “False” is the statement is not accurate. The definition for “Pants on Fire” is the statement is not accurate and makes a ridiculous claim. So, we have a vote by the editors and the line between “False” and “Pants on Fire” is just, you know, sometimes we decide one way and sometimes decide the other.”
I have not known PolitiFact to offer information about its “Pants on Fire” rating that would counter the idea that the rating is fundamentally subjective. All of the available evidence points to the conclusion that the rating is just as subjective as PolitiFact’s statement of principles makes it appear.
If the “Pants on Fire” rating is subjective, it follows that each “Pants on Fire” rating counts as a subjective opinion judgment on the set of claims receiving that rating.
It would also follow that an analysis of how PolitiFact applies that subjective rating by party could reveal something significant about ideological bias at PolitiFact.
The numbers make for a fascinating study. During its early years, PolitiFact National had little trouble finding false (“False” plus “Pants on Fire”) ratings for Democrats. Over time, PolitiFact National would find fewer and fewer falsehoods from Democrats. That trend could potentially have something to do with Republicans lying more, but statistically would suggest that Democrats lie less today than they did 10 years ago. Is it a crowding effect? Or simple favoritism toward Democrats?
In terms of the bias in PolitiFact’s application of its “Pants on Fire” rating, a statement PolitiFact regards as false (rated “False” or “Pants on Fire”) is over 50 percent more likely to receive a “Pants on Fire” rating if it comes from a Republican.
Reminder: The evidence says that’s an opinion measure. It cannot be explained as an objective outworking of “Republicans lie more” without showing it is not an opinion measure.
In 2020, PolitiFact revamped its website. The tagging system made dividing up PolitiFact’s findings by franchise quite a bit trickier, so the 2020 data represents all of PolitiFact and not just PolitiFact National. In 2020 a statement PolitiFact regards as false was over four times more likely to receive a “Pants on Fire” rating if it came from a Republican instead of a Democrat.
Reminder: The evidence says that’s an opinion measure.
Interestingly, the percentage of “Pants on Fire” ratings for Republicans was well within established norms for PolitiFact. That percentage, after 2007, has stayed near the range of 25 to 35 percent in any given year. Democrats have never exceeded 25 percent after 2007 and the running average falls at about 17 percent (“Totals” tab). The year 2020, as of Dec. 18, 2020, established a new low for Democrats, as less than 7 percent (6.25) of Democrat claims perceived as false received a “Pants on Fire” rating. That’s three “Pants on Fire” ratings out of 45 total perceived as false. For comparison, PolitiFact National found seven “Pants on Fire” statements from Democrats among 40 perceived as false back in 2008.
As for Republicans, in 2020 PolitiFact rated “Pants on Fire” 70 of 248 claims it perceived as false. That percentage falls very slightly above the average for PolitiFact National since it started in 2007.
The consistency in the numbers for Republicans strongly suggests an outcome expected by those issuing the ratings, which they bring to pass by assigning the ratings accordingly. It matters not whether the match occurs intentionally. The bias exists either way.
Still, the truly remarkable feature of the data is PolitiFact’s increasing reluctance to apply its subjective “Pants on Fire” rating to Democrats. That reluctance makes PolitiFact’s typical treatment of Republican claims viewed as false stand out that much more obviously.
The evidence says PolitiFact exercises a partisan bias in its application of the “Pants on Fire” rating. This case ought to push the International Fact-Checking Network to examine carefully its requirements for non-partisanship.
Further, the IFCN should consider whether the use of subjective rating scales is consistent with an objective approach to fact-checking. It is not fair to expect fact-checkers to appear non-partisan while instructing them to assign subjective truth value ratings to political claims.
In summary, PolitiFact appears to fail the test of non-partisanship and fairness by applying a significantly tougher subjective standard to Republicans than to Democrats.
Note: PolitiFact may add a few more “False” or “Pants on Fire” ratings by the end of the year, but those are not likely to significantly affect these findings.
Thanks for your attention to this matter, and feel free to contact me with any questions about the methods applied for this study or its conclusions.
Pre-republication update Feb 8, 2026: The Seabrook interview of Holan disappeared from SoundCloud, but we located it on Spotify, replaced the URL and added the timestamp in brackets. Of course those changes were never sent to the IFCN, but they help readers today understand the evidence made available to the IFCN.