The International Fact-Checking Network’s Epic Fail on Fact-Checking Accountability
It’s a scandal when an accountability organization fails to provide accountability.
After years of finding fault with mainstream media fact checkers, especially PolitiFact, it came as welcome news in 2018 that the International Fact-Checking Network would institute a formal system for registering complaints about its stable of fact-checking organizations.
A properly implemented system of complaint monitoring would offer member organizations an excellent means of identifying and fixing problems. Alternatively, an ineffective complaint monitoring system might serve as a fig leaf exaggerating the reliability of fact-checking organizations.
We ended up with a scandalously tiny fig leaf.
Why the Fig Leaf Matters
The IFCN developed its accountability system partly in response to Facebook.
This followed the announcement by Facebook that being a signatory to this code is a minimum condition for being accepted as a third-party fact-checker on the social network (more about that in this Q&A) and a consultation among existing signatories.
Facebook wanted a way to deal with “fake news” and misinformation. It struck on the idea of partnering with fact checking organizations but wanted some kind of public assurance that it was picking competent ones.
The IFCN came up with the “verified signatory” of its Code of Principles at least partly in response to Facebook.
Instagram recently adopted a similar method, which isn’t surprising given that Facebook owns Instagram.
IFCN accountability undergirds the authority by which fact checkers can limit the distribution of content on social media.
That’s a prodigious power, even if that effort has barely started.
It’s no place for fig leaf accountability.
Overview of the IFCN Complaint System
The IFCN augmented its accountability system in 2018 with a complaint system. The complaint system offered a means of helping the IFCN keep its members compliant with the IFCN’s code of principles. In late 2018 the IFCN offered the following explanation above its complaint submission form:
One of the commitments of the IFCN code of principles is to [a] robust corrections policies with clear channels of communication. Complaints about specific fact checks, [sic] should therefore be first submitted through the specific channels offered by the individual verified signatories.
The IFCN does monitor potentially significant violations to the code of principles. Those infractions could be, but are not limited to: complete lack of sources, fabrication of content, lack of funding information, open advocacy for a political cause, and not having a public methodology. Please use the form below to alert us to such infractions.
The IFCN committed to forwarding complaints to its signatory organizations, but decided against making complaints available to the public.
Zebra Fact Check was likely among the first to send formal complaints through the IFCN in early 2018.
Why Focus on PolitiFact?
Zebra Fact Check focused its complaints on PolitiFact for the sake of convenience and to test an IFCN vulnerability.
We found choosing PolitiFact convenient thanks to our long history of critiquing its work. Indeed, our knowledge of PolitiFact’s spotty history on correcting its mistakes offered a ready-made reason to question whether PolitiFact lives up to the IFCN Code of Principles it agreed to follow. The IFCN’s signatories pledge to “scrupulously” follow an open and honest corrections policy.
The non-profit Poynter Institute houses the International Fact-Checking Network while owning PolitiFact, producing a natural vulnerability to criticism. A built-in conflict of interest may discourage the IFCN from holding PolitiFact to account. We judged that potential conflict of interest would pressure the IFCN to respond carefully to complaints about PolitiFact.
We resolved to find clear errors in PolitiFact’s work, contact PolitiFact about them and notify the IFCN if PolitiFact neglected to act.
Zebra Fact Check envisioned a constructive purpose for the criticism. PolitiFact might have responded to criticism of its corrections policy practices by improving those practices, for example.
Complaints: Round One
In January 2018 Zebra Fact Check caught PolitiFact incorrectly describing the federal government’s CHIP insurance program. Pointing out the mistake to PolitiFact resulted in no correction, so we alerted the IFCN.
In April 2018, Zebra Fact Check caught PolitiFact in another mistake.
The fact checkers failed to distinguish between cutting a program relative to its present-day baseline and its projected costs. Again, PolitiFact made no correction. Again, we alerted the IFCN.
The 2018 PolitiFact Assessment
PolitiFact came due early in 2018 for its annual IFCN assessment. The IFCN maintains a stable of “independent assessors” and assigns one of them to perform initial and annual reviews.
The University of Wisconsin-Madison’s Michael Wagner performed PolitiFact’s 2018 assessment. Wagner mentioned the two complaints from Zebra Fact Check, but they did not appear to impact his judgment that PolitiFact complies with IFCN principles.
On the other hand, Wagner noted the importance of responding appropriately to complaints:
In addition to a bit more transparency on funding, my assessment offered some ‘food for thought’ for PolitiFact in terms of its ratings system, processes for dealing with complaints and with respect to sharing the transcripts of on-the-record interviews for fact-checks. It might also better serve PolitiFact’s readers, and help PolitiFact’s own overall transparency, to develop a protocol for dealing with complaints about ratings. Some complaints will surely be ideological screeds. However, others will debate the rating on the merits. This second category of complaints deserve some kind of response, whether individually, publicly, or via some other mechanism.
Wagner’s 2018 assessment was no scandal. In fact, it encouraged us to believe that the 2019 assessment might help improve PolitiFact’s adherence to its corrections policy.
The 2019 PolitiFact Assessment
PolitiFact’s 2019 annual assessment was due in April. By April 15, 2019 we lodged at least six formal complaints about PolitiFact.
Note: As the IFCN uses a complaint submission form instead of accepting complaints via email, we occasionally failed to preserve the exact submission date. The IFCN so far does not keep a public record of submitted complaints.
When the 2019 assessment failed to appear in April, we elected to keep right on sending complaints. This might allow us to see whether the assessor took account of everything received before the date of the assessment. We added complaints on May 3, May 10, May 20(?), June 4, June 5, June 13, and June 20.
PolitiFact’s 2019 assessment did not appear online until June 2019, carrying the date June 20, 2019. The assessor should have had at least eight complaints to consider.
But the assessment made no mention at all of any complaints about PolitiFact. Instead, it presented PolitiFact as meeting all the criteria from the IFCN Code of Principles (bold emphasis added):
PolitiFact is an excellent, transparent, industry-leading fact-checking site owned by Poynter. It complies with each of the IFCN criteria in clear, easy-to-find ways. I recommend that PolitiFact’s appplication [sic] be accepted. I also recommend that PolitiFact consider informing readers about the number and nature of the corrections requests they receive. I also recommend that PolitiFact considers either making the distinctions between “half true” and “mostly false” on the Truth-o-Meter more distinct or further specifying how decisions are made that place checks on claims in one category over the other. Perhaps a link to an “anatomy of a fact check” page that explains why claim A was half true and claim B was mostly false, for example.
The 2019 assessment made it appear that neither the IFCN nor Wagner were aware of any evidence PolitiFact failed to scrupulously follow an open and honest policy on corrections.
That’s a scandal.
What Happened to the Complaints?
Zebra Fact Check inquired of the IFCN regarding the missing complaints with a June 21, 2019 email. By July 26, 2019, IFCN Director Baybars Örsek replied to explain a “technical error” led to the failure to forward the complaints.
Responding to our follow up questions, Örsek confirmed that the PolitiFact assessment was the only one affected by the problem:
We will contact the assessor of Politifact and share the complaints and check with him if he thinks the assessment needs to be reconsidered by the board. If that would be the case, we will open new voting and review the application.
Regarding other applications that might possibly have [been] affected by the technical problem (halted notifications in June and July), due to the increased number of applications and the Global Fact [event] in late June, we did only have two renewing organizations assessed (Maldito and Science Feedback) and after you raised up the technical problem we have checked and confirmed that neither of those two has received an official complaint during their year of signatory status.
So the IFCN said it would get the complaints belatedly to Wagner, and he would decide whether to take action.
“No need to change the assessment’s recommendation”
On Sept. 23, 2019, Örsek responded the same day to an outreach email, saying the complaints did not alter Wagner’s assessment:
The assessor of Politifact’s last application has informed the IFCN on August 22 that there was no need to change the assessment’s recommendation to the board, therefore no action has been taken against Politifact.
Naturally, we were curious as to how the assessor could reach that conclusion. We promptly responded to Örsek to suggest that some sort of public disclosure was in order:
Will the public receive any explanation at all regarding the content of the allegations against PolitiFact or a transparent explanation as to why the assessor decided there was no need to change the recommendation in the assessment?
We never received a response from Örsek. But we got the information directly from Wagner.