Transparency statement from IFCN lacks transparency

The IFCN’s 2020 transparency statement about its “Code of Principles” enforcement strangely lacks transparency. The IFCN does a particularly bad job of explaining how it handles complaints about its signatory organizations.

The IFCN first allowed the public to lodge complaints about its signatory fact-checking organizations in 2018. Little fanfare accompanied the move, continuing through 2019. But apparently word got out in 2020 and the IFCN received 167 complaints about its signatories. The IFCN detailed the number of complaints in its 2020 “transparency report” on its Code of Principles:

Effectively from 1st of April 2020, ​167 complaints on 11 verified signatories have been recorded by IFCN.

The transparency report did not identify the 11 verified signatories receiving complaints. But it did reveal it found the majority of them obviously irrelevant for not alleging a transgression of the Code:

The complaints that are identified as relevant to verified signatories’ compliance to Code of Principles have been shared with the assessors during the application process for their review. Out of those 167 complaints, 14 of them have been identified as legitimate complaints, meaning they should be shared with the assessors during the renewal assessments.

The transparency statement offers no details about the 14 complaints deemed relevant to signatories’ compliance with the Code of Principles.

Durin(g) 2020, a very significant amount of complaints have been submitted (by) one single source against one specific verified signatory, therefore, [sic] encouraging us to explore how to navigate such overwhelming attempts without allowing the system to be abused while making sure legitimate complaints are recorded.

The transparency statement does not identify the “one single source” complaining about “one specific verified signatory.” But we are at liberty to suggest it very likely refers to this author and Zebra Fact Check. And the signatory was PolitiFact, though we also submitted complaints about Science Feedback and the Washington Post Fact Checker.

Abusing the System?

The statement hints that our complaints were dubious, and the abundance of them might make it harder to record legitimate complaints. The transparency statement offers no details helping the reader to judge whether our complaints were somehow less than legitimate. Nor has the IFCN taken any steps we can see to help us craft more appropriate complaints. We literally have no idea what the IFCN did with our complaints this year. It claims to review them and perhaps forward the relevant ones to the appropriate outside assessor. But the IFCN produced no evidence of a review process, like a written summary of findings. And none of our complaints received mention in the annual reviews for 2020 (Science Feedback, Washington Post Fact Checker).

Perhaps the annual review of PolitiFact, due in June 2020 and now over six months overdue, will contain something telling us what happened to the complaints. And perhaps the IFCN will one day respond to our message asking why the complaints about Science Feedback and the Washington Post Fact Checker received no mention in their verification assessments.

Echoes of 2019

It’s worth remembering that in 2019 the IFCN admitted it failed to forward our PolitiFact complaints to assessor Michael Wagner. When Wagner belatedly received them, he wrongly dismissed them based on his analysis of one complaint. That flawed analysis was based on his misunderstanding of Kenneth Thorpe’s estimate of the costs associated with Bernie Sanders’ Medicare For All plan. We never received any response to our request to the IFCN for an explanation. We know about Wagner’s method of assessment because he told us about it. He was transparent.

So the IFCN intimates our complaints were not legitimate but has yet to publicly and convincingly demonstrate the illegitimacy of any complaint against one of its verified signatories. Zebra Fact Check regards the IFCN’s investigation of two complaints, one against AltNews and one against Science Feedback, as respectively unconvincing and botched.

In summary, the IFCN’s process for dealing with complaints offers almost no transparency at all. The IFCN would tell us how many complaints it received and how many it deemed worthy of passing on to the relevant outside assessors. But we don’t know the subject matter of the complaints, who sent them, the target of the complaint or whether and why legitimate complaints affected re-verification decisions.

The IFCN’s transparency statement helps make clear the great extent to which the complaints process counts as a black box–the opposite of transparent.

More about what we don’t know

We do not know how many complaints the IFCN credits to Zebra Fact Check. Nor do we know how many, if any, of our complaints it deemed not relevant.

Of course the IFCN’s non-disclosure explains our ignorance.

We know that every one of our complaints we identified an IFCN Code transgression and gave supporting evidence. It may well be that most or all of the 14 relevant complaints had their origin with Zebra Fact Check.

In one of its rare moves offering greater transparency about complaints, the IFCN created a message in response to new complaints that includes a URL leading to the submitted complaint form. We lobbied the IFCN to create some system for acknowledging the receipt of specific complaints, so we applaud that policy. But the IFCN intended the system for use by the complainant only. It serves broader public transparency poorly.

Mystery Markings

The system of URLs does give us some information about the handling of our complaints. Some of them the IFCN marked “Approved.” All but one of those marked “Approved” shares the mark “Unresolved,” including the complaints about Science Feedback and the Washington Post Fact Checker. The approved complaint marked as resolved was apparently marked resolved by me in error. The IFCN has yet to make good on its commitment to change it back to “Unresolved.”

We know of one complaint we submitted without saving the verification URL. Not counting that one, we can confirm a total of eight “Approved” complaints.

What does “Approved” mean on the complaint form? We don’t know. The IFCN has not explained it. It may mean the IFCN found the complaint relevant enough to send on to the outside assessor. Or it may mean something else.

Of the eight “Approved” complaints, seven of eight remain listed as “unresolved.” Perhaps that means the signatory did too little to fix the problem. But again, we just don’t know for sure that’s what it means. The IFCN does not explain it.

And the four (minimum) complaints we sent that the IFCN marked as “Not Approved”? We don’t know much about those, either. The IFCN approved most of those on the same day in August. Nothing after August 2020 received approval. Each complaint marked “Not Approved” specifies a breach in the IFCN Code and offers supporting evidence.

The Benefits of Transparency

Perhaps one day it will occur to the IFCN that providing guidance to complainants may help generate better quality complaints. After all, it offers guidance to would-be IFCN signatories to help those organizations submit better applications.

The IFCN, to my knowledge, has never contacted me regarding the content of any of the complaints I have submitted. The IFCN apparently does not send out messages such as “Your complaint was not approved because ___________.”

So we know complainants should identify a breach of the IFCN’s Code of Principles. Other than that, approval is a black box.