The International Fact-Checking Network’s Epic Fail on Fact-Checking Accountability
It’s a scandal when an accountability organization fails to provide accountability.
After years of finding fault with mainstream media fact checkers, especially PolitiFact, it came as welcome news in 2018 that the International Fact-Checking Network would institute a formal system for registering complaints about its stable of fact-checking organizations.
A properly implemented system of complaint monitoring would offer member organizations an excellent means of identifying and fixing problems. Alternatively, an ineffective complaint monitoring system might serve as a fig leaf exaggerating the reliability of fact-checking organizations.
We ended up with a scandalously tiny fig leaf.
Why the Fig Leaf Matters
The IFCN developed its accountability system partly in response to Facebook.
This code of principles was launched on Sep. 15, 2016. On Jan. 17, 2017, the IFCN introduced an application and vetting process.
This followed the announcement by Facebook that being a signatory to this code is a minimum condition for being accepted as a third-party fact-checker on the social network (more about that in this Q&A) and a consultation among existing signatories.
Facebook wanted a way to deal with “fake news” and misinformation. It struck on the idea of partnering with fact checking organizations but wanted some kind of public assurance that it was picking competent ones.
The IFCN came up with the “verified signatory” of its Code of Principles at least partly in response to Facebook.
Instagram recently adopted a similar method, which isn’t surprising given that Facebook owns Instagram.
IFCN accountability undergirds the authority by which fact checkers can limit the distribution of content on social media.
That’s a prodigious power, even if that effort has barely started.
It’s no place for fig leaf accountability.
Overview of the IFCN Complaint System
The IFCN augmented its accountability system in 2018 with a complaint system. The complaint system offered a means of helping the IFCN keep its members compliant with the IFCN’s code of principles. In late 2018 the IFCN offered the following explanation above its complaint submission form:
One of the commitments of the IFCN code of principles is to [a] robust corrections policies with clear channels of communication. Complaints about specific fact checks, [sic] should therefore be first submitted through the specific channels offered by the individual verified signatories.
The IFCN does monitor potentially significant violations to the code of principles. Those infractions could be, but are not limited to: complete lack of sources, fabrication of content, lack of funding information, open advocacy for a political cause, and not having a public methodology. Please use the form below to alert us to such infractions.
The IFCN committed to forwarding complaints to its signatory organizations, but decided against making complaints available to the public.
Zebra Fact Check was likely among the first to send formal complaints through the IFCN in early 2018.
Why Focus on PolitiFact?
Zebra Fact Check focused its complaints on PolitiFact for the sake of convenience and to test an IFCN vulnerability.
We found choosing PolitiFact convenient thanks to our long history of critiquing its work. Indeed, our knowledge of PolitiFact’s spotty history on correcting its mistakes offered a ready-made reason to question whether PolitiFact lives up to the IFCN Code of Principles it agreed to follow. The IFCN’s signatories pledge to “scrupulously” follow an open and honest corrections policy.
The non-profit Poynter Institute houses the International Fact-Checking Network while owning PolitiFact, producing a natural vulnerability to criticism. A built-in conflict of interest may discourage the IFCN from holding PolitiFact to account. We judged that potential conflict of interest would pressure the IFCN to respond carefully to complaints about PolitiFact.
We resolved to find clear errors in PolitiFact’s work, contact PolitiFact about them and notify the IFCN if PolitiFact neglected to act.
Zebra Fact Check envisioned a constructive purpose for the criticism. PolitiFact might have responded to criticism of its corrections policy practices by improving those practices, for example.
Complaints: Round One
In January 2018 Zebra Fact Check caught PolitiFact incorrectly describing the federal government’s CHIP insurance program. Pointing out the mistake to PolitiFact resulted in no correction, so we alerted the IFCN.
In April 2018, Zebra Fact Check caught PolitiFact in another mistake.
The fact checkers failed to distinguish between cutting a program relative to its present-day baseline and its projected costs. Again, PolitiFact made no correction. Again, we alerted the IFCN.
The 2018 PolitiFact Assessment
PolitiFact came due early in 2018 for its annual IFCN assessment. The IFCN maintains a stable of “independent assessors” and assigns one of them to perform initial and annual reviews.
The University of Wisconsin-Madison’s Michael Wagner performed PolitiFact’s 2018 assessment. Wagner mentioned the two complaints from Zebra Fact Check, but they did not appear to impact his judgment that PolitiFact complies with IFCN principles.
On the other hand, Wagner noted the importance of responding appropriately to complaints:
In addition to a bit more transparency on funding, my assessment offered some ‘food for thought’ for PolitiFact in terms of its ratings system, processes for dealing with complaints and with respect to sharing the transcripts of on-the-record interviews for fact-checks. It might also better serve PolitiFact’s readers, and help PolitiFact’s own overall transparency, to develop a protocol for dealing with complaints about ratings. Some complaints will surely be ideological screeds. However, others will debate the rating on the merits. This second category of complaints deserve some kind of response, whether individually, publicly, or via some other mechanism.
Wagner’s 2018 assessment was no scandal. In fact, it encouraged us to believe that the 2019 assessment might help improve PolitiFact’s adherence to its corrections policy.
The 2019 PolitiFact Assessment
PolitiFact’s 2019 annual assessment was due in April. By April 15, 2019 we lodged at least six formal complaints about PolitiFact.
Note: As the IFCN uses a complaint submission form instead of accepting complaints via email, we occasionally failed to preserve the exact submission date. The IFCN so far does not keep a public record of submitted complaints.
When the 2019 assessment failed to appear in April, we elected to keep right on sending complaints. This might allow us to see whether the assessor took account of everything received before the date of the assessment. We added complaints on May 3, May 10, May 20(?), June 4, June 5, June 13, and June 20.
PolitiFact’s 2019 assessment did not appear online until June 2019, carrying the date June 20, 2019. The assessor should have had at least eight complaints to consider.
But the assessment made no mention at all of any complaints about PolitiFact. Instead, it presented PolitiFact as meeting all the criteria from the IFCN Code of Principles (bold emphasis added):
PolitiFact is an excellent, transparent, industry-leading fact-checking site owned by Poynter. It complies with each of the IFCN criteria in clear, easy-to-find ways. I recommend that PolitiFact’s appplication [sic] be accepted. I also recommend that PolitiFact consider informing readers about the number and nature of the corrections requests they receive. I also recommend that PolitiFact considers either making the distinctions between “half true” and “mostly false” on the Truth-o-Meter more distinct or further specifying how decisions are made that place checks on claims in one category over the other. Perhaps a link to an “anatomy of a fact check” page that explains why claim A was half true and claim B was mostly false, for example.
The 2019 assessment made it appear that neither the IFCN nor Wagner were aware of any evidence PolitiFact failed to scrupulously follow an open and honest policy on corrections.
That’s a scandal.
What Happened to the Complaints?
Zebra Fact Check inquired of the IFCN regarding the missing complaints with a June 21, 2019 email. By July 26, 2019, IFCN Director Baybars Örsek replied to explain a “technical error” led to the failure to forward the complaints.
Responding to our follow up questions, Örsek confirmed that the PolitiFact assessment was the only one affected by the problem:
We will contact the assessor of Politifact and share the complaints and check with him if he thinks the assessment needs to be reconsidered by the board. If that would be the case, we will open new voting and review the application.
Regarding other applications that might possibly have [been] affected by the technical problem (halted notifications in June and July), due to the increased number of applications and the Global Fact [event] in late June, we did only have two renewing organizations assessed (Maldito and Science Feedback) and after you raised up the technical problem we have checked and confirmed that neither of those two has received an official complaint during their year of signatory status.
So the IFCN said it would get the complaints belatedly to Wagner, and he would decide whether to take action.
“No need to change the assessment’s recommendation”
On Sept. 23, 2019, Örsek responded the same day to an outreach email, saying the complaints did not alter Wagner’s assessment:
The assessor of Politifact’s last application has informed the IFCN on August 22 that there was no need to change the assessment’s recommendation to the board, therefore no action has been taken against Politifact.
Naturally, we were curious as to how the assessor could reach that conclusion. We promptly responded to Örsek to suggest that some sort of public disclosure was in order:
Will the public receive any explanation at all regarding the content of the allegations against PolitiFact or a transparent explanation as to why the assessor decided there was no need to change the recommendation in the assessment?
We never received a response from Örsek. But we got the information directly from Wagner.
Champion of Transparency, Michael Wagner
Without Michael Wagner we could not tell this part of the story about accountability at the IFCN. Only Wagner’s dedication to transparency made available the reasoning behind the IFCN’s kid gloves treatment of PolitiFact.
Though the following analysis of Wagner’s reasoning fails to flatter him, he remains one of the heroes of the story.
Why No Need to Alter the Assessment of PolitiFact?
When we asked Wagner what complaints he received from the IFCN, we got both more and less than we hoped.
On the one hand, Wagner’s assurance he received “a number of emails” told us little. After all, we had no sure way of knowing the IFCN received every complaint that was sent.
But Wagner sent us a copy of the message he sent the IFCN about the complaints he received.
“I told Baybars Orsek I would read all of the emails (which I did),” Wagner wrote, “and comment on the ones from 2019 since they were more directly relevant to the renewal application.”
Why were corrections failures from late 2018 not particularly relevant to an annual assessment due in early 2019?
Our most charitable view of Wagner’s reasoning marks it down as a method for saving time. The IFCN pays its outside assessors for their assessments. We doubt Wagner was offered a bonus for reviewing the complaints the IFCN failed to deliver on time. He likely had little incentive to spend time on it.
All the Eggs in One Basket?
Though Wagner said he would review “the ones from 2019,” he wrote about only one complaint, the one from Feb. 26, 2019.
Wagner erred consistently in his evaluation of the complaint.
Existing Costs vs. Added Costs
Wagner explained his reasoning:
First, one message, relating to the Feb. 19 story about Medicare For All, describes a message sent to the Politifact reporter asking, in part, about the origin of the $2 billion [sic] figure in the story. The $2 billion [sic] refers to a description of how Sen. Bernie Sanders proposes to pay for his Medicare for all proposal. The sentence in the story, “Sanders offered some possibilities” includes a hyperlink to a page on Sen. Sanders’ website. The second sentence in the linked piece from Sanders’ website provides the $2 billion [sic] figure. It is reasonable to assume a fact-checker, like the one who contacted the Politifact reporter, should’ve found that figure.
Wagner’s description, while not technically incorrect (other than substituting “billion” for “trillion”), offered a misleading impression.
Wagner hinted that the complainant could not find the cited number. But the complaint information we sent the IFCN clearly showed that we located the $2 trillion figure and that we questioned whether it qualified as a means Sanders offered of paying the cost of his Medicare For All (M4A) plan.
I used Twitter to ask the author, Jon Greenberg, about the origin of the supposed $2 trillion per year that would help pay down added costs of Medicare For All. No response resulted.
The linked tweet shows $2 trillion figure and questions PolitiFact’s description. The surrounding context shows that it could not qualify as a means of paying the added cost of M4A, for Sanders was simply noting the present total national cost (private and government) of medical care. Total (private and government) costs cannot be used as a figure paying for added federal government costs because total costs already include all costs. The result resembles double counting. It’s like trying to use the down payment on a new car to pay the monthly payments. The down payment is gone by the time the monthly payments come due. In like manner, added federal costs are what remains after paying the existing federal costs.
Sanders’ $2 trillion could work as a funding source for M4A cost estimates offered in the form of total costs. But that isn’t what PolitiFact did.
Thorpe’s Estimate of Added Costs
As we see next, Wagner incorrectly believed PolitiFact’s fact check used at least one such estimate of total costs (bold emphasis added):
The emailer also claimed it was a mistake in reasoning to apply the Sanders explanation for paying for his proposal to the numbers from one estimate of the costs of Sanders’ proposal (roughly $32 billion [sic]), as that estimate claimed the $32 billion [sic] was added cost to health care spending by the government. However, the other estimate Politifact cited, roughly $25 billion [sic], did not make such a claim.
The $25 trillion (not “billion”) estimate, from economist Kenneth Thorpe, was an estimate of added government costs, even though Thorpe did a poor job of communicating it. In our complaint to the IFCN, we correctly asserted a “person familiar with the Mercatus and Thorpe studies should know that the respective predictions represent estimates of added costs, that is, expenses added to what the federal government could expect to pay for medical services.”
FactCheck.org and the nonpartisan think tank Committee for a Responsible Federal Budget reached the same conclusion we did about the Thorpe estimate.
Using Falsehood to Correct the Truth
Wagner’s explanation continued:
Indeed, the Politifact article noted that, with respect to the cost of the Sanders proposal, “This is the great unknown.” Then, it notes two examples of competing claims. The article would have been more thorough to report that the $32 billion [sic] estimate was an estimate of added costs while the $25 [trillion] estimate was an estimate of total costs – but in that case the article should also have noted that the first estimate included costs not included in the second estimate (in my view, the 2nd estimate is better as it is more of an apples to apples analysis than the $32 billion estimate is).
Contrary to what Wagner wrote, PolitiFact would have been wrong to report the $25 trillion Thorpe estimate as an estimate of total costs. Including another explicitly false statement would not have improved PolitiFact’s fact check.
Wagner concluded:
The important points for the story to convey on this matter are 1) the uncertainty of the costs of the plan and 2) how Sanders said he would pay for it. The story does those two things.
So, I do not feel the need to change my overall recommendation.
Readers who shared Wagner’s understanding of PolitiFact’s fact check received a wildly inaccurate view of the differences in the Thorpe and Mercatus Center estimates. Both estimates referred to added federal government costs and they differed by a fairly modest $7 trillion. And at no point did Sanders suggest using the current cost to the government of medical care to pay for the government’s added costs under his M4A plan.
Wagner was wrong on both points he emphasized.
Wagner based his evaluation on just one of more than a half-dozen complaints he likely received and dropped the ball on that one evaluation.
Yet the IFCN accepted the reasoning Wagner offered. That’s a scandal.
Why did the IFCN accept Wagner’s reasoning, especially after we pointed out its problems?
And why did two other IFCN treatments of complaints turn out so strangely?
A Pair of Peculiar Complaint Results
AltNews
In June 2019 the IFCN announced one of its verified signatories from India, AltNews, was accused of breaching the IFCN Code of Principles.
Accusers said one of the founders of AltNews, Pratik Sinha, sustains membership in the Jan Sangharsh Manch organization. The critics said that relationship violated the IFCN principle stipulating political neutrality.
The IFCN investigated by asking four questions of AltNews. Based on the answers to those questions and undisclosed deliberations, the IFCN decided that involvement with Jan Sangharsh Manch did not count as political involvement:
The key reason? Jan Sangharsh Manch “is not a registered body or a political organization linked to a political party in India.”
Is an unregistered organization therefore non-political? Jan Sangharsh Manch grew out of a group of labor unions. And some of its members founded India’s “New Socialist Party.”
The justification the IFCN published for its decision could have passed as subjective. The IFCN never released more information about its “in-depth consideration” or “consultations with the assessor.”
The IFCN should have worked harder to lay questions to rest.
Science Feedback/Health Feedback defines “abortion”
In August 2019 a fact check from Facebook fact-checking partner and IFCN-verified signatory Science Feedback, under its “Health Feedback” banner, temporarily resulted in restricted distribution of a video featuring Lila Rose, a “pro-life” activist.
The group Live Action had posted the video to its Facebook page and submitted objections to Facebook after Health Feedback labeled its video “Inaccurate.”
Facebook looked to the IFCN to settle the complaint. The IFCN split the baby in half, finding the fact check accurate but below the IFCN standard for failing to disclose the bias of some of the medical experts it cited.
Facebook lifted the restrictions on the Live Action video and some on the left objected (ex: Rolling Stone).
Zebra Fact Check reviewed the fact check and the IFCN review (conducted by outside assessor Sarphan Uzunoglu) and found it overlooked obvious signs of bias. For example, the fact check insisted on defining “abortion” as a medical procedure to “end a pregnancy” regardless of the survival of the fetus. The rigged definition played a central role in the supposed debunking of the Live Action video.
Health Feedback (Science Feedback):
(A)bortion is medically defined as a procedure to end a pregnancy – this definition does not change depending on the reasons for an abortion, i.e. whether the procedure is motivated by an unwanted pregnancy or medical emergency or some other situation has no effect on its medical definition
That definition effectively included induced delivery as well as cesarean section. When have “anti-abortionists” ever objected to either?
A definition of “abortion” that included the idea of destroying the fetus would have worked to support Rose’s argument in the video. Those definitions exist but Science Feedback elected not to use them.
The IFCN review overlooked multiple problems that should have publicly discredited all the fact check’s key findings.
That’s a scandal.
What Next?
Upon finding these problems with the IFCN process, of course it made sense to share them with the IFCN and work toward solutions.
As the next section shows, we failed to obtain any acknowledgment of any problem from the IFCN.
Michael Wagner, Champion of Transparency II
We have noted the role of outside assessor Michael Wagner as a heroic figure practicing the transparency needed to bring this story to light. Wagner also used his 2018 and 2019 reviews of PolitiFact to make valuable suggestions for improving transparency at PolitiFact.
In his 2018 assessment of PolitiFact Wagner wrote:
(I)t might be worth considering including a page that publishes complaints and questions about the work PolitiFact does. Of course, many of these complaints will be of a nature that does not merit a response, but the transparency of noting the questions the organization is asked and the comments the organization receives would serve to counter some of the more conspiracy-minded complaints made about the important work PolitiFact does.
Zebra Fact Check applauds the type of policy Wagner recommended. Answering criticism defuses it and models the principle of transparency.
One caveat: This assumes that the organization can sensibly answer the criticism.
Wagner repeated his recommendation for increased transparency in his 2019 assessment of PolitiFact:
I also recommend that PolitiFact consider informing readers about the number and nature of the corrections requests they receive.
Regardless of whether PolitiFact has considered Wagner’s policy recommendation, PolitiFact’s website offers no visible sign it has adopted any such policy.
If it’s Good for PolitiFact, is it Good for the IFCN?
We hold that that the IFCN may receive the same gains from transparency that a fact-checking organization like PolitiFact might enjoy. Imagine the benefit to fact-checking if complaints submitted through the IFCN resulted in better practices by fact-checks like PolitiFact.
Alternatively, well-crafted rebuttals of would-be critics may likewise benefit fact checkers and the IFCN. As Wagner put it, such rebuttals could “serve to counter some of the more conspiracy-minded complaints made about the important work” fact checkers do.
A Shocking and Stony Silence
Surprisingly, mainstream fact-checking’s transparency advocates have offered nearly nothing in response to our inquiries.
On Oct. 6, 2019 IFCN Director Baybars Örsek confirmed the explanation we received from Wagner matched the explanation Wagner sent the IFCN.
On Oct. 14, 2019 Örsek affirmed the IFCN sustains a policy of sharing complaints it receives with the relevant fact-checking organizations. If the IFCN receives a formal complaint about PolitiFact, for example, the IFCN sends PolitiFact a copy of the complaint.
Zebra Fact Check has contacted the IFCN and the Poynter Institute to assert the IFCN owes the public either an admission of error with a plan for fixing the problems or else a sound rebuttal of the charges of error.
This Nov. 19, 2019 message to the Poynter Institute serves to illustrate:
The IFCN has two public complaint situations from the past year and botched at least one of them (involving Science Feedback). And the IFCN has stayed publicly silent about another set of valid complaints involving PolitiFact.
Poynter and the IFCN cannot earn a reputation for public trust by making decisions based on false information.
The public deserves an explanation for what occurred regarding PolitiFact. If Wagner is right and I am wrong, then please publicly disgrace me and my complaint. If I am right, on the other hand, I expect the IFCN to make a statement for publication about what it intends to do to rectify the situation.
What would explain the decisions by the IFCN and the Poynter Institute not to respond at all to multiple requests for an explanation?
What explanation would keep their silence from counting as another scandal?
Conclusion: Accountability Failed
Whatever the reason behind it, we find accountability for error lacking in the fact-checking community. We count it as alarming to encounter that lack of accountability from the IFCN, which bills itself in part as an agent for accountability.
Zebra Fact Check calls on the IFCN to address the scandal by living up to the standards of transparency and accountability it says it requires from fact-checking organizations.
It cannot accept an error-filled justification for declining to follow up on complaints about one of its “verified signatories.”
If in India only registered political groups count as political groups, then the IFCN should clearly explain how that works.
It cannot complete a review of a complaint about a fact check by one of its signatories and miss that the fact check used a rigged definition in its finding, among other problems.
Those who would hold others accountable for accuracy and transparency need to model accuracy and transparency in their work.