Accountability? The International Fact-Checking Network Re-verifies PolitiFact

Does the International Fact-Checking Network effectively confirm that its list of “verified” signatories follows its code of principles? The task makes for a tall order, and we see plenty of reason to doubt.

Over the years PolitiFact has impressed us with its ability to ignore well-reasoned criticisms sent to its staff in response to its invitation to find problems with its fact checks. In this article we recount our experience with trying to help put some teeth in the IFCN’s verification process.

What Counts as an “Open and Honest” Corrections Policy?

The IFCN’s code of principles, the code that sets the minimum standard for a third-party fact checker participating in Facebook’s efforts to fight “fake news,” calls for an “open and honest” corrections policy:

A COMMITMENT TO OPEN AND HONEST CORRECTIONS We publish our corrections policy and follow it scrupulously. We correct clearly and transparently in line with our corrections policy, seeking so far as possible to ensure that readers see the corrected version.

By signing the IFCN’s set of principles, PolitiFact commits to “open and honest corrections” by scrupulously following its corrections policy.

PolitiFact publishes its corrections policy and, aside from some early trouble linking to its 2018 update to that policy, makes it easy to find. So far, so good:

How we correct our mistakes

Mistakes happen. PolitiFact corrects errors as quickly as possible and with appropriate transparency.

Major errors of fact – A serious error that results in a new rating or otherwise changes the general outlook of the fact-check receives a mark of correction at the top of the fact-check.

The text of the fact-check is updated with the new information, and an archived copy of the previous fact-check is preserved and linked to. Additionally, the link text for the item is marked as updated. Corrected fact-checks receive a tag of “Corrections and updates.”

Errors of fact – Errors of fact that do not impact the rating or do not change the general outlook of the fact-check receive a mark of correction at the bottom of the fact-check.

The text of the fact-check is updated with the new information. The correction states the correct information that has been added to the report. If necessary for clarity, it repeats the incorrect information. Corrected fact-checks receive a tag of “Corrections and updates.”

Typos, grammatical errors, misspellings – We correct typos, grammatical errors, misspellings, transpositions and other small errors without a mark of correction or tag and as soon as they are brought to our attention.

Updates – From time to time, we add additional information to stories and fact-checks after they’ve published, not as a correction but as a service to readers. Examples include a response from the speaker we received after publication (that did not change the conclusion of the report), or breaking news after publication that is relevant to the check. Updates can be made parenthetically within the text with a date, or at the end of the report. Updated fact-checks receive a tag of “Corrections and updates.”

Explanatory editor’s notes – Sometimes we alert readers to other information that would be helpful, without changing the original report, such as an outpouring of reader response. In those cases, we post an editor’s note, either at the top or the bottom of the report, as appropriate. Editor’s notes are sometimes used on initial publication to explain a special report’s purpose or outlook

 

IFCN Policy on Complaints

As we mentioned earlier, we have noticed PolitiFact frequently ignoring reasonable calls to correct or clarify its fact checks. Over the past year or so we have encouraged IFCN Director Alexios Mantzarlis to improve the IFCN’s approach to accountability for its list of verified signatories (using an example featuring PolitiFact to drive the point). Mantzarlis responded on Feb. 9, 2018 with an encouraging new policy proposal. When the IFCN receives complaints about an organization it will share those complaints with that organization. The IFCN will collect all complaints about an organization in a file. The IFCN will turn that file over to the independent assessor reviewing applications for re-verification.

We viewed that response as a step in the right direction, though we think fact-checking would benefit from the transparency of making all submitted complaints available to the public.

We received permission from Mantzarlis to share the information about the new policy but we held off in the expectation that the IFCN would soon publish it. Mantzarlis later let us know that the IFCN would announce the policy with a major revamping of its website. Delays with that update pushed back a formal announcement of the policy change.

At Zebra Fact Check we had the information on the new policy and an inclination to act on it.

Test Case: PolitiFact

The IFCN received our Jan. 26, 2018 complaint about PolitiFact and implicitly affirmed it would go into the file of the independent assessor when PolitiFact reapplied for verification.

In that example PolitiFact claimed Nancy Pelosi was right that CHIP renewal over 10 years would save $6 billion. PolitiFact, paraphrasing a line from a Congressional Budget Office report, said the savings occurred because “the reauthorization of CHIP would essentially take the place of more expensive programs.” But a closer reading of the report showed that the projected savings came from income and payroll taxes on income working parents would receive in lieu of using employer-provided insurance. PolitiFact’s fact check downplayed that fact and produced instead the false impression the government ran the CHIP program more efficiently than the alternatives.

We decided to add to the file showing PolitiFact’s failures to act on information that ought to warrant either a correction or a clarification.

In March, we found a Trump-O-Meter article where two different entries continued PolitiFact’s history of leaning away from the term “cut” when Democrats cut the growth of spending while leaning toward the term “cut” when Republicans do the same thing. One of PolitiFact’s entries actually called a cut to spending growth a decrease “compared with current levels.” On March 29, 2018 we sent an email to PolitiFact calling for a correction or clarification. When PolitiFact followed its pattern of non-response we forwarded that message to the IFCN with additional comments on April 4, 2018.

An email from the IFCN affirmed it added the complaint to PolitiFact’s file.

PolitiFact’s Verification Status Renewed

When we noticed the IFCN renewed PolitiFact’s verified status on April 23, 2018, we looked for mention of our complaints.

The IFCN assigned independent assessor Michael W. Wagner, an assistant professor with the University of Wisconsin-Madison, to PolitiFact’s application. Wagner mentioned our complaints in section 2 of the assessment. That section addresses nonpartisanship and fairness:

2) Nonpartisanship and fairness

Principle 1 “We fact-check claims using the same standard for every fact check. We do not concentrate our fact-checking on any one side. We follow the same process for every fact check and let the evidence dictate our conclusions. We do not advocate or take policy positions on the issues we fact-check.”

Given that one of our complaints notes that PolitiFact approached the same issue using two different standards, Wagner had a point treating the complaints in section 2. PolitiFact’s use of two different standards for claims about cutting future growth of spending directly contradicts the first part of the IFCN principle. Interestingly, the instructions given the assessor for judging compliance do not mention the use of two different standards. Instead, the assessment instructs judges to rule on compliance based on “The signatory demonstrates that fact checks cover a variety of subjects and speakers and do not unduly concentrate on one side of the topic/context they fact check.”

We suggest judging compliance with the principle of treating all claims by the same standard by looking at whether the organization judges all claims by the same standard.

After a paragraph describing how PolitiFact fulfilled the requirements sought by the IFCN, Wagner mentioned our complaints:

Note: Two complaints to IFCN about PolitiFact – from the same source –  were brought to my attention. In both cases, the complaints rest upon disagreements about the editorial judgment PolitiFact made in rendering Truth-O-Meter ratings. In one case, the issue related to what constitutes a cut to a public policy program and in the other case the issue related to whether it is appropriate to infer credit claiming by a political elite when a precise claim of credit was not made but obviously inferred. In the first case, the complaint stands on firmer ground than the complaint in the second case. Regardless, my own content analysis of about 50 claims concluded that there is not systematic professional or ideological bias in terms of the facts checked or ratings rendered.

Even so, the person who lodged the complaint, a fact-checker himself, also claimed to have received no response from PolitiFact. I’m not sure of the volume of complaints the site receives, but the critiques leveled by the reader merited attention.

We were pleased that Wagner judged the complaints deserved some attention from PolitiFact. But we think he described our complaints inaccurately by calling them “disagreements about the editorial judgment PolitiFact made in rendering Truth-O-Meter ratings.” Ours were not disagreements with PolitiFact over a conclusion drawn from its description of the facts. Ours were disagreements over PolitiFact’s descriptions of the facts. We showed PolitiFact making a clear misstatement of fact on budget cuts, and we separately showed PolitiFact emphasizing budget savings on CHIP reauthorization at the expense of the origin (tax revenue!) of those budget savings—to the point of leaving a very misleading impression with PolitiFact’s readers.

Open and Honest

Through this stage of the experiment, we have two challenges to PolitiFact’s descriptions, both sent to PolitiFact.

The IFCN received both challenges and reportedly sent them to PolitiFact again.

The independent assessor judged the challenges held merit and warranted a response. Presumably editors at PolitiFact have had a chance to read the assessment.

Confronted three times with this evidence, PolitiFact still has not changed either one of the flawed stories. And we still have received no comment or explanation from PolitiFact about its refusal to address the complaints.

How could this possibly qualify as openness and honesty about corrections? We don’t see how any fact checker could think carefully about this paragraph from PolitiFact’s “Trump-O-Meter” story and come away thinking either sentence is true:

The 2018 White House budget proposal released in May left Medicare benefits largely untouched compared with Medicaid, which would see a more than $600 billion decrease over 10 years compared to current spending levels. Still, Medicare spending would decrease by more than $50 billion in the next decade compared with current levels.

Both sentences are flatly false. The White House budget proposal cuts spending on Medicaid and Medicare compared to current-law projections, not “current levels.” Spending keeps going up under the budget proposal.

There’s no excuse for failing to run a correction. And so long as PolitiFact ignores these types of obviously needed corrections it counts as a sham to call its corrections policy open and honest. It’s certainly not open, and claiming otherwise isn’t honest.

Where To From Here?

The IFCN had in its possession evidence of PolitiFact using different standards for the same claim and failing to act on needed corrections.

Despite that, PolitiFact passed the IFCN’s verification process on the standard for nonpartisanship/fairness and the standard for open and honest corrections.

We think that’s a problem.

We wrote to Wagner and asked how he would recommend the IFCN improve its compliance assessment for corrections policies.

Wagner graciously responded to our question:

(M)y assessment of PolitiFact provides the beginnings of an answer in the open and honest corrections section where I wrote, “given the prominence and quality of PolitiFact, it might be worth considering including a page that publishes complaints and questions about the work PolitiFact does.” This statement operates under the assumption that corrections will likely stem from complaints and/or questions about PolitiFact’s work on any given fact-check or series of fact-checks. More specifically to your questions, then, IFCN might consider suggesting or requiring fact-checking organizations publish some or all of the complaints and questions they receive about their stories.

We find merit in Wagner’s suggestion.

Given the low state of trust in media fact checkers we recommend the IFCN make its standard guideline the use of a “Report An Error” feature as popularized by the Report An Error Alliance and implemented on the Zebra Fact Check website.

Signatories should commit to addressing a number of reported error tickets dictated by a formula. Organizations might use a variety of methods to allow focus on the best and most important complaints in cases where the volume of complaints precludes answering all of them.

We look forward to the announcement of new improvements when the IFCN unveils the revamped version of its website.

As things stand, however, the IFCN’s verification process provides little guarantee of fact checker quality.