Do fact checkers follow their principles?

A look at the International Fact-Checking Network’s effort to enforce its code of principles

In 2016 the mainstream media and its fact-checking brethren discovered the threat of “fake news” and started moving to protect people from the threat. Facebook acknowledged the problem by arranging for fact-checkers to flag intentionally false “news” stories. To help quell public concerns that biased fact-checkers might influence the flow of news on Facebook, approved fact checkers would need to follow the principles set forth by the International Fact-Checking Network.

What could possibly go wrong?

Two main problems might hinder Facebook’s fact-checking safeguards. First, the code of principles might do little to ensure quality even if fact-checkers followed them to the letter. We touched on that with an article that served as our signing statement for the IFCN’s statement of principles. In this article we place our focus on the other problem: What if the fact checkers sign the statement but do not follow the principles to which they agreed?

The ones acting to protect the people from fake news recognized the problem: Anybody could claim to abide by the IFCN’s principles. So Facebook and the IFCN agreed that only “verified” signers of the IFCN’s code of principles might qualify to help police fake news on Facebook. The IFCN, headquartered at St. Petersburg’s Poynter Institute*, came up with a method for vetting organizations that have signed on to its code of principles.

According to the IFCN’s descriptions, the IFCN charges applicants a fee of $200. It then appoints an independent assessor to judge the applicant’s compliance based on a standardized form. Assessors receive $350 for conducting the assessment.

Foreseeing the problems

Zebra Fact Check saw going in that adherence to principle might prove challenging to verify in some cases. It is easy enough, for example, to see whether a fact-checking site has made good on its commitment to conspicuously post a link to its corrections policy. If one can easily find a link to the corrections policy then compliance is obvious. But how does one measure following through on a commitment to “scrupulously” follow the corrections policy?

We posted an article calling for the IFCN to make public the applications of would-be signatories to its statement of principles. We also called for public disclosure of the names of the appointed assessors. As we had noted that a governing board would vote on whether to accept the recommendations of assessors, we also called for public disclosure of the votes of board members.

We received public assurances that the IFCN would publish its assessments and identify the assessors. Curiously, that commitment only extends to organizations that obtain verified status.

With 16 organizations now owning verified status, we have enough information to deepen our critique of the IFCN’s process.

Principles? “The code is more what you’d call ‘guidelines’ than actual rules”

Not long after the IFCN started posting links to review assessments on its page of principles, we noticed that organizations could obtain verified status without meeting all of the standards. See Chart A below, showing how assessors used the IFCN’s form to rate compliance.

We used Twitter to question some of these outcomes, receiving no responses from two assessors. After the director of the IFCN, Alexios Mantzarlis, started responding to our questions on Twitter, we sought a conversation via email.

We emphasized to Mantzarlis that the IFCN process leads observers to think it has confirmed that organizations on its “verified” signatory list comply with its checklist. For example, when France’s Libération Désintox appears on the list, readers infer that an assessor confirmed the organization complies with requirement 4b: “The signatory explains its fact-checking methodology publicly and clearly in an accessible place.” But despite counting Libération Désintox as verified, the only information we have from the IFCN about it shows Libération Désintox does not comply with 4b.

In one of his responses, Mantzarlis asked how we can question the journalistic expertise of the assessor who approved Libération Désintox’s application.

We noted to Mantzarlis that substituting the judgment of assessors for objective compliance with the IFCN’s standards makes a mockery of its standards, comparing the situation to the “pirate’s code” in the Disney film “Pirates of the Caribbean.” The pirate’s code, we learned in the movie, “is more what you’d call ‘guidelines’ than actual rules.”  The IFCN set of standards for fact-checkers works the same way. The expert assessor may recommend verifying compliance with the IFCN’s standards despite shortcomings, and the fact-checking experts who vote to accept that recommendation may likewise overlook the fact of non-compliance when declaring to the public that they have “verified” that an organization meets IFCN standards.

The IFCN’s process presents itself as a ground floor of objective standards for approving fact-checkers. In practice, the process allows experts who are part of the journalistic tribe to non-transparently approve organizations that fail to meet the standards.

While the IFCN does offer some comments indicating organizations had to make changes before receiving verified status, those explanations commonly fail to account for all the deficiencies identified by the assessors.

Is the IFCN transparent enough?

The IFCN and its members, perhaps with exceptions, do not appear to understand the purpose and necessity of transparency.

Journalists, including fact checkers, do not enjoy a high level of trust from the public. And aside from tricking people into extending their trust, transparency serves as the only method for  building trust. People require the means to know journalists are giving the straight story.

Seemingly, journalists associated with the IFCN think cutting corners on transparency can move journalism toward the role of serving as a trusted primary custodian of high quality information.

Our message to the IFCN: A trustworthy fact-checker can hardly go to excess in terms of transparency. The same principle applies to the means the IFCN uses to verify compliance with its code of principles. We suggest three high-priority changes to the IFCN’s current process.

No. 1:  Eliminate any pretense that compliance with its code of standards is “verified”

As noted above, the current IFCN system sets forth standards that “verified” organizations apparently need not fulfill. The IFCN may obtain transparency in this situation in either of two ways.

The first option involves explaining that the IFCN code of principles is aspirational. That is, it serves as an ideal that fact checkers ought to work to achieve. Explaining this to the public will involve making clear that the IFCN cannot verify aspirations in any definite sort of way. This option severely devalues the IFCN’s statement of standards as a means of establishing trust.

The second option involves making a serious effort to establish the IFCN code of principles as a true minimum standard. If the IFCN principles call for standard 4a and a fact-checking organization does not meet standard 4a, then do not allow the public to think that compliance with 4a is “verified.” If verifying compliance with certain principles proves very difficult in some cases, explain the difficulties clearly and transparently.

No. 2: Put Board members’ votes on the public record

When an assessor recommends that the IFCN verify a signatory to its code of principles, a board then votes on whether to accept the recommendation. We can think of no good reason why the IFCN should keep such votes confidential, if it aims to build public trust in its process. If a board member votes to accept as “verified” an organization that has failed compliance with one or more of the IFCN standards, does it legitimately build trust to keep that information secret?

No. 3: Make all applications and assessments public

Under its current policy, the IFCN does not publish applications or assessments until its Board votes to accept an applicant as “verified.” In practice, that means organizations that never receive IFCN approval can depend on those assessments staying out of the public sphere.

Why? What kind of public trust does the IFCN build by keeping the details from the public?

When we first asked Mantzarlis about this policy, we thought his reply reflected IFCN concern over liability under the law if organizations contested the finding. But Mantzarlis at the time said he was replying hastily and later suggested we pay more attention to this explanation:

We are committed to the fullest transparency possible in the verification process of the fact-checkers’ code of principles. For that reason, we publish on our site a detailed explanation of the process, the template of the checklist we ask assessors to use to evaluate aspiring signatories, the names of the assessors, the names of the IFCN Interim Board members, the full applications of verified signatories and the full evaluation from the external assessors (at this link).

We are not publishing the assessments of rejected applications, for reasons both practical and philosophical. Our primary goal remains to encourage organizations to follow the principles and highlight the ones that have been found to respect them. Readers can easily determine whether an organization has not been verified by seeing whether it is on our public list.

 

As a matter both practical and philosophical, not publishing rejected applications undermines public trust in the IFCN. If the IFCN does not prioritize building legitimate public trust in fact-checking, then perhaps it should shuffle its priorities.

How would the public know whether an unverified organization had applied for verification and paid its associated fee?

How would the public know on what grounds an unverified organization failed its test for verification?

Based on Mantzarlis’ response, the IFCN asks the public to trust the IFCN to tell them whether fact-checking organizations are trustworthy. But what is the difference between Libération Désintox, found lacking in five out of 11 criteria, compared to any one of the unverified organizations? If the public cannot see how the IFCN distinguishes between the two, then what is the basis for trust in the IFCN’s determinations?

The problem of verification nightmares

We have touched on the inherent difficulty of verifying compliance with some of the IFCN’s principles. We will expand on one example, for experience tells us that fact checkers tend to correct their mistakes reluctantly. The fifth of the IFCN’s principles has signatories pledging an open and honest corrections policy:

A COMMITMENT TO OPEN AND HONEST CORRECTIONS
We publish our corrections policy and follow it scrupulously. We correct clearly and transparently in line with our corrections policy, seeking so far as possible to ensure that readers see the corrected version.

 

How does one verify scrupulous compliance with an corrections policy? The current IFCN system asks the applicant to provide a link to its corrections policy and offer one or two examples of recent corrections. We noted in our correspondence with Mantzarlis that any unscrupulous fact checker could likely achieve that standard of compliance. As such, the IFCN’s list of “verified” signatories tells readers virtually nothing about whether organizations on the list follow their policies scrupulously. It mainly tells readers that an organization posts its policy where people can find it.

But the best corrections policy in the world isn’t much good if the organization does not follow its policy, right?

The IFCN promises the public that it will continue to assess the effectiveness of its verification process. But people have no reason to trust that promise if the verification process right out of the box cannot begin to assure compliance with a given principle. From the start, it was glaringly obvious that verifying compliance with a corrections policy carries all kinds of problems. Rather than saying it would monitor the process for its effectiveness, the IFCN should have admitted from the first that it cannot promise truly effective verification on corrections policy compliance.

We offered Mantzarlis a suggestion to improve verification in this difficult area, using corporate “Mystery Shopper” programs as a model. We have no indication yet whether the IFCN would consider a system that has a chance of partially working.

Page 2 of this article offers an example of a corrections issue the IFCN process would tend to overlook.

 

*The Poynter Institute owns the PolitiFact fact-checking franchise through the Tampa Bay Times newspaper

Clarification/Correction May 13, 2017: Substituted “hinder” for “render” in the third paragraph

Correction May 13, 2017: Added, as originally intended, a more obvious hotlink to the article’s second page

Pages: 1 2