Review: ‘Heroes or hacks: The partisan divide over fact-checking’

Does a partisan divide exist in attitudes toward political fact checkers?

Bill Adair and Rebecca Iannucci of Duke University and the Duke Reporter’s Lab took up the question in a June 17, 2017 report, Heroes or hacks: The partisan divide over fact-checking, hereafter HOH.

We will review HOH in two sections, first critiquing the report itself and second exploring the context of the report. We find HOH complements survey data from 2016 and helps show the need for an explanatory hypothesis.

A partisan divide?

The researchers said they were drawn toward their research question by anecdotal reports:

(F)act-checkers have seen anecdotal evidence that conservatives and liberals have different
views.

…

But is there truly a partisan divide? Or do partisans just practice a form of conditional love –
supporting fact-checking when it helps their candidates and attacking when it doesn’t?

 

The researchers chose five conservative websites and five liberal websites to help test their hypothesis. This phase of the research creates questions about the reliability of the data.

We selected 10 sites – five liberal (Huffington Post, Daily Kos, Occupy Democrats, Talking
Points Memo and Media Matters) and five conservative (Breitbart, The Daily Caller, Newsmax,
National Review Online and Media Research Center). They were chosen based on their overall
traffic from ComScore and how much original content they generate.

 

Are the selected websites properly representative of conservative and liberal points of view? Who knows? The researchers offer no real assurance. To their credit, however, they dedicate a few words to the problem in the appendix. We fail to see the advantage HOH’s approach holds over polling people about their attitudes toward fact-checking. In fact-checking terms, HOH prefers the secondary source to the primary source, using websites as proxies for public opinion.

With the websites selected, the researchers reported finding 722 statements about fact-checking. Students coded the statements according to a set of instructions. One of the researchers resolved any discrepancies in students’ coding.

The findings? The group of five conservative websites was more critical of fact-checking than the five liberal websites.

HOH interpreted the results as confirmation of a hypothesis (bold emphasis added):

The data confirmed our hypothesis and revealed some sharp differences:

● Liberal websites were far more likely to cite fact-checks to make their points than conservative sites were.
● Conservative sites were much more likely to criticize fact-checks and to allege partisan bias.
● When student researchers categorized the tone of mentions, we found liberal sites made most of the positive references while the negative references came primarily from the right.
● Conservative sites made the most critical comments about fact-checking, occasionally using quotation marks (“fact-checking”) to imply it wasn’t legitimate. One even likened a fact-checker to a Bangkok prostitute.

 

Claims that research data “confirm” a hypothesis pass as hyperbole. HOH has as its working hypothesis the idea that conservatives and liberals have different attitudes toward fact-checking. A survey of conservative and liberal publications showing the two groups expressing differing attitudes toward fact-checking does not “confirm” that hypothesis in any hard sense. Rather, the data support the hypothesis in a soft sense. That is,  they do not falsify the hypothesis and fulfill a reasonable prediction based on the hypothesis–that the selected conservative and liberal publications would reflect differing attitudes toward fact-checking.

That’s how science works.

A problem of ambiguity

HOH presents the reader with an undefined and ambiguous idea of “fact-checking.” The survey results were likely to address attitudes toward the mainstream fact checkers, not toward “fact-checking” in the abstract. The term “fact-checking” encourages readers to interpret the results in terms of fact-checking in the abstract. This ambiguity in HOH results in biased messaging. A person who views conservatives as anti-science might conclude that HOH helps show conservatives do not care much for facts. But the ambiguity in HOH makes the drawing of such conclusions premature.

On one hand, HOH hints that fact checkers properly represent fact-checking in the abstract:

The results are significant because the fact-checkers have always emphasized their independence. Washington Post fact-checker Glenn Kessler says his approach is to be “dispassionate and nonpartisan, drawing attention to inaccurate statements on both left and right.” PolitiFact describes itself as “an independent, nonpartisan news organization (that is) not beholden to any government, political party or corporate interest.” FactCheck.org calls itself “a nonpartisan, nonprofit ‘consumer advocate’ for voters that aims to reduce the level of deception and confusion in U.S. politics.”

 

On the other hand, HOH barely entertains the notion that the conservative critique is broadly on target (bold emphasis added):

Our findings are limited to how fact-checking is portrayed by liberal and conservative writers. We are not assessing whether there is in fact any partisan slant by the fact-checkers. But we believe the dramatic difference in how fact-checkers are portrayed shows they need to strengthen their outreach to conservative journalists and, particularly, to conservative audiences. The fact-checkers need to understand the reasons for the partisan divide and find ways to broaden the acceptance of their work.

 

We suggest that finding ways to broaden acceptance of fact-checking puts the cart before the horse. Finding and eliminating any existing bias serves as the top priority, then finding ways to broaden acceptance. Finding ways to gain greater acceptance for biased fact-checking would befit the Ministry of Truth from George Orwell’s classic novel “1984.”

 

The Context of the “Partisan Divide”

We noted in the first section how HOH creates an ambiguity around the term “fact-checking.” In this section we will explain how the wider context of fact-checking makes HOH another argument helping to show that mainstream fact checkers lean left.

In late 2016, a couple of surveys emerged suggesting that the public places little trust in mainstream fact checkers. We wrote about the surveys back in October 2016. The YouGov poll produced striking results showing that only 16 percent of moderates placed a high degree of trust in fact checkers. Sixteen percent of moderates placed a high degree of trust in fact checkers. It is not just a partisan divide. Liberals appear to like the fact checkers considerably more than do moderates.

HOH aside, the polls make a strong case that conservatives and liberals hold differing attitudes toward mainstream fact checkers. And when we consider the low amount of trust moderates (see middle thumbnail above) place in the fact checkers it strengthens the inference that the fact checkers do, in fact, favor the left over the right. Survey data also show very low numbers of self-identified Republicans among the ranks of journalists.  Combined, these data help point the direction of future research. Researchers should look first at the possibility of fact checker bias. Lacking evidence of bias, it will then make sense to look at whether conservatives and moderates simply have more trouble accepting the truth than do liberals.

No credible researcher should assume fact checkers perform unbiased work.

Summary

HOH supports the notion that conservatives and liberals view fact checking differently. It correctly identifies the trust gap as a problem for fact checkers. But it fails to explicitly call for research into the most obvious potential cause for the partisan divide it alleges: journalistic bias.

We continue to advocate using the fact checkers’ own rating systems to explore the extent to which fact-checking journalists allow ideology to affect their work. HOH helps justify that call for research, perhaps fulfilling the prediction that liberals would welcome left-leaning fact checks more than would conservatives.