Study: You can fool some of the people some of the time

A critical review of Jordan Humphreys’ “When Lies Matter:  The Effect of Media Coverage of Misinformation on Public Opinion in the Health Care Reform Debate”


Zebra Fact Check started in part to help express my concern about the state of high-quality information.  The sources normally deemed worthy of trust, like journalists and academia, tend to have liberals in control.  That’s not necessarily a bad thing if journalists and scholars successfully set aside ideological bias while reporting and researching.  But we face the danger of a rigged game when it comes to citing sources of supposedly high-quality information.

I examine with interest scholarly attempts at figuring out why some people don’t accept the plain truth.  Why don’t people display the good sense possessed by the scholars?  I read the scholars’ papers and ask “Are these scholars immune to the effects that keep people from displaying the good sense possessed by the scholars?”

On to another such study I recently encountered, “When Lies Matter:  The Effect of Media Coverage of Misinformation on Public Opinion in the Health Care Reform Debate,” published in the Fall 2013 issue of Undergraduate Research Journal.

Author Jordan Humphreys places his focus on the public’s belief in “death panels” during the run up to the 2010 election:

With all of the news on the issue, with Democrats and Republicans weighing in on either side, and with non-partisan fact-checkers striving to set the record straight, 1 in 3 Americans ended up believing “death panels” had a place in health care plan in August 2009. Did this affect their votes in the November 2010 election?


We need to somehow help those dunderheads who believe the “death panel” myth, right?

Pew Research death panel questionsAs I’ve pointed out in earlier writings, the “death panel” line came from Sarah Palin’s personalized illustration of economist Thomas Sowell’s description of the economic effect of increased government control of health care.  Fact checkers tended to ignore Sowell’s role in Palin’s writing on the subject.  The Pew Research poll question Humphreys mentions in his paper also completely fails to make the connection to Sowell’s argument.  The version of the follow-up question found at the Pew Research Poll database (see inserted image) is also missing a key word, making the question grammatically incorrect.  Should respondents with knowledge of Sowell’s argument address the question according to the straw man it contains, or answer the question as it should have been asked:  Does the health care law create a “death panel” through rationing, as Palin noted in accordance with Sowell’s reasoning?  The fact checkers had already rigged the game by distorting Palin’s statement.  Pew Research reproduces the distortion in framing the survey question.  They’re playing the telephone game with facts.

Humphreys’ paper opens with an ironic example of a fact: the media-perpetuated myth that  Palin’s illustration of “death panels” was explicit in the health reform law.  Humphreys, while citing the research of Matthew Shafer and Regina Lawrence, goes on to laud the journalistic malpractice that enabled the error:

This ritual of he-said, she-said coverage is called procedural  objectivity by the researchers. Journalists here are seeking to include both sides of the issue in attempts to remain objective—they position their reporting in the middle of a debate so that flak  comes from both sides fitting  with the reporter’s adage that ‘you know you’ve done a good job once everyone’s complaining.’

However, this obedience to objectivity irresponsibly gives validity to both sides in the argument—in this way, a gamut of scientific research on the existence of global warming on one side can be reduced to just an opinion when opposed with findings from fringe researchers that the phenomenon doesn’t exist. In cases of fact (or well tested theory), journalists do their readers a disservice by  presenting both sides as an argument in procedural objectivity.


Humphreys says journalists fail their readers by presenting both sides when one side is factually correct.  But do we want journalists deciding what is fact and what isn’t?  The “he said—she said” model exists for a reason:  Journalists once recognized their limitations.  Making determinations of truth where experts disagree draws journalists into non-objective behavior.

PolitiFact transcends the limitations of journalists past?

Humphreys goes on to claim that fact checkers such as PolitiFact do enough research to settle issues of fact.

We’ve done enough research to place Humphreys’ claim under severe duress.  The “death panel” claim serves as the obvious parry to Humphreys’ thrust.  The fact checkers failed to pay attention to the context of Palin’s “death panel” claim.  They presented one side of the argument, their own, called it a fact and proceeded to sell a bill of goods later bought by Humphreys.  Not realizing he was duped, Humphreys wants more of the same from fact checkers.

Humphreys’ vacuous experiment

Humphreys’ experiment establishes the possibility of fooling some of the people some of the time.

He devised an experiment to test how different approaches to reporting a story might affect readers’ belief in so-called “misinformation,” specifically whether readers believed “the Affordable Care Act allowed the government to dictate care for patients.”  Knowing the background of the “death panel” claim as described above, the statement of misinformation comes across ambiguously.  It might mean the government loosely dictates care for patients, as by manipulating reimbursement rates according to Independent Payment Advisory Board recommendations.  And it could represent the media narrative, that the health care law establishes a body that explicitly makes health care decisions for individuals.  We’ll see that Humphreys’ experiment steered his experimental groups toward the latter.

To address the specific issue of misinformation, Humphreys used a statement from Michele Bachmann about the IPAB, where Bachmann stated the board would be in charge of all medical decisions for 300 million Americans.  Bachmann’s claim varies dramatically from Palin’s, the former strongly emphasizing that the IPAB would directly control individual health care decisions.

Here’s how Humphreys did his experiment:

  1. Humphreys altered an opinion article from a Florida news source, changing a story about Republican presidential candidate Bachmann into a story about fellow Republican candidate Mitt Romney.  Humphreys apparently did not realize the original article was not a news story.
  2. Humphreys altered the appearance of the story, crediting it to Reuters and a freelance writer, adding a photo of Mitt Romney and presenting the result to study subjects as a news story.
  3. Humphreys created a control group that read the faux news story he created
  4. Humphreys created experimental groups by adding to the story a claim from Bachmann about the IPAB, attributed to Romney, along with a correction using the words of PolitiFact.
  5. One experimental group received only the original story plus Bachmann’s claim attributed to Romney.
  6. A second experimental group received the added Bachmann claim (attributed to Romney) along with the PolitiFact correction attributed to PolitiFact, including the stipulation that PolitiFact is “a non-partisan fact-checking organization.”
  7. The third experimental group received the added Bachmann claim (attributed to Romney) along with the PolitiFact correction attributed to White House Press Secretary Jay Carney.

The results were predictable.  The control group affirmed belief that “the Affordable Care Act allowed the government to dictate care for patients” at a 40 percent rate.   The group exposed to Bachmann’s words placed in Romney’s mouth affirmed belief at a 50.8 percent rate.  The group exposed to the “misinformation” as well as the correction attributed to PolitiFact affirmed belief at a 35.8 percent rate.  The group exposed to the “misinformation” along with correction attributed to Jay Carney affirmed belief at a 44 percent rate.

Let’s review the researcher’s primary research question:  “Does media coverage dissuade citizens from or encourage them to believe misinformation?”

The answer is “yes.”  The media encourage belief in misinformation and discourage belief in misinformation.  Which side the media take in any given situation is anybody’s guess, though it’s likely they’ll be tougher on conservative thought.

At its base, Humphreys’ experiment examines what happens when people receive new information from a trusted source.  Of course information from trusted sources carries a greater effect on their thinking, and that will tend to hold true whether the information is true of false.

Given the attention granted to the subject of “death panels” during the election, Humphreys chose an inadequate control group.  Persons in the control group may have had familiarity with the IPAB in the sense Palin had criticized, and Humphreys’ survey question would have measured beliefs based on that.  At the same time, Humphreys’ experimental groups framed the question according to the dominant media narrative, ignoring the economic argument from Sowell to focus on whether the health care law gives the government meticulous control over individual health care decisions.

Humphreys concludes by recommending a subpar fact checking organization, PolitiFact, as an authoritative voice in judging the accuracy of political speech:

So what do we do to combat opinion swings by a misinformed citizenry? Stop misinforming them. This study has proved that we have effective measures for combatting [sic] some misinformation in our arsenal. With the exception of those covering a debate live or online and reporters putting together a quick summary of events, PolitiFact is testing and compiling the information necessary to adequately debunk misinformation claims in the hours after they are made (the examples used in this study were corrected by PolitiFact by the midnight after the debate), so there are options for journalists to inform citizens without risking misinforming them.


Like Brendan Nyhan and Jason Reifler before him, Jordan Humphreys allows his acceptance of misinformation to corrupt his experiment.  He ends up proving only that you can fool some of the people some of the time.  Scholars and researchers aren’t exempt.

1 Comment

  1. Jack Adams

    another example of real scholarship when it comes to checking facts and not opinions. You rock, Zebra.


Leave a Comment

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.