FiveThirtyEight and atheists in prison

FiveThirtyEight“Atheists are underrepresented [in federal prison].”

Mona Chalabi, in a March 12, 2015 article for FiveThirtyEight

 

Overview

FiveThirtyEight uses data inadequate for the task of estimating the percentage of atheists in federal prison. FiveThirtyEight’s argument won’t even support a probabilistic case that atheists are underrepresented in the federal prison population.

The Facts

FiveThirtyEight’s Mona Chalabi received a request from a reader to look into the claim atheists are underrepresented in prison.

Data on religion in U.S. prisons is hard to find and usually comes from biased sources. Back in 1997, a blog post appeared on HolySmoke.org, a now-dormant anti-religion website. Beneath the headline, an author using the name Rod Swift published statistics apparently received from the Federal Bureau of Prisons (BOP) showing the religious affiliations of inmates. One number jumped out at Swift: 0.2 percent of the prison population was atheist.

That number became a routinely cited statistic that wasn’t really challenged or updated — until 2013. Hemant Mehta (a writer who, seven years earlier, had become famous for selling his soul on eBay) issued a new Freedom of Information Act request to the Federal Bureau of Prisons asking for updated figures about the religious affiliation of prisoners. Today, those numbers are still the most detailed ones available.

 

Chalabi’s work is okay through this point, though the claim about the number of atheists in prison has faced challenges over the years. We’ll confront Chalabi with the traditional counter, which she attracts with her description of her methods (bold emphasis added):

The data here is self-reported, so it’s also dependent on whether prisoners were willing to disclose their religion. Seventeen percent of inmates listed “no preference” for their religion, but the Bureau of Prisons couldn’t clarify how that is different from the 3 percent of prisoners who described themselves as “Other,” so I decided to cut it out of the data.

 

We found it very surprising to see this description from FiveThirtyEight, which has built a pretty good reputation for handling facts and figures.

First, it’s always wise to design a questionnaire to accurately measure the critical data. If the questionnaire is not explicitly designed to capture the critical data then the researcher needs to consider very carefully whether the method in question meets the need.

Second, one needs a solid rationale for ignoring portions of the data.

The right tool for the job?

There’s a great deal we don’t know about the methods the prison system used to collect its information. Was the question of religious preference open-ended? We think that’s likely, given the variety in the response, and at least one other clue supports the hypothesis: Seventeen inmates named “Science” as their religion.

The big problem in the data set stems from the large number of ambiguous responses. Over 37,000 inmates listed their religion as “no preference.” Over 6,000 listed their religion as “Other.” Another 7500+ listed their religion as “Unknown.”

Given these facts, the prison survey serves as something other than a precision tool for this task. Atheists may have elected to list their religion as “no preference,” “Other” or “Unknown.” The atheist who looked into the issue in 2013, Hemant Mehta, did a better job than Chalabi in sniffing out potential problems:

Finally, it’s also important to note that 17% of prisoners reported no religious preference. They’re not necessarily atheists and may even believe in a higher power. We really don’t know. 3% were “Other” and 3.44% were “Unknown.” We can’t assume these people are atheists or Christian or anything else. However, if you combined the Atheist/No Religious Preference groups and lumped them together as “Nones,” as some sociologists do, you’d get 17% of the prison population… I’m not sure that tells you anything useful, though, because of the murkiness of the labels.

 

Mehta’s right about the murkiness of the labels, and that high degree of murkiness throws conclusions about the smaller groups, such as atheism, into much doubt. Combining all the ambiguous categories, including “Other,” the percentage of atheists in the prison population could reach a maximum of 23.6 percent. The truth falls between .1 percent and 23.6 percent, and we don’t know where based on the prison survey. Without knowing the figure with some precision, there’s no way to know whether it is higher or lower than the percentage of atheists in the general population (Chalabi uses 1 percent, Mehta 0.7 percent).

Ignoring the data

Chalabi provides an extraordinarily thin rationale for discarding the “no preference” and “other” groups from the data she uses for her calculations. As we noted above, the existence of the ambiguous groups throws great doubt on the measurements. Discarding the data accomplishes little more than constructing an illusion of certainty around uncertain data. Chalabi’s result counts as a fallacy of fake precision.

Summary

We don’t know whether atheists are underrepresented in prison, and neither does Chalabi. The prison survey doesn’t offer enough information to answer the question to any reasonable level of certainty.

“Atheists are underrepresented [in federal prison].”

generic fallacy button

Ignoring ambiguous results that encompass over 17 percent of the data results in a fallacy of fake precision. We don’t know whether atheists are underrepresented in the federal prison population, so we refrain from calling Chalabi’s claim either true or false. Instead, we simply say Chalabi has failed to marshal the evidence needed to even probabilistically support her claim.

 

References

Chalabi, Mona. “Are Prisoners Less Likely To Be Atheists?FiveThirtyEight. ESPN Internet Ventures, 12 Mar. 2015. Web. 13 Mar. 2015.

Mehta, Hemant. “What Percentage of Prisoners Are Atheists? It’s a Lot Smaller Than We Ever Imagined.” Patheos.com. Patheos, 16 July 2013. Web. 13 Mar. 2015.

2 Comments

  1. D2E

    Your summary at the top of the page reads:

    “… data inadequate for the task of estimating the percentage of atheists in federal prison. … argument won’t even support a probabilistic case that atheists are underrepresented in the federal prison population.”

    This wording may suggest to some readers (especially those unfamiliar with interpreting statistical data) that because:

    “FiveThirtyEight’s argument won’t even support a probabilistic case that atheists are underrepresented in the federal prison”

    They can feel safe in assuming that the number is not underrepresented.

    I understand you clarify near the bottom:

    “We don’t know whether atheists are underrepresented in the federal prison population, so we refrain from calling Chalabi’s claim either true or false.

    Instead, we simply say Chalabi has failed to marshal the evidence needed to even probabilistically[sic] support her claim.”

    But this summary really should be near the top. Especially the line reading:
    “we refrain from calling Chalabi’s claim either true or false.”

    Presenting the information the way you do can lead to
    a reaffirmation of bias due to anchoring.
    https://en.wikipedia.org/wiki/Anchoring

    In essence, because you don’t provide wording that outright contradicts anyone’s beliefs in the beginning, they will assume that the ambiguity in data will support their beliefs.

    It is not unreasonable to assume many of your readers may be conservative (because of your stated political affiliation), so the article appears at first sight to reinforce whatever bias they may have.

    I would suggest language like this:
    “Based on the available data, it would be unwarranted to state any percentage meant to represent the atheist prison population”.

    I think some social context would be helpful too.
    Namely that, atheist communities use this statistic to support the idea(s) that it is possible to be ‘Good without God’ and that ‘Religion does not improve morality and may hinder it’.

    While religious communities tend to believe the opposite.
    (http://www.pewglobal.org/2014/03/13/worldwide-many-see-belief-in-god-as-essential-to-morality/)

    You might want to suggest to the reader that their own biases might influence their own perceptions about the ambiguity in the data and that they should not use the available prison data to form an opinion one way or the other, do to it being unreliable.

    Reply
    1. Bryan W. White (Post author)

      D2E,

      I have committed this site to a high level of transparency on corrections and clarifications. While your suggestions are not objectionable, they represent a path toward a cluttered presentation (multiple correction/clarification notices and archived earlier versions) and a drain on time spent producing new fact check material and other research.

      Thanks for taking the time to comment. I understand the comments are motivated by good will, representing a desire to see good information communicated more effectively.

      Reply

Leave a reply here or on our Facebook page

This site uses Akismet to reduce spam. Learn how your comment data is processed.