Mona Chalabi, in a March 12, 2015 article for FiveThirtyEight
FiveThirtyEight uses data inadequate for the task of estimating the percentage of atheists in federal prison. FiveThirtyEight’s argument won’t even support a probabilistic case that atheists are underrepresented in the federal prison population.
FiveThirtyEight’s Mona Chalabi received a request from a reader to look into the claim atheists are underrepresented in prison.
Data on religion in U.S. prisons is hard to find and usually comes from biased sources. Back in 1997, a blog post appeared on HolySmoke.org, a now-dormant anti-religion website. Beneath the headline, an author using the name Rod Swift published statistics apparently received from the Federal Bureau of Prisons (BOP) showing the religious affiliations of inmates. One number jumped out at Swift: 0.2 percent of the prison population was atheist.
That number became a routinely cited statistic that wasn’t really challenged or updated — until 2013. Hemant Mehta (a writer who, seven years earlier, had become famous for selling his soul on eBay) issued a new Freedom of Information Act request to the Federal Bureau of Prisons asking for updated figures about the religious affiliation of prisoners. Today, those numbers are still the most detailed ones available.
Chalabi’s work is okay through this point, though the claim about the number of atheists in prison has faced challenges over the years. We’ll confront Chalabi with the traditional counter, which she attracts with her description of her methods (bold emphasis added):
The data here is self-reported, so it’s also dependent on whether prisoners were willing to disclose their religion. Seventeen percent of inmates listed “no preference” for their religion, but the Bureau of Prisons couldn’t clarify how that is different from the 3 percent of prisoners who described themselves as “Other,” so I decided to cut it out of the data.
We found it very surprising to see this description from FiveThirtyEight, which has built a pretty good reputation for handling facts and figures.
First, it’s always wise to design a questionnaire to accurately measure the critical data. If the questionnaire is not explicitly designed to capture the critical data then the researcher needs to consider very carefully whether the method in question meets the need.
Second, one needs a solid rationale for ignoring portions of the data.
The right tool for the job?
There’s a great deal we don’t know about the methods the prison system used to collect its information. Was the question of religious preference open-ended? We think that’s likely, given the variety in the response, and at least one other clue supports the hypothesis: Seventeen inmates named “Science” as their religion.
The big problem in the data set stems from the large number of ambiguous responses. Over 37,000 inmates listed their religion as “no preference.” Over 6,000 listed their religion as “Other.” Another 7500+ listed their religion as “Unknown.”
Given these facts, the prison survey serves as something other than a precision tool for this task. Atheists may have elected to list their religion as “no preference,” “Other” or “Unknown.” The atheist who looked into the issue in 2013, Hemant Mehta, did a better job than Chalabi in sniffing out potential problems:
Finally, it’s also important to note that 17% of prisoners reported no religious preference. They’re not necessarily atheists and may even believe in a higher power. We really don’t know. 3% were “Other” and 3.44% were “Unknown.” We can’t assume these people are atheists or Christian or anything else. However, if you combined the Atheist/No Religious Preference groups and lumped them together as “Nones,” as some sociologists do, you’d get 17% of the prison population… I’m not sure that tells you anything useful, though, because of the murkiness of the labels.
Mehta’s right about the murkiness of the labels, and that high degree of murkiness throws conclusions about the smaller groups, such as atheism, into much doubt. Combining all the ambiguous categories, including “Other,” the percentage of atheists in the prison population could reach a maximum of 23.6 percent. The truth falls between .1 percent and 23.6 percent, and we don’t know where based on the prison survey. Without knowing the figure with some precision, there’s no way to know whether it is higher or lower than the percentage of atheists in the general population (Chalabi uses 1 percent, Mehta 0.7 percent).
Ignoring the data
Chalabi provides an extraordinarily thin rationale for discarding the “no preference” and “other” groups from the data she uses for her calculations. As we noted above, the existence of the ambiguous groups throws great doubt on the measurements. Discarding the data accomplishes little more than constructing an illusion of certainty around uncertain data. Chalabi’s result counts as a fallacy of fake precision.
We don’t know whether atheists are underrepresented in prison, and neither does Chalabi. The prison survey doesn’t offer enough information to answer the question to any reasonable level of certainty.
“Atheists are underrepresented [in federal prison].”
Ignoring ambiguous results that encompass over 17 percent of the data results in a fallacy of fake precision. We don’t know whether atheists are underrepresented in the federal prison population, so we refrain from calling Chalabi’s claim either true or false. Instead, we simply say Chalabi has failed to marshal the evidence needed to even probabilistically support her claim.
Chalabi, Mona. “Are Prisoners Less Likely To Be Atheists?” FiveThirtyEight. ESPN Internet Ventures, 12 Mar. 2015. Web. 13 Mar. 2015.
Mehta, Hemant. “What Percentage of Prisoners Are Atheists? It’s a Lot Smaller Than We Ever Imagined.” Patheos.com. Patheos, 16 July 2013. Web. 13 Mar. 2015.