Sentience Institute Survey Analysis
Sentience Institute (SI)1 is a relatively new think tank within the effective altruism movement. According to SI’s website, the organization was founded in June 2017 with the stated mission of “expanding humanity’s moral circle” through researching the most effective strategies to achieve this goal, with an initial focus on farmed animals. They have released a few publications on this topic—one of which is their “Survey of US Attitudes Towards Animal Farming and Animal-Free Food October 2017.” Here, we review this survey in order to better understand the extent to which we can draw conclusions from the survey’s results. This is important, as the survey produced several surprising results. Assessing the validity of those results will be helpful in determining to what extent our beliefs should change. SI plans to use this initial survey to provide baseline data for the same survey in future years, in order to monitor how attitudes toward animal farming and animal-free food shift as the animal advocacy movement grows. By adequately monitoring changes in these attitudes, this survey could be a valuable resource for annually assessing the extent to which the animal advocacy movement is accomplishing its goals.
Overview
In October 2017 Sentience Institute—in collaboration with the survey company Ipsos—sent a survey of 17 questions to 1,0942 U.S. adults to assess their attitudes toward “animal farming” and “animal-free foods.”3 As this was an exploratory survey, there were no statistical hypotheses SI was looking to confirm. They did however have three main goals:
- to provide baseline data for future research using the same survey
- to test predictions of how adults will respond
- to conduct exploratory analyses on the associations between responses and demographics
SI found that, overall, U.S. adults showed a stronger opposition toward animal farming and more significant support for animal-free food than expected—with 33% supporting a ban on animal farming and 54% trying to consume fewer animal-based foods and more plant-based foods. It is worth noting that the numbers highlighted in SI’s report exclude those who chose “no opinion.”4 After the survey, SI conducted an exploratory analysis to better understand the association of respondent demographics with responses. Although this analysis did suggest some associations as likely,5 SI noted that these results “should be taken with large grains of salt” because the analysis was not pre-registered, and multiple comparisons weren’t accounted for.6 They do, however, make the raw data and code publicly accessible (and they encourage its use in future research). Since this section of SI’s report focuses more on potential avenues for future research, this blog post will primarily discuss the strengths and limitations of the goals, methods, and conclusions of the survey itself.
Strengths
This survey is not entirely novel—previous Researchers have conducted similar surveys. This includes Faunalytics, who has been collecting information on opinions and behavior relating to animals since 2009. A quick look through the Faunalytics archive of publishings on attitudes toward animals (in addition to a search through Google Scholar7 ) yielded many publications on the topic of attitudes toward animal welfare. However, most of these studies8 seem to focus strictly on animal welfare rather than on the overall concept of raising and slaughtering animals for food (and the potential banning of such practices). SI’s survey seems to be one of the first surveys to directly ask questions regarding these concepts, making it a very important and relevant addition to the field. It is worth noting that this survey was repeated almost identically by the Department of Agriculture at Oklahoma State University in January 2018. The substantially similar results attained by OSU provide significant validation of SI’s results.
From a methodological perspective, SI took steps to ensure the external validity9 of the results by using a census-balanced audience, and took steps to avoid publication bias10 by committing to publish their results before running the survey with the Open Science Framework. They also provided the raw data and code for their exploratory analysis, allowing other Researchers to repeat the analysis. In order to increase the validity of the survey results, SI attempted to avoid social desirability bias11 by not mentioning their name in the survey’s introduction and by using a Likert scale to minimize possible indications of a desired response. Additionally, according to the first footnote within the report, a disqualifying question was used to ensure that all responses used in the analysis were from individuals who had carefully read each question.12
The remainder of this post mainly focuses on a few apparent limitations of SI’s survey and their interpretation of its results. Despite these limitations, however, we feel that this research is an important addition to the field of effective animal advocacy.
Limitations
Survey Questions
SI does not offer an explanation of the rationale that led them to choose the items used in the survey. We believe that doing this would have been quite helpful. Only three of the 17 items include references to previous research,13 so we expect that readers may wonder about the origin of the remaining 14 questions. If the other questions are original to SI and/or if there were important reasons for their selection, that would be useful information to know—especially for those who may like to build on this research in future.
Although SI took steps to reduce the risk of bias and measurement errors, there is one question where these risks seem to have been overlooked. The “personal choice” question seems to be a double-barreled question, which may significantly diminish the value of its responses. The first barrel states, “Whether to eat animals or be vegetarian is a personal choice,” and the second states, “nobody has the right to tell me which one they think I should do.” These are two completely different concepts. The first addresses personal opinions, whereas the second addresses whether people have the right to voice those opinions. If most of the respondents agree with second part of the statement, it somewhat contradicts other responses to the survey.14 For example, if something like a third of adults in the U.S. support a ban on animal farming,15 then some of the same adults who indicated that they believe vegetarianism is a personal choice favor a ban that to a significant extent seems to impose upon that choice. In the next section, we will discuss the tension between the responses to different items in the survey in more detail.
Survey Results and Conclusion
The results of this survey showed some surprising and interesting statistics, especially with regard to slaughterhouses, animal farming, and perceptions of animal welfare. At the beginning of the report the author lists several results that seem most compelling to SI, accompanied by commentary on some of the implications they believe such statistics may have. Some of the results could have benefited from more discussion, such as the following two:
97% of US adults agree “Whether to eat animals or be vegetarian is a personal choice, and nobody has the right to tell me which one they think I should do.”
49% of US adults support a ban on factory farming, 47% support a ban on slaughterhouses, and 33% support a ban on animal farming. The support for the latter two exceeded even Researcher expectations. This suggests that animal-free food advocates might be able to succeed with stronger proposals than they currently use, though readers should take into consideration social desirability bias in this type of survey.
These results seem contradictory—if there were no animal farming or slaughterhouses, people would not be able to make their own decisions about how to eat. Though the author does attribute these surprising results to potential social desirability bias, the lack of definitions for terms used in the survey16 might mean that people are responding to the question(s) using their own concepts for those terms, which may be substantially different from the meaning intended by the survey authors. For instance, perhaps most respondents thought that they could still easily eat animals if there was a ban on slaughterhouses. That said, the 2018 survey replication asked respondents who agreed with a ban on slaughterhouses a follow-up question with a brief description of slaughterhouses17 and approximately 73% of those who agreed with the ban indicated that they knew that a ban would mean “you would not be able to consume meat.”18 This leads us to believe that the discrepancy in the results previously highlighted should be attributed to other factors besides respondents’ lack of knowledge of the relevant concepts. The discrepancy may be largely attributable to social desirability bias or to the fact that people’s beliefs do not always match their individual behaviors.19
The conclusions associated with the following results also seem to merit closer examination:
58% of US adults think “most farmed animals are treated well,”20 despite over a decade of undercover investigations in the US and USDA data suggesting over 99% of farmed animals live on factory farms.21 This suggests either we have insufficient awareness of factory farming, or people have refused to accept the evidence, perhaps due to the emphasis on individual diet change and subsequent cognitive dissonance with their own eating habits.
75% of US adults say they usually buy animal products “from animals that are treated humanely,” despite estimates suggesting fewer than 1% of US farmed animals live on non-factory farms.22 This suggests a psychological refuge effect where people justify their animal product consumption by incorrectly assuming they are eating ethically-produced food.
When reading these two results separately, they offer very interesting conclusions—especially given the USDA data SI provides. However, when reading them together, they seem to offer very similar results. It would make sense that if 58% of U.S. adults agree that most farmed animals are treated well, then those same adults would believe they are purchasing humanely-raised animal products. Also note that for the reported 75%, SI did not include those who responded “no opinion” when summarizing these results. Since 20% of U.S. adults responded “no opinion” to this statement, the overall percentage of those who agreed with the statement is 60%.23 SI offers two distinct conclusions for these statistics, and, given that these results have similar implications, the conclusions could apply to either. For example, if U.S. adults think most farmed animals are treated well simply because they have a lack of knowledge on the topic, then they may believe the meat they are buying is humanely-raised because they lack knowledge of the fact that it is likely not humanely-raised. Similarly, if U.S. adults think this way due to cognitive dissonance, this could be the same reason they believe that their meat comes from a humane source. SI suggests the latter statistic may be due to a “psychological refuge effect.” While they do describe this concept in a couple of words, there is no reference provided for the reader to further investigate.24 There seems to be room for SI to expand on the conclusions they draw here, provide references for the psychological concepts used, and better explain their reasoning for suggesting different conclusions.
Conclusion
The most surprising result from this survey is the large proportion of respondents who seemed to be in favor of banning factory farming, slaughterhouses, or animal farming. Though the 2018 replication certainly helps to validate SI’s results, the apparent suboptimal wording of the personal choice question still remains a limitation. If SI is to repeat this survey without any changes, it may be helpful to further explain their original design choices and analysis in order for those following this research to better understand its implications. If they do address some of the limitations, it will be important to explain any divergences from their original design—so as to not detract from the usefulness of a repeated survey as a way to track trends over time.
SI’s survey had several important methodological strengths, including census-balancing, pre-registration, attempts to avoid social desirability bias, and a disqualifying question to ensure that responses used in the analysis were from individuals who had carefully read each question. Ultimately, given the methodological strengths of the survey—even in light of the limitations mentioned—we feel that this research has the potential to be an important asset in the assessment of the extent to which the animal advocacy movement as a whole is accomplishing its goals.
The author of the Sentience Institute survey previously worked as a Researcher at ACE. Additionally, some ACE staff are friends with some staff at Sentience Institute. While we do not think that these associations have impacted our impartiality on this particular piece, we would like to note the potential for a conflict of interest.
From Sentience Institute: “This is the number of complete responses which passed the awareness check. An unusually large number of respondents failed the awareness check, 46.3% of the 2,438 total submissions. We did have a more challenging awareness check than many surveys, which was after the ‘public demonstration’ question and read: ‘Suppose you were given $10 and allowed to donate any amount of it to an effective non-profit organization that works to help farmed animals, keeping the rest for yourself. If you read this question, please select four dollars. How much of this $10 would you donate?'”
“Animal farming” and “animal-free foods” are phrases used by Sentience Institute in this survey.
This exclusion of those who reported no opinion had varying impacts on differences between the reported proportions in SI’s visual and their table. For instance, according to the visual, 53% of respondents agree with the statement “I am currently trying to consume…” Since SI excluded those who had no opinion, the 53% was out of 98% rather than 100%, resulting in the 54% reported in the table. For the question on whether respondents purchase animal products from animals who are treated humanely, according to the visual, 20% of respondents had no opinion and 60% agreed. In contrast, the table reports that 75.4% of people agreed with that question.
“US adults who identify as Female are more opposed to animal farming than those who identify as Male (14.5% more likely to agree with the average Likert-scale statement), Black (Non-Hispanic) and Hispanic (all races) are more opposed than White (Non-Hispanic) (13.7% and 19.7%), and Northeastern are more opposed than Southern (5.5%).” See footnotes 10 and 11 in Sentience Institute’s survey.
The significance threshold for p-values reported by SI did not adjust for multiple hypothesis testing.
Cornish, A., Raubenheimer, D., and McGreevy, P. (2016). “What We Know about the Public’s Level of Concern for Farm Animal Welfare in Food Production in Developed Countries.” Animals, 6(11): 74.
Tawse, J. (2010). Consumer attitudes towards farm animals and their welfare: a pig production case study. Bioscience Horizons: The International Journal of Student Research, 3(2): 156–165.
Lusk, J. L. and Norwood, F. B. (2008). A survey to determine public opinion about the ethics and governance of farm animal welfare. J Am Vet Med Assoc, 233(7): 1121–1126.
McKendree, M. G., Croney, C.C., and Widmar, N.J. (2014). Effects of demographic factors and information sources on United States consumer perceptions of animal welfare. Journal of Animal Science, 92(7): 3161–3173.GiveWell describes external validity as determining how the results seen in a given study will translate to other settings and larger-scale programs.
GiveWell describes publication bias as “a broad term for factors that systematically bias final, published results in the direction that the Researchers and publishers (consciously or unconsciously) wish them to point.”
Social desirability bias is a risk in which the respondents provide the response for which they believe the surveyor is looking.
“Suppose you were given $10 and allowed to donate any amount of it to an effective non-profit organization that works to help farmed animals, keeping the rest for yourself. If you read this question, please select four dollars. How much of this $10 would you donate?” 46.3% of respondents responded inaccurately and were disqualified.
These three questions are: (i) “I am currently trying to consume fewer animal-based foods (meat, dairy, and/or eggs) and more plant-based foods (fruits, grains, beans, and/or vegetables).” (ii) “Farmed animals have roughly the same ability to feel pain and discomfort as humans.” (iii) “Most farmed animals are treated well. For example, the animals are given enough space and kept in good health.”
These responses include questions to the following questions: (i) “People should consume fewer animal-based foods (meat, dairy, and/or eggs) and more plant-based foods (fruits, grains, beans, and/or vegetables).” (ii) “Please consider the following information before proceeding: In recent years, scientists and chefs have started using plants to create “animal-free” foods that are very similar to meat, dairy, and eggs in their taste, texture, and nutritional profile. When these foods are the same price as animal-based foods, people should eat more of these foods and fewer animal-based foods.” (iii) “Please consider the following information before proceeding: Scientists and chefs are also using technology to grow real meat from animal cells without animal slaughter, by feeding those cells with plant-based nutrients so they grow and form muscle fibers, just like they would in an animal’s body. When these foods are the same price as animal-based foods, people should eat more of these foods and fewer animal-based foods.”
This was another interesting statistic found by this survey. It is discussed in more detail in this blog post in the Survey Results and Conclusion section.
For example, The survey often refers to “animal farming” and “slaughterhouses” without explicit definition.
“Were you aware that slaughterhouses are where livestock are killed and processed into meat, such that, without them, you would not be able to consume meat?”
“Approximately 73% of participants stated, yes, they are aware that slaughterhouses are where livestock are killed and processed into meat.”
The replication of SI’s survey in January 2018 notes: “Participants who agreed with this statement were asked a follow-up question: “Were you aware that slaughterhouses are where livestock are killed and processed into meat, such that, without them, you would not be able to consume meat?” Approximately 73% of participants stated, yes… That still leaves about 34% of Americans saying they wish to ban slaughterhouses… Even though most Americans eat meat, they also do not like the idea of slaughterhouses.”
Reese notes that, “We made a mistake in saying “treated well” in this question but “treated humanely” in the other. We should have used the same word in both.”
See footnote 4 in Sentience Institute’s survey.
See footnote 5 in Sentience Institute’s survey.
The overall percentage of those who agreed with the “most animals are treated well” item was 52%. Excluding those who responded “no opinion,” it was 58%.
A google search of the term “psychological refuge effect” produced no results with these exact words or a description of the concept. Rather, the search produced links discussing the “Prospect Refuge Theory,” which seems different from what SI is describing.