Some Thoughts About Vegan Outreach’s Leaflet Study
Introduction
We’re always excited to see new research on effective animal advocacy, so of course the recent Vegan Outreach leaflet study has caught our attention. The study has many merits. For example, it investigates participants’ self-reported dietary choices, which might contribute to our understanding of how to best help farmed animals. Participants were assigned randomly to treatment conditions and given both a pre-test and a post-test questionnaire. While the Researchers at Vegan Outreach did not pre-register an analysis plan, they did refrain from testing more hypotheses than they reported; we can therefore take the results with more confidence than we would had they tested every possible comparison.
We feel the study has some limitations, however, including the lack of a control group and relatively small sample size for the comparisons Vegan Outreach was originally interested in. Overall, we found the study interesting and useful but we recommend caution in interpreting its findings.
Summary of Method and Findings
Vegan Outreach conducted a Pay Per Read study on Amazon’s Mechanical Turk (MTurk) to investigate the effects of leaflets on readers’ dietary habits. Subjects were given a short survey with a dietary assessment component. Then they were randomly assigned to read one of four leaflets and asked a few more questions, some of which tested comprehension. Three months later, participants were contacted again and asked to complete a follow-up survey.
Vegan Outreach was primarily interested in “single-week avoidance” of foods. For example, they measured the number of participants who refrained from eating beef for at least a week by combining the number who reported eating beef “less than 1 time per week” and “never.”
Vegan Outreach cites three primary findings from the study. First, they found that for every 75 participants who read a leaflet, at least 1 reduced their consumption of all non-vegan foods to less than once per week. Second, they found that participants who reduced beef or pork consumption after reading a leaflet were also likely to reduce their consumption of chicken and other animal products. Third, Vegan Outreach claims that providing health information in the leaflets, especially encouraging participants to eat legumes, likely leads to readers feeling healthier. They did not find any statistically significant differences in the effects of the leaflets relative to one another.
The Leaflets: Between-Groups Effects and Pooled Effects
We have two primary concerns about Vegan Outreach’s analysis of the effects of the leaflets on participants’ diets. First, the study had relatively low power, which makes it difficult to draw conclusions about the relative effects of individual leaflets. Second, there was no control group, which makes it difficult to draw conclusions about the pooled effects of all the leaflets.
If Vegan Outreach was strictly interested in measuring differences in the effects between leaflets, they would not have needed a control group but they would have needed a larger sample size to attain higher statistical power. According to a power analysis they conducted prior to collecting data, they would have needed over 2000 participants to read each leaflet and complete the pre- and post-test questionnaires in order to have an 80% chance of detecting effects of the expected size for individual animal product consumption categories. They had 356–404 participants per leaflet who completed both questionnaires. Finding enough participants to achieve the desired power is often difficult, especially when participants are required to complete both a pre-test and a post-test. We understand that Vegan Outreach included as many participants as they could with the funds they had available. Still, when possible, we recommend designing studies with the goal of achieving at least an 80% chance of detecting the expected effect size or the effect size of interest.
If Vegan Outreach was strictly interested in measuring the pooled effects of all leaflets, they may not have needed a larger sample size but they would have needed a control group. Pooling the effects of all leaflets led them to the conclusion that about 1 out of 75 participants reduced their animal product consumption after reading a leaflet. However, without a control group, we can’t know whether participants’ dietary change was a result of reading a leaflet. Perhaps 1 out of 75 individuals reduce their animal product consumption every 3–5 months regardless of whether they read a Vegan Outreach leaflet. Perhaps participants’ dietary habits were affected by the holidays or some other event that occurred during the months in which the study took place. In those cases, the pooled effects of the booklets may be different than Vegan Outreach estimates.
It seems that Vegan Outreach chose not to use a control group because their first priority was to assign as many participants as possible to each treatment group in order to maximize the statistical power of their study. However, since they ended up testing both between-groups effects and pooled effects, an ideal survey design would have included both a large enough sample size and a control group.
Are Beef and Pork Reduction Associated with Chicken Reduction?
We are excited about Vegan Outreach’s finding that participants who began avoiding beef and pork did not seem to increase their chicken consumption as a substitute. We wonder, though, about the extent to which Vegan Outreach’s finding generalizes to all recent beef and pork avoiders. It may be that readers of the Vegan Outreach leaflets were more likely than other recent animal product avoiders to reduce their chicken consumption.
One of the Vegan Outreach leaflets specifically advocated eating fewer chickens. All four of the Vegan Outreach leaflets included the following message:
Because many more chickens are killed to produce the same amount of meat as from cows and pigs, you’ll prevent more animal suffering by first eliminating chickens.
If the chicken leaflet or the message about preventing suffering affected participants’ chicken consumption, they’re good materials for animal advocates to include in their literature. If they did not, it may be true that all recent beef and pork reducers are also likely to reduce their chicken consumption.
The Value of Providing Health Information
Vegan Outreach performed chi-squared tests to check for associations between which information participants looked up, which foods they avoided, and how healthy they felt. They helpfully provide a link to an Excel document with the results. They conclude from their analysis that providing nutrition information, especially promoting legumes, is helpful for an “important segment of people.”
It seems very plausible to us that providing nutrition information is an effective way to help people begin and maintain a veg diet. However, it’s not clear to us that Vegan Outreach’s analysis supports that claim. One reason for our uncertainty is that they ran 44 chi-squared tests. Running multiple tests increases the risk that some results will be statistically significant by chance. To mitigate the higher risk of getting “false positives,” Researchers running multiple tests should use stricter criteria for what they consider to be a significant result. In this case, we suggest using a Sidak-adjusted alpha level of about 0.001 for n = 44 tests. Some of the associations that Vegan Outreach mentions are not significant under the stricter criteria. For example, feeling healthier is not associated with eating beans or with avoiding beef at a significant level.
Other Considerations
Dietary Assessment Methodology
Vegan Outreach used a Food Frequency Questionnaire (FFQ) to assess changes in participants’ dietary habits, which is a method we feature in our Survey Guidelines. FFQs are commonly used in nutritional and medical studies and they can be useful for gathering a lot of information. However, they lack the sensitivity to detect small changes in dietary patterns. Also, participants may not be able to reliably report their dietary habits over the past 1–3 months; they would likely give more accurate answers when asked to report their dietary habits over the past 1 or 2 days.
In the time since Vegan Outreach conducted its study, we have updated our Survey Guidelines. We now recommend using a 24 Hour Recall (particularly the ASA 24) to measure small dietary changes in response to interventions. When it is not possible to use a 24 Hour Recall, we recommend using an FFQ that is modified to measure consumption over the past 1–2 days.
Possible Sources of Bias and How They Were Addressed
Selection bias, non-response bias, and social desirability bias are frequent sources of concern in animal advocacy research. We appreciate the steps that Vegan Outreach has taken to address each of them.
Vegan Outreach tells us that MTurk users were recruited with the phrase: “Read pamphlet and answer survey on diet, ~12 min.” By excluding phrases like “animal welfare” from the survey title, Vegan Outreach may have avoided selection bias; in other words, they may have avoided disproportionately recruiting MTurk users who were already interested in animal welfare and thus more likely to be affected by the booklets.
Non-response bias might have affected the study results, especially given that 37.3% of pre-test respondents did not complete the post-test survey at the time of analysis. Subjects who failed to complete the post-test were plausibly less affected by reading the leaflet, so leaving them out of the analysis might lead to an over-estimate of the leaflet’s effects. However, Vegan Outreach assures its readers that, after running analyses that included only participants who completed both tests, they ran the same analyses again including all participants. They assumed that participants who did not complete the post-test did not change their dietary habits after reading the leaflet. They claim that “nearly all of the previously significant effects remained so” when they ran the second set of analyses. We think this was a clever way to show that their results were not dramatically affected by non-response bias.
Social desirability bias is a particularly big problem in advocacy research. To some extent, it’s impossible to avoid. Even when Researchers take steps to obscure the purpose of a study, there is always a chance that participants might figure out that the study is being conducted by an animal advocacy group and give answers that they presume Researchers want to hear.
Several features of Vegan Outreach’s survey design may have successfully reduced the effects of social desirability bias. In the pre-test survey, we notice that they asked participants about their dietary habits prior to presenting the leaflet. We also notice that they included vegan products as well as animal products in the list of foods on the survey. Both of these decisions may have prevented participants from drawing conclusions about which answers were “desirable.”
It’s also possible that the post-test survey was not considerably affected by social desirability bias. Since it was conducted 3–5 months after participants read a leaflet, many participants may have forgotten the content of the leaflet and the presumed purpose of the study. However, in the post-test questionnaire, Vegan Outreach asked participants to identify which leaflet they read from a list prior to presenting the FFQ, which may have influenced responses. Ideally, the FFQ should have been presented prior to the reminder about the leaflet.
Conclusions
We are happy to see new research on the effects of animal advocacy interventions. Vegan Outreach’s pre-test/post-test design was strong and they dealt well with possible sources of bias. However, we recommend that Researchers design studies with the goal of achieving at least an 80% chance of detecting the effects of interest. We also recommend using control groups for any studies that may be used to support claims about the effects of an intervention. We thank Vegan Outreach for their work and for providing us with their raw data and survey materials.
