“Survey respondents who report having received either Vegan Outreach leaflet, even if they report also having received the control leaflet, will be considered part of the experimental group. Respondents who report only having received the control leaflet will be considered part of the control group. Respondents who do not report having received any leaflet we distributed will not be considered part of either group.
The primary goal of analysis will be to determine whether rates of conversion to vegetarianism and veganism during the study period are higher among subjects in the experimental group than among subjects in the control group. Depending on the final survey design, the analysis may also consider whether respondents in the experimental group have reduced their rates of meat and animal product consumption more than respondents in the control group, even among those respondents who do consume these products at the time of the survey.” — ACE pre-analysis plan
Analysis of Study Results
The results of the study were analyzed by volunteers from Statistics Without Borders, in accordance with the analysis plan composed before the study was undertaken. Here, we summarize the most important findings.
Leaflets Received
As in the pre-analysis plan, participants were categorized by whether they reported receiving an experimental leaflet (the leaflet produced by Vegan Outreach that was distributed at their school), a control leaflet about puppy mills only, or no leaflet. This categorization placed 123 people in the experimental group, 23 in the control leaflet group, and 477 in the no leaflet group. Because this resulted in an extremely small control group, further analysis compared all three groups to better estimate effect sizes.
Because we knew which leaflets had been distributed at each school, we were also able to identify a total of 101 people who reported receiving leaflets not known to be distributed at their school during the semester in question. Some of these participants may have received the leaflets they reported during the time frame indicated in the question through means we did not account for. However, this raises significant doubts about the categorization of participants into groups in this and other studies using self-report to determine exposure to outreach materials.
Conversions to Vegetarianism
Large studies have shown frequent misreporting of vegetarianism when respondents are asked to indicate whether they are vegetarian or consume a vegetarian diet, as compared to when they are asked whether they consume specific meat products. Response patterns to our survey were similar to those on other surveys, with about 45% of respondents who indicated they were currently vegetarian or vegan also indicating that they ate one or more of red meat, poultry, or fish and seafood. We therefore used the questions about frequency of consumption of particular foods as our indicators of conversion to vegetarianism or veganism. Specifically, we considered the number of people who marked that they consumed a type of food three months before being surveyed, but had stopped by the time of the survey.
Of the 123 people in the experimental group, 7 had stopped eating red meat and 5 had stopped eating poultry at the time of the survey that had been eating these meats 3 months prior. This was statistically significant compared to the group that received no flyer, but not compared to the (very small) group that only received a control flyer. In fact, when looking at changes in who “never” ate a particular type of meat, the two non-experimental groups combined had one more person eating red meat at the end of the study period than at the beginning, and the number eating poultry in those groups remained constant. Results for fish did not appear to depend on which flyer, if any, was received.
Overall Meat Reduction
We also considered whether respondents had increased or decreased the frequency with which they consumed the listed meats. Two methods were used. Respondents had indicated how frequently they ate each type of food three months prior to the survey and at the time of the survey. The first method compared these frequencies for each respondent, categorizing respondents by whether their consumption increased, decreased, or remained the same. The proportion of respondents in each of these groups was not found to vary by whether the respondents had received a flyer. The second method looked at the size of the increase or decrease, measured by the number of times per week that the respondent consumed a particular food. This was also not found to depend on whether the respondent had received a flyer.
Previous surveys have found that a large part of the effectiveness of interventions is in decreases in meat consumption by respondents who do not stop eating meat altogether. While we did not find that meat consumption decreased more for respondents who reported receiving a leaflet, we did find an overall reduction in reported meat consumption throughout the whole sample. These results suggest that decreases found in other studies may simply be part of a background pattern and not attributable to any particular intervention. For instance, it could be the case that people in general are more likely to believe they are reducing their meat consumption than increasing it at any given time. Or college students might really eat less meat at the end of the semester than at the beginning, as they get bored of their original choices in the dining hall.
Conclusions
This study provides weak evidence that leafleting is effective in convincing college students to stop eating red meat and poultry. Because our efforts to provide a relevant control group were not fully successful, it is difficult to estimate the effect size, but it is probably bounded above by 5%. This study does not support claims that leaflets also cause incremental changes in meat consumption by recipients who do not stop eating one or more types of meat entirely. More detailed analysis will be coming later this week. [Ed.: The full analysis and dataset is now available.]
About Allison Smith
Charity Feedback from 2022 Evaluations
Highlights From Our 2022 Reddit AMA
Our Approach to Assessing Programs
Nice work!
What does the 5% number relate to? Meat consumption fell <5%? as measured in pounds or animals?
The 5% number is a suggested upper bound on the percentage of people who will stop eating a given type of meat (red meat or poultry) in response to receiving a leaflet.
Our estimated effect size in terms of pounds or animals consumed by the overall population is close to 0%, since when we looked at overall reduction we found no association between people receiving a leaflet and the amount their consumption was reduced by. This doesn’t contradict the other finding, because so many more people didn’t quit eating any type of meat than did that they dominate the reduction calculation. Also, intuitively, the people who quit eating meat may have been eating disproportionately little of it to begin with.
Is it possible to make the dataset publicly available?
The full analysis and dataset is now available.
We plan to make the dataset publicly available with the expanded analysis.
This is really disappointing to read
While we’ve found that leafleting has less of an effect than some people would expect, it’s important to keep in mind that we did find some positive results. In fact, the results we found still suggest that leafleting is about as effective as we’d conservatively estimated it to be before our study. We haven’t changed our Leafleting Impact Calculator yet, but because we only used the results from a previous study that had to do with people who gave up certain animal products entirely, we still think it gives reasonable results. (We’re not so confident about the implications it describes for fish, laying hens, and dairy cattle, but a large majority of the impact it takes into account is from red meat and poultry consumption, and we think those results are about right, taking in account both this study and what we knew beforehand.)
We were also a bit disappointed that we didn’t get a larger control group of people who’d taken the flyers that weren’t about farm animals. But survey research is complicated to do, and we’re heartened by the fact that the large control group was well-matched with the experimental group on factors we know are related to diet change like gender, age, and how many people reported being vegetarian before the point when flyers were distributed.