“Survey respondents who report having received either Vegan Outreach leaflet, even if they report also having received the control leaflet, will be considered part of the experimental group. Respondents who report only having received the control leaflet will be considered part of the control group. Respondents who do not report having received any leaflet we distributed will not be considered part of either group.
The primary goal of analysis will be to determine whether rates of conversion to vegetarianism and veganism during the study period are higher among subjects in the experimental group than among subjects in the control group. Depending on the final survey design, the analysis may also consider whether respondents in the experimental group have reduced their rates of meat and animal product consumption more than respondents in the control group, even among those respondents who do consume these products at the time of the survey.” — ACE pre-analysis plan
Analysis of Study Results
The results of the study were analyzed by volunteers from Statistics Without Borders, in accordance with the analysis plan composed before the study was undertaken. Here, we summarize the most important findings.
As in the pre-analysis plan, participants were categorized by whether they reported receiving an experimental leaflet (the leaflet produced by Vegan Outreach that was distributed at their school), a control leaflet about puppy mills only, or no leaflet. This categorization placed 123 people in the experimental group, 23 in the control leaflet group, and 477 in the no leaflet group. Because this resulted in an extremely small control group, further analysis compared all three groups to better estimate effect sizes.
Because we knew which leaflets had been distributed at each school, we were also able to identify a total of 101 people who reported receiving leaflets not known to be distributed at their school during the semester in question. Some of these participants may have received the leaflets they reported during the time frame indicated in the question through means we did not account for. However, this raises significant doubts about the categorization of participants into groups in this and other studies using self-report to determine exposure to outreach materials.
Conversions to Vegetarianism
Large studies have shown frequent misreporting of vegetarianism when respondents are asked to indicate whether they are vegetarian or consume a vegetarian diet, as compared to when they are asked whether they consume specific meat products. Response patterns to our survey were similar to those on other surveys, with about 45% of respondents who indicated they were currently vegetarian or vegan also indicating that they ate one or more of red meat, poultry, or fish and seafood. We therefore used the questions about frequency of consumption of particular foods as our indicators of conversion to vegetarianism or veganism. Specifically, we considered the number of people who marked that they consumed a type of food three months before being surveyed, but had stopped by the time of the survey.
Of the 123 people in the experimental group, 7 had stopped eating red meat and 5 had stopped eating poultry at the time of the survey that had been eating these meats 3 months prior. This was statistically significant compared to the group that received no flyer, but not compared to the (very small) group that only received a control flyer. In fact, when looking at changes in who “never” ate a particular type of meat, the two non-experimental groups combined had one more person eating red meat at the end of the study period than at the beginning, and the number eating poultry in those groups remained constant. Results for fish did not appear to depend on which flyer, if any, was received.
Overall Meat Reduction
We also considered whether respondents had increased or decreased the frequency with which they consumed the listed meats. Two methods were used. Respondents had indicated how frequently they ate each type of food three months prior to the survey and at the time of the survey. The first method compared these frequencies for each respondent, categorizing respondents by whether their consumption increased, decreased, or remained the same. The proportion of respondents in each of these groups was not found to vary by whether the respondents had received a flyer. The second method looked at the size of the increase or decrease, measured by the number of times per week that the respondent consumed a particular food. This was also not found to depend on whether the respondent had received a flyer.
Previous surveys have found that a large part of the effectiveness of interventions is in decreases in meat consumption by respondents who do not stop eating meat altogether. While we did not find that meat consumption decreased more for respondents who reported receiving a leaflet, we did find an overall reduction in reported meat consumption throughout the whole sample. These results suggest that decreases found in other studies may simply be part of a background pattern and not attributable to any particular intervention. For instance, it could be the case that people in general are more likely to believe they are reducing their meat consumption than increasing it at any given time. Or college students might really eat less meat at the end of the semester than at the beginning, as they get bored of their original choices in the dining hall.
This study provides weak evidence that leafleting is effective in convincing college students to stop eating red meat and poultry. Because our efforts to provide a relevant control group were not fully successful, it is difficult to estimate the effect size, but it is probably bounded above by 5%. This study does not support claims that leaflets also cause incremental changes in meat consumption by recipients who do not stop eating one or more types of meat entirely. More detailed analysis will be coming later this week. [Ed.: The full analysis and dataset is now available.]