ACE Highlight: Updated Leafleting Intervention Report

Leafleting is a common intervention in farmed animal advocacy that is carried out by many organizations. In 2015, Vegan Outreach alone reported distributing close to 3 million leaflets,¹ most of which were handed out on college campuses.² In order to evaluate the effectiveness of leafleting, ACE completed an intervention report on leafleting in 2014. However, the quickly growing field of animal advocacy research—as well as a decline in our confidence regarding the findings from some of the previously cited studies—gave us good reason to believe that the 2014 report no longer accurately reflected the state of the evidence regarding leafleting, nor our current views on it.³ In fact, since December 2016 we’ve noted that it was a mistake to wait as long as we did to update our 2014 leafleting intervention report and to fail to provide sufficient disclaimers in the meantime.

We have now published an updated leafleting intervention report, which incorporates a variety of methods not used in our previous report and reaches quite different conclusions. In particular, based on a meta-analysis combining data from several studies on the short-term effects of leafleting, we conclude that those trials don’t provide clear evidence that leafleting decreases recipients’ animal product consumption in the first few months after distribution. In fact, the results suggest leafleting is about as likely—or perhaps even more likely—to actually cause increases in animal product consumption during this time period. Other evidence⁴ also supports the conclusion that leafleting is probably less effective than some other promising farmed animal advocacy interventions.

This updated report improves on our previous leafleting report by including, among other things:

• A more systematic literature search
• A detailed assessment of the various risks of bias in the most pertinent animal advocacy studies of leafleting
• A meta-analysis and in-depth discussion of the limitations of those key animal advocacy studies
• A greater discussion of relevant historical evidence and literature from the social sciences
• The incorporation of model uncertainty and a Bayesian prior in our cost-effectiveness estimates
• Information from multiple case studies, as well as interviews with advocates who have a considerable amount of experience with leafleting

An input and some outputs from the the meta-analysis that seem noteworthy are:

• Each of the six randomized controlled field trials included in the meta-analysis was judged to have at least one domain in which the risk of bias was substantial enough to limit our confidence in its results.
• The 95% confidence interval for each of the summary estimates from the meta-analysis overlaps with a point estimate of an effect size of zero. By following the conventionally applied frequentist framework of statistical inference, we would fail to reject the null hypothesis in all cases. In other words, we would not reject the hypotheses that leaflets have no impact on consumption of any of the particular animal products included in the meta-analysis (namely red meat, poultry, fish, eggs, and dairy).
• In most cases, the meta-analysis estimates of change in animal product consumption suggest that leaflets are more likely to cause increases in animal product consumption than decreases in animal product consumption. We think these results support the case that leaflets are likely not very effective; however, we do not think they should be taken as robust estimates of the effects of leafleting.

While the meta-analysis completed in the report was crucial for evaluating the evidence from the particularly relevant field trials and had a large impact on the report overall, it certainly wasn’t the only source of relevant information in the report. Some other points from the updated report that seem noteworthy are:

• If leaflets do have a significant favorable effect on short-term diet changes, then it seems that there is reason to expect that online ads would be more cost-effective than leaflets.
• It is unclear to us just how large a role leaflets have played in social movements where they have been employed. Our limited impression is that relevant Researchers generally don’t seem to think that leaflets played a large role in most social movements, and that leaflets have not been widely considered as an essential part of any other contemporary social movement’s recent work.
• Our cost-effectiveness estimate for leafleting, which is highly uncertain and should be interpreted carefully, had 90% subjective confidence intervals of a decrease in supply of 3 to an increase in supply of 10 farmed animals and a corresponding change of -2 to 2 farmed animal years per dollar spent on leafleting.

Readers are encouraged to refer to the full report to further understand ACE’s current views and reasoning about the effectiveness of leafleting.

1. In 2015, Vegan Outreach sent out 3.3 million leaflets, about 80% of which (i.e., about 2.6 million) were reported as being handed out. This information can be found in ACE’s Conversation with Jack Norris of Vegan Outreach (2016). 

2. In 2015, Vegan Outreach’s Adopt a College program distributed 2,367,516 leaflets total on various school campuses. This number includes high school campuses, although our impression is that the majority of these leaflets were distributed at colleges; for example, none of the program’s 40 biggest school leafleting events for 2015 took place at high schools. For the source of this information, see the Leafleting Statistics page on the Adopt a College website. 

3. Some of our further thoughts on the problems with our previous leaflet report are available in the following blog posts:

   • Responses to common critiques
   • Response to a recent critique of our research

    

4. For instance, interviews of animal advocates with experience leafleting and reasoning about the comparative advantages of different farmed animal advocacy interventions both support this conclusion. See Part Five and Part Three of the report, respectively, for further discussion of what these sources of information can tell us about the effectiveness of leafleting.