Overview

Our 2017 leafleting intervention report is guided by our recently updated intervention evaluation process, which provides a new framework for using multiple sources of evidence to evaluate an intervention’s effectiveness. As such, this report incorporates a variety of methods not used in our previous report, as well as the results of several more recent studies, and reaches quite different conclusions. In particular, based on a meta-analysis combining data from several studies on the short-term effects of leafleting, we conclude that those trials don’t provide clear evidence that leafleting decreases recipients’ animal product consumption in the first few months after distribution. In fact, the results suggest leafleting is about as likely—or perhaps even more likely—to actually cause increases in animal product consumption during this time period. Other evidence¹ also supports the conclusion that leafleting is probably less effective than some other promising farmed animal advocacy interventions.

This updated report improves on our previous leafleting report by including, among other things:

• A more systematic literature search
• A detailed assessment of the various risks of bias in the most pertinent animal advocacy studies of leafleting
• A meta-analysis and in-depth discussion of the limitations of those key animal advocacy studies
• A greater discussion of relevant historical evidence and literature from the social sciences
• The incorporation of model uncertainty and a Bayesian prior in our cost-effectiveness estimates
• Information from multiple case studies, as well as interviews with advocates who have a considerable amount of experience with leafleting

An input and some outputs from the the meta-analysis that seem noteworthy are:

• Each of the six randomized controlled field trials included in the meta-analysis was judged to have at least one domain in which the risk of bias was substantial enough to limit our confidence in its results.
• The 95% confidence interval for each of the summary estimates from the meta-analysis overlaps with a point estimate of an effect size of zero. By following the conventionally applied frequentist framework of statistical inference, we would fail to reject the null hypothesis in all cases. In other words, we would not reject the hypotheses that leaflets have no impact on consumption of any of the particular animal products included in the meta-analysis (namely red meat, poultry, fish, eggs, and dairy).
• In most cases, the meta-analysis estimates of change in animal product consumption suggest that leaflets are more likely to cause increases in animal product consumption than decreases in animal product consumption. We think these results support the case that leaflets are likely not very effective; however, we do not think they should be taken as robust estimates of the effects of leafleting.

While the meta-analysis completed in the report was crucial for evaluating the evidence from the particularly relevant field trials and had a large impact on the report overall, it certainly wasn’t the only source of relevant information in the report. Some other points from the updated report that seem noteworthy are:

• If leaflets do have a significant favorable effect on short-term diet changes, then it seems that there is reason to expect that online ads would be more cost-effective than leaflets.
• It is unclear to us just how large a role leaflets have played in social movements where they have been employed. Our limited impression is that relevant Researchers generally don’t seem to think that leaflets played a large role in most social movements, and that leaflets have not been widely considered as an essential part of any other contemporary social movement’s recent work.
• Our cost-effectiveness estimate for leafleting, which is highly uncertain and should be interpreted carefully, had 90% subjective confidence intervals of a decrease in supply of 3 to an increase in supply of 10 farmed animals and a corresponding change of -2 to 2 farmed animal years per dollar spent on leafleting.

Readers are encouraged to refer to the full report to further understand ACE’s current views and reasoning about the effectiveness of leafleting.

---

1. For instance, interviews of animal advocates with experience leafleting and reasoning about the comparative advantages of different farmed animal advocacy interventions both support this conclusion. See Part Five and Part Three of the report, respectively, for further discussion of what these sources of information can tell us about the effectiveness of leafleting. 

View Online Download PDF

Report

The organization of this report may differ slightly from the one described in our intervention evaluation guide, as the two projects were developed concurrently.

Table of Contents

• Overview
• Report
  • Part One: Intervention description and a brief outline of a theory of change
  • Part Two: Evidence from particularly relevant field randomized controlled trials
  • Part Three: Some reasoning about the effectiveness of leaflets as an animal advocacy intervention
  • Part Four: Case study analysis / Cost-effectiveness estimate
  • Part Five: Interviews in the field
• Conclusions

Part One: Intervention description and a brief outline of a theory of change

The animal advocacy intervention of leafleting is a form of individual outreach. It is primarily carried out on college campuses, but to an extent it is also done at concerts and other large social events where the attendees are either teenagers or young adults, and/or liberal/progressive leaning.¹ Leafleters usually say a short sentence² while offering leaflets³ to many of the people who pass by. Leaflets are usually 5–20 page printed documents that attempt to persuade individuals to avoid or reduce animal product consumption.⁴ Leafleters can be paid staff of one of various animal advocacy organizations, or they can be volunteers. Leafleting is a common animal advocacy intervention used by various organizations, and several million animal advocacy leaflets are reportedly handed out in the U.S. annually. We would guess that most animal advocacy leaflets handed out in the U.S. are designed, printed, and handed out by Vegan Outreach. In 2015, Vegan Outreach reported distributing almost 3 million leaflets each year, most of which were handed out on college campuses.^{5, 6}

This intervention report focuses on the effectiveness of these typical animal advocacy leaflets⁷ when claimed best practice approaches⁸ to leafleting are used. This means that this intervention evaluation does not directly consider the effectiveness of leaflets that only focus on quite specific aspects of animal advocacy such as limiting dairy consumption or chicken consumption, nor does it directly address leaflets that only focus on antispeciesism or wild animal suffering.

The following table offers a very brief theory of change for typical animal advocacy leaflets in the short to medium term.

Table 1. Some of the more likely possible short and medium term outcomes from typical animal advocacy leaflets

Some of the more likely possible outcomes in the short term9

Leaflet recipients change their consumption of animal products. This may cause a slight increase in consumption of animal product alternatives and/or animal products sourced from operations which claim to meet higher welfare standards. Leaflet recipients change their attitudes towards the treatment of animals. This may cause a slight increase in consumption of animal product alternatives and/or animal products sourced from operations which claim to meet higher welfare standards. Leafleting contributes to the growth of the animal advocacy movement by providing a low-commitment, unspecialized way to begin advocating for animals.

Some of the more likely possible outcomes in the medium term10

Changes in consumption caused by leaflets may cause supermarkets to increase or decrease their supply of animal product alternatives and/or animal products sourced through claimed higher welfare programs. Leaflets may positively contribute to the growth of the animal advocacy movement by aiding in the causal chain that results in some people, including future movement leaders, joining the animal advocacy movement who otherwise wouldn’t have done so.11 Leaflets may contribute to increased public awareness of factory farming and/or increased quantity of desirable opinions pertaining to farmed animal advocacy issues (which causes other promising farmed animal advocacy interventions to be more likely to succeed). E.g., Corporate campaigns/outreach and ballot measures may become more likely to succeed. There could also be greater consumer acceptance of cultured animal products.

Part Two: Evidence from particularly relevant field randomized controlled trials

Methods and outcomes of literature search

Before completing this intervention report, we already knew of multiple studies from the animal advocacy community pertaining to the effectiveness of typical animal advocacy leafleting. This knowledge came from the general research we had completed over the past few years, as well as from conversations we had with leaders in the field. While completing the research required for this report, we became aware of some other evidence and research through a moderate literature search on Google Scholar in mid-late June 2017.¹² Numerous pieces of identified evidence were excluded from further analysis because we thought that the interventions that they provided information about were different enough from those we studied that the interpretation of these results wouldn’t lead to meaningful updates about the effects of typical animal advocacy leafleting.¹³ Furthermore, including results produced by significantly different interventions in our leafleting meta-analysis could meaningfully distort our estimate of the effects of leafleting. In addition, one study conducted by two collaborating organizations was reported twice, once by each organization.¹⁴ A proposal for further study was identified that, as far as we know, isn’t going to be implemented in the near future.¹⁵ We also know of some relevant forthcoming studies that could further inform our views about the effectiveness of leafleting.¹⁶

An overview of particularly relevant field randomized controlled trials

Table 2 provides an initial overview of particularly relevant field randomized controlled trials. View the spreadsheet summary of findings.

Table 2 includes key information about aspects such as sample size, question wording, and food groups included in the assessment of animal product consumption, as well as sample characteristics. Before assessing the results of these studies we will first quickly assess the risk of potential biases in the studies. We didn’t commit ahead of time as to how this review would assess or deal with bias¹⁷ and the project leader wasn’t blinded to the authors or the institutions associated when assessing the risk of bias in these studies.¹⁸ It is important to assess the risk of bias in studies because it is a threat to their internal¹⁹ and external validity.²⁰ Less biased studies are more likely to yield results that are closer to the truth and more biased studies are less likely to yield results that are closer to the truth. At times it was difficult for us to assess the risk of bias associated with studies because of the suboptimal amount of relevant information provided by the reports.²¹ Still, enough information was available that useful bias assessments could be completed. To perform these assessments, we used the Cochrane Risk of Bias tool, which involves assessing studies’ risk of being affected by six categories of bias: (i) selection bias,²² (ii) performance bias,²³ (iii) detection bias,²⁴ (iv) attrition bias,²⁵ (v) reporting bias,²⁶ and a catch-all item called “other sources of bias.” The presence of any of the first five items in a randomized trial has been found to often bias the resulting estimates of an intervention’s effectiveness.²⁷ The bias assessments in each of those domains for the individual studies we analyzed, along with brief justifications, are available here. A summary of the bias assessments is presented in Table 4 and information for interpreting Table 4 is shown in Table 3.

Table 3. Legend for summary of bias assessments table

Symbol	Interpretation in terms of risk of bias28, 29
	Low risk of bias (i.e., plausible level of bias unlikely to seriously alter the results.)
	Unclear risk of bias (i.e., plausible bias that raises some doubt about the results)
	High risk of bias (i.e., plausible bias that seriously weakens confidence in the results)

Table 4: Summary of bias in main leafleting studies

Study	Selection Bias	Performance Bias	Detection Bias	Attrition Bias	Reporting Bias	Other Bias
Humane League Labs (2014) What elements make a vegetarian leaflet more effective?
Animal Charity Evaluators (2013) 2013 ACE Leafleting Study
Humane League Labs (2015) Which request creates the most diet change?
Hennessy (2016) The Impact of Information on Animal Product Consumption
Flens et al (2017) The Effectiveness of Leafleting on Reducing the Consumption of Animal Products in Dutch Students
Animal Equality Spain (2014)

As can be seen in Table 4, each of the studies was judged to have at least one domain in which there was a risk of bias substantial enough that our confidence in its results is very limited.

A meta-analysis of the six particularly relevant field randomized controlled trials

The data for this meta-analysis was extracted from the raw data of the six particularly relevant randomized controlled trials (RCTs) by the project leader.³⁰ All else equal, pooling results of studies is likely to increase the precision of the estimate of the effect being studied (i.e., pooling results seems likely to narrow confidence intervals), and results with higher precision can better guide advocacy. Meta-analysis techniques involve pooling results by weighing studies according to the amount of information they contribute (more specifically, by the inverse variances of their effect estimates). This gives studies with greater statistical power more weight in the aggregated estimate, though it doesn’t account for the possible different biases and the extent of these biases in the results of each study.

In this meta-analysis, missing data from participants was ignored. Some measures were in interval censored form, providing information that an individual’s consumption of a product was within a given range of values, but not specifying a precise level of consumption. These variables were transformed to continuous outcomes by an algorithm which equally extended adjacent intervals such that there were no gaps between them and then computed the midpoint of the extended intervals to estimate the average value of that interval.^{31, 32} Other ordinal measures were treated as interval measures,³³ and the clustering of participants was ignored.³⁴ The meta package in R was used to create the below forest plots; the R code for generating the forest plots or for the extraction of data from individual studies is available upon request. There were also a number of specific data-aggregation decisions that were relevant to the interpretation of individual studies.³⁵

Most of the below forest plots show confidence intervals for Hedges’ g estimates of the standardized difference between the control group and the treatment group’s mean self-reported consumption of dairy, poultry, red meat, eggs, and fish. Standardized differences in means are used when studies do not yield directly comparable data, such as when studies assess the same outcome³⁶ with different measures.³⁷ Hedges’ g calculates the difference between the experimental and control groups, divided by the pooled standard deviation of these groups. Since in this case the standard deviations are generally about the same as mean consumption, thinking of our standardized mean differences (SMDs) as being roughly equal to the percent difference between the control and treatment groups’ means gives a general sense of the magnitude of the estimated effects. The endpoints of the lines for each study in the below forest plots indicate the 95% confidence intervals of the standardized effect estimate, which estimates the difference in animal product consumption between leaflet recipients and control group members—we would expect a negative effect estimate if leaflets are associated with reduced consumption of a given animal product, and a positive effect estimate if they are associated with increased consumption. The statistical power of each study is indicated by the size of the box around the point estimate of the effect size; bigger boxes correspond to studies with greater power, whose results are more meaningful and hence are weighted more heavily in the meta analysis. The diamond summary estimates (fixed effects model and random effects model) aggregate the results from the individual trials into an estimate and 95% confidence interval by accounting for both the differing estimated effect sizes and the differing statistical power of the experiments.

Table 5. Meta-analysis consumption results from the fixed-effects and random-effects models

Animal Product	Estimated standardized mean difference from meta-analysis as 95% CI (fixed effects model)	Estimated standardized mean difference from meta-analysis as 95% CI (random effects model)
Red meat41	[-0.03, 0.14]42	[-0.03, 0.14]43
Poultry44	[-0.06, 0.12]	[-0.07, 0.16]
Fish45	[-0.10, 0.07]	[-0.12, 0.10]
Eggs	[-0.10, 0.07]	[-0.10, 0.07]
Dairy	[-0.07, 0.11]	[-0.07, 0.11]

The fixed effects model is based on the assumption that the differences between the studies’ findings are due to sampling error.⁴⁶ In contrast, the random effects model allows for the possibility that the true effect size differs from study to study in part because of real differences in the effect of the treatment in different circumstances, also known as study heterogeneity.^{47, 48} Our understanding is that there is no consensus on which model is preferable to use, but there is some evidence that fixed effects models overestimate confidence in treatment effects more than random effects models do, and that at least some Researchers believe there are only very limited circumstances in which a fixed-effects model is appropriate.⁴⁹ Since there were potentially-significant differences in the circumstances and methodology of these studies, we will refer to the random effects model estimate when considering the results of this meta-analysis.

One noticeable result from this meta-analysis is that the 95% confidence interval for all the summary estimates overlaps with a point estimate of an effect size of zero. By following the conventionally applied frequentist framework of statistical inference, we would fail to reject the null hypothesis in all cases, and so would not reject the hypotheses that leaflets have no impact on consumption of any of these particular animal products. Another noticeable result from this meta-analysis is that in most cases the summary estimate is positive, suggesting that leaflets are more likely to cause increases in animal product consumption than decreases in animal product consumption. That is, in most cases it seems that the majority of the probability density function for the estimated effect of leaflets on animal product consumption suggests that leaflets cause increases in animal product consumption. This is the case for dairy, poultry, and red meat in both the fixed-effects and random-effects models, as well as for fish in the fixed-effects model, though (as mentioned above) the estimate is imprecise in all cases, and all of our results are consistent with leaflets having no effect on consumption.

The following forest plot shows confidence intervals for the effect size estimate on lacto-vegetarian⁵⁰ prevalence. For binary outcomes, effect sizes are often expressed as an odds ratio. In this case, the odds ratio represents the odds an individual in the treatment group would report a lacto-vegetarian diet divided by the odds that a participant in the control group would report a lacto-vegetarian diet. An odds ratio significantly greater than one indicates that, relative to assignment to the control group, assignment to the treatment group increases the probability of a participant reporting a lacto-vegetarian diet.

Table 6. Odds Ratio for Estimated Effect of Leafleting on lacto-vegetarian prevalence⁵²

Model	Estimate	95% Confidence Interval
HLL (2014)	0.36	0.09-1.45
ACE (2013)	1.79	0.59-5.47
HLL (2015)	1.16	0.54-2.48
Hennessy (2016)	0.59	0.27-1.29
Flens et al (2017)	0.71	0.04-13.6
Fixed Effects Model	0.86	0.55-1.36
Random Effects Model	0.85	0.50-1.46

Again, one noticeable result from the meta-analyses pertaining to lacto-vegetarian prevalence is that the 95% confidence interval for all the summary estimates overlaps with a point estimate of an effect size of zero.⁵³ By following the conventionally applied frequentist framework of statistical inference, we would fail to reject the null hypothesis in all cases, and so would not reject the hypothesis that leaflets have no impact on lacto-vegetarian prevalence. Another noticeable result from this meta-analysis that the summary estimate suggests that leaflets seem more likely to cause decreases in lacto-vegetarian prevalence than increases in lacto-vegetarian prevalence. That is, it seems that the majority of the probability density function for the estimated effect of leaflets on lacto-vegetarian prevalence suggests that leaflets cause decreases in lacto-vegetarian prevalence.

Differences in the estimate of the effect of leafleting on different animal products and/or lacto-vegetarian prevalence seem to be too uncertain and not large enough to be worth discussing in detail. Given the high risk of bias in all the studies included in this meta-analysis, we think it is likely that the summary estimate 95% confidence intervals are not accurate 95% confidence intervals. There is some evidence that trials at high risk of bias tend to overestimate treatment effects (Moher et. al, 2010, Schulz 1995b, Odgaard-Jensen 2010) and we would guess that a similar conclusion would hold for the present meta-analysis but are somewhat uncertain about it. Several severe limitations to the available particularly relevant randomized controlled trials—including the limited scope of effects studied, the risk of several forms of bias, and possible variations in how intervention was carried out—further contribute to our uncertainty about these summary estimates. Furthermore, these confidence intervals don’t take into account our uncertainty in model selection during the meta-analysis and our uncertainty about which trials to include in the meta-analysis. As frequentist-style confidence intervals, these intervals don’t use Bayesian reasoning that incorporates our prior thoughts on the effectiveness of leafleting, which intervals representing our current beliefs on the subject arguably ought to do.⁵⁴ For all these reasons, we treat these confidence intervals as very tentative estimates that need to be further updated to better estimate the effect of leaflets. (We estimate leaflets’ cost-effectiveness, and adjust for some external concerns, in Part Four of this report.) Despite our uncertainty, we do think that these estimates give relatively good information about the order of magnitude of the effect we should expect.

Some further elaboration on the limitations to the evidence from the particularly relevant field randomized controlled trials

There are numerous further limitations to the available evidence from the particularly relevant randomized controlled field trials apart from the previously mentioned risks of bias. The main additional limitations seem to be:

• The possibility of publication bias
• Possibly important differences in the way in which the intervention was implemented in the trials and the way in which it is usually implemented by animal advocacy organizations
• The evidence from the field trials still allows for significant uncertainty regarding the effect of leaflets on a variety of important outcomes (e.g., welfare reforms, ballot measures, demand for cultured animal products, and demand for animal products sourced through higher welfare methods)

We now briefly discuss each of these points in turn.

The set of available field trials may be constrained by publication bias among the groups who conduct such trials, which would make results that indicate that leaflets decrease animal product consumption more likely to be published than results that indicate that leaflets have no effect on, or cause increases in, animal product consumption. For instance, Animal Equality’s Spain and U.K. studies’ results have only recently been made public, several years after the experiments were performed. We would speculate that these delays in dissemination are at least in part because the results of the trials didn’t indicate that leaflets caused a significant decrease in animal product consumption (a result that the organizations involved had at least some incentive to demonstrate), and note there is some empirical evidence of a similar phenomenon in other fields (Guyatt et al, 2011). We know of at least some cases in which an organization chose not to formally analyze results from a leafleting study, though these may have been due to low response rates and hence low sample sizes rather than low effect sizes.⁵⁵ There are also certain characteristics of leafleting trials that seem to increase the risk of publication bias. For instance, all of the studies are small in scale and appear to have been funded by the leafleting “industry”—i.e., by organizations that carry out or support leafleting. Meta-analyses that exclude unpublished results are at risk of overestimating intervention effects (Higgins & Green, 2011). ACE research staff think it is unlikely that other randomized controlled field trials of leafleting exist—this is because, due to our contacts at most animal advocacy organizations that conduct research, it is likely that we would know of these trials if they did exist.⁵⁶ Still, it is possible that some trials were excluded because of a publication bias. If that were true, it would seem to make it more likely that this meta-analysis overestimates the possibility that leaflets cause a decrease in animal product consumption.

There may be a number of important ways that the manner in which the leafleting was performed in these highly relevant randomized controlled field trials differs from the way in which animal advocacy leafleting is usually performed. For instance, the leafleting in studies may be different than leafleting implemented by some large animal advocacy organizations, because these organizations often attempt to leaflet approximately 10% or 15% of a campus,⁵⁷ while the studies we consider in the meta-analysis may have aimed to reach a different proportion—more likely a smaller proportion—of students on campus. The leafleters involved with the studies may also be less experienced than those who usually leaflet in the field. They may also give different instructions to recipients than they normally would.⁵⁸ It is also possible that the leaflets themselves were meaningfully different in some way from the leaflets that are usually used in the field.⁵⁹ Differences in how well leafleting was received in different areas could also impact studies that are trying to generalize about the absolute efficacy of leafleting.

There are also a number of important outcomes that these trials don’t measure. For instance, the trials don’t measure outcomes such as support for various actions that are likely to help animals, including welfare reforms, ballot measures, demand for cultured animal products, and demand for animal products sourced through higher welfare methods. There is also the possibility that leaflet recipients could share leaflets with people who weren’t included in the study and those people may have changed their behavior. Furthermore, the self-reported consumption measures used in these trials may meaningfully differ from actual consumption.⁶⁰ More fundamentally, the outcome of interest is actually the change in the number of animals supplied through industrial agriculture rather than the change in the number of animals demanded.⁶¹ (Though even that change in supply may importantly differ from morally desirable outcomes.)⁶² Anecdotally, several major figures in the animal advocacy movement were reportedly converted by leaflets,⁶³ which—if true—suggests leafleting may have important effects on building the animal advocacy movement. There are also some possible positive effects of leaflets on movement building, and perhaps some examples of leaflets being directly associated with large-scale outcomes that seem quite positive.⁶⁴ The particularly relevant randomized controlled field trials also don’t account for how the effects of other interventions may impact the effectiveness of leafleting.^{65, 66} Given the lack of significant decreases in short-term animal product consumption detected by the meta-analysis, we lower our estimate of the likelihood that leaflets cause significant changes in recipients’ opinions or dietary behaviors.

Summary

Our analysis of evidence from animal advocacy had a number of Researcher degrees of freedom⁶⁷ and was completed in a limited amount of time.⁶⁸ When applicable, high-power and high-quality randomized controlled trials provide the most reliable evidence on the effect of animal advocacy interventions. However, in this case all of these trials were judged to be at high risk of at least one type of bias, and we would guess that this means that they are probably more likely to overestimate the decrease in animal product consumption caused by leaflets than to underestimate that effect (Moher et. al, 2010, Schulz 1995b, Odgaard-Jensen 2010). There is a risk that the overall evidence body is tainted by a publication bias, which again seems more likely to lead to overestimating the effects of leaflets, since a majority of these studies were conducted by organizations engaged in leafleting. There are also issues with the implementation of leaflets in the trials likely being at least slightly different from usual practice. Lastly, the results of the trials simply do not speak to the effects of leaflets on a number of important outcomes.

Overall, we have very little confidence in the point estimate of leaflets’ effect on consumption of various animal products. The estimated effects seem very likely to be somewhat inaccurate, possibly in a way that significantly affects their implications for our position on leaflets.⁶⁹ Most of the potential biases in our data, such as publication bias across leafleting analyses published by animal advocacy organizations, seem more likely to lead us to underestimate than to overestimate the change in animal product consumption associated with leaflets. Thus, the lower bounds of our confidence intervals, corresponding to bounds on the amount by which leaflets might correlate with a decrease in animal product consumption, seem more likely to provide a reasonable bound on the effect size than the upper bounds. If the body of data we are using is indeed biased toward leaflets, it seems likely that there will actually be a greater than 2.5% chance that the true effect increases consumption by more than the upper bound of our 95% confidence interval. With the available evidence, we certainly can’t reject the null hypothesis that leaflets have no effect on short-term consumption of animal products. If anything, our meta-analysis of the available evidence instead causes us to update towards thinking that leaflets may actually cause short-term increases in animal product consumption rather than decreases. For those reasons we think that, as a body, the available evidence from particularly relevant randomized controlled trials seems to very poorly (if at all) support the case for allocation of resources towards leafleting, either in absolute terms or when it is compared to other promising farmed animal advocacy interventions.

Poor	Weak	Moderate	Strong
There is little to no evidence to support this choice of intervention. Or, the evidence suggests an intervention may have no effect or a negative impact.	There is weak evidence to support this intervention but it is either exploratory in nature, weak in effect, or the studies are of low quality.	There is moderate evidence to suggest this choice of intervention.	There is strong, high quality evidence to support this choice of intervention.

Part Three: Some reasoning about the effectiveness of leaflets as an animal advocacy intervention⁷⁰

To aid in our evaluation of leafleting, we now very briefly examine:

• Possible places where leafleting has a comparative advantage over other animal advocacy interventions
• Some relevant social movement evidence
• Some possible macro indicators of leaflet effectiveness
• The effectiveness of individual outreach interventions in other areas which seem analogous to leafleting
• Some speculative reasoning about the long-term effects of leafleting

Possible places where leafleting has a comparative advantage over other animal advocacy interventions

Leafleting compares moderately favorably to other animal advocacy interventions in ease of scaling, as each additional leaflet is relatively cheap⁷¹ and large numbers of leaflets can be distributed by volunteers with little or no training. The fact that little or no training is involved with distributing leaflets also means that distributing leaflets provides a way for interested individuals to easily become more involved in animal advocacy.⁷² It seems clear that spending on leafleting results in relatively large numbers of people being exposed to farmed animal advocacy, and that favorable dietary change on their part may well result, in expectation, in a large benefit for farmed animals. However, we are very uncertain about not only the extent to which dietary change occurs due to leaflets, but, increasingly, whether the short-term diet changes associated with leaflets are in fact positive.

Leaflets have some potential advantages over online ads: they are a physical piece of literature recipients can keep and show others; the leafleter interacts with the recipient and there is the opportunity for a meaningful conversation; people often see dozens of online ads daily but are rarely handed a leaflet. However, if leaflets do have a significant favorable effect on short-term diet changes, then it seems that there is reason to expect that online ads would be more cost-effective than leaflets. This is because the mechanism for change behind the two interventions is very similar (i.e., short exposure to persuasive informational content) and online ads do seem to have a variety of meaningful advantages over leaflets. As Cooney (2014) notes, online ads can more narrowly target demographics (e.g., age, gender, interests) that are most receptive to animal advocacy messaging, and they require fewer financial resources. Online ad campaigns also appear to require less volunteer time than leafleting, so if organizations switched from leafleting to online ads, their volunteers’ time might be able to be used on other activities. Another possible comparative advantage is that if the ad provider’s payment method is per click on the online ads, then the advertiser will usually only pay for clicks from those who have an interest in the content, as opposed to paying to give leaflets to a substantial number of people who don’t seem likely to be affected by them. Online ad campaigns can yield a large amount of data, which is now often available through sources like Google Analytics and Facebook Insights; advocates can use this large volume of feedback to better refine their efforts.⁷³ In addition, the cost per impression for online ads is substantially cheaper than that for leaflets, to the extent that, unless a leafleting impression is several orders of magnitude more effective than an online ad impression, it seems likely that online ads are more cost-effective.⁷⁴ We would also guess that the short-term effects of undercover investigations may be more promising than leafleting because of the weak correlational evidence suggesting that negative media coverage is associated with a decrease in animal product consumption.

Our limited impression is that leaflets don’t seem to compare very favorably in a number of important areas when compared to other promising farmed animal advocacy interventions. In short, however, we are still quite uncertain about this. It appears that in many important areas, there is another promising farmed animal advocacy intervention that is likely preferable to leafleting. For instance, as previously mentioned, if leaflets do have a significant favorable effect on short-term diet changes, then it seems that there is reason to think that online ads would be more cost-effective than leaflets. Based on the available evidence from the particularly relevant RCTs, the effects of leafleting that are more easily measurable don’t seem as promising as those effects of corporate outreach that are easily measurable. We would guess that leaflets have weaker effects on building the animal advocacy movement and increasing the likelihood of a major social shift away from factory farming than either (i) interventions featuring more in-depth materials, such as documentaries and books sympathetic to animal advocacy, or (ii) interventions more directly attempting to increase consumer acceptance of cell-cultured or plant-based animal product substitutes.

Some relevant social movement evidence

Given the time constraints involved with this intervention report, we only briefly examined some of the social movement evidence pertaining to leafleting, and we think that it has notable limitations.⁷⁵

Leaflets could be broadly categorized as an individual consumer-action approach to causing social change. Critics argue that such approaches are not supported by historical examples or other empirical evidence.^{76, 77} It appears that promoting the individual consumer approach of recycling was successful,⁷⁸ but this was carried out in large proportion by beverage corporations (Elmore, 2012), making the analogy to animal advocacy leafleting tenuous. We are also aware of some limited evidence of the use of leaflets in other social movements, such as the women’s rights, slavery abolition, and children’s rights movements.⁷⁹ However, we are unaware of any detailed analysis of the role that leaflets played in those movements. For this reason, and because inferring causality from history is inherently difficult, it is unclear to us just how large a role leaflets have played in social movements where they have been employed. Our limited impression is that:

• Relevant Researchers generally don’t seem to think that leaflets played a large role in most social movements.⁸⁰
• Leaflets have not been widely considered as an essential part in any other contemporary social movement’s recent work.

Some possible macro indicators of leaflet effectiveness

Another approach to evaluating the effectiveness of leaflets is through what GiveWell defines as “macro” evidence: “evidence from programs carried out on a large scale (regional, national, or multinational) without separating people into “treatment groups” and “control groups.”⁸¹ Leaflets and similar forms of outreach could have contributed to the current perception in the U.S. that farm animals are frequently mistreated for commercial gain,⁸² and/or they could have contributed to, or could be contributing to, the growing awareness of veganism.⁸³ There are other interventions which may have had more of an effect—investigations, for example, which often achieve broad media coverage—but it is difficult to tease out causation from observation of social trends.⁸⁴ There is also some reason to think that leaflets may have played an important part in building the animal welfare movement by generating publicity and attracting future movement leaders. However, that reasoning seems to suggest that investigations and influential books and/or documentaries played a larger role.^{85, 86} At one point in the animal advocacy movement’s history these individual outreach interventions may have been a good course of action, but it is now no longer clear whether these individual approaches are the optimal course of action. Indeed, a small survey of some effective animal advocacy Researchers seemed to indicate that there was a consensus favoring institutional approaches over individual approaches.⁸⁷

One could check calculations of the vegetarian conversion rate of leaflets against a back-of-the-envelope calculation that uses estimates of:

• The proportion of vegetarians
• The average vegetarian adherence length
• The number of leaflets handed out annually
• The proportion of the vegetarians created by leafleting⁸⁸

This doesn’t seem like a useful check for our cost-effectiveness estimates. This is because (i) we don’t focus on vegetarian conversions and (ii) we don’t have particularly good data for some of the important parameters in that calculation. Still, it is worth noting that even some quite preliminary models seem to relatively strongly conflict with high estimates of both vegetarian conversion (e.g., >5%) and adherence (e.g., >5 years) as a result of the leaflets. What may be more informative is looking at changes in vegetarianism and veganism rates in countries that Vegan Outreach (or other leafleting organizations) have recently expanded into, especially if they decide to leaflet there such that a relatively large amount of leafleting occurs per person. In this ideal location where non-leaflet efforts aren’t increased also, comparing the number of new vegetarians and vegans to the number of leaflets distributed could then give us an upper bound on the rate of diet conversions from leaflets.

The effectiveness of individual outreach interventions in other areas which seem analogous to leafleting

It may be possible to draw some lessons about the effects of leafleting from studies of increasing voter turnout. Perhaps the most studied behavior change that is reasonably similar to that asked for in animal advocacy leaflets is turning out to vote in U.S. elections. In a 2016 conversation with Josh Kalla, a PhD student in political science at University of California, Berkeley, we were told that a review of hundreds of experiments on different get out the vote (GOTV) tactics indicated that the more personal the communication, the larger the effect size. The review in question also estimates that one vote can be gathered per 189 leaflets handed out, and estimates that the cost per vote through direct mailings⁸⁹ is $67 (Green & Gerber, 2004).⁹⁰ However, there are significant differences between reduced animal product consumption and voting: “voting seems to be more widely agreed upon as the right thing to do than the agreement about the ethics of reducing animal product consumption, reduced animal product consumption is a sustained effort whereas voting is a one-time event, [and] there are more GOTV campaigns, so vegetarian ads may face lower-hanging fruit […] images of animals may or may not be more effective than GOTV reminders/arguments.”⁹¹ We are unsure if the GOTV reminders are part of the best reference class to use in forecasts concerning animal advocacy leaflet effectiveness. Since we have a limited idea of what the appropriate reference class is for forecasting the effectiveness of leaflets, and a limited idea of the outcomes for that reference class, the performance of this one analog alone currently does not result in a significant change in the performance we expect for leaflets.

Some speculative reasoning about the long-term effects of leafleting

It seems likely that the vast majority of animals whose lives we can affect will live in the far future. If we were certain about the long-term effects of interventions, this would probably be our primary consideration in deciding what to promote. However, although the far future is very important, we don’t have enough confidence in our predictions to base all our decisions on what we believe about it. There is extremely limited evidence available to suggest how leaflets may shift conditions for farmed animals over the very long term,⁹² and any long-term predictions about this seem highly uncertain.⁹³ We aren’t aware of anyone who has attempted to study this subject explicitly, and there are few clear parallels to past situations in other movements.⁹⁴ In the absence of strong evidence, multiple contradictory theories are possible regarding how leaflets could affect the situation for animals in the far future, and it seems that we have very little evidence available here beyond speculative reasoning.

Animal advocacy leafleting of the type considered in this report seems to have an incremental, consumer-change based approach to social change, which may or may not be desirable. This form of leafleting seems less focused on building a mass movement than some other animal advocacy interventions such as animal advocacy protests or legal reform initiatives—although that is a tentative conclusion. If effective, leaflets address both dietary and attitudinal change, but seem to focus to a greater extent on dietary change, and so may be more likely than other approaches to lead to immediate behavior change that directly spares animals. If people simply changed their attitudes with respect to farmed animals, that might not lead to actual impact for animals. After all, many people currently care about animals, but relatively few are vegetarian or vegan.⁹⁵ It might actually be easier to change individuals’ diets without making a moral appeal (e.g., by promoting cultured meat). Acceptance of the moral arguments against meat eating may follow, rather than precede, diet change.

It is unclear what impacts, if any, leaflets have on the broader state of affairs for animal agriculture and how animals are viewed in society. We know of little, if any, historical precedent that shows major social change—perhaps roughly equivalent to a greater than 20% reduction in the number of animals raised for food—being caused by individual advocacy and/or changes in personal consumption. This means that, although leaflets might cause some more immediate changes in the demand for animal products, they might not have as much impact on animal welfare in the long run as other interventions. Such other interventions might include protests or legal reform, both of which have some historical precedents to provide evidence that they can lead to social change. In their focus on individual dietary change, typical animal advocacy leaflets tend to focus on vegetarianism or a reduction in consumption of animal products, rather than complete veganism. Some contest this approach, claiming it dilutes the message of animal advocates and makes it more difficult to convince people of the seriousness of animal suffering in the long run.⁹⁶ We also recognize this concern, but we haven’t seen convincing evidence that these undesirable effects clearly outweigh the potential benefits of reducetarian/vegetarian messaging, such as a lower barrier to entry for the animal advocacy movement that could increase total support and momentum. Critics also believe it is difficult to build a mass movement when the perceived criteria for acceptance in the movement is a lifestyle change. Additionally, they believe that a consumer focus provokes less moral outrage than focusing on the institution of factory farming—and that therefore this approach is missing an important driver of activism and subsequent social change.⁹⁷ Those beliefs don’t seem clearly unreasonable, but they don’t seem to be strongly supported by evidence. Even within that framework, leafleting could still function as an effective complement to other interventions.⁹⁸

Overall, we do not think most organizations engaging in leafleting view it as part of a coherent strategy to provide benefits to animals in the far future. Its long-term effects are not well understood, and could be either positive or negative. As with most interventions performed by animal advocates, we think the long-term effects are more likely to be positive than negative, because promoting concern for animals’ interests is so important that in the absence of strong reasons to believe the effects are negative, we think it is likely that the effects are positive on balance. We don’t think the long-term effects of any animal advocacy intervention are extremely well understood—though some seem clearer than the long-term effects of leaflets—and so we put limited weight on this consideration in our overall understanding of how effective or ineffective any intervention is.⁹⁹

Summary

The comparative advantages that leafleting has over a number of other farmed animal advocacy interventions mainly seem to be its (i) ease of scaling, (ii) lower bar for involvement of newer advocates who lack experience and/or training, and (iii) the large number of leaflet recipients exposed to animal advocacy. The available historical/social movement evidence doesn’t seem to either strongly support the use of leaflets or to strongly weaken the case for leafleting being a highly effective animal advocacy intervention. The available “macro” evidence is rather inconclusive, but does seem to make it unlikely that leafleting has a high rate of converting people to vegetarian diets which they adhere to for multiple years. We also have a limited idea of what the appropriate reference class for forecasting the effectiveness of leaflets is and what the outcomes for that reference class would be; as a result, currently the results of interventions we consider to be in this reference class don’t play a large role in our reasoning. Finally, there is extremely limited evidence available to suggest how leaflets may shift conditions for farmed animals over the very long term. Still, our tentative conclusion is that the comparative advantages of leafleting seem to be outweighed by the disadvantages, when compared to promising farmed animal advocacy interventions. Important considerations include short-term diet effects, short-term effects on farmed animal welfare, contributions to movement building, and plausible long-term effects; in each of these areas, we have reason to think that another intervention outperforms leafleting. In short, our limited impression is that qualitative concerns do not generally favor leafleting, and in some areas indicate that other interventions may be preferable.

Poor	Weak	Moderate	Strong
Little is understood about the effectiveness of this intervention.	There is some understanding of the effectiveness of this intervention, weakly supported by evidence.	There is a good understanding of the effectiveness of this intervention, moderately supported by evidence.	There is a strong understanding of the effectiveness of this intervention, tested and supported by evidence.

Part Four: Case study analysis/Cost-effectiveness estimate

Case Study Analysis

Case Study Analysis of The Humane League Site Visit 2015: San Diego Warped Tour

The original notes on this case study were written by Jon Bockman, ACE’s Executive Director. The Humane League’s San Diego grassroots Director, Beau Broughton, seems to have been the primary organizer for the leafleting event that Jon observed. The event involved Humane League staff, interns, and volunteers handing out leaflets at the Warped Tour music festival. This event was originally selected for a case study by Bockman in order to observe THL’s operations, rather than to observe a representative leafleting event.

The leafleting at this event seemed well organized: leafleters appeared professional, they were well received (in that no negative reactions were recorded), they were able to connect their message¹⁰⁰ to the vegetarianism of some band members, and they were leafleting a receptive demographic (14–20 year-olds with some interest in ideas or culture outside the mainstream).¹⁰¹ This case study seems to present a successful example of leafleting at an event where many passersby would be of a targeted demographic.

Bockman noted that the take rate of the leaflets was very high, with around 90% of the people offered a leafleting accepting it. This provided some further evidence that the leafleting was well executed. A likely explanation for this high take rate was the successful targeting of a quite suitable demographic. Additionally, Bockman thought that the leafleters were given reasonable, standard advice before the leafleting commenced.¹⁰²

Of the eight leafleters, five were staff or interns, and three were volunteers (including Bockman). This meant that it was mostly staff time being spent on the intervention. However, Broughton mentioned that the number of volunteers is variable—he gave an example of a time when there were 15 volunteers and five staff members at an event.

The group of ten leafleters distributed an estimated 7,383 leaflets in two hours. This is over twice the rate at which leaflets were handed out in our second case study. Broughton said that looking at how many leaflets are discarded is more trouble than it’s worth as a method of trying to assess how effective the leafleting was.

No one who received a leaflet came back to talk to Bockman. Bockman thought that this was probably because they were on their way out of the event at this time, and so would not naturally have walked by the leafleters a second time.

Bockman also did not report any conversations he had with the audience, other than some mentioning that they had received many of these leaflets before. Jon thought these people were a small minority and had probably received these leaflets before because they had come to the Warped Tour before. This was the only evidence of possible audience saturation in any of the three case studies.

This leafleting event seemed to be the most well-organized of the three described case studies in this report.

Case Study Analysis of The Humane League Site Visit 2015: Leafleting and Meeting with Board Members

The original notes on this case study were by Allison Smith, ACE’s Director of Research. Rachel Black, then the Humane League’s Philadelphia Director, was the primary organizer for the event. The event involved a Humane League staff member, two interns, and a volunteer handing out leaflets outside the library at the Community College of Philadelphia. This event was originally selected for a case study by Smith in order to observe THL’s operations, rather than to observe a representative leafleting event.

At this event, Smith observed that the leaflets were handed out quite quickly (846 per hour by the four leafleters, by her count). Black was very impressed by the take rate and noted that she would have been only 100 leaflets short of meeting her goal for the entire semester after this event. Only one of 40 people Smith observed discarded their leaflet.

Black considered cancelling the leafleting event because it was raining, and she had to adapt her plans in order to go forward with the event. She leafleted in a less familiar area because of this. The leafleters expected that the event would have gone better (including attracting more volunteers) if it had not been raining. Some of the audience had conversations with Black, but Smith did not hear what was said in them.

This case study provides a similar level of evidence as the previous case study, though the previous case study seemed to provide a greater level of detail.

Case Study Analysis of Moving Mountains: Animal Rights Organizations, Emotion, and the Autodidactic Frame Alignment, Appendix C: Observation Report

The original notes on this case study (p.146–149) were taken by Lee Jarvis as part of a dissertation on the use of emotional messaging in the animal advocacy movement. Jarvis only refers to the leafleters as volunteers and it isn’t clear to us which individual organized the event, but the organization involved was Vegan Outreach. The event took place on a commercial street in Miami Beach, Florida.

In addition to the leafleting component of this event, the volunteers were using a video to attempt to motivate people to reduce their animal product consumption and they also had a table where some further resources were available. This means that it isn’t clear whether some of the observed evidence ought to be attributed to the leafleting or to the other facets of this event.¹⁰³

Of the three case studies, this case study provides the most information about the psychological responses of the people who receive leaflets. Of the four people who Jarvis observed stopping and talking to the volunteers, two were vegetarian or vegan. The two others were large, athletic men who were concerned about staying healthy and fit on a vegan diet. That provides some very weak evidence that athletic men may be an underserved demographic. The volunteers at this event did seem to be prepared; they had some special leaflets geared towards athletes as well as a poster on their booth that catered to athletes.¹⁰⁴ The fact that Vegan Outreach had these special leaflets and posters ready provide some evidence of their good judgement in the planning of this event. That this event took place near a beach may mean that athletic men were overrepresented relative to the average leafleting audience. This event seemed to be less well targeted at what seem likely to be the demographics most receptive to leaflets, relative to the other two case studies.

Jarvis did observe that many of the audience appeared to be in their 20s, but we would guess they were possibly less concentrated in that demographic than the other two leafleting events, and they were probably less likely to be liberal or have some interest in ideas or culture outside the mainstream. These factors may have been responsible for the lower take rate at this event; around half of those offered a leaflet took one. One of the volunteers at this event suggested to Jarvis that for one night with a three-person booth, typically 2–10 people stop to talk to them or ask questions.¹⁰⁵

This was the only one of the three case studies where any negative reactions were reported¹⁰⁶ by the authors, though there were still very few negative reactions reported.

Summary of Case Studies

Case studies can provide valuable information about interventions. This could be particularly true for leafleting, given the lack of high-quality research investigating the intervention. The lack of negative reactions in these case studies could be seen as evidence of effectiveness, though it could also be attributed to a lack of engagement from the audience. The information about the different “take rates” across the case studies is suggestive of there perhaps being significant variance in this factor across leafleting events.¹⁰⁷ The case studies also indicate that it is worth considering the audience of leafleting and tailoring messages to that audience where possible. Lastly, it is at least somewhat informative that the final case study mentioned that leafleters felt that people watching videos shouldn’t be interrupted, possibly suggesting that they think videos are more persuasive than a leaflet.

However, in order for these case studies to have provided moderate or strong evidence about the effectiveness of leafleting, evidence about the long-term behavior of those receiving leaflets would be needed. Some insight into the psychological state of those receiving leaflets might have given some evidence about this behavior change, but there was little opportunity in any of the case studies for positive or negative evidence about that. This is because not many conversations with the audience were recorded. No cost-effectiveness estimate was attempted for any of the three case studies because the amount of resources used in them was not clear, and the level of behavior change achieved was also not clear.

Of the 15 people in total who were leafleting at these events, eight were staff or interns, and seven were volunteers. Since one of the advantages that ACE sees in leafleting is relative ease of volunteer involvement, this perhaps low number of volunteers could be somewhat concerning. However, these case studies may not be representative of the ratio of staff and interns to volunteers at leafleting events in general. This is because the events at which Bockman and Smith completed case studies were specifically selected to observe THL’s operations, rather than to observe a representative leafleting event.

In sum, these three case studies provide poor or no evidence in favor of the proposition that leaflets are more cost-effective than other promising animal advocacy interventions.

Poor	Weak	Moderate	Strong
Development of case studies did not provide evidence to support this intervention choice.	The case studies provided weak evidence to support this intervention choice.	The case studies provided moderate evidence to support this intervention choice.	The case studies provides strong evidence to support this intervention choice.

Cost-effectiveness estimate¹⁰⁸

The limitations of our cost-effectiveness estimate

This cost-effectiveness estimate is an approximation of the costs and benefits of leafleting. It is highly uncertain, because the evidence pertaining to the effects of typical animal advocacy leaflets is at a high risk of bias and has other severe limitations. We worry that readers may think that we have a higher degree of confidence in this cost-effectiveness estimate than we actually do. To be clear, this is a very tentative cost-effectiveness estimate and it plays only a limited role in our overall opinions of which charities and interventions are most effective. We are more confident that the values of the bounds of the interval, and the corresponding confidence levels, represent reasonable estimates than we are that the average value of the estimate provides an accurate point estimate. However, we still feel uncomfortable putting too much weight on the bounds because they involve quantifying very difficult to quantify sources of uncertainty. There are also a number of parameters involved in this estimate that add additional uncertainty above and beyond that which we already had about the evidence from the particularly relevant quantitative trials. In some cases we have assigned quantitative values according to our best judgment and reasoning.

This cost-effectiveness estimate of leafleting is an approximation that does not take into account the possible positive or negative consequences of leaflets on a variety of outcomes. It only attempts to estimate the direct short-term effects of leaflets on the consumption of some animal products; all other possible effects of leaflets have been excluded. For instance, the possible effects of leaflets on shellfish consumption and wild fish consumption have not been included, nor have the possible indirect effects of leaflets on the short-term consumption of animal products (e.g., through possibly making other animal advocacy interventions more likely to succeed). This estimate also assumes that changes in consumption linearly correspond to changes in the self-reported number of meals containing a given animal product. In addition, it makes use of simple assumptions about the costs involved with leafleting by, for instance, not attempting to incorporate the opportunity costs for the volunteers involved. There could also be meaningful differences in moral status between different farmed animal species, and this model doesn’t attempt to account for them. Additionally, there are potentially meaningful differences in the average welfare of different species of farmed animals, and this model doesn’t attempt to account for that either. Again, this cost-effectiveness estimate is an approximation and it relies on flawed data and extremely difficult to quantify sources of uncertainty. Those severely suboptimal features contribute to the reasons why cost-effectiveness estimates are far from the only factor we consider when we evaluate interventions and charities. The following estimate should be interpreted carefully—it is a rough estimate, and not a precise calculation of cost effectiveness.

Reasoning behind our cost-effectiveness estimate

We are aware of a number of different estimates of the cost per leaflet distributed. For instance, a number of estimates suggest that it costs less than $0.10 per leaflet distributed,¹⁰⁹ while others suggest that the cost per leaflet distributed is slightly more than $0.10.¹¹⁰ These estimates appear only to account for the cost of purchasing leaflets, and not for further costs involved with their distribution. To better approximate the cost per leaflet distributed, we will use information from a 2014 conversation with Jack Norris of Vegan Outreach, in which we were told that the estimated costs that Vegan Outreach pays when they print and hand out literature at colleges are between $0.25 and $0.50, depending on how far they have to travel. Similarly, in ACE’s 2016 review of Vegan Outreach, we estimated that the cost per leaflet distributed was $0.31 to $0.42. The marginal cost per leaflet may be much lower than this, but it is useful to consider the total costs of a leafleting program when evaluating the overall cost-effectiveness of an organization, and these overall numbers also help guide our thinking as to whether organizations should pursue leafleting at all. Even these total cost numbers have a large amount of uncertainty. We don’t feel highly confident in those estimates and would guess that it is not highly unlikely that they could be misleading by a factor of 3 or 4—but it seems highly unlikely that they are incorrect by an order of magnitude. To incorporate that uncertainty in our cost-effectiveness estimate, we will use a lognormal distribution with a 90% subjective confidence interval¹¹¹ of $0.16–$1.60 per leaflet distributed for leafleting in general.

As for the change in animal products consumed per leaflet distributed, we will use the results of the meta-analysis for the six particularly relevant field randomized controlled trials as one key parameter in our initial model to estimate them. The estimated standardized differences in group means from the meta-analysis are shown in Table 5. As described in Part Two of this report, these differences were calculated by pooling the results of the particularly relevant field randomized controlled trials (with inverse variance weighting) and evaluating Hedges’ g of the pooled results. In order to convert those standardized mean difference estimates into estimates of change in animal product consumption, we first need to translate them back into one of the dietary change dependent variables used in some of the randomized controlled trials. We will translate the pooled estimates into consumption numbers using the meals-per-week results from the two HLL field trials. We chose these variables mainly because a greater number of respondents across the analyzed studies were asked to describe their consumption in this format. Our estimates of the effects of leafleting will take into account the reported consumption of all participants at baseline and the control participants at endline to calculate the mean of diet change under control conditions, as well as the standard deviation of consumption for all participants. This information allows us to estimate what difference in meals per week corresponds to the estimated standardized difference in means.¹¹² We pooled the results from the HLL field trials for leafleting to give an estimate of the number of meals per week that the respondents reported consuming animal products (in a typical week), and to calculate the standard deviation of overall consumption. These values are shown in Table 7.

Table 7. Estimated standardized mean difference from the meta-analysis presented in Part 2 and pooled estimates from the HLL field trials of meals per week involving certain animal products

Animal Product	Estimated standardized mean difference from meta-analysis as 95% CI (random effects model)	Mean number of meals containing this in a typical week (estimated standard deviation in parentheses)	Estimated standard error of the mean number of meals in a typical week containing this animal product
Red meat113	[-0.03, 0.14]114	4.38 (3.88)	0.050
Poultry115	[-0.07, 0.16]	5.91 (3.76)	0.048
Fish116	[-0.12, 0.10]	2.07 (2.57)	0.033
Eggs	[-0.10, 0.07]	4.31 (3.52)	0.045
Dairy	[-0.07, 0.11]	7.21 (4.41)	0.083

Knowing this estimated mean and standard deviation in consumption, and the estimated standardized mean difference between the treatment and control groups’ consumption, we can then estimate the percentage increase in the number of meals associated with being in the treatment group. We then use that percentage increase to estimate the change in demand for farmed animal products. Using estimates of the cumulative elasticity factors for these products, we also estimate the corresponding change in supply per dollar spent on leafleting, both in terms of animals spared and years of animal suffering averted.¹¹⁷ We also have to estimate the duration of the change in consumption associated with leafleting. For specifics, please see the full model in Guesstimate.

Somewhat counterintuitively, this estimate—which, again, doesn’t take into account the possible indirect short-term effects and is based on a very limited evidence base—suggests that leaflets’ direct short-term effect is to increase farmed animal suffering. The result of the cost-effectiveness model is a 90% subjective confidence interval for the change in supply of farmed animals per dollar ranging from a decrease in supply of 0.3 animals to an increase in supply of 10 animals¹¹⁸ and a corresponding change in the number of years of farmed animal life of -0.4 to 1 years.¹¹⁹ Note that the probability distributions within these subjective confidence intervals are not symmetric; instead, they are positively or right skewed. A lot of the variation in these values, as well as the reason the ranges are more positive than negative, is due to the estimated effect of leaflets on broiler chicken consumption. That is, the main reason that the estimate is positive is because the estimated standardized mean difference from the meta-analysis was symmetrically distributed with most of its probability mass on leaflets causing an increase in poultry consumption. We didn’t complete a sensitivity analysis of the results from the meta-analysis. It is possible that slightly different assumptions would have caused the point estimate for the standardized mean difference for poultry to in fact be negative, and that could have have led to the point estimate for leaflets’ effect to be a decrease in farmed animals raised and farmed animal life-years rather than an increase.

We make subjective adjustments to this cost-effectiveness estimate to incorporate the following information:

• There is animal advocacy evidence from other randomized controlled field trials and randomized field trials that seem relevant. Upon initial inspection the results of some other relevant trials seem to be more positive,¹²⁰ but the results from a randomized controlled field trial of online advertising are similarly negative.¹²¹
• Our general understanding of psychology and advertising makes us think that it is somewhat unlikely that leaflets will increase animal product consumption in the short term.¹²²
• We would probably expect that the high risk of bias in the six particularly relevant field randomized controlled trials would lead to favorable overestimates of the effects of leaflets.
• The estimate is quite sensitive to small changes in the estimate of the effects of leaflets on broiler consumption and our uncertainty in model selection during the meta-analysis and cost-effectiveness estimate.¹²³

In order to better agree with the first three factors, we believe that our cost-effectiveness estimate should account for a rough estimate of our Bayesian prior for the how effective leaflets are. We guess that our prior for the estimated SMD caused by leaflets is normally distributed with a mean of -0.005 and a standard deviation of 0.1.¹²⁴ Note that we didn’t commit ahead of time to that Bayesian prior, and that it is based on rough subjective judgments.¹²⁵ We will also attempt to incorporate a factor to account for general model uncertainty by slightly biasing our estimates of the SMDs upwards and multiplying their standard deviations by 1.75.¹²⁶ This is an attempt to account for the possibility that different reasonable analytic decisions in the meta-analysis could have led to different results. Note that the value of the factor to account for general model uncertainty is based on our rough subjective judgement. These adjustments weren’t pre-committed to before our viewing the results of the meta-analysis and the resulting unadjusted cost-effectiveness estimate. It is possible that our reasoning for including these subjective adjustments was suboptimal, and we encourage readers to examine these subjective adjustments closely.

The results from the cost-effectiveness model initially were a 90% subjective confidence interval for the change in supply of farmed animals per dollar ranging from a decrease in supply of 0.5 animals to an increase in supply of 10 animals¹²⁷ and a corresponding change in the number of years of farmed animal life of -0.2 to 1 years.¹²⁸ With the above-mentioned subjective adjustments, our 90% confidence interval now becomes a decrease in supply of 3 to an increase in supply of 10 animals¹²⁹ and a corresponding change of -2 to 2 farmed animal years per dollar spent on leafleting.¹³⁰ Note that the probability distributions within these intervals are not symmetric; instead, they are positively or right skewed. Those who disagree with us about the subjective adjustments might want to consider only the original estimate, or possibly make their own subjective adjustments to that original estimate. Given the very high uncertainty involved in making this estimate, and the relatively small part that this estimate plays in our overall evaluations, we don’t think it is worth speculating further about the shape of our subjective probability distribution for the effect of leaflets on short-term consumption of some animal products.

Poor	Weak	Moderate	Strong
Development of cost-effectiveness estimate did not provide evidence to support this intervention choice.	The cost-effectiveness provided weak evidence to support this intervention choice.	The cost-effectiveness provided moderate evidence to support this intervention choice.	The cost-effectiveness provides strong evidence to support this intervention choice.

Summary

The case studies we considered provided limited evidence about the effectiveness of leafleting, because they didn’t address whether there were changes in the short- or long-term behavior of those receiving leaflets. Our cost-effectiveness estimate didn’t incorporate information from any of the three case studies, because the amount of resources spent on each of those leafleting events was not clear—and, more importantly, the level of behavior change achieved through these events was not at all clear. This cost-effectiveness estimate appears to indicate that, compared to the short-term calculable effects of other animal advocacy interventions, leafleting is likely to be less cost-effective—and may even be actively counterproductive, at least in terms of these effects. Note that because this cost-effectiveness estimate is largely based on the meta-analysis completed earlier in the report, it shouldn’t be taken as separate or corroborating evidence of that meta-analysis. Still, it is worth reiterating that it is fairly concerning that for some animal products the cost-effectiveness estimate assigns less probability to short-term decreases in consumption than to to short-term increases in consumption. We believe that a qualitative analysis that considered the results of this cost-effectiveness estimate along with the variety of factors mentioned in Part Three of this report would moderately update us towards the conclusion that the effects of leaflets are positive in expectation. For instance, since leaflets seem a priori more likely than not to promote concern for farmed animals, they could complement other farmed animal advocacy interventions in ways that may accelerate and make the success of those other interventions more likely. Leaflets also appear to have played a non-trivial role in building the contemporary animal advocacy movement. Our impression is that if such qualitative effects of leaflets were incorporated into the cost-effectiveness estimate, then the impact of leaflets would be estimated to be positive in expectation.

Part Five: Interviews in the field

Matt Ball of One Step for Animals, Rachel Black of The Humane League, and Jon Camp of The Humane League were interviewed as part of this report.¹³¹

In their interviews, Ball and Camp both noted that in their experience many people responded to leaflets that asked for full veganism by stating that they “could never go vegan.” This contributed to both of them favoring requests to reduce animal product consumption. Ball and Camp noted that, over the past two decades, leaflets had moved from originally being something like an academic position statement, to now being an attempt to persuade people as effectively as possible with simple writing, pictures, and a more positive focus.

Black and Camp had similar views on a number of points relevant to leafleting. These included that:

• Communication and coordination between the main animal advocacy organizations that practice leafleting is very good.
• Experience is not very important for being good at leafleting; the personality of the leafleter is at least as important.
• Some colleges and universities were much more receptive to leafleting than others. Ball also mentioned differences in how well leafleting was received in different areas, and thought this could bias studies of local populations that were trying to generalize about the absolute efficacy of leafleting.

All of the interviewees had noticed that within the past decade or two leaflet recipients had become more familiar with the ideas presented in the leaflets and seemed to become more sympathetic to them. The extent to which leafleting is responsible for this change is unclear. Ball noted that although people were more receptive, total animal consumption in the United States had still increased. Ball thought that leafleting could potentially have a net-negative effect by encouraging people to substitute chicken for other animal products (therefore increasing the total number of animals subject to factory farming). Part of his reasoning for why this could occur was that the arguments made in leaflets for reducing animal product consumption seem to apply more—or even much more—strongly to red meat than to chicken.

All the interviewees thought that leafleting seemed less effective than online ads and corporate campaigns. They were all excited by the fact that online ads to be highly targeted, and by the ease with which organizations can quickly test and improve online ads. Camp and Ball also mentioned that they were impressed by how easy it was to scale online ads, and that there seemed to be a growing amount of agreement that online ads appear to be more cost-effective than leafleting. All of the interviewees thought that some leafleting should still be done: Black and Ball thought that less than the current amount should be done and Camp thought that there was currently about an optimal amount of leafleting. All the interviewees mentioned that leafleting can provide a good option for people who have recently started their farmed animal advocacy. Black viewed leafleting as important for getting support for corporate campaigns. Camp and Black thought that some people in the movement might be biased towards leafleting as an intervention because it is a method that some groups in the movement have used for decades. Camp also believed that people may be underestimating the ability of leafleting to get new activists involved.

Overall, these interviews provide weak evidence against leafleting being as cost-effective as other promising farmed animal advocacy interventions.

Poor	Weak	Moderate	Strong
Our field conversations do not provide evidence to support this intervention choice.	Our field conversations provide weak evidence to support this intervention choice.	Our field conversations provide moderate evidence to support this intervention choice.	Our field conversations provide strong evidence to support this intervention choice.

---

1. For instance, THL primarily targets college campuses, concerts, and other large social events where the attendees are either teenagers or young adults, or they are more liberal or progressive. This information can be found in ACE’s Conversation with Andrea Gunn of The Humane League (2015). 

2. For example:

   • “Help animals”
   • “Info to help animals”
   • “Info on compassionate eating”
   • “Info against animal cruelty”
   • “Brochure against factory farming”
   • “Hello”
   • “Info about where your food comes from”
   • “Info on helping animals and the environment”
   • “Did you get one of these?”

   —Leafleting Tips for Pros. (n.d.). In Adopt a College, A Project of Vegan Outreach. 

3. Example leaflets:

   • Vegan Outreach’s “Compassionate Choices”
   • Vegan Outreach’s “Your Choice”
   • Vegan Outreach’s “Even If You Like Meat…”
   • Farm Sanctuary’s “Something Better”
   • A list of leaflets available from MFA’s online store
   • Animal Equality’s “Make A Difference”
   • FARM’s “Have We Been Lied To?”

    

4. Different leaflets are intended to be used when targeting slightly different audiences. For example, Andrea Gunn, then the national grassroots Director at The Humane League, has told us that “the three versions of Vegan Outreach leaflets target different audiences: “Your Choice” is aimed at college students, “Compassionate Choices” is used for more diverse audience age groups, and “Compassionate Athletes” is specifically for athletes. Vegan Outreach is largely responsible for updating the leaflets. Vegan Outreach has specifically requested that their leaflets only be used at events where there is a targeted demographic, so for non-targeted leafleting (e.g., leafleting in a busy intersection or outside a subway station), THL uses Mercy For Animals’ “FRESH” leaflet. Mercy For Animals has a special version of the “FRESH” leaflet for Gay Pride festivals (only changes are adding LGBT people on the front and back cover), and THL uses this booklet for all Pride events.” —Conversation with Andrea Gunn of The Humane League (2015) 

5. In 2015, Vegan Outreach sent out 3.3 million leaflets, about 80% of which (i.e., about 2.6 million) were reported as being handed out. This information can be found in ACE’s Conversation with Jack Norris of Vegan Outreach (2016). 

6. In 2015, Vegan Outreach’s Adopt a College program distributed 2,367,516 leaflets total on various school campuses. This number includes high school campuses, although our impression is that the majority of these leaflets were distributed at colleges; for example, none of the program’s 40 biggest school leafleting events for 2015 took place at high schools. For the source of this information, see the Leafleting Statistics page on the Adopt a College website. 

7. Example leaflets:

   • Vegan Outreach’s “Compassionate Choices”
   • Vegan Outreach’s “Your Choice”
   • Vegan Outreach’s “Even If You Like Meat…”
   • Farm Sanctuary’s “Something Better”
   • A list of leaflets available from MFA’s online store
   • Animal Equality’s “Make A Difference”
   • FARM’s “Have We Been Lied To?”

    

8. Examples of claimed best practices can be found in the following resources:

   • MFA’s Guide to Leafleting
   • Compassionate Action for Animals’ Guide to Effective Leafleting
   • Vegan Outreach’s Leafleting Tips from the Pros!

    

9. In this context, by short term we mean roughly less than half a year. 

10. In this context by medium-term we mean roughly more than half a year but less than five years. 

11. This could also be the result of leaflets contributing to the growth of the animal advocacy movement by providing a low bar to entry for those who wish to increase their animal advocacy efforts. 

12. This literature search was completed by the project leader. Important search terms and the extent of the search were recorded. Search terms and the extent of searching included:

    • ” “leaflet” ” (first 400 results)
    • ” “flyer” meta analysis systematic review” (first 100 results)
    • “”leaflet” meta analysis systematic review diet” (first 20 results)

    Note that putting a word in quotations means that the results have to contain that word. Studies were initially screened based on title and the accompanying text visible for Google Scholar results. Eligibility was assessed by the project leader who read relevant sections of the specific literature in question. This eligibility assessment wasn’t checked by other reviewers of this piece. The inclusion/exclusion criteria were not clearly prespecified and were instead left to the judgment of the project leader. The project leader made a judgement call about when to stop for any given search term based on when they felt it was sufficiently unlikely that anything useful would be found. 

13. The following is a list of some of the evidence rejected on these grounds, grouped by reason for rejection.
    Results reported were not from a randomized controlled trial in the field:

    • “For every 100 ‘Why Vegan’ or ‘Vegetarian Starter Kit’ type leaflets handed out—each containing half a dozen or more graphic images and descriptions of animals suffering in IFAFs—Friedrich estimates that 1 recipient will change his or her eating habits (Ball & Friedrich:19).” (Quoted text from Gunther, 2006).
    • “PETA surveyed people who received their vegetarian starter guide, and responses indicated that more than eighty percent of non-vegans changed their diet, with twenty-three percent going from meat-eater to an entirely vegan diet after reading the guide. (Ball & Friedrich, pp.18-19)” (Quoted text from Gunther, 2006).
    • Broughton, B. (2014). Meat, Media, and Morals: Furrowing in on Effective Food Activism in the Animal Protection Movement.
    • Freeman, C. P. (2008). Struggling for ideological integrity in the social movement framing process: How US animal rights organizations frame values and ethical ideology in food advocacy communication. University of Oregon.

    Results reported were not from a randomized controlled trial in the field, and multiple studies used the same participant group:

    • The following five Humane League Lab studies were excluded because they completed on the same group of participants and they weren’t field trials: (i) What cover photos make people most interested in reading pro-veg literature?, (ii) Is one message or multiple messages more effective?, (iii) Which vegan meals do omnivores find most appetizing and accessible?, (iv) Which farm animal photos are most likely to inspire people to eat vegan?, and (v) Is animal cruelty or abolitionist messaging more effective?

    Results reported were not from a randomized controlled trial in the field, and the intervention studied was judged to be significantly different from leafleting:

    • Fehrenbach, K. S. (2015). Designing Messages to Reduce Meat Consumption: A Test of the Extended Parallel Process Model. Arizona State University.
    • Arndt, C. A. S. (2016). Tailoring feedback and messages to encourage meat consumption reduction. Kansas State University.

    The intervention studied was judged to be significantly different from leafleting:

    • Baker et al., (2010). An RCT study to evaluate a targeted, theory driven healthy eating leaflet. Social Science & Medicine, 71(11), 1916-1920.
    • Kellar, I., & Abraham, C. (2005). Randomized controlled trial of a brief research-based intervention promoting fruit and vegetable consumption. British Journal of Health Psychology, 10(4), 543-558.

    The form of leafleting studied was judged to be significantly different from common leafleting practice:

    • Animal Equality U.K. (2014). In private communication we were told that more than 90% of the leaflets had been left in bulk in libraries or included in newspapers instead of handed out. We thought the results were not of much interest, as they didn’t reflect the results of leafleting as it is commonly carried out.

    The study had a small sample size:

    • Several small scale Vegan Outreach field trials mentioned in this Vegan Outreach blog post.

     

14. The 2013 Farm Sanctuary study/Humane League Labs study is separately reported by Farm Sanctuary and by Humane League Labs here and here. 

15. For example see this proposed methodology on the Effective Altruism forum. 

16. For instance, some relevant trials that haven’t been released yet but will likely be informative are:

    • Testing the effects of humane education: Pilot research for a randomized controlled trial
    • Meet your meat: Using virtual and real contact to reduce meat consumption
    • Nudging for good: An experimental analysis of moral aversion to the consumption of animal products

     

17. Bias is defined here as being any process at any stage of inference tending to produce results that differ systematically from the true values (Egger, 2001). 

18. Only the project leader assessed the risk of bias in these studies. Other team members didn’t do this independently, but did have the opportunity to peruse and provide feedback on the project leader’s assessments while reviewing this report. 

19. Internal validity can be briefly defined as the extent to which observed changes can be attributed to the intervention and not to other possible causes. 

20. External validity can be briefly defined as the applicability of the results of a study to other populations, settings, treatments, and measurement variables. 

21. For instance, a number of the studies didn’t seem to clearly meet a number of criteria from the CONSORT 2010 statement on guidelines for reporting randomized trials, including:

    • Item 2b. Trials didn’t really specifiy objectives or hypotheses
    • Item 4a. Eligibility criteria for participants
    • Item 4b. Settings and locations where the data were collected
    • Item 5. The interventions for each group with sufficient details to allow replication, including how and when they were actually administered
    • Item 6a. Completely defined pre-specified primary and secondary outcome measures, including how and when they were assessed
    • Item 6b. Any changes to trial outcomes after the trial commenced, with reasons

    Other criteria not met by some of these studies included items 7a, 8a, 9, 11b, 13, 13a, 13b, 14a, 15, 16, 17a, 17b, 18, 20, 21, 22, and 25. 

22. According to the Cochrane Handbook for Systematic Reviews of Interventions, “selection bias refers to systematic differences between baseline characteristics of the groups that are compared. The unique strength of randomization is that, if successfully accomplished, it prevents selection bias in allocating interventions to participants. Its success in this respect depends on fulfilling several interrelated processes. A rule for allocating interventions to participants must be specified, based on some chance (random) process. We call this sequence generation. Furthermore, steps must be taken to secure strict implementation of that schedule of random assignments by preventing foreknowledge of the forthcoming allocations. This process if often termed allocation concealment, although could more accurately be described as allocation sequence concealment. Thus, one suitable method for assigning interventions would be to use a simple random (and therefore unpredictable) sequence, and to conceal the upcoming allocations from those involved in enrolment into the trial.” —Introduction to sources of bias in clinical trials. (2011) In J.P.T. Higgins and S. Green (Eds.), Cochrane Handbook for Systematic Reviews of Interventions (Version 5.1.0). The Cochrane Collaboration. 

23. “Performance bias refers to systematic differences between groups in the care that is provided, or in exposure to factors other than the interventions of interest. After enrolment into the study, blinding (or masking) of study participants and personnel may reduce the risk that knowledge of which intervention was received, rather than the intervention itself, affects outcomes. Effective blinding can also ensure that the compared groups receive a similar amount of attention, ancillary treatment, and diagnostic investigations. Blinding is not always possible, however. For example, it is usually impossible to blind people to whether or not major surgery has been undertaken.” —Introduction to sources of bias in clinical trials. (2011) In J.P.T. Higgins and S. Green (Eds.), Cochrane Handbook for Systematic Reviews of Interventions (Version 5.1.0). The Cochrane Collaboration. 

24. “Detection bias refers to systematic differences between groups in how outcomes are determined. Blinding (or masking) of outcome assessors may reduce the risk that knowledge of which intervention was received, rather than the intervention itself, affects outcome measurement. Blinding of outcome assessors can be especially important for assessment of subjective outcomes, such as degree of postoperative pain.” —Introduction to sources of bias in clinical trials. (2011) In J.P.T. Higgins and S. Green (Eds.), Cochrane Handbook for Systematic Reviews of Interventions (Version 5.1.0). The Cochrane Collaboration. 

25. “Attrition bias refers to systematic differences between groups in withdrawals from a study. Withdrawals from the study lead to incomplete outcome data. There are two reasons for withdrawals or incomplete outcome data in clinical trials. Exclusions refer to situations in which some participants are omitted from reports of analyses, despite outcome data being available to the trialists. Attrition refers to situations in which outcome data are not available.” —Introduction to sources of bias in clinical trials. (2011) In J.P.T. Higgins and S. Green (Eds.), Cochrane Handbook for Systematic Reviews of Interventions (Version 5.1.0). The Cochrane Collaboration. 

26. “Reporting bias refers to systematic differences between reported and unreported findings. Within a published report those analyses with statistically significant differences between intervention groups are more likely to be reported than non-significant differences. This sort of ‘within-study publication bias’ is usually known as outcome reporting bias or selective reporting bias, and may be one of the most substantial biases affecting results from individual studies (Chan 2005).” —Introduction to sources of bias in clinical trials. (2011) In J.P.T. Higgins and S. Green (Eds.), Cochrane Handbook for Systematic Reviews of Interventions (Version 5.1.0). The Cochrane Collaboration. 

27. See sections 8.9 through 8.14 of the Cochrane Handbook for Systematic Reviews of Interventions (Version 5.1.0) for discussions of the evidence indicating the importance of each of these factors. 

28. The results of a study may in fact be unbiased despite a methodological flaw; therefore, it is more appropriate to consider risk of bias. 

29. These symbols and interpretations are derived, respectively, from Presentation of assessments of risk of bias and Summary assessments of risk of bias. (2011) In J.P.T. Higgins and S. Green (Eds.), Cochrane Handbook for Systematic Reviews of Interventions (Version 5.1.0). The Cochrane Collaboration. See Figures 8.6.c and 8.7.a respectively. 

30. Reviewers didn’t check to ensure that the data had been correctly extracted. 

31. The logic behind using the midpoint with extended intervals transformation is the belief that if a respondent’s consumption falls between two categories then they will report the category of consumption to which they are closest to falling into. It follows that it is better to consider the true ranges of adjacent categories with gaps between them as the range found by extending the intervals of adjacent categories so that there is no gap between them. For example, a commonly-used food frequency questionnaire has the following categories: never, less than 1 time per week, 1–6 times per week, 1–3 times per day, and 4 or more times per day. These categories with extended intervals are: never, less than 1 time per week, 1–6.5 times per week, 6.5–24.5 times per week and 24.5 or more times per week. By then taking the midpoint of these extended intervals or using the justification provided above the frequency of consumption per week for each category respectively would be: 0, 0.5, 3.75, 15.5, and 28. 

32. To give a better sense of how common a continuous transformation is in this literature we will now summarize the methods used to analyze studies which employed an ordinal Food Frequency Questionnaire (FFQ) and were included in the recent systematic review of dietary interventions amongst college students completed by Dielens et al (2016). Twenty studies were included in this systematic review and nine of these studies likely used ordinal FFQs. Six studies that used ordinal FFQs convert the ordinal data to be treated as continuous data (Clifford et al (2009), Franko et al (2008), LaChuasse (2012), Zhang and Crooke (2012), McClain et al (2012)). For three studies it’s not clear if an ordinal FFQ was used; if it was used, then these studies also converted the data from ordinal to continuous (Kattelman et al (2014), Richards et al (2006)). Brown et al (2014b) treated the ordinal FFQ as ordinal data. However, the dependent variable in that study is an ordinal FFQ with the four response options: Does not apply, Rarely/Never, Sometimes, and Usually/Often. These response options are harder to convert than the ordinal categories with some numeric information found in animal advocacy FFQs, such as those describing whether a product was eaten daily, weekly, or monthly.
    Further evidence that the continuous transformation is quite common in the literature that deals with analysis of FFQs is noted by Doidge & Segal (2012), p. 237; by Pollard et al (1998), p. 167–168; and by Salvini et al. (1989), p. 859, who say that for frequencies presented as a range, transforming to the midpoint is often used. The following studies all quantify FFQ categories by taking the midpoint of the interval: Vereecken & Maes (2003), p. 584; Plesko et al (2000), p. 203; Mikkelsen et al (2006), p. 772; Vereecken et al. (2008), p. 299; Dehghan et al (2005); Beck et al (2012), p. 1178; McCrory et al (1999), p. 565; Wolk et al (1994), p. 571; and Frances et al (2002) p. 215. The National Cancer Institute (2016) scoring instructions for a fruit and vegetable screener approximately suggest transformations for frequencies presented as a range to the midpoint. Pollard et al (1998) suggest that “4 or more times per day” can be considered to represent 4.5 times daily, whereas Salvini et al (1989) suggest it should be transformed to 4 times per day. 

33. There is evidence that supports treating similar ordinal measures as interval measures. For example:

    • Sullivan, G. M., & Artino Jr, A. R. (2013). Analyzing and interpreting data from Likert-type scales. Journal of graduate medical education, 5(4), 541–542.

     

34. For the sake of simplicity, analysis by school/other clusters wasn’t taken into account during the analysis. That will lead to some underestimation of the standard errors. Sometimes information about this clustering wasn’t readily available. 

35. For the data taken from the 2017 trial performed by Flens et. al., we excluded participants from the analysis if they didn’t remember being given a leaflet, or if they didn’t answer the question about having been given a leaflet. This is a slightly different exclusion procedure than the one used by Flens et. al. in their analysis. 

36. It is possible that these studies are measuring somewhat different variables. For example, reductions in reported number of meals of some animal product might not linearly correspond to reductions in reported consumption of that animal product, either mechanistically (people reduce number of meals but up amount per meal) or because people are more likely to misreport one of these. 

37. This is, for instance, noted on the Wikipedia page on effect sizes. 

38. Note that the entry for Henessy (2016) in this forest plot is based on meat consumption. The consumption measurement question was phrased as “On about how many days in the past seven days did you consume meat? Include chicken, beef, pork, fish, seafood, etc.” 

39. Note that the entry for Henessy (2016) in this forest plot is based on meat consumption. The consumption measurement question was phrased as “On about how many days in the past seven days did you consume meat? Include chicken, beef, pork, fish, seafood, etc.” 

40. Note that the entry for Henessy (2016) in this forest plot is based on meat consumption. The consumption measurement question was phrased as “On about how many days in the past seven days did you consume meat? Include chicken, beef, pork, fish, seafood, etc.” 

41. The project leader used their judgment to derive the estimate for red meat consumption from estimates of cow consumption and pig consumption. 

42. This was found by combining the SMD for individual consumption of pig and cow in the studies involved in the meta-analysis. 

43. This was found by combining the SMD for individual consumption of pig and cow in the studies involved in the meta-analysis. 

44. The project leader used their judgement to convert the estimate for poultry consumption into an estimate for chicken and turkey consumption. 

45. Note that approximately 40% of fish consumed are farmed fish, both in the U.S. and in the world as a whole. According to 2016 data from the Food and Agriculture Organization (FAO), 44.1% of world fish production by mass (p.4), and about 8% of U.S. fish production by mass (p.173), comes from aquaculture. In 2015, the U.S. produced about 7.75 billion pounds of edible fishery products, 6.94 billion of which it exported, while it imported 11 billion pounds of such products. Thus, about 93% of U.S. fish consumption was of imported fish products, and we think it is reasonable to assume that about 44% of the fish killed in the production of these imported products were farmed fish. (We assume that farmed fish have an equal average weight to wild-caught fish, and that farmed fish products are as likely to be imported as wild-caught fish products. We also assume that the same proportion of farmed fish and wild fish are caught for consumption; according to the 2016 FAO data, a large majority of fish products by weight are consumed as food, so errors arising from this assumption are likely to be small.)
    The National Oceanic and Atmospheric Administration also states that, based on 2011 data, “[a]bout half the seafood we eat is wild-caught; the other half is farm-raised, that is, from aquaculture.”—The Surprising Sources of Your Favorite Seafoods. (September 3, 2012). National Oceanic and Atmospheric Administration Office of Aquaculture. 

46. “Use of a fixed effect meta-analysis model assumes all studies are estimating the same (common) treatment effect. In other words, there is no between study heterogeneity in the true treatment effect. The implication of this model is that the observed treatment effect estimates vary only because of chance differences created from sampling patients. Hypothetically, if all studies had an infinite sample size, there would be no differences due to chance and the differences in study estimates would completely disappear.” —Riley, R. D., Higgins, J. P., & Deeks, J. J. (2011). Interpretation of random effects meta-analyses. The BMJ, 342, d549. 

47. “A random-effects meta-analysis model assumes the observed estimates of treatment effect can vary across studies because of real differences in the treatment effect in each study as well as sampling variability (chance). Thus, even if all studies had an infinitely large sample size, the observed study effects would still vary because of the real differences in treatment effects. Such heterogeneity in treatment effects is caused by differences in study populations (such as age of patients), interventions received (such as dose of drug), follow-up length, and other factors.” —Riley, R. D., Higgins, J. P., & Deeks, J. J. (2011). Interpretation of random effects meta-analyses. The BMJ, 342, d549. 

48. Some variation in the results of different studies could be due to chance alone. Heterogeneity reflects the true differences in the results of studies, that is, the differences which would remain if variation due to chance alone could be eliminated (Liberati et al, 2009). The I2 statistic attempts to quantify the amount of variation in results across studies beyond that expected by chance, with its value being an estimate of the percentage of the total variation due to heterogeneity rather than chance (Higgins & Thompson, 2002). One proposed set of criteria for interpreting I2 estimates is that an I2 less than 40% is low, 30–60% may be moderate, 50–90% may be substantial, and 75–100% is considerable (Schünemann, 2013). 

49. Hunter and Schmidt found in 2000 that “FE [fixed effects] models typically manifest a substantial Type I bias in significance tests for mean effect sizes and for moderator variables (interactions), while RE [random effects] models do not. Likewise, FE models, but not RE models, yield confidence intervals for mean effect sizes that are narrower than their nominal width, thereby overstating the degree of precision in meta-analysis findings.” —Hunter, J. E., & Schmidt, F. L. (2000). Fixed effects vs. random effects meta‐analysis models: Implications for cumulative research knowledge. International Journal of Selection and Assessment, 8(4), 275-292.
    A later analysis by Schmidt et. al. found that fixed-effects and random-effects models overstated the precision of their results to a similar degree. However, the authors also asked “Are there any circumstances in which the choice of the FE model would be appropriate?” and responded as follows:

    These circumstances would appear to be very limited. The FE model would be appropriate if one had strong evidence that the primary studies to be included in the meta-analysis were virtually identical, i.e. they are all literal or operational replications of each other (Aronson, Ellsworth, Carlsmith, & Gonzales, 1990). That is, if the studies drew their samples from the same population (e.g. college sophomores), tested exactly the same hypotheses with exactly the same study design, treatment strength (if an experimental study), measures, instructions, time limits, etc, then one might assume a priori that the same population parameter was estimated in all the primary studies […] Such a situation would be expected to occur only rarely (Aronson et al., 1990). In any other situation, an FE model would be inappropriate and the recommendation would be that any meta-analysis conducted using the FE model should be reanalysed using an RE model[.]”

    —Schmidt, F. L., Oh, I. S., & Hayes, T. L. (2009). Fixed‐versus random‐effects models in meta‐analysis: Model properties and an empirical comparison of differences in results. British Journal of Mathematical and Statistical Psychology, 62(1), 97-128. 

50. Lacto-vegetarian prevalence was estimated in these studies by using the proportion of people who reported not consuming any animal products except dairy in their self-reported consumption. 

51. Animal Equality Spain (2014) was excluded from this meta-analysis because the dependent variable in that study could not reliably indicate changes in lacto-vegetarian status. 

52. Animal Equality Spain (2014) was excluded from this meta-analysis because the dependent variable in that study was unable to reliably detect changes in lacto-vegetarian status. 

53. Note that an odds ratio of one is equivalent to an effect size of zero. 

54. Nor do these confidence intervals include results of other randomized controlled field trials and randomized field trials that seem relevant but that we didn’t include in the meta-analysis. 

55. For instance, Pilot Trials One, Two, and Three mentioned in this Vegan Outreach blog post had low response rates and appear not to have been analyzed; the blog post notes that post-test results for Pilot Trial One in particular were not analyzed due to concerns about the small sample size caused by the trial’s low response rate. 

56. The fact that we were able to include the study by Flens et al (2017), and to access the Animal Equality results prior to their publication, provides evidence in this direction. 

57. “On each visit Vegan Outreach tries to give out leaflets to about 15% of the students at a school.” —Conversation with Jack Norris of Vegan Outreach (2014)
    “There is some standardization for [The Humane League’s] leafleting practices at colleges: they aim to distribute leaflets to a minimum of ten percent of the student population.” —Conversation with Andrea Gunn of The Humane League (2015) 

58. For example, “[a]fter each respondent was provided with a booklet, they were told to read the booklet for as long or short of a time as they wanted.” —Doebel, Sabine and Susan Gabriel. (2015). Does Encouraging The Public To “Eat Vegan,” “Eat Vegetarian,” “Eat Less Meat,” or “Cut Out Or Cut Back On” Meat And Other Animal Products Lead To The Most Diet Change? Humane League Labs. 

59. For example, “[t]he treatment is a double-sided trifold leaflet instead of a longer leaflet, which may either reduce or enhance the impact.” —Hennessy, S. R. (2016). The impact of information on animal product consumption. (Doctoral dissertation). 

60. For further information, see the National Cancer Institute’s overview of evaluating the effect of an intervention on diet. 

61. The change in supply corresponding to some change in demand can be estimated by using the cumulative elasticity factor. 

62. Leaflets perhaps primarily seek to reduce consumption of animal products, which, if successful, would most likely lead to fewer animals being bred and raised for food production. Most animal advocates see this as beneficial, because farmed animals live in circumstances which appear to be extremely unpleasant and demonstrate a variety of stress behaviors in response to these conditions. However, it should be noted that population ethics is difficult even when applied to humans who can actually express a preference between their current circumstances and never having been born, and there is necessarily more uncertainty with regard to the preferences of animals who cannot communicate about abstract topics.
    Reduced demand for animal products could reduce the pollution and environmental damage that results from most animal agriculture practices. It could also increase habitat (and hence wild animal population) as less farmland is used for feed, housing, and other agricultural activities. There is substantial research on the environmental impact of animal agriculture, but little on how it affects wild animal welfare. 

63. For example, it has been said that a leaflet played a significant role in Joe Espinosa’s conversion. Espinosa has now become perhaps the world’s leading leafleter. In fact, ACE Executive Director Jon Bockman reports that a leaflet from Espinosa played a significant role in his own journey. 

64. For instance:

    • Reportedly, a single leaflet played an important role in Ruth Harrison’s animal advocacy journey. “In 1961, Ruth’s life irrevocably changed when the animal rights group ‘Crusade Against All Cruelty to Animals’ slipped her a small leaflet about the plight of animals raised for food such as veal calves, broilers, and laying hens under her door (photo 2).” (van de Weerd & Sandilands, 2008). Harrison authored Animal Machines and Peter Singer reports that book had an important influence on him (Singer 1986, p.149). In addition, Animal Machines is widely thought to have played an important role in the Brambell report, in which the five freedoms were stated.
    • “Time and again in this study, when I asked what it was that got informants started in the movement, the response was that it came in the form of a leaflet, advertisement or an arresting image.” (Munro 2005, p.143)
    • “John Bryant advocates social change via leafleting and notes in his Fettered Kingdoms (Bryant, 1982) that the great strength of the animal rights movement lies with the supporters who hand out leaflets every week: ‘The leaflet is our media. In nearly twenty years in animal welfare and rights I have rarely found a campaigner who did not join the movement after being handed a leaflet—usually in the street’ (Bryant, 1982, p. 88).” (Munro, 2011)
    • The leaflet is therefore one of the oldest tactics in the social movement’s repertoire. For many activists like John Bryant, it is the medium of the animal movement. The political potency of the leaflet can be gauged by its impact in the McLibel episode when vegetarian, animal rights activists distributed a short critique of McDonald’s in the form of a leaflet which subsequently led to the widely publicized libel trial in London’s High Court in 1996. (Munro, 2011)
    • “In fact, some of the McLibel Two’s colleagues had agreed to apologize to the company for distributing an offending leaflet in order to avoid litigation and possible financial ruin. Morris and Steel, the “animal rights vegetarian activists from Hell,” were the exception in that they were prepared to go to court to defend their right to free speech. The McLibel trial turned out to be the longest trial of its kind in British history. Although McDonald’s prevailed, some of the activists’ most important charges were upheld (Vidal, 1997).” (Munro, 1999)
    • “I quickly wrote a leaflet and circulated it around Oxford. Receiving no replies I reprinted it with an illustration of a poor little chimpanzee experimentally infected with syphilis, asked a friend David Wood to add his name so the leaflet would have a university address on it, and sent it around all the university colleges. This time I had some replies. One of the recipients was a young Australian philosopher called Peter Singer. Within months he was in touch with me. A lot has happened since then.” (Ryder, 2010)

     

65. A Shapley value can be used to attribute impact to different factors in some causal chains. It may be possible to use such a technique to assign impact to the different interventions that in combination result in some outcome. 

66. An increased availability of foods not sourced from industrial animal agriculture could make the target audience more likely to change their diets based on the suggestions of the leaflet. Corporate outreach and investigations can promote discussion and concern for the treatment of farmed animals. In the long term, this can increase the chances of success for other advocacy efforts, such as leafleting and ads, corporate policy change, and legal change. An increased public interest in vegetarian and vegan diets and in helping farm animals could lead to increasing returns as people take more interest in the leaflets, and feel more comfortable changing their diet or attitudes in response. It could also potentially saturate the market and target audience, but it seems unlikely to us that this will be the case in the near future. 

67. We didn’t pre-commit to the protocol that would be followed prior to this review. We didn’t pre-commit to inclusion/exclusion criteria, bias assessments, or meta-analytic techniques. This analysis wasn’t pre-registered and we didn’t pre-commit to a method of data analysis. We also didn’t pre-commit to which dependent variables we would analyze in the meta-analysis, although we did have some rationale for choosing to study the specific outcomes we selected. 

68. The main staff time allocated to this project was approximately three months of time from the project leader. For reference, note, that the Centre for Reviews and Dissemination at York estimates that a systematic review will take a team 9–24 months. 

69. For reference, discrepancies between results of meta-analyses of small studies and subsequent large trials may occur as often as 20% of the time. See Cappelleri, J. C., et. al. (1996). Large trials vs meta-analysis of smaller trials: how do their results compare? JAMA, 276(16), 1,332-1,338. 

70. The organization of this intervention report may differ slightly from the one described in our intervention evaluation guide, as the two projects were developed concurrently. 

71. See Part Four of this report for specific estimates. 

72. Compassionate Action for Animals’ Guide to Effective Leafleting says that “[o]ne person can hand out hundreds of leaflets in a short amount of time, it doesn’t require a lot of planning, it can be mobile, and having friendly outgoing people in the streets for animals is great for our image.” 

73. Cooney also notes the further reasons: that online ads provide a point of contact with interested individuals, and the greater potential for animal advocacy content to go viral. —Cooney, N. (2014). Well-Planned Facebook Ads: The Most Cost-Effective Way To Create New Vegans, Vegetarians and Meat Reducers? Reducing Suffering. 

74. Section V.3.1 of ACE’s online ads report includes ACE staff’s subjective estimates of the online ad costs per impression. As a pessimistic estimate, that report estimates a cost of $5.66 for 2,630 impressions, or $.0022 per impression. In Part Four of this leafleting report we estimate an average cost of $.16-1.60 per leaflet distributed. 

75. As a general note on social movements, there appear to be very few examples of social movements that achieved widespread changes in behaviors as deeply ingrained as eating habits. Regarding those that did (e.g., abolitionism), there do seem to be noticeable differences between them and the animal advocacy movement. Furthermore, the nature of most historical research also makes it very difficult to establish causation, and the available research may place too much focus on the movements that succeeded, while neglecting failed movements. 

76. “In any event, consumer action alone is unlikely to constitute the sole, or even the greatest, response to the animal welfare issue.” —Anderson, J. (2011). Protection for the Powerless: Political Economy History Lessons for the Animal Welfare Movement. Stanford Journal of Animal Law and Policy 4(1).
    “Moreover, the movement’s focus on mass consumer dietary change has little historical or empirical basis, despite being our movement’s main strategy.” —Burns, B. Why Beyoncé Going Vegan is Bad for Animals. Direct Action Everywhere. 

77. The free produce movement, a boycott of goods produced by slave labor, provides a historical example of a social movement focused on individual changes in consumption. Our impression is that the movement had limited success outside the Quaker community in which it began, and that it faced criticism from other abolitionists.
    Elizabeth O’Donnell writes of the British free produce movement that “despite the many and widespread attempts to encourage and sustain ethical consumerism in the battle against slavery, most historians agree that it had little impact in the drive towards abolition.” —O’Donnell, E. A. (2009). ‘There’s Death in the Pot!’* The British Free Produce Movement and the Religious Society of Friends, With Particular Reference to the North-East of England. Quaker Studies, 13(2), 185.
    Frederick Tolles, writing about the history of the movement in 1943, noted that “[a]t its height the movement embraced many non-Quaker abolitionists, including William Lloyd Garrison; but as abolitionism moved on to more radical measures, the free produce movement tended to become again an exclusively Quaker affair […] Like colonization, the quiet boycott of slave-grown products fell into the background as the abolition movement approached flood tide. A non-violent method of opposing slavery, it was not drastic enough to satisfy […] abolitionist zealots[.]” —Tolles, F. (1943). Review of “The Free Produce Movement: A Quaker Protest against Slavery.” The New England Quarterly, 16(2), 351. 

78. For instance, in ACE’s case study on environmentalism, we noted that “[t]he proportion of solid municipal waste that Americans recycled rose from 9.3% in 1975 (one year after the introduction of the first curbside recycling program) to 34.3% in 2013, a 3.7-fold increase. Even more impressive is the breadth of participation: as of 2014, 76% of Americans said they recycle all or most of the time, with 96% recycling at least occasionally. One study found that while having the option to recycle changes consumption patterns, the effect is the same regardless of green attitude. This suggests that recycling is broadly understood by environmentalists and non-environmentalists alike.
    The environmental benefits of this widespread adoption have been substantial. For example, recycling releases less carbon into the atmosphere than would disposal of that waste in a landfill. In 2013, these carbon savings amounted to taking 39 million cars—about 15% of the nation’s vehicles—off the road for a year.” 

79. We are aware of reports or examples of leafleting playing a role in the children’s rights movement, the the slavery abolition movement, and the American revolution. For instance:

    • Pamphlets were used to inform Swedes about the 1979 Swedish corporal punishment ban. “The ban was well-publicized by the media, but more importantly, a 1-page color pamphlet explaining the the reason for the law and providing alternatives to corporal punishment was given to every household with a young child. These pamphlets were also distributed through media offices and child care centers and translated into all immigrant languages […] Further, for 2 months, information about the law was printed on milk cartons, to ensure that it was present at family mealtimes when parents and children could discuss the issue together.” —Durrant, J. (1996) The Swedish Ban on Corporal Punishment: Its History and Effects. Frehesse, D. Horn, W. Bussman, K. (Eds.). in Family Violence Against Children: A Challenge for Society, Berlin, Walter de Gruyter, 22.
    • Anthony Benezet’s The Case of Our Fellow-Creatures, a pamphlet describing the moral case against slavery
    • “Thomas Paine’s Common Sense, which was released in 1776, articulated many of the ideas that were gaining ground in favor of the American Revolution. Paine advocated for rejecting the tyranny of the British government and for adopting a more egalitarian system of governance. Common Sense reached hundreds of thousands of American Colonists and played an instrumental role in galvanizing them to take up the cause of the Revolution.” —Jon Camp, Vegan Outreach. (January 2, 2014). Vegan Publishers blog.

     

80. It is possible that leaflets did contribute to past social movements, but weren’t implemented on a large enough scale to have a visible macro impact. 

81. GiveWell also notes that “‘[m]acro’ evidence inherently does not come from carefully controlled studies, and so it can always potentially be wrongly attributing impact to a program—for example, one might observe that child mortality fell sharply following the introduction of a vaccination program, when in fact other factors (such as generally improving standards of living) had more to do with the decline. However, we consider a ‘macro’ story to be an important indicator that a program can work on a large scale.” —GiveWell. (2010). Criteria for Evaluating Programs—2009–2011. GiveWell. 

82. Although we haven’t performed a full analysis, it appears that public awareness of poor conditions for farmed animals has increased over the last 10–20 years. For instance, we think there has been more discussion of these issues in the media and through documentaries, and there has been more public reaction through seeking out “humane” alternatives and supporting legal and corporate policies that restrict certain practices. We would guess that most people who work on farmed animal advocacy would agree with this claim based on their personal experiences interacting with the public. There is also evidence supporting this general point in the conversations described in Part Five of this report. 

83. Google Trends search for “vegan” in the United States 

84. Our limited impression is that these changes in public awareness seem to correspond to the growth of undercover investigations as a tactic, and the reactions of farmers, corporations, and the media seem to indicate that these investigations have been an important driver of changing attitudes. 

85. “While Lewis [Bollard of The Open Philanthropy Project] isn’t highly confident in his views on this, he thinks the most important factors in building the animal welfare movement have included:

    • Writing and advocacy by Peter Singer and other philosophers. Animal Liberation sold millions of copies, and Lewis has heard the modern animal welfare movement described as “the first social movement started by a philosopher.” Early on in his position at the Open Philanthropy Project, Lewis asked roughly 40 current leaders in the animal welfare movement (e.g., Nathan Runkle, Paul Shapiro) what had originally influenced them to get involved, and over half mentioned Animal Liberation.
    • Publicity generated by undercover investigations, and to some extent, leafleting and other forms of outreach and activism. (After Animal Liberation, these types of activism were the next-most cited influences by the movement leaders Lewis spoke to.)
    • PETA, which played a particularly important role in getting publicity and mobilizing the movement early on.
    • Prop 2 in California, which seems to have served as more of a rallying point for the movement than other ballot measures. (Some of the movement leaders Lewis asked about their influences mentioned Prop 2.)” Muehlhauser, L. (Interviewer) and Bollard, L. (Interviewee). (February 23, 2017). A conversation with Lewis Bollard, February 23, 2017.

     

86. Similarly, some correlational evidence suggests that only 8% of respondents who had converted to vegetarianism or veganism listed leaflets as one of the top three reasons they initially changed their diets, compared to (for example) 43% for documentaries and 38% for books. Leaflets were recognized much less often as one of the factors that often played a role in conversions to animal product reducers or eliminators. See Large-scale survey of vegans, vegetarians, and meat reducers. (2014). Humane League Labs. 

87. Note that three of ACE’s staff and one of ACE’s former staff were a part of the 21 person sample initially contacted to complete this survey. There have also been some criticisms of the representativeness of the Researchers included in this survey. Also, the consensus of a sample of Researchers does not necessarily indicate the truth. 

88. Some correlational evidence suggests that only 8% of respondents who had converted to vegetarianism or veganism listed leaflets as one of the top three reasons they initially changed their diets, compared to (for example) 43% for documentaries and 38% for books. Leaflets were recognized much less often as one of the factors that often played a role in conversions to animal product reducers or eliminators. See Large-scale survey of vegans, vegetarians, and meat reducers. (2014). Humane League Labs. 

89. This seems to be the most informative of the estimated costs per vote to estimates concerning leaflet cost- effectiveness. 

90. See page 139. Green & Gerber noted that this estimate was only on the borderline of being statistically significant. 

91. Carl Shulman. (June 12, 2013). Re: Effective Altruism Through Advertising Vegetarianism? 

92. In this context, by long term we mean more than a decade. 

93. Predicting many long-term effects seems plagued by uncertainty. For example, immediately after World War I, popular opinion held that the war had been so terrible that no one would risk another such conflict with modern weaponry, and leaders tried to establish international structures that would prevent another such conflict. But historians now believe that the terms of the peace settlement contributed to the rise of the Nazi party and the outbreak of World War II. It is often difficult to determine the impacts of specific policy decisions or action groups even in retrospect. 

94. One possible parallel that does exist is to the antislavery movement and boycotts of slave-produced goods (with related support for competing industries) and the similar recent “Fair Trade” movement, which also seeks to promote products made by companies who treat human workers fairly. While boycotts of slave-produced goods in the late 1700s and early 1800s did not end slavery as an institution, we haven’t seen much evidence that they hampered the more radical movements which led to the end of chattel slavery in British territories and in the U.S. The ultimate effects of the Fair Trade movement are far less clear, since it is ongoing at the present time. 

95. A Gallup poll conducted in the U.S. in 2015 found that 32% of respondents supported animals having the same rights as people, while an additional 62% said they should have some protection. But the best estimates for the percentage of Americans who are vegetarian are much lower, around 1–2%. 

96. “We have our greatest success in helping others go vegan if we discuss the implications of what we do to animals; the ethical argument is by far our strongest one. Large advocacy groups with access to considerable resources and large followings tell others that we should be asking people to reduce—rather than end—their exploitation, diluting our collective vegan message of social justice and undermining the ethical argument. Too many people are being taught that animal exploitation is okay in moderation, and that the best approach to having a vegan world is to not talk about veganism at all. Be aware of the source of this wrong-headed advocacy approach: pandering by large animal advocacy groups to those engaging in the exploitation.” —Taft, C. (2015). Mainstream Animal Advocacy Messages Framed By Those Doing The Harm. Vegan Publishers. 

97. “Moral outrage is also described as “a response to the behavior of others, never one’s own.” It seems natural that institutional messaging would be more likely to spark the emotion because it puts the blame for the issue on an outside institution or one that the audience member is only a small part of, usually the animal agriculture industry or society as a whole.” —Reese, J. (2016). The Animal-Free Food Movement Should Move towards an Institutional Message. Medium. 

98. It seems somewhat relevant that our limited impression is that moderate and mainstream strategies can succeed despite the criticism from more radical flanks of the movement. That impression is at least weakly supported in ACE’s case study on environmentalism. 

99. We don’t have a formal method for weighing multiple considerations against one another. Since it seems likely that the vast majority of animals we can affect will live in the far future, however, if we were certain about the long-term effects of many interventions, this would probably be our primary consideration in deciding what to promote. However, it is not because although it is very important, we don’t have enough confidence in our predictions to base all our decisions on what we believe about the far future. 

100. “The Humane League and Vegan Outreach put out a special leaflet for the Warped Tour that has pictures and quotes of vegetarian band members from the groups playing the tour. When they pass out the literature they say “message from the bands” instead of “flyer to help animals” or something similar, because some of the material in the leaflets is from band members and because it makes people more likely to take the booklet.” —The Humane League Site Visit 2015: San Diego Warped Tour 

101. This judgement is based on their attendance at the Warped Tour music festival, which is an alternative music festival. 

102. “Vic Sjodin from Vegan Outreach was there giving people tips on how to leaflet based on his experience. Jon thought Vic’s advice to newer leafleters was good; it was fairly standard advice, such as making sure to extend the hand with the leaflet and always smiling.” —The Humane League Site Visit 2015: San Diego Warped Tour 

103. The case study also mentions that Vegan Outreach volunteers had been told not to interrupt video watchers because they (or at least some long-term leafleters) felt the video conveyed their message more effectively than any conversation would. —Jarvis Jr, L. C. (2016). Moving mountains: Animal rights organizations, emotion, and autodidactic frame alignment. Florida Atlantic University, p.148. 

104. They also had Spanish-language leaflets on hand to help reach out to the Latino community in South Florida. See Jarvis Jr, L. C. (2016). Moving mountains: Animal rights organizations, emotion, and autodidactic frame alignment. Florida Atlantic University, p.147. 

105. “She suggested that, on any given night, they may have anywhere from 2 to 10 people stop to ask questions about animal cruelty or a vegan diet.” —Jarvis Jr, L. C. (2016). Moving mountains: Animal rights organizations, emotion, and autodidactic frame alignment. Florida Atlantic University, p.148. 

106. In Lee’s words he “witnessed one young man shouting ‘oh shit!’ at the documentary and then immediately saying ‘nuh uh’ to a volunteer reaching out a pamphlet to him. (OC—the ‘oh shit’ was sarcastic and the ‘nuh uh’ was, for lack of a better term, ‘smart aleck-y’).” —Jarvis Jr, L. C. (2016). Moving mountains: Animal rights organizations, emotion, and autodidactic frame alignment. Florida Atlantic University, p.149. 

107. In Bockman’s case study the take rate was roughly 369 leaflets per person hour. In Smith’s case study the take rate was 219 leaflets per person hour. 

108. In our work to identify the most effective ways to help animals, ACE employs both qualitative and quantitative strategies. One way that we evaluate programs (or groups of programs) quantitatively is by assigning numerical values to their immediate costs and benefits in order to model their cost effectiveness. For instance, we might estimate the number of animals helped by a particular ad campaign and the costs that were invested in that campaign. We then use those numbers to calculate a cost-effectiveness estimate (CEE) in terms of “lives spared per dollar” or “years of suffering averted per dollar.” These estimates allow us to directly compare different programs and charities, which helps us decide which of these programs and charities affect the most animals per dollar. Cost-effectiveness estimates are only one of ACE’s criteria; the other six are:

     • The charity has room for more funding and concrete plans for growth.
     • The charity engages in programs that seem likely to be highly impactful.
     • The charity possesses a strong track record of success.
     • The charity identifies areas of success and failure and responds appropriately.
     • The charity has strong leadership and a well-developed strategic vision.
     • The charity has a healthy culture and a sustainable structure.

      

109. For instance:

     • In their order form, Vegan Outreach lists costs at about $0.07 per leaflet.
     • We were told that “not including the costs of printing, it costs them 2¢ to distribute a Veg Starter Kit and 4–6¢ to distribute a leaflet.”
     • We were also told: “MFA expect[s] to distribute about 1.4 million booklets by the end of the year. Overall, this program costs roughly $100–130k.”

      

110. For instance:

     • We have been told: “[e]ach additional leaflet is fairly cheap; currently, Vegan Outreach asks for $0.07–$0.25 depending on the specific booklet for at-cost distribution of their materials. Factoring in both groups’ costs for THL distributing Vegan Outreach leaflets, the cost per leaflet rises to about 12¢.”
     • This report from Vegan Outreach puts the marginal cost of a leaflet at $0.11.

      

111. An SCI is a range of values that communicates a subjective estimate of an unknown quantity at a particular confidence level (expressed as a percentage). We generally use 90% SCIs, which we construct such that we believe the unknown quantity is 90% likely to be within the given interval and equally likely to be above or below the given interval. 

112. When calculating our SMD of Hedges’ g for each study, we took the difference in consumption change between the experimental and control groups and divided by a pooled standard deviation. While this would typically be derived from the standard deviation of the changes, in some cases we only had information about consumption after receiving the leaflet, and hence used the standard deviation of consumption for that single period of time. Since our calculation of the pooled SMD combined effect sizes estimated by these two different methods, we believe either would have been a reasonable choice for converting the SMD to a measure of consumption. For our cost-effectiveness estimate, we estimate the change in consumption associated with leaflets as the SMD, multiplied by the standard deviation of all participants’ consumption, added to the change in consumption for the control groups. 

113. The project leader used their judgment to convert the estimate for red meat consumption into cow consumption and pig consumption. 

114. This was found by combining the SMD for individual consumption of pig and cow in the studies involved in the meta-analysis. 

115. The project leader used their judgement to convert the estimate for poultry consumption into an estimate for chicken and turkey consumption. 

116. Note that approximately 40% of fish consumed are farmed fish, both in the U.S. and in the world as a whole. According to 2016 data from the Food and Agriculture Organization (FAO), 44.1% of world fish production by mass (p. 4), and about 8% of U.S. fish production by mass (p. 173), comes from aquaculture. In 2015, the U.S. produced about 7.75 billion pounds of edible fishery products, 6.94 billion of which it exported, while it imported 11 billion pounds of such products. Thus, about 93% of U.S. fish consumption was of imported fish products, and we think it is reasonable to assume that about 44% of the fish killed in the production of these imported products were farmed fish. (We assume that farmed fish have an equal average weight to wild-caught fish, and that farmed fish products are as likely to be imported as wild-caught fish products. We also assume that the same proportion of farmed fish and wild fish are caught for consumption; according to the 2016 FAO data, a large majority of fish products by weight are consumed as food, so errors arising from this assumption are likely to be small.)
     The National Oceanic and Atmospheric Administration also states that, based on 2011 data, “[a]bout half the seafood we eat is wild-caught; the other half is farm-raised, that is, from aquaculture.”—The Surprising Sources of Your Favorite Seafoods. (2012). National Oceanic and Atmospheric Administration Office ofAquaculture. 

117. The standardized mean difference (SMD) provides an estimate of the ratio between the change in animal product consumption associated with leaflets and the standard deviation of animal product consumption within the population. Using data from a particular study, in this case the one by THL, we can multiply the SMD by the standard deviation of consumption of a particular product, say eggs, to obtain the expected change in egg consumption associated with leaflets. Since the THL data is in terms of the number of meals eaten per week, rather than number of animals consumed, we divide this result by the mean of the THL data to obtain the percent change in consumption associated with leaflets. We can then multiply this by the average number of eggs consumed per year to provide an estimate of the change in eggs consumed that is associated with leaflets. Multiplying by an elasticity factor, which relates demand and supply for a given product, then gives us an estimate for the change in number of animal products produced by the animal agriculture industry due to the change in demand associated with leaflets.
     Note that this estimate makes some assumptions which may be inaccurate. In particular, it assumes that a change in, say, number of meals consumed containing a product is a good proxy for change in the total amount of that product consumed by a person. It also assumes that changing behavior in response to leaflets is uncorrelated with previous consumption rates. 

118. The ranges from five computations from the Guesstimate model were: -0.29 to 6.4, -0.27 to 7.9, -0.27 to 5.6, -0.27 to 8.5, -0.26 to 5.8 animals. The method we use does calculations using Monte Carlo sampling. This means that results can vary slightly based on the sample drawn. Unless otherwise noted, we have run the calculations five times and rounded to the point needed to provide consistent results. For instance, if sometimes a value appears as 28 and sometimes it appears as 29, our review gives it as 30. 

119. The ranges from five computations from the Guesstimate model were: -0.34 to 0.62, -0.4 to 0.85, -0.29 to 0.73, -.31 to 0.66, and -.27 to 0.77 years of farmed animal life. The method we use does calculations using Monte Carlo sampling. This means that results can vary slightly based on the sample drawn. Unless otherwise noted, we have run the calculations five times and rounded to the point needed to provide consistent results. For instance, if sometimes a value appears as 28 and sometimes it appears as 29, our review gives it as 30. 

120. For example:

     • Together, these figures show that the reduce and eliminate messages decreased self-reported meat consumption over the preceding 30 days, on the order of 1.1 servings of meat per week for the reduce appeal (p= 0.028) and 0.90 servings of meat per week for the Eliminate appeal (p= 0.002). (Macdonald, Caldwell & Boese, 2016)
     • Vegan Outreach reported that based on their randomized trial on MTurk the data suggested that for every 75 booklets read that could result in one person reducing their consumption of all non-vegan foods to less than once per week.

      

121. The 90% confidence interval for the effect size is a decrease of consumption by 0.3% to an increase of 6.6% in the treatment group. See Our Initial Thoughts on the Mercy For Animals Facebook Ads Study. ((The 90% confidence interval for the effect size is a decrease of consumption by 0.3% to an increase of 6.6% in the treatment group. See Our Initial Thoughts on the Mercy For Animals Facebook Ads Study. 

122. Still, leaflets may cause a shift away from consumption of larger farmed land animals (e.g., cows and pigs) to greater consumption of broiler chickens. 

123. This is meant to include the possibility that data extraction errors biased the estimate. 

124. In comparison, the estimated standard deviations for each estimated SMD, once accounting for general model uncertainty, were:

     • Poultry 0.098
     • Red meat 0.080
     • Fish 0.098
     • Eggs 0.071
     • Dairy 0.080

      

125. For example, it is likely that, while we felt fairly uncertain about the effects of leaflets prior to this analysis, our priors would have had less probability mass on a positive SMD. That is, before we conducted this analysis, we might not have thought it was reasonable to estimate a roughly even chance of leaflets causing net increases in animal product consumption. Our mean may also have been lower, reflecting an expectation that leafleting would cause a substantial decrease in animal product consumption. For the purpose of this analysis, we chose to use a fairly conservative prior which mainly incorporated a high level of uncertainty without making strong assumptions about the magnitude or direction of the dietary changes associated with leaflets. 

126. In the future, we would like to use a more formal process to correct for model uncertainty, as we have with prior knowledge. Due to time constraints, we instead made an adjustment to SMD variance according to our best judgment. 

127. The ranges from five computations from the Guesstimate model were: -0.29 to 6.4, -0.27 to 7.9, -0.27 to 5.6, -0.27 to 8.5, -0.26 to 5.8 animals. The method we use does calculations using Monte Carlo sampling. This means that results can vary slightly based on the sample drawn. Unless otherwise noted, we have run the calculations five times and rounded to the point needed to provide consistent results. For instance, if sometimes a value appears as 28 and sometimes it appears as 29, our review gives it as 30. 

128. The ranges from five computations from the Guesstimate model were: -0.34 to 0.62, -0.4 to 0.85, -0.29 to 0.73, -.31 to 0.66, and -.27 to 0.77 years of farmed animal life. The method we use does calculations using Monte Carlo sampling. This means that results can vary slightly based on the sample drawn. Unless otherwise noted, we have run the calculations five times and rounded to the point needed to provide consistent results. For instance, if sometimes a value appears as 28 and sometimes it appears as 29, our review gives it as 30. 

129. The ranges from five computations from the Guesstimate model were: -1.3 to 7, -1.4 to 9.4, -1.4 to 6.6, -1.4 to 6, -1 to 9.2 animals. The method we use does calculations using Monte Carlo sampling. This means that results can vary slightly based on the sample drawn. Unless otherwise noted, we have run the calculations five times and rounded to the point needed to provide consistent results. For instance, if sometimes a value appears as 28 and sometimes it appears as 29, our review gives it as 30. 

130. The ranges from five computations from the Guesstimate model were: -1.5 to 1.2, -1.3 to 2.3, -1.8 to 1.3, -1.4 to 1.4, and -1 to 2.2 years of farmed animal life. The method we use does calculations using Monte Carlo sampling. This means that results can vary slightly based on the sample drawn. Unless otherwise noted, we have run the calculations five times and rounded to the point needed to provide consistent results. For instance, if sometimes a value appears as 28 and sometimes it appears as 29, our review gives it as 30. 

131. Matt Ball, Rachel Black, and Jon Camp all have extensive experience with animal advocacy leafleting, which was a driving factor in ACE’s decision to interview them for this report. 

Conclusions

Overall assessment

Below are a number of suggestions that may improve the effectiveness of leafleting. These suggestions are based almost entirely on anecdotal evidence, but anecdotal evidence may still be useful in this case. For example, people who are leafleting will have real-time feedback about what does and does not make a difference to the take rate, and over a long period of time they may form quite accurate opinions about this.

In order to prepare properly for the leafleting event, consider the following suggestions:

• Suggest leafleting as an option for people who are looking to volunteer. Leafleting does not require specialized skills or costly resources and so it can be a good opportunity for people to first become involved in animal advocacy. It could be important that leafleters are not put off of further participation by their experience.
• Carry the right type of leaflet for your audience. If you expect to encounter many people who are distinct demographic groups, then carrying stocks of leaflets targeted to each of those groups may be a good idea.
• Give leafleters basic advice about leafleting before starting. It may take only a few minutes to give people a lot of what they need to know.¹
• Dressing well and being well-groomed is often recommended.

When planning where to hold the leafleting event, consider the following suggestions:

• Consider areas or events where people are more likely to be young and socially progressive.²
• Pick areas with moderate to high foot traffic. This will allow you to hand out more leaflets in the same amount of time. Areas with too much traffic can be bad because the take rate can decrease and you may be less sure that you are handing everyone a leaflet.³
• Pick the right time to do the leafleting. Certain times of day, days of the week, or times of the year may be more effective than others. For example, approximately 8 AM has been described as the best start time for leafleting on campuses.⁴
• Know what areas have already been leafleted and inform your organization or other leafleters of your leafleting plans. Coordination and collaboration between organizations is important.⁵
• Try to leaflet in public areas such as on college or university campuses, or on the sidewalk near events, to avoid any trouble with the authorities. You may still be asked to leave. Often people who make such requests are ignorant of the laws and regulations around leafleting, but to avoid any significant negative incidents (e.g., attempts to have leafleters banned from a campus) it’s probably a good idea to leave.

In order to improve the take rate while leafleting, consider the following suggestions:

• Be (or try to act) confident and friendly. Personality may play a large role in how effective a leafleter is, so selecting people with appropriate personalities for it is a good idea.⁶
• Say a short phrase as you are presenting the leaflet to someone, such as “help animals.” Say this as a statement rather than as a question.⁷
• Use appropriate body language. For example, extending your arm fully and leaning towards the person when presenting a leaflet to someone is important. Body language is very important in improving the take rate.⁸
• Be gracious whether or not the person takes the leaflet.⁹

Evaluative questions

To what extent is this intervention cost-effective when compared to other interventions we have evaluated?

The cost-effectiveness estimate completed in Part Four appears to indicate that compared to the short-term calculable effects of other animal advocacy interventions, leaflets appear less cost-effective. It is worth reiterating that it is fairly concerning that for some animal products the cost-effectiveness estimate assigns less probability to short-term decreases in consumption than to to short-term increases in consumption. We believe that a qualitative analysis that considered the results of this cost-effectiveness estimate along with the variety of factors mentioned in Part Three of this report would moderately update us towards the conclusion that the effects of leaflets are positive in expectation. For instance, since leaflets seem a priori more likely than not to promote concern for farmed animals, they could complement other farmed animal advocacy interventions in ways that may accelerate and make the success of those other interventions more likely. Leaflets also appear to have played a non-trivial role in building the contemporary animal advocacy movement. The indirect effects of leaflets don’t seem to compare particularly well with the indirect effects we would expect from other animal advocacy interventions. As with most interventions performed by animal advocates, we think the long-term effects are more likely to be positive than negative, because promoting concern for animals’ interests is so important that in the absence of strong reasons to believe the effects are negative, we think it is likely that the effects are positive on balance. We don’t think the long-term effects of any animal advocacy intervention are extremely well understood, though some seem clearer than the long-term effects of leaflets, and so we put limited weight on this consideration in our overall understanding of how effective or ineffective any intervention is. It seems the most important considerations include short-term diet effects, short-term effects on farmed animal welfare, contributions to movement building, and plausible long-term effects; in each of these areas, we have reason to think that another intervention outperforms leafleting. In short, our limited impression is that qualitative concerns do not generally favor leafleting, and in some areas indicate that other interventions are preferable.
Scale:
1	2	3	4	5
This intervention is not cost-effective compared to other interventions we have evaluated.		This intervention is comparable to the other interventions we have evaluated, in terms of cost-effectiveness.		This intervention is cost-effective compared to other interventions we have evaluated.
Level of Certainty:
1	2	3	4	5
We are highly uncertain about the cost-effectiveness of this intervention.		We are moderately certain about the cost-effectiveness of this intervention.		We are highly certain about the cost-effectiveness of this intervention.

To what extent does this intervention achieve positive outcomes for animals?

Leafleting is a relatively well-studied animal advocacy intervention, but the state of the evidence regarding its effects is still very weak. Before conducting our meta-analysis, we would have guessed that the primary effects of leaflets in the short term would be reductions in animal product consumption. Despite the high risk of bias and the severe limitations of the six particularly relevant field randomized controlled trials, the evidence that they produce appears to us to be the strongest evidence about the effects of leaflets in the short term. A meta-analysis of those trials suggested that leaflets have a very small effect on short-term consumption of animal products, if any. Judging from those results alone, the leaflets seem about as likely to increase short-term consumption of animal products as they are to decrease short-term consumption of animal products. However, leaflets seem more likely than not to promote concern for farmed animals, they do appear to complement other farmed animal advocacy interventions in ways that may accelerate and make the success of those other interventions more likely; leaflets also appear to have played a non-trivial role in building the contemporary animal advocacy movement.
Scale:
1	2	3	4	5
This intervention creates no net positive change (and might even create net negative change) for animals.		This intervention creates some net positive change for animals.		This intervention creates significant net positive change for animals.
Level of Certainty:
1	2	3	4	5
We are highly uncertain about the impact this intervention has for animals.		We are moderately certain about the impact this intervention has for animals.		We are highly certain about the impact this intervention has for animals.

Should the animal advocacy movement continue to devote the same amount of resources to leaflets than it does currently?

The short-term and indirect effects of leaflets don’t seem to compare particularly well with the indirect effects we would expect from other animal advocacy interventions. We would guess that the long-term effects of leafleting would also not compare particularly well to those of other interventions. While ACE encourages leafleting for its good potential to involve new activists, we encourage groups with the ability to do so to carry out other programs, such as corporate outreach and undercover investigations, which seem more reliably effective overall. We currently don’t recommend that organizations create new leafleting programs or expand existing programs when that funding could be used for more promising interventions. We feel that the main benefits of leaflets are that they can quickly reach a large number of people at a low cost, and that they provide an easy way for novice advocates to become further involved. However, we feel that these benefits are outweighed by the concerns of uncertain impact per recipient and the potentially net-negative short-term effects on consumption. Our limited understanding of the indirect effects and our very limited estimate of the long-term effects also lead us to suggest that the movement not devote more resources to leafleting.
Scale:
1	2	3	4	5
The animal advocacy movement should devote far fewer resources to protests than it does currently.		The animal advocacy movement should continue to devote the same amount of resources to protests that it does currently.		The animal advocacy movement should devote far greater resources to protests than it does currently.
Level of Certainty:
1	2	3	4	5
We are highly uncertain about the amount of resources that the animal advocacy movement should devote to this intervention.		We are moderately certain about the amount of resources that the animal advocacy movement should devote to this intervention.		We are highly certain about the amount of resources that the animal advocacy movement should devote to this intervention.

Questions for further consideration

We hope that future research will further investigate the effects of leaflets, as well as the relative effectiveness of different kinds of leaflets. In particular, we would like to see the following questions addressed:

• To what extent do the specific arguments, visuals, or asks in a leaflet influence the effects it has? Are some arguments, visuals, or asks much more effective than others?
• To what extent does the impact of leaflets vary depending on the country where they are distributed? Is leafleting in some countries much more effective than leafleting in other countries?
• To what extent does the impact of leafleting vary depending on the demographics of the recipient? Is leafleting some demographics much more effective than leafleting other demographics?
• What does research into the effectiveness of individual outreach interventions that seem analogous to leafleting, such as online ads, suggest about the effectiveness of leafleting?
• The case studies analyzed in Part Four of this report may not represent all common types of leafleting events. What do case studies of representative leafleting events suggest about the effectiveness of leafleting?
• Should future meta-analyses evaluating the effect of leaflets attempt to incorporate the results from non-randomized field trials or from randomized Amazon Mechanical Turk trials?
• How can we better assess the impact of leafleting’s more indirect effects? For instance, how can we better assess the effect of leaflets on a variety of important outcomes, such as support for welfare reforms and ballot measures, demand for cultured animal products, and demand for animal products sourced through higher-welfare methods?
• How can we better assess the medium- and long-term effects of interventions in general?
• How should the cost-effectiveness estimate for leafleting impact our cost-effectiveness estimates for similar interventions like online ads?
• More generally, how should the results of our meta-analysis and cost-effectiveness estimate for leafleting impact our understanding of the effectiveness of other interventions that aim to change individuals’ attitudes and behaviors?

---

1. “[Jon] sees training for basic leafleting scales as taking ‘a minute or two’ and experience as making very little difference.” —Conversation with Jon Camp of The Humane League 

2. See ACE’s Conversation with Andrea Gunn of The Humane League. 

3. According to Vegan Outreach’s Leafleting Tips from the Pros! “When traffic gets heavy, the turndown rate sometimes gets high […] But even with a lower acceptance rate, you can still give out a lot over the course of an hour in high traffic.” 

4. Jon sees the start of the semester as the ideal time to leaflet colleges […] Jon believed, based on experience with Vegan Outreach, that the best time for leafleting at colleges was from around 7:55 AM., shortly before the 8:00 class changes, to about 2 or 3.” —Conversation with Jon Camp of The Humane League 

5. “[Jon] sees both organizations [THL and Vegan Outreach] as seeing coordination to avoid duplication as essential.” —Conversation with Jon Camp of The Humane League 

6. “Having a high confidence level, being positive, smiling, and being energetic are key characteristics of a good leafleter. Even if you aren’t outgoing, you should want to be able to fake that for the hour that you’re leafleting, because it makes a big difference.” —Conversation with Rachel Black of the Humane League 

7. “[Jon] has found that short statements like ‘help animals’ work better than questions or statements which imply doubt like ‘info to help animals.”‘ —Conversation with Jon Camp of The Humane League 

8. Sjodin, Vic. (2013). How to leaflet? Basic leafletting [sic] techniques by Vic Sjodin [Video file]. 

9. “Even if a leaflet is rejected, Jon sees presenting the interaction as positive to onlookers as helpful, for example by saying “have a good day,” rather than allowing an air of rejection.” —Conversation with Jon Camp of The Humane League