Our Approach to Evaluating Animal Advocacy Interventions
ACE reviews evidence on the efficacy of interventions in animal advocacy in order to help determine which are likely to be most useful. Our research on interventions serves multiple purposes: not only do we use our intervention evaluations to help shape our charity evaluations, but they inform animal advocates so they can plan their campaigns with evidence in mind. Not every implementation of an intervention is equally effective, so in our intervention evaluations we try to consider representative implementations, and we discuss the factors that are most likely to affect results when they vary.
Selection and Prioritization
Our selection of interventions is guided by two factors. First, we seek to conduct evaluations that will provide relevant evidence to animal advocacy organizations and to potential donors. Interventions should be implemented by at least one animal advocacy group working in a high-potential-impact area (for instance, farm animal advocacy or wild animal suffering) to be considered. Interventions that form a large part of some organization’s activities are particularly relevant to examine from a donor’s perspective, as donations to these organizations then support mostly or exclusively that intervention, as with leafleting and Vegan Outreach.
Second, we prioritize evaluations of interventions based on the probability that after the evaluation, we will be able to conclude that the intervention is either better or worse than average. The more information that is available about an intervention, the more likely our evaluation will be to end with a clear recommendation regarding it. For instance, we will likely have a difficult time evaluating the effectiveness of campaigns to promote legislative change, relative to campaigns like leafleting for individual diet change that produce more incremental results and whose effectiveness can thus be measured at any point in time.1
This section describes the process for our evaluation work starting in early 2014.
ACE’s Director of Research works with staff and qualified volunteers to conduct evaluations. ACE staff and volunteers working on evaluations will be referred to below as ‘evaluators’. When possible, two evaluations of the same intervention are conducted independently, and evaluators reconcile their findings.
We use an evaluation template to ensure that crucial factors are considered for each intervention. Evaluators ask questions of individuals who use the intervention; review financial reports; perform searches of academic, industry, and activist literature; and sometimes conduct original research. Evaluators document conversations and any original research so that sources can be provided in the final report.
In our evaluations, we seek to consider both quantitative and qualitative evidence. We also consider evidence directly connected to the intervention in question (for instance, studies of a particular implementation in an animal advocacy context) as well as evidence more distantly related (for instance, from general psychological studies or from sociological work done on other movements for social change). However, not all evidence is equally strong. Characteristics we consider important in judging the strength of evidence include the following.
- The source of the evidence. The strongest evidence is that produced by an impartial and reputable source. Evidence that argues against its source’s interests is also particularly strong.
- The representativeness of the evidence. Evidence that includes all relevant instances or an apparently random sample of them is stronger than evidence that has been or may have been curated to make a specific point.
- The length of the inference chain of which the evidence is a part. Long chains of inference magnify the possibility for error, even if each step in the chain is relatively well understood.
Ideally, we consider evidence showing whether interventions caused significant change, but in many cases, the available evidence is only evidence about whether interventions are correlated with change. Most interventions on behalf of animals have not been tested with large scale randomized controlled trials; indeed doing so would prove difficult. Therefore, in addition to considering evidence that directly suggests causation, we consider evidence that suggests correlation when paired with a potential causal mechanism. In this case, there should also be evidence that the causal mechanism is plausible, such as randomized controlled trials that show an analogous mechanism or parts of the mechanism working elsewhere.
The finished product of each evaluation is a narrative report summarizing the evaluator’s research and conclusions. One component of this report is a detailed cost-effectiveness estimate, but it is important to note that such estimates are subject to many and varying sources of error and should not be used in isolation.
We publish the results of finished evaluations along with relevant supporting materials. When we have used materials that we do not have the right to reproduce, we summarize and cite the sources we have used.
We revise individual intervention evaluations as we become aware of evidence that has the potential to affect our opinion of the intervention in question. Due to time constraints, we may sometimes wait for multiple new pieces of evidence to accumulate before revising an evaluation. If you are aware of evidence we haven’t considered that is relevant to an intervention evaluation, please contact us.
We are committed to reconsidering our recommendations every year in December. While we update individual evaluations on a rolling basis and some may not be revised in a given year, this gives us a concrete time to make sure that our overall recommendations are consistent with our latest research.
GiveWell has an interesting series of posts explaining their understanding of the effectiveness of policy-oriented philanthropy, as compared to developing-world aid. It is notable that they conclude that determining whether philanthropy has a strong track record of influencing public policy would require “an enormous, long-term effort.”
Although these interventions are arranged roughly according to our confidence in their cost-effectiveness, the evidence available varies by type. Therefore, some of the interventions whose cost-effectiveness we have not been able to estimate with confidence may nevertheless be highly cost-effective and have large impacts.
Corporate Outreach: Organizations work with restaurant chains, supermarkets, and other businesses to strengthen a variety of possible animal welfare policies. These include cage-free-egg campaigns, but also campaigns to increase the availability of vegetarian and vegan food options and campaigns against gestation crates and other particularly cruel practices.
Leafleting: Organizations provide veg advocacy literature and/or send teams to distribute that literature on sidewalks and college campuses. Individuals can also easily obtain leaflets to distribute on their own and it takes only a small time commitment to do so, making leafleting alone or with a group a promising volunteering activity.
Online Ads: Organizations show ads on Facebook or other websites that link to a webpage with pro-veg/anti-meat information, often a video or text page that emphasizes a Vegetarian Starter Guide, and encourage the users to pledge to go vegetarian or enter their email address for more information. Currently, ACE does not recommend that organizations create new online ads programs or expand existing programs, at least when that funding could be used for more promising interventions such as corporate outreach and undercover investigations.
Humane Education: Humane education is a form of animal advocacy in which speakers visit high school and college classes and give presentations on the effects of factory farming on animals, the environment, health, and other areas of concern. Currently, ACE does not have enough strong evidence to recommend humane education as an intervention.
Undercover Investigations: Undercover Investigations: We use the term “undercover investigation” to refer to any project where activists obtain documentation (e.g. photos and videos) of the treatment of animals without the explicit cooperation of the people or organizations using the animals. We focus primarily on investigations of farms and other animal agriculture facilities.
Interventions we are considering but which we have not yet evaluated are listed below. For some interventions, the evaluation is currently in process. Other interventions are listed although we have not yet begun formally evaluating them.
Advertising: Organizations pay for advertising on billboards, commercials (television and online), and other advertising platforms. Advertisements use limited space and time to educate viewers about industrial agriculture, farmed animals, and vegetarian or vegan eating, often subject to content restrictions for the platform.
Animal Welfare Food Labels: Organizations promote the use of clear food labeling that informs consumers about the conditions of animals used in producing food items. Alternatively, organizations oppose the use of misleading labels.
Boycotts: Organizations or individuals refuse to buy certain products or to do business with certain companies in protest of their policies. Veganism can be thought of to some extent as a boycott of animal agriculture, but boycotts are typically more organized and include clearer sets of demands indicating under what circumstances the boycott would end.
Cage-Free-Egg Campaigns: Organizations lobby college food service providers and other businesses to commit to purchasing only cage-free eggs or to increasing the proportion of their egg purchases that are cage-free. Some campaigns are conducted privately between organization representatives and decision-makers, while others involve petitions and public pressure tactics.
Demonstrations: Organizations or coalitions of individuals air their grievances in public space. Demonstrations target systemic problems rather than specific policies of individual companies or agencies, and can range from a few individuals holding signs to mass marches or creative protests.
Farm Animal Rescue: Organizations and individuals rescue animals from farms or slaughterhouses. Rescues sometimes involve taking videos or photographs of the conditions the animals were living in and generally involve rehabilitating them and either placing them for adoption or offering them sanctuary for the remainder of their lives.
Grassroots Political Campaigning: Individuals and groups affect the political process in many ways, including letter-writing campaigns and demonstrations, to encourage the passage of bills that would benefit animals or the defeat of bills that would harm them. Grassroots campaigns involve the participation of many interested individuals, often at a local level or through networks not deeply embedded in the political power structure.
Humane Farming Promotion: Organizations research how farming practices can be improved to increase animal welfare and promote such improvements. Means of promotion include grants to farmers to implement new practices and educational and publicity efforts supportive of certain uses of animals.
Institutional Meat Reduction Campaigns: Organizations work with institutions including school districts and hospitals to implement a variety of meat reduction techniques such as Meatless Mondays and increased availability of vegetarian and vegan options in cafeterias.
Cultured Meat Research: Scientists and companies perform basic and applied research intended to lead to the availability of meat grown in labs as a competitor to industrial animal agriculture. Non-profits help researchers secure funding, gain publicity, and make connections.
Letter-Writing Campaigns: Individuals or groups send letters addressing animal issues to government officials or to newspaper editors or publications. Organizations orchestrate campaigns of this type by distributing sample letters to their members, encouraging them to adapt them and send them to their elected representatives or local papers.
Lobbying: Organizations work with lobbying firms or meet with state or national elected officials on their own behalf to influence the legislative process, specifically legislation affecting animals. They help write bills and find co-sponsors, or explain how they think the officials should vote and why.
Meat Substitute Creation: Scientists and companies develop plant-based meat substitutes and market them to compete with the products of animal agriculture. Non-profits and businesses help promote and publicize these products through activities such as feed-ins.
Media Campaigns: Organizations and individuals disseminate information about industrial agriculture, farmed animals, and vegetarian and vegan diets through traditional media. For example, organizations produce videos of their investigations of farm conditions which are shown on the news media, sometimes along with interviews of people involved in the organization.
Pay-Per-View Video Outreach: Organizations and volunteers bring video equipment to areas with heavy foot traffic and pay passers-by small amounts of money to watch brief videos about farmed animals and industrial agriculture. They also provide resources to support diet changes inspired by the videos.
Protests: Organizations and coalitions of individuals object to the policies of specific companies and institutions. Protests can range from a few people holding signs to mass marches and other coordinated actions.
Veg Starter Guide Stands: Organizations and individuals distribute Veg Starter Guides and other literature using newsracks or by placing the stacks of the literature in coffee shops and other businesses that allow this. Individuals in the area then have immediate access to the guides’ information.
VegFests: Organizations organize celebrations of veg eating ranging in length from a day to a week. Events include food vendors and samples, educational experiences such as movies, lectures, and reading material, and community building.