Our Approach to Assessing Programs

Introduction

To identify animal advocacy organizations working in the most effective areas, we include Programs as one of our four charity evaluation criteria. In this criterion, we assess the expected effectiveness of charities’ programs without considering their actual achievements. During our assessment, we analyze the groups of animals a charity’s programs affect, the countries in which they take place, the outcomes they work toward, and the interventions they use to achieve those outcomes.

Our 2022 Programs Assessment

Gathering Information

At the beginning of our evaluation process, we select a group of animal advocacy organizations to invite to participate in this year’s evaluations. Selection begins with ACE’s comprehensive list of animal advocacy organizations, which already contains some information about the four factors we assess in this criterion: animal groups, countries, outcomes, and interventions.¹ We narrowed down this list using our quantitative model,² and our evaluations committee then used a process of iterative discussion and voting to determine which charities to invite.

For the organizations that accepted our invitation in 2022, we requested further details about each of their programs, including their expenditures per program. Due to already narrowing down our list of charities and inviting the most promising ones, we expected that those who accepted our invitation already work in high-priority program areas; therefore, they all generally performed well in this criterion.

Prioritization of key factors

In 2022, we used a scoring framework to assess the effectiveness of charities’ programs. We used the Scale, Tractability, and Neglectedness (STN) framework to score the priority levels of different types of animal groups, countries, outcomes, and interventions; for countries, we also included an assessment of global influence.

Individual ACE research team members scored various types of animal groups, countries, outcomes, and interventions on a 1–5 scale, using their own percentage weights for STN. We then averaged these scores and percentage weights to calculate an overall priority level score for each type. In each subsection below we link to spreadsheets that illustrate our 1–5 scoring and STN weightings.

Animal groups

We prioritize groups of animals that are affected in larger numbers, whose situation seems tractable, and who receive relatively little attention in animal advocacy. We consider farmed animal advocacy high priority because of the large scale of animal suffering involved and its high tractability and neglectedness relative to other cause areas. Among farmed animals, we prioritize farmed fishes and farmed chickens, as well as programs targeting farmed invertebrates. Given the large number of wild animals and the small number of organizations working on their welfare, we believe that wild animal advocacy also has potential for high impact despite its lower tractability. For more details on how we prioritized different groups of animals in 2022, see this spreadsheet.

Countries

We prioritize counties with relatively large animal agricultural industries, few other charities engaged in similar work, and in which animal advocacy is likely to be feasible and have a lasting impact. Additionally, we consider global influence as a fourth factor in our country prioritization. Our methodology for scoring countries uses Mercy For Animals’ (MFA) Farmed Animal Opportunity Index (FAOI) for scale, tractability, and global influence. However, we use different weightings for each of these factors, and we assess neglectedness by comparing our own data on the number of farmed animal organizations working in each country to the human population (in millions) of that country. For more details on how we prioritized countries in 2022, see this spreadsheet.

Outcomes

We categorize the work of animal advocacy charities into six outcomes. We prioritize those outcomes using the STN framework, and we consider long-term impacts as an additional factor in our prioritization. As a result of using our framework, we consider the following outcomes to be higher priority: improving welfare standards, increasing the availability of animal-free products, and strengthening the animal advocacy movement. We give lower priority to the following outcomes: decreasing the consumption of animal products, increasing the prevalence of anti-speciesist values, and providing direct help to animals. For more details on how we prioritized outcomes in 2022, see this spreadsheet.

Interventions

We currently categorize the interventions charities use into 16 types. Similar to our prioritization of outcomes, we prioritize those intervention types using the STN framework and consider long-term impacts as an additional factor in our prioritization. In line with our commitment to following empirical evidence and logical reasoning, we use existing research to inform our assessments and explain our thinking about the effectiveness of different interventions. Where available, we used relevant research and our research briefs to inform our scores and corresponding priority levels for each intervention type. For more details on how we prioritized intervention types in 2022, see this spreadsheet.

Calculation of overall scores

We used information supplied by the charity to estimate the percentage of program funding spent on different types of animal groups, countries, outcomes, and interventions. Using those estimates and our priority level scores, we arrived at a single program score for each charity, which represented the expected effectiveness of the charity’s collective programs.

Changes from Previous Years

Over the years, we have considered similar factors when assessing charities’ programs, such as the expected effectiveness of interventions and outcomes. Although we have scored interventions and other program elements internally for several years, we haven’t consistently published these scores—partly because we have favored a more qualitative approach to this criterion. This decision has created a lack of transparency in our evaluative claims and a lack of clarity about how various charities compare to each other.

To increase transparency and facilitate comparison between charities, in 2020, we started to take a more systematic approach to assessing programs by clearly identifying various assessment factors (animal groups, countries, outcomes, and interventions) and being more consistent across different charities. In 2022, we took a more quantitative approach to this criterion and introduced a scoring framework that builds on the same four factors we used in 2021. With the scoring framework, we introduced spreadsheets with priority-level scores for different factors, estimates of charities’ spending toward each of those factors, and a single program score for each charity.

This year, we also tried to more clearly distinguish the Programs and Cost Effectiveness criteria and make them more complementary to each other. To achieve this, we (i) avoided considering particular program achievements in the Programs criterion and (ii) incorporated the intervention score in the Cost Effectiveness criterion. We expect to continue differentiating these two criteria to avoid unnecessary double-counting of factors.

Finally, this year we included more explicit considerations of long-term effects in our scoring framework. When assigning priority-level scores to intervention types and outcomes, we factored in considerations of scale over the long, medium, and short term.

Limitations and Future Improvements

Our Programs assessment framework has several limitations and areas for improvement. We consider the following to be the most important.

A limitation of our 2022 programs assessment and related spreadsheets is the lack of explanation or justification for each score. In each review, we referenced some sources explaining why we prioritize work targeting certain animal groups. We also provided the data we used to prioritize countries and a summary of research on intervention effectiveness. However, we did not provide specific details on the reasoning behind each score due to our own capacity constraints. We want to improve this in the next evaluation cycle.

Another limitation of our 2022 framework is that each factor (groups of animals, countries, interventions, outcomes) was scored in isolation. The score assigned to each factor type did not change depending on other variables. For example, while an intervention like individual outreach may be more effective in regions where farmed animal advocacy is highly neglected, the intervention score for individual outreach was the same regardless of the country where it was implemented. Similarly, our country scores did not change regardless of the interventions being implemented there. Our method seems to account for all the factors and avoid double-counting, but it lacks a more tailored approach where scores are more interdependent. We will consider how to overcome this issue in future evaluations.

Finally, our prioritization of countries has been one of the most controversial aspects of the Programs criterion, and it’s undoubtedly one of the areas with the most room for improvement. Because we used MFA’s FAOI data as a reference for scoring scale, tractability, and global influence of countries, many limitations that apply to MFA’s method would also indirectly apply to our method. For example, the FAOI only includes 60 countries, excludes various key variables (e.g., attitudes toward animal issues, market trends, cultural norms, history of animal farming, prevalence of plant-based options, impact of COVID-19, etc.), and was published in 2021, leaving out more recent developments. In addition, we only considered one indicator of neglectedness, and the list of farmed animal advocacy organizations we used might be incomplete and outdated. We expect to update this method for the next evaluation cycle. We may also change how we estimate neglectedness, such as by considering Animal Protection Index (API) data.

1. Because most information we have about charities comes from their websites and other online sources, the information we rely on at the charity selection stage is often incomplete and not up to date. For charities that agree to participate in our evaluation process, we send information requests to obtain more accurate information. 

2. For more information about this quantitative model, see our blog post detailing the process leading to 2021 our recommendations.