Recommendation Criteria for Animal Advocacy Charities
Before we select which organizations to recommend, we establish the criteria on which we will evaluate them and how we intend to assess those criteria. Below is a summary of the recommendation criteria we use to evaluate charities.
Criterion 1: Programs
In this criterion, we assess the expected effectiveness of a charity’s programs without considering their particular achievements. During our assessment, we analyze the groups of animals the charity’s programs affect, the countries in which they take place, the outcomes they work toward, and the interventions they use to achieve those outcomes, as well as how the charity allocates their spending toward different programs. A charity that performs well on this criterion has programs that are expected to be highly effective in reducing the suffering of animals.
Criterion 2: Cost Effectiveness
In this criterion, we assess the effectiveness of a charity’s approaches to implementing interventions, their recent achievements, and the costs associated with those achievements. By conducting this assessment, we seek to gain insight into a charity’s overall impact on animals given the resources they used to achieve their results. A charity that performs well on this criterion likely utilizes their available resources in a cost-effective manner.
Criterion 3: Room for More Funding
In this criterion, we investigate whether a charity would be able to absorb and effectively utilize funding that a new or renewed recommendation may bring in. Our room for more funding (RFMF) assessment ultimately guides our recommendation decision; charities are ineligible to receive a particular recommendation status if they would be unable to absorb and effectively utilize the subsequent funding. A charity that performs well on this criterion has feasible plans to effectively utilize funding, and the barriers to accomplishing those plans are primarily monetary—not due to lack of time, lack of qualified personnel, difficulty recruiting and onboarding talent, or other non-monetary issues.
Criterion 4: Leadership and Culture
In this criterion, we assess whether organizations seem to have leadership and culture issues that are substantial enough to affect our confidence in their effectiveness and stability. We review aspects of organizational leadership and culture by examining information provided by top leadership staff (as defined by each charity) and by capturing staff perspectives via our culture survey.
Criteria for Evaluating Potential Charities
Below, we outline and elaborate upon each of the four criteria we use to evaluate charities and how we assess those criteria. This is the most recent version of our charity evaluation criteria. Previous versions are available in our Process Archive.
ACE has learned a great deal from conducting charity evaluations over the past several years. These updated criteria reflect new information that is relevant to our process, and we expect that they will continue to change as we gain more experience evaluating charities.
Our current approach uses a scoring framework for Programs and Cost Effectiveness. The idea behind scoring these criteria is to make ACE’s internal process for assessing programs more transparent and less subjective. It also gives us a way to compare charities with each other. We assess Room for More Funding by calculating dollar estimates, and we evaluate Leadership and Culture by distributing a survey to staff and examining information provided by the charity’s leadership.
- Criterion 1: Programs
- Criterion 2: Cost Effectiveness
- Criterion 3: Room for More Funding
- Criterion 4: Leadership and Culture
Criteria Descriptions
Criterion 1: Programs
How do we identify highly effective programs?
ACE characterizes effective programs as those that (i) target high-priority animal groups, (ii) work in high-priority countries, (iii) work toward high-priority outcomes, and/or (iv) pursue interventions that are expected to be highly effective. This year, we used a scoring framework to assess the effectiveness of charities’ programs.
We use the Scale, Tractability, Neglectedness (STN) framework to score the priority levels of different groups in these four categories:
- Animal groups. By using the STN framework, we aim to prioritize programs targeting animal groups that are affected in large numbers,1 whose situation seems tractable, and who receive relatively little attention in animal advocacy. We consider farmed animal advocacy high priority because of the large scale of animal suffering involved and its high tractability and neglectedness relative to other cause areas. Among farmed animals, we currently prioritize farmed fishes and farmed chickens, as well as programs targeting farmed invertebrates. Given the large number of wild animals and the small number of organizations working on their welfare, we believe that wild animal advocacy also has potential to be high impact despite its lower tractability. For more details on how we currently prioritize animals, see this spreadsheet.
- Countries. By using the STN framework, we aim to prioritize countries with relatively large animal agricultural industries, few other charities engaged in similar work, and in which animal advocacy is likely to be feasible and have a lasting impact. Additionally, we consider global influence as a fourth factor in our prioritization. Our methodology for scoring countries uses Mercy For Animals’ (MFA) Farmed Animal Opportunity Index (FAOI) for scale, tractability, and global influence.2 To assess neglectedness, we compare our own data on the number of farmed animal organizations working in each country to the human population (in millions) of that country. For more details on how we currently prioritize countries, see this spreadsheet.
- Outcomes. We categorize the work of animal advocacy charities into six outcomes. We prioritize those outcomes using the STN framework, and we consider long-term impacts as an additional factor in our prioritization. Currently, we give higher priority to the following outcomes: improving welfare standards, increasing the availability of animal-free products, and strengthening the animal advocacy movement. We give lower priority to the following outcomes: decreasing the consumption of animal products, increasing the prevalence of anti-speciesist values, or providing direct help to animals. For more details on how we currently prioritize outcomes, see this spreadsheet.
- Interventions. We currently categorize the interventions charities use into 16 types. We we prioritize those intervention types using the STN framework, and we consider long-term impacts as an additional factor in our prioritization. In line with our commitment to following empirical evidence and logical reasoning, we use existing research to inform our assessments and explain our thinking about the effectiveness of different interventions. When available, we use relevant research and our research briefs to inform our scores and corresponding priority levels for each intervention type. For more details on how we currently prioritize interventions, see this spreadsheet.
We use information supplied by charities to estimate the percentage of program funding spent on different groups in each of the above categories (animal groups, countries, outcomes, and interventions). We then use those estimates and our priority level scores to calculate at a singular program score for each charity, representing the expected effectiveness of their collective programs.
What if a charity has programs with both high and low impact?
It’s important to note that charity donations may be used to fund all of a charity’s projects, not strictly its most effective ones. This problem can not always be solved by restricting donations to specific programs due to the problem of fungibility. Suppose a charity receives a restricted donation for an effective project. It can then use that donation to free up the unrestricted funding that would have gone toward that effective project and move that unrestricted funding to a different, less effective project.
The Programs criterion is currently designed to assess all programs charities work on as a whole rather than each individual program. We expect this provides a better picture of how programs spending will be used overall.
Is pursuing a project with low impact grounds for disqualification?
One might consider charities that pursue low-impact projects as unable to commit to cause prioritization. However, what is considered high-impact is not well-established or agreed upon across the movement. In some cases, programs that seem to have a low impact—such as those that are not focused on farmed or wild animals—are effective in context of a charity’s larger goals; for example, an organization may run a companion animal program for the purpose of engaging the community and expanding its farmed animal programs. Additionally, projects that can be considered low-impact may have positive indirect impacts, such as generating significant funding from donors or garnering positive press.
Criterion 2: Cost Effectiveness
What goes into a cost-effectiveness assessment?
A charity’s recent cost effectiveness provides insight into their overall impact on animals given the resources used to achieve their results, and is a useful component in understanding how cost effective future donations to the charity might be.
In a perfect world with no uncertainty, a cost-effectiveness calculation would be all that is needed to evaluate a charity. However, for reasons discussed by GiveWell in “Why We Can’t Take Expected Value Estimates Literally” and by Peter Hurford in “Why I’m Skeptical About Unproven Causes,” it is necessary to take a more nuanced approach.
From 2014–2018, we used quantitative cost-effectiveness models that compared charities’ outcomes to the expenditures of each of their programs and attempted to estimate the number of animals spared per dollar spent. After considering the limitations of such purely quantitative cost-effectiveness models, we transitioned in 2019 to a qualitative approach that, analyzed the resources used and outputs achieved for each intervention type.
In 2021, we analyzed charities’ use of resources over the past 18 months and compared that to the results of each of their main programs during that time. The goal was to understand if charities had successfully implemented their programs and whether their past successes were achieved at a reasonable cost. Our overall cost effectiveness assessment was a binary assessment of whether there were any causes for concern.
In 2022, we sought to improve our cost-effectiveness assessment in three main ways. First, we wanted to distinguish more clearly between the Programs and Cost-Effectiveness criteria and reassess how they should best work together to contribute to a charity’s overall evaluation. We now assess the overall expected impact of a charity’s programs (considering animal groups, countries, outcomes, and interventions) within the Programs criterion. We then use the intervention score to further analyze a charity’s cost effectiveness.
Second, we aimed to provide a more fine-grained cost-effectiveness estimate rather than a binary assessment of whether we thought there was cause for concern. To this end, we analyzed a charity’s key achievements in each of their programs and compared how cost-effectively different charities implemented the interventions they used. The goal was to assign each charity a cost-effectiveness score that could be used to rank charities relative to each other. In line with these changes, we updated our definition of cost effectiveness from “how well the charity has made use of its available resources” to “a charity’s overall impact on animals given the resources they used to achieve their results.”
Finally, we aimed for as much transparency about our process as possible so that donors who disagree with parts of our model can still make use of our intermediary estimates. To this end, we publish the complete list of achievements provided by charities and the cost-effectiveness model for each charity alongside their review.
Criterion 3: Room for More Funding
In this criterion, we examine whether a charity has room for more funding and plans for growth. We aim to recommend high-impact and scalable work, and we seek to ensure that a charity can absorb and effectively utilize funding that an ACE recommendation may bring in. If a charity has a prior recommendation status, we examine whether it could continue to effectively absorb funding that comes from our recommendation.
Why might a charity not have room for more funding?
Charities may have non-monetary bottlenecks to expanding their operations, e.g., difficulty hiring the talent needed to expand, government restrictions, or a lack of necessary partnerships with other groups. Given the variety of potential problems, we need to verify that a charity’s effective interventions are bottlenecked specifically by money. Otherwise, donations might not be what a charity needs, so we would not want to recommend it to donors.
How do we evaluate a charity’s room for more funding?
First, we ask the charity to estimate their projected revenue, expenditures, and plans for expansion, assuming that their ACE recommendation status and the amount of ACE-influenced funding they receive will stay the same. We then assess our level of confidence in the charity’s projections and ability to carry out their plans. We express high confidence in the charity’s plans if their growth rate is consistent with past revenue and expenditures, and the primary barrier to accomplishing their plans is strictly monetary. We express lower confidence in a charity’s plans if their growth rate is overly ambitious or unrealistic, or if they may encounter nonfinancial barriers to the scalability of their programs (e.g., lack of experience managing organizational change, difficulty recruiting and onboarding talent, entry into regions outside of their core expertise, or government restrictions).
Next, we ask the charity to indicate how they would spend additional, unexpected funding and rate the feasibility of their plans. In 2022, we estimated that the additional funding a charity receives due to an ACE recommendation is roughly $200,000 for Standout Charities and $1,000,000 for Top Charities per year.
Finally, we use our ratings to estimate charities’ RFMF. For charities that are already recommended and are being re-evaluated, our RFMF analysis focuses on whether they could continue to effectively absorb funding that comes from our recommendation. These final estimates may include an adjustment based on the status of the charity’s reserves. Since it is common practice to hold more funds than needed for current expenses in order to be able to withstand changes in the business cycle or external shocks that may affect incoming revenue, we encourage charities to use their ACE-influenced donations to build their reserves and factor that into our RFMF calculations.
How do we monitor a charity’s room for more funding?
It is possible that a charity could run out of room for more funding more quickly than we expect or that they could come up with good ways to use funding beyond what we expect. If a charity receives a recommendation as Top Charity, we check in mid-year about the funding they’ve received since the release of our recommendations, and we use the estimates presented above to indicate whether we still expect them to be able to effectively absorb additional funding at that time.
Criterion 4: Leadership and Culture
The way an organization is led affects its organizational culture, which in turn can impact the organization’s effectiveness and stability. In this criterion, our main goal is to assess whether organizations seem to have leadership and culture issues that are substantial enough to affect our confidence in their effectiveness and stability.
How do we identify concerns about leadership and culture?
We review aspects of organizational leadership and culture by examining information provided by top leadership staff (as defined by each charity) and by capturing staff perspectives via our culture survey. At a charity’s request, we also distribute the survey to volunteers working at least five hours per week.
We recognize that surveying staff and/or volunteers could lead to inaccuracies due to selection bias, and that surveys may not reflect employees’ true opinions as they are aware that their responses could influence ACE’s evaluation of their employer. In our experience, it is easier to assess issues with an organization’s culture than it is to assess the strength of an organization’s culture. Therefore, we focus on determining whether there might be issues in the organization’s culture that have a negative impact on staff productivity and wellbeing.
We collect information about staff members’ perceptions of leadership and satisfaction with their compensation. To capture whether an organization also provides non-material incentives, we assess staff engagement using the Gallup Q12 survey. We consider an average engagement score below the median value of the scale a potential concern. We also ask staff about whether experiences of harassment or discrimination in the workplace have been addressed appropriately.
Additionally, we request charity leadership to provide information about human resources policies, including policies regarding representation/diversity, equity, and inclusion (REI).
We consider how each charity performs on each of these factors, identify areas of potential concern, ask follow-up questions to leadership to clarify potential concerns, and discuss and vote on whether concerns are substantial enough to affect our confidence in a charity’s effectiveness and stability.
Why do we not assess how strong leadership and culture are?
In our experience assessing charities’ leadership and culture, we have realized that ACE needs to be in a better position to evaluate how strong charities’ leadership and culture are. We do not have the capacity to conduct a thorough enough investigation to get an accurate picture of these organizational components. When we took that approach in the past, we made mistakes in evaluating charities very highly in this criterion when events later showed that the reality was completely different.
In 2020, we changed our approach to this criterion. We now focus on determining whether there might be issues in the organization’s leadership and culture that have a negative impact on staff productivity and wellbeing. We try to avoid making unnecessary positive statements about charities’ culture and leadership that are difficult to verify and maintain a non-evaluative perspective when reporting culture survey responses and information provided by leadership. We then consider all collected data to identify concerns and point out some positive findings.
We don’t consider the number of individuals as the only relevant characteristic for scale, and we don’t necessarily believe that groups of animals or species should be prioritized solely based on scale. However, the number of animals in a group or species is one characteristic of scale that we use for prioritization.
We acknowledge that using Mercy For Animals’ FAOI scores can bias us toward their own considerations of the most important countries to focus on.
