Evaluation Criteria
RECOMMENDATION CRITERIA FOR ANIMAL ADVOCACY CHARITIES
Before we select which organizations to recommend, we establish the criteria on which we will evaluate them and how we intend to assess those criteria. Below is a summary of the criteria we use to evaluate charities.
Criterion 1: Impact Potential1
With this criterion,2 we assess the impact potential (IP) of a charity’s programs without considering their specific program achievements. During our assessment, we analyze the groups of animals the charity’s programs target, the countries where they take place, and the intervention types they use. We also examine how the charity allocates expenditures among different animal groups, countries, and interventions. A charity that performs well on this criterion has programs with great potential to reduce animal suffering or improve animal wellbeing.
Criterion 2: Cost Effectiveness
With this criterion, we assess the effectiveness of a charity’s approach to implementing interventions, their achievements, and the costs associated with those achievements. Charities that perform well on this criterion likely use their available resources in a cost-effective manner.
Criterion 3: Room for More Funding
A recommendation from ACE could lead to a large increase in a charity’s funding. With this criterion, we investigate whether a charity would be able to absorb the funding that a new or renewed recommendation may bring, and the extent to which we believe that their future uses of funding will be as effective as their past work.
Criterion 4: Organizational Health
With this criterion,3 we assess whether any aspects of an organization’s leadership or workplace culture pose a risk to its effectiveness or stability, thereby reducing its potential to help animals. Problems with leadership and workplace culture could also negatively affect the reputation of the broader animal advocacy movement, as well as employees’ wellbeing and their willingness to remain in the movement.
CRITERIA FOR EVALUATING POTENTIAL CHARITIES
Below, we outline and elaborate upon each of the four criteria we use to evaluate charities and how we assess those criteria. This is the most recent version of our charity evaluation criteria. Previous versions are available in our Process Archive.
ACE has learned a great deal from conducting charity evaluations over the past several years. These updated criteria reflect new information that is relevant to our charity evaluation process, and we expect that they will continue to change as we gain more experience evaluating charities.
Our current approach uses a scoring framework for Impact Potential and Cost Effectiveness. The idea behind scoring these criteria is to make ACE’s internal process for assessing programs more transparent and less subjective. It also gives us a way to compare charities with each other. We assess Room for More Funding by calculating dollar estimates, and we evaluate Organizational Health by distributing a survey to staff and examining information provided by the charity’s leadership to identify any potentially significant management issues.
- Criterion 1: Programs
- Criterion 2: Cost Effectiveness
- Criterion 3: Room for More Funding
- Criterion 4: Leadership and Culture
This criterion is complementary to Cost Effectiveness and reflects the impact of a charity’s work in a way that is not fully captured by examining their past achievements and costs.
This criterion was called Programs from 2020 to 2022. We decided to rename it Impact Potential to better reflect its focus on assessing the effectiveness of charities’ programs without considering their implementation. This name is more specific and less confusing internally, especially since we recently changed the name of our research team to the Programs team.
This criterion was called Leadership and Culture from 2020 to 2022. We found that ‘leadership’ was often misunderstood as referring solely to the qualities of individual leaders and that ‘culture’ was understood in very different ways across countries and demographics. With the new name Organizational Health, we intend to highlight the broad focus of this criterion and to clarify that its goal is to identify any significant risks to the organization’s effectiveness and stability.
Criterion 1 – Impact Potential: How promising are the charity’s programs?
How do we identify promising programs?
ACE characterizes promising programs as those that (i) target high-priority animal groups, (ii) work in high-priority countries, and/or (iii) use interventions that work toward high-priority outcomes. We used a version of the Scale, Tractability, and Neglectedness framework to score the impact potential of charities’ programmatic work in three categories: animal groups, countries, and interventions. This year, we also introduced synergy scores to account for the unique impact that a combination of factors can create.
- Animal groups. We used the Scale, Tractability, and Neglectedness (STN) framework to score 16 animal groups relative to each other. By using this framework, we aim to prioritize groups of animals who are affected on a large magnitude, where there appear to be tractable solutions to improve their situation, and whose situation is relatively neglected in animal advocacy. We consider farmed animal advocacy high priority because of the large scale of animal suffering involved and its high tractability and neglectedness relative to other cause areas. Among farmed animals, we currently prioritize farmed fishes and farmed chickens, as well as programs targeting farmed invertebrates. Given the large number of wild animals and the small number of organizations working on their welfare, we believe that wild animal advocacy also has potential to be high impact despite its lower tractability. For more details on how we currently prioritize animals, see the animal-relative scores spreadsheet.
- Countries. The countries and regions in which a charity operates can affect the impact potential of their work. In the case of farmed animal organizations,1 we used the STN framework to score 196 countries relative to each other. By using this framework, we aim to prioritize countries with relatively large animal agriculture industries, few other charities engaged in similar work, and where animal advocacy is likely to be feasible and have a lasting impact. Additionally, we consider a country’s Global Influence as a fourth factor in prioritizing countries. Our method of scoring countries was inspired by Mercy For Animals’ Farmed Animal Opportunity Index (FAOI), using most of their proxies for Scale, Tractability, and Global Influence. We also considered Neglectedness as a factor. To assess neglectedness, we calculated the ratio of the human population to the number of farmed animal organizations in each country. Specifically, we compared our own data on the number of farmed animal organizations (excluding farmed animal sanctuaries) working in each country to the human population (in millions) of that country. For more details on how we currently prioritize countries, see the country-relative scores spreadsheet.
- Interventions. We categorized the interventions animal advocacy charities use into 26 types and the main outcomes they work toward into eight types. Using the STN framework, we calculated different IP scores for 77 combinations of interventions and expected outcomes. In line with our commitment to following empirical evidence and logical reasoning, we used existing research to inform our assessments and explain our thinking about the impact potential of different interventions. We compiled research about the effectiveness of each intervention using information from our research newsletter, research library, and research briefs. Additionally, we calculated an uncertainty score for each combination of interventions and expected outcomes. For more details on how we currently prioritize interventions, see this intervention-relative scores spreadsheet.
- Synergy. These scores represent the impact of a specific combination of factors (intervention, animal group, and country) and aim to capture a dimension of charities’ impact that is different from the one reflected in the animal-relative, country-relative, and intervention-relative scores. With the synergy scores, we intend to analyze charities’ work in a way that is (to some extent) sensitive to each charity’s particular context as well as consistent across charities. For more details on how we scored synergy impacts, see this synergy scores spreadsheet.
What if a charity has programs that are likely to have both high and low impact?
It’s important to note that charity donations may be used to fund all of a charity’s projects, not strictly its most effective ones. This problem cannot always be solved by restricting donations to specific programs due to the problem of fungibility. Suppose a charity receives a restricted donation for an effective project. It can then use that donation to free up the unrestricted funding that would have gone toward that effective project and move that unrestricted funding to a different, less effective project.
The Impact Potential criterion is currently designed to assess all programs charities work on as a whole rather than each individual program. We expect this provides a better picture of how program spending will be used overall.
Is pursuing a project with low impact grounds for disqualification?
One might consider charities that pursue low-impact projects as unable to commit to cause prioritization. However, what is considered high-impact is not well-established or agreed upon across the movement. In some cases, programs that seem to have a low impact—such as those that are not focused on farmed or wild animals—are effective in the context of a charity’s larger goals; for example, an organization may run a companion animal program for the purpose of engaging the community and expanding its farmed animal programs. Additionally, projects that can be considered low-impact may have positive indirect impacts, such as generating significant funding from donors or garnering positive press.
The framework we used to prioritize countries only applies to farmed animal advocacy. We have not developed a framework to prioritize wild animal welfare work because there are very few organizations that work on wild animal welfare, and those we have considered so far are focused on indirect work such as research and academic development, which is less country-specific.
Criterion 2 – Cost Effectiveness: How much has the charity achieved through their programs?
What goes into a cost-effectiveness assessment?
A charity’s recent cost effectiveness provides insight into their overall impact on animals given the resources used to achieve their results and is a useful component in understanding how cost effective future donations to the charity might be.
In a perfect world with no uncertainty, a cost-effectiveness calculation would be all that is needed to evaluate a charity. However, for reasons discussed by GiveWell in “Why We Can’t Take Expected Value Estimates Literally” and by Peter Hurford in “Why I’m Skeptical About Unproven Causes,” it is necessary to take a more nuanced approach.
Our methods for assessing cost effectiveness have changed over the years that ACE has conducted charity evaluations. To learn more about our past methods please refer to our Process Archive.
In 2023, we conducted our analysis by comparing a charity’s reported expenditures over 12 months to their reported achievements in each intervention category during that time.
We asked charities to report up to 10 key achievements per intervention category, alongside their estimated expenditures for each achievement.
During our evaluation, we verified a subset of the charity’s reported achievements. Specifically, we identified the three intervention categories for which the charity reported the highest expenditures and selected up to three key claims per intervention category to verify. We aimed to identify the claims with the highest reported expenditures, ideally by finding at least one independent source to confirm the claim. When we were unable to do so, we directed follow-up questions to the charity to verify their achievements.
We used a Weighted Factor Model1 (WFM) approach to calculate the charity’s final cost-effectiveness score (see image below) based on their achievement scores. The achievement scores represent a combination of the intervention impact potential (IP) score and the implementation score for each key achievement.
- For each key achievement, we assigned the respective intervention IP score which was scored as part of the Impact Potential criterion. We also computed the achievement quantity, i.e., how much the charity accomplished per U.S. dollar or per $100,000. We applied discounts to the achievement quantities if a charity collaborated with other organizations and in certain other cases, e.g., if a charity influenced funding rather than directly providing it, if they summarized and disseminated research rather than conducting original studies, or if a corporate campaign did not result in any commitments. We then normalized the achievement quantity against other achievements in the same intervention category and with the same unit and mapped it onto a 1—7 scale.
- We then used a rubric to score the achievement quality (i.e., how well the charity had implemented the intervention). The rubric included the respective animal-relative IP score for the animal group targeted by the achievement, as well as intervention-specific factors. Where an achievement targeted multiple animal groups, we used the average animal score.2
- We calculated a weighted average of the achievement quantity per dollar/$100k and the achievement quality to arrive at the implementation score.
- We then multiplied the intervention IP score by the implementation score and normalized and mapped the resulting score onto a 1–7 scale to arrive at the achievement score.
- The final cost-effectiveness score for each charity is the average of all of their achievement scores, weighted by their expenditures. This score indicates, on a 1–7 scale, how cost effective we think the charity has been during the reporting period, with higher scores indicating higher cost effectiveness. Note that charities were benchmarked against other charities under evaluation rather than against all charities, and the standard of effectiveness in the charities we selected to evaluate is likely high. Low cost-effectiveness scores therefore don’t necessarily indicate low cost effectiveness in absolute terms.
We also rated our level of uncertainty in each achievement score based on the intervention uncertainty score, the number of missing scores in the achievement scoring rubric, and whether we were able to successfully verify a charity’s achievements. Uncertainty was scored on a 1–7 scale, with higher scores indicating higher uncertainty. The final uncertainty score for each charity is the average of the achievement uncertainty scores, weighted by the relative achievement expenditures. We consider a score to represent “high uncertainty” when it is equal to or greater than the median of all charities under evaluation.
For more information about Weighted Factor Models, see Charity Entrepreneurship (2019).
See here for the full rubric. Two researchers scored each achievement on the rubric, and discussed significant disagreements before a second round of revising scores. We averaged the two researchers’ scores for each factor. Where we did not have enough information to score an achievement, we set the corresponding factor weight to zero.
Criterion 3 – Room for More Funding: How much additional money can the charity effectively use in the next two years?
In this criterion, we examine whether a charity has room for more funding and plans for growth. We aim to recommend high-impact and scalable work, and we seek to ensure that a charity can absorb and effectively utilize funding that an ACE recommendation may bring in. If a charity has a prior recommendation status, we examine whether it could continue to effectively absorb funding that comes from our recommendation.
Why might a charity not have room for more funding (RFMF)?
Charities may have non-monetary bottlenecks to expanding their operations, e.g., difficulty hiring the talent needed to expand, government restrictions, inexperience in managing international expansion and/or organizational change, or a lack of necessary partnerships with other groups. Given the variety of potential problems, we need to verify that a charity’s effective interventions are bottlenecked specifically by money. Otherwise, donations might not be what a charity needs, so we would not want to recommend it to donors.
How do we evaluate a charity’s room for more funding?
To estimate charities’ RFMF, we request their financial and staffing records from 2020 onwards and ask them to forecast their revenue and staff size through 2025. We also ask them to report how their projections are allocated across different interventions, animal groups, and countries. We then assess our overall level of uncertainty in the charity’s projected revenue, expenditures, and hiring plans. This assessment is based on factors such as sustainability of growth,1 alignment with projections from our previous evaluation (if applicable), and our uncertainty in charities’ ability to find and train planned new hires in the projected timeframe based on our understanding of the talent landscape.2
Our focus is to determine whether additional resources will likely be used for programs with high impact potential and/or that we have assessed as highly cost effective, or other beneficial organizational changes. The latter may include investments into infrastructure and staff retention, both of which we think are important for sustainable growth.
Next, we ask charities to indicate how they would spend additional, unexpected funding that an ACE recommendation may bring. Because previously recommended charities tend to receive more ACE-influenced funding over time, we also ask those charities to specify how they would use additional funding.
We then assess our level of uncertainty in the effectiveness of the charity’s plans in 2024 and 2025 and estimate their RFMF for those years. To do this, we assign an uncertainty score on a 1–7 scale, with higher scores indicating higher uncertainty. We use these uncertainty scores and the charity’s revenue and expenditure trajectory to define two RFMF dollar estimates that represent the amount beyond the charity’s projected revenue in 2024 and 2025 that we believe they could effectively use. If the charity has plans for a large amount of unexpected funding that are likely to be as effective as their past work, they will receive higher RFMF estimates.
These final estimates may include an adjustment based on the status of the charity’s reserves. Since it is common practice to hold more funds than needed for current expenses in order to be able to withstand changes in the business cycle or external shocks that may affect incoming revenue, we encourage charities to use their ACE-influenced donations to build their reserves and factor that into our RFMF calculations.
How do we monitor a charity’s room for more funding?
It is possible that a charity could run out of room for more funding more quickly than we expect or that they could come up with good ways to use funding beyond what we expect. Our RFMF estimates are intended to identify the point in time at which we would want to check in with a charity to ensure that they have used their funds effectively and can still absorb additional funding. Therefore, we check in with our recommended charities twice a year leading up to our Recommended Charity Fund distributions to update our understanding of their RFMF.
Criterion 4 – Organizational Health: Are there any management issues substantial enough to affect the charity’s effectiveness and stability?
The way an organization is led affects its organizational culture, which in turn can impact the organization’s effectiveness and stability, thereby reducing its potential to help animals. In this criterion, our main goal is to assess whether organizations seem to have leadership and culture issues that are substantial enough to affect our confidence in their effectiveness and stability.
How do we identify concerns about organizational health?
One part of our assessment consists of examining information provided by top leadership staff. We ask charities to report the people policies and processes they have in place, including policies around compensation, workplace conditions, and representation, equity, and inclusion (REI), as well as how leadership staff make these policies accessible to their employees. We also consider leadership’s commitment to transparency, in terms of both their communications with ACE and the organizational information that they make publicly available on their website.
We also ask charities about their board’s membership and functions so that we can understand the extent to which these align with the Compliance Practices set out in BoardSource’s Recommended Governance Practices.
We recognize that not all of the policies and processes that we ask charity leadership about will be common or relevant in all countries and situations. Where there are indications that important policies and processes may be lacking, we follow up with the charity to gain a better understanding of the context.
The other part of our assessment consists of soliciting staff and volunteer perspectives via our engagement survey. We developed this survey in collaboration with organizational consultants Scarlet Spark. To help ensure that our questions were reliable predictors of organizational health, we based them where possible on recognized frameworks, such as the cross-culturally validated Gallup Q12 Employee Engagement Survey, the Maslach Burnout Inventory, Google’s Project Oxygen, and cross-cultural research by Culture Amp.
We require at least 65% of the charity’s staff to respond to the survey to ensure that we have a representative sample of responses. There is no participation threshold for volunteers, recognizing that most organizations do not have a fixed number of volunteers as their participation tends to fluctuate.
The engagement survey also contains a link to an anonymous Whistleblower Form,1 developed with support from legal experts at Animal Defense Partnership, for any staff or volunteers who wish to report issues of harassment and discrimination. In most cases where we decide to take action based on such reports, this consists of sharing relevant non-confidential information with the leadership of the organization in question and hearing their perspective. We do this to improve our understanding of what happened, whether the leadership members were aware, and what measures they took, if relevant. We then factor this information into our overall Organizational Health assessment.
Poor performance on a single aspect of the organizational health assessment does not automatically lead to an unfavorable overall outcome. Instead, we seek to assess organizational health from multiple perspectives to arrive at the most appropriate conclusion within the time available based on the information we have. This includes our follow-up conversations with the charity’s leadership.
Once we have identified areas of potential concern and asked follow-up questions to leadership, we discuss and vote on whether any concerns are substantial enough to affect our confidence in a charity’s effectiveness and stability.
Why do we focus on organizational health risks, rather than strengths?
In practice, gaining an accurate picture of charities’ organizational health is very difficult. For example, we recognize that our engagement surveys could produce inaccuracies due to selection bias and may also not reflect employees’ true opinions, as they are aware that their responses could influence ACE’s evaluation of their employer.
In some of our past approaches to this criterion, we sought to give a more comprehensive assessment of charities’ strengths. However, we realized that we were not able to conduct a sufficiently thorough investigation to be confident in these assessments, some of which later turned out to be very different from reality.
In 2020, we changed our approach to this criterion. We now focus on determining whether issues in the organization’s leadership and culture might significantly negatively impact staff productivity and wellbeing or could be detrimental to the organization’s effectiveness and stability. While we do still note positive elements of charities’ culture and leadership, this is not the focus of our assessments.
The publicly accessible version of this form can be found via ACE’s Third-Party Whistleblower Policy on our website.