Updates to Our Charity Evaluation Criteria in 2023

Introduction

Animal Charity Evaluators is entering our 2023 charity evaluation season! This is the time of year when we work to identify charities that can do the most good for animals with two years of additional funding. To provide more transparency and insight into our evaluation process, we wanted to share some changes we are making to our four charity evaluation criteria since last year. These changes reflect our current thinking, but experience has taught us that when methodological changes are applied in practice, things don’t always go the way we planned. Therefore, these changes might be refined as we navigate the coming evaluation season. Additionally, we are strongly considering renaming our evaluation criteria. If we do, the new names will be reflected when we publish our charity reviews in November.

Programs

The Programs criterion assesses the expected effectiveness of a charity’s programs based on their focus areas without considering the specific results of those programs. We plan to evaluate this criterion by analyzing three factors: (i) the groups of animals a charity’s programs affect, (ii) the countries in which their programs take place; and (iii) the interventions they use to work toward highly promising outcomes. We will use more proxies for each of these factors this year than we have previously.

Within the scoring framework we use to calculate the priority level of particular animal groups, we will now take into account global population numbers, animal welfare ranges (based on a composite score that gives some weight to the output of Rethink Priorities’ Moral Weight Project), and typical welfare situation (i.e., the level of suffering typically experienced by animals in a group).

Last year, for charities working on farmed animals, we used Mercy For Animals’ Farmed Animal Opportunity Index (FAOI) to calculate the priority level scores of work in particular countries, adding our own proxy for neglectedness (human population per farmed animal advocacy organization). This year, we are creating our own country prioritization framework, adapting the FAOI’s methodology to meet ACE’s requirements. This allows us to use the latest data and include countries not covered in the FAOI. To improve our proxy for neglectedness, we are developing a more robust list of charities that we are aware of around the world. Finally, we are using both the current animal population and the estimated animal population by 2050 as proxies for scale.

We are collecting and organizing relevant research to update our intervention scores, and are updating our Menu of Outcomes and Menu of Interventions. We plan to publish the 2023 versions in the coming months.

Finally, we are creating “synergy scores” to represent the additional expected impact of working in specific combinations of animal groups, countries, and interventions. For example, if evidence shows that targeting certain animal groups is particularly promising in certain countries, our new framework would allow us to account for these idiosyncrasies.

Cost Effectiveness

This criterion assesses a charity’s achievements and the costs associated with those achievements. This year, we are using a weighted factor model to combine a variety of objective and subjective factors into a single score. These factors are divided into two categories: those that relate to the effectiveness of the intervention itself, and those that relate to the charity’s implementation of the intervention.

The factors that make up the first category are the scale, tractability, and neglectedness of the interventions used. We believe that these factors drive a large portion of the charity’s impact and consider them as comprising the baseline score, which is then adjusted by the charity’s implementation.

The factors that make up the second category are aligned with past achievements and are either objective (e.g., the number of social media impressions per dollar spent) or subjective (e.g., the type of engagement generated). All of these factors come from a rubric, reducing subjectivity in scoring. We also allow charities to group different achievements together to better capture the interdependence of work done across their programs.

This system allows for particularly strong implementations to overcome lower scores for interventions that we don’t assess as being typically as promising as others. Since we don’t think quantified expected impact on animal figures (e.g., animal years affected per dollar spent) are currently useful based on the information we have,¹ the weighted factor model allows our qualitative assessments to be informed by as much quantitative information as possible.

Notes on the relationship between Programs and Cost Effectiveness

Programs and Cost Effectiveness will both include uncertainty estimates based on factors such as:

• Standard deviation of scores between different ACE researchers
• Number and quality of publications providing evidence on a topic
• Amount of missing information
• Confidence in the theory of change from the ultimate outcome of reducing animal suffering

Since the two criteria are complementary aspects of charities’ impact, our tentative plan is to weigh each of them based on our level of uncertainty. This would mean that if we have high uncertainty in our scoring of their programs but low uncertainty in our scoring of their cost effectiveness, we would put more weight on our scoring of the latter.

Room for More Funding

This criterion assesses how much money a charity can effectively use in the next two years based on its historical financials and plans for growth. Our method of calculating Room for More Funding is largely staying the same, but we will be sharing more detailed uncertainty estimates and transparency into our confidence levels in a charity’s plans.

Leadership and Culture

This criterion assesses organizational factors that can influence a charity’s effectiveness and stability. This year, we engaged nonprofit accelerator Scarlet Spark to help refine the way we assess this criterion. Their expert support has informed some changes to how we carry out this assessment.

The first component of our assessment examines a charity’s policies and processes, such as those related to workplace conditions, leadership and strategy, and staff engagement. This year, we expanded the list of policies that we take into consideration. Our primary intent for examining policies is to help charities identify practices that might work for them, rather than solely comparing them against a checklist that we consider to be best practice. The assessment will now also encompass financial health and accountability.

The second component of our assessment involves an engagement survey that is filled out by the charity’s staff (and occasionally volunteers). We modified our survey questions to reduce their focus on Western cultures and piloted the questions with charities from different global regions to help make sure this is the case.² We also sought to align our questions with the latest evidence on predictors of organizational effectiveness. Additionally, we will ask questions to account for risk factors such as burnout and organizational instability.

Additionally, we are considering changes to our approach to soliciting information on harassment and discrimination in the engagement survey. This would help ensure that claimants understand the implications of providing such information, improve the comprehensiveness of any such information that we receive, and better identify the level of detail we are able to share with the leadership of the charity in question.

Finally, we will continue to explore how we can most effectively help charities improve based on the results of our assessment of leadership and culture.

Acknowledgements

Thank you to the numerous experts in the animal advocacy and effective altruism communities who have graciously provided their feedback and guidance on our methodology, including:

• Programs: Bob Fischer (Rethink Priorities), Andy Burnham (ACE volunteer), Dennis Powers, and Chandler Rombes (Mercy For Animals’ Farmed Animal Opportunity Index team)
• Cost Effectiveness: Saulius Simcikas, Vicky Cox (Charity Entrepreneurship), Nuño Sempere (Quantified Uncertainty Research Institute)
• Leadership and Culture: Andrea Gunn (ACE Board of Directors and Sharpen Strategy), Andreas Setzer (Know Your Elephants), Katie Stanford and Mathias Nestore (The Life You Can Save), Krista Hiddema, Zane McNeill (RARA)
• Members of last year’s evaluated charities, especially Haven King-Nobles (Fish Welfare Initiative), Javiera Mayorga (Vegetarianos Hoy), and Jakub Stencel (Anima International)

Special thanks to Tania Luna and Alyssa Greene-Crow from Scarlet Spark for sharing their expertise and providing valuable ongoing support on our Leadership and Culture assessment.

1. The charities we evaluate employ a wide range of interventions (e.g., corporate welfare campaigns vs. attempts to build an academic field), and we lack the empirical research that would enable us to make informed, quantified estimates of those interventions’ impact on animals. The resulting uncertainties would be significant, limiting the usefulness of the estimates. Additionally, cost effectiveness estimates are more straightforward for direct effects, but indirect and long-term effects (e.g., downstream effects of corporate campaigns on animal product consumption) are harder to estimate but potentially more significant. 

2. We recognize that the survey may still be biased toward cultures our team is most familiar with. The ACE team is committed to minimizing this bias as much as possible.