2021 Evaluation Process - Animal Charity Evaluators

Overview

Once a year, ACE evaluates charities and updates our recommendations as a result of those evaluations. In 2021, our formal evaluation process took place from June through November. Prior to November 2021, our most recent recommendation update took place in November 2020.

We made several improvements to our charity evaluation process this year, including a more rigorous selection process, updated criteria for assessing charities, and an expanded and more systematic verification process.

We updated our 2021 charity selection process to be more comprehensive and rigorous by compiling a master list of about 2,800 charities from around the world and using a quantitative model to select charities based on important factors. We also updated our criteria for evaluating charities with the intention of better capturing and communicating the reasoning behind our recommendation decisions. To increase transparency in our assessments and improve readability, we decided to focus on the most relevant criteria for our recommendation decisions and reduce the number of criteria included in our reviews. In addition, we divided each criterion into the three main sections—Method, Information and Analysis, and Our Assessment—to clarify which areas of the review explain our method, our analysis, and our assessment of each criterion.

ACE prioritizes using accurate and reliable evidence in our work. This year, we continued to intensify our verification process and devoted a slightly larger proportion of our time to verifying claims reported by charities. We relied on publicly available information, internal documents, media reports, and independent sources to verify claims, and we also followed up with charities for further information or details. For each charity, we verified at least one key result per program.

The general timeline of our 2021 evaluation process was as follows:

Early June: Selected charities to invite to be reviewed
Mid June to late October: Invited charities to participate, gathered information from charities, drafted our comprehensive reviews, and solicited feedback from ACE’s board and staff
Late October to mid November: Sent charities completed drafts of the reviews and finalized our recommendation decisions
Mid November: Addressed charities’ feedback on our drafts and solicited charities’ approval to publish
Late November: Published our recommendations on November 23

ACE’s evaluations committee completed most of the work on the reviews. Board members and the Executive Director provided feedback on the drafts, and the Copy Editor edited the reviews and all associated content. ACE’s communications team created the images that appear in our reviews, published the reviews on our website, and announced our 2021 recommendation decisions.

2021 charity evaluation process flowchart

Fig. 1: Flowchart depicting ACE’s 2021 evaluation process

Our Selection Process

We began our 2021 evaluation process by compiling an internal list of charities to consider. The list included the following:

Charities that requested to be evaluated
Charities that ACE staff and board members suggested evaluating
Charities that third parties asked us to evaluate
Charities we had considered or evaluated in the past that:
- were selected as Top Charities in 2020 (we reevaluate our Top Charities every year)
- were selected as Standout Charities in 2019 (we reevaluate our Standout Charities every two years)
- had been close to the threshold for further investigation but had been excluded for some reason
- we wanted to reconsider due to changes in their programming or our understanding of their activities
International charities that we had not previously evaluated and that we understood to have significant influence in their home countries
Lists of animal advocacy organizations provided by other membership organizations, some of which were shared with us in confidence

Following this initial step, we generated a comprehensive list of about 2,800 charities to consider evaluating in 2021. We filtered down the list by removing charities with key words in their name that indicated they were unlikely to work in a high-priority cause area. We then collected additional data on the remaining charities (e.g., whether they work to reduce the suffering of farmed animals, whether they operate a sanctuary, whether they work to help wild animals, or whether they do none of those activities). From this process, we identified around 700 farmed animal charities and around 40 wild animal charities.

In order to further refine the list of farmed animal charities, we used a quantitative model that prioritized charities based on the outcomes they work toward, the regions they work in, and the specific animal group(s) their programs target. Once we ran the model, we performed some rationality checks to make sure the model was working as expected. The evaluations committee then voted on which of the top 200 charities we should consider for selection, and through a process of iterative discussion and voting, selected 25 charities to invite to review. These charities were selected based on factors such as how well we thought they would perform on our evaluation criteria and how useful we thought the knowledge acquired and potentially published from the comprehensive evaluation would be.

We sent each of the 25 charities a copy of our Charity Evaluation Handbook and formally invited them to participate in the review process. Twelve charities declined to be reviewed,¹ resulting in a total of 13 charities to evaluate: our four Top Charities from 2020, two Standout Charities that were last evaluated in 2019, and seven other charities that we had not evaluated in at least three years. None of these 13 charities withdrew from our review process.

Evaluations

We conducted evaluations according to our general process for comprehensive reviews. As part of the evaluation process, we asked each charity that agreed to participate to provide information and documentation about their ongoing programs, accomplishments, finances, leadership, and culture. To assess workplace culture, we distributed a survey to each charity’s staff members and volunteers.

The evaluations team solicited feedback from ACE’s Executive Director and two board members while drafting the reviews. Once drafted, we sent the reviews to the charities for feedback and approval. Before granting approval, charities were given the opportunity to request edits, including removing confidential information or correcting factual errors. That said, all of our reviews represent our own understanding and opinions, which are not necessarily those of the charities reviewed. This year, all 13 charities for which we drafted reviews agreed to have their reviews published.

Recommendations

After drafting each comprehensive review (but before receiving approval from charities), the evaluations committee held several meetings to discuss the selection of Top and Standout Charities.² Initially, they held a meeting per charity to discuss each one in depth, identify areas of concern or disagreement, and hold a vote to gauge a tentative recommendation decision. Once the committee had held meetings for each charity, they held another meeting to look at all of the tentative recommendation decisions and ensure that each charity had been considered fairly. There was substantial initial agreement on the status of some charities but not others. The evaluations committee then presented their findings to the rest of staff and board members to gather feedback on the tentative decisions as well as some particular areas of uncertainty. Following this discussion, the evaluations committee held a couple of additional meetings to integrate the staff and board’s feedback and finalize the recommendation decisions, revisiting decisions for which the votes were split.

In the end, we selected three Top Charities. We think that, overall, each of our Top Charities perform well on our evaluation criteria. They each conduct effective programs, are able to make use of additional funding, and have a sustainable work culture. The evaluations committee had either consensus or a majority of supporting votes for each of the selected Top Charities.

We also selected seven Standout Charities. Our Standout Charities are those that we did not select for a top recommendation, but nonetheless chose to recommend to our supporters because we think they are promising. We think that donations to these charities seem likely to have a relatively high expected value. The evaluations committee had a majority of supporting votes for each of the selected Standout Charities.

Additional Information

Of the charities that declined the opportunity to participate, most responded that they were too busy at the time and/or that they preferred to wait until 2022 to be evaluated.
The one exception is one member of the research team who was not at all involved in the evaluation process for one charity due to a conflict of interest.