2020 Evaluation Process - Animal Charity Evaluators

Overview

Once a year, ACE evaluates charities and updates our recommendations as a result of those evaluations. In 2020, our formal evaluation process took place from June through November. Prior to November 2020, our most recent recommendation update took place in December 2019.

This year, we updated our menu of outcomes with the intention of better capturing and communicating both how the animal advocacy movement works as a whole and how individual charities operate within the movement. This update was reflected throughout the reviews, especially in our theory of change diagrams and our evaluation of charities’ programs and cost effectiveness. We also intensified our verification process and devoted a slightly larger proportion of our time to verifying claims reported by charities. To verify claims, we relied on publicly available information, internal documents, media reports, and independent sources, and we often followed up with charities for further information or details. For each charity, we verified at least one key result per program.

Additionally, we made several process improvements that we believe increased the efficiency of our work. Regarding interactions with charities, we asked the organizations fewer questions to limit the burden placed on them, and we assigned a single person to handle communication with all charities. We also replaced the conversation calls with charities’ leadership with written questionnaires, relieving our capacity by not having to produce call summaries. Internally, we implemented an agile operational model that allowed the evaluations team to cooperate more closely on drafting reviews and adjust our process continually.

The general timeline of our evaluation process was as follows:

Late June: Selected charities to invite to be reviewed
Early July through early November: Invited charities to participate, gathered information from charities, drafted our comprehensive reviews, and solicited feedback from ACE’s board and Executive Director
Early November: Sent charities completed drafts of the reviews and finalized our recommendation decisions
Mid November: Addressed charities’ feedback on our drafts and solicited charities’ approval to publish
Late November: Published our recommendations on November 24, 2020

Six members of ACE’s team completed most of the work on the reviews. Four board members and ACE’s Executive Director provided feedback on the drafts of the reviews. The Managing Editor and Copy Editor worked to copy edit the reviews and all associated content. ACE’s communications team created the images that appear in our reviews as well as published and announced the results of our evaluations.

**Fig. 1:** Flowchart depicting ACE’s 2020 charity evaluation process

Our Selection Process

We began our 2020 evaluation process by compiling an internal list of charities to consider evaluating. The list included the following:

Charities that requested to be evaluated
Charities that ACE staff and board members suggested evaluating
Charities that third parties asked us to evaluate
Charities we had considered or evaluated in the past that:
- were selected as Top Charities in 2019 (we reevaluate our Top Charities every year)
- were selected as Standout Charities in 2018 (we reevaluate our Standout Charities every two years)
- had been close to the threshold for further investigation but had been excluded for some reason
- we wanted to reconsider due to changes in their programming or our understanding of their activities
International charities that we had not previously evaluated and that we understood to have significant influence in their home countries

Following this initial step, we generated a list of 105 charities to consider evaluating in 2020.

Six members of ACE’s research team and the Managing Editor worked together to select which of the 105 charities under consideration would be reviewed. First, each team member was to use the charities’ websites, social media pages, and any other relevant sources of information, such as news stories or past reviews, to vote for the charities most fit to be reviewed. The team then met and discussed reasons for and against selecting specific charities until consensus was reached and 22 charities were selected. These charities were selected based on factors such as (i) how likely we thought each charity was to perform well on our evaluation criteria, and (ii) how useful we thought the knowledge we would acquire and potentially publish from the comprehensive evaluation would be.

We sent each of these 22 charities a copy of our Charity Evaluation Handbook and formally invited them to participate in the review process. Six of the 22 charities declined to be reviewed,¹ and one charity did not respond to our request to be evaluated. We thus ended up with a total of 15 charities to evaluate: our four Top Charities from 2019, two Standout Charities we last evaluated in 2018, and nine charities we had not evaluated in at least the past three years. One of these 15 charities withdrew before the review process started, one withdrew midway through the review process, and one withdrew after receiving the draft of their review—leaving us with a total of 12 published charity reviews.

Evaluations

We conducted evaluations according to our general process for comprehensive reviews. As part of the evaluation process, we asked each charity that agreed to participate to provide information and documentation about their ongoing programs, accomplishments, finances, and strategy. This year, we also asked charities about the effects of the COVID-19 pandemic on their work. To assess workplace culture, we distributed a survey to each charity’s staff.

While drafting the reviews, we solicited feedback on each criterion from ACE’s Executive Director and three board members. Once drafted, we sent the reviews to the charities for feedback and approval. Before approval, charities had the opportunity to request edits, including requests to remove confidential information or to correct factual errors. Nonetheless, all reviews represent our understanding and opinions, which are not necessarily those of the charities evaluated. This year, 12 of 13 charities for which we drafted reviews agreed to have their reviews published.

Recommendations

After the comprehensive reviews were drafted, but before they were approved by charities, seven members of ACE’s team and ACE’s Executive Director held several meetings to discuss the selection of Top and Standout Charities. In preparation for these meetings, staff read and considered all information in each charity’s review and provided initial impressions of the recommendation decisions.² We observed substantial initial agreement on the status of several charities, but not others. We subsequently discussed the strengths and weaknesses of each charity in-depth, candidly challenged each other’s assumptions and reasoning, and considered feedback and impressions from ACE’s board.

In the end, we selected four Top Charities. We think that, overall, each of our Top Charities performs well on our evaluation criteria. They each conduct effective programs, can make use of additional funding, and have a sustainable work culture. We had either consensus or a majority of supporting votes among the evaluation team that each of these four groups should be selected as Top Charities.

We also selected six Standout Charities. Our Standout Charities are those that we did not select for a top recommendation but nonetheless wanted to call to the attention of our readers because we think they are promising. We think that donations to these charities still seem likely to have a relatively high expected value. We had a majority of supporting votes among the evaluation team that each of these six groups should be selected as Standout Charities.

Additional Information

Of the charities that declined the opportunity to participate, most responded that they were too busy at the time and/or that they preferred to wait until 2021 to be evaluated.
One member of the research team was not involved in the evaluation process for one charity due to a conflict of interest.