2019 Evaluation Process - Animal Charity Evaluators

Overview

Once a year, ACE evaluates charities and updates our recommendations as a result of those evaluations. Prior to December 2019, our most recent recommendation update took place in November 2018.

During the first few months of 2019, we made some updates to our charity evaluation process and conducted some of the foundational and intervention research that informs our reviews. Our formal evaluation process took place from June to December.

This year, we included ratings of each charity’s performance on each criterion for the first time. These ratings are visual representations intended to help our readers quickly compare charities’ performance and easily identify how confident we feel about our appraisal. We also increased the number of visual aids in each review in order to make our charity evaluations more accessible to a wider audience.

Additionally, we made changes to our 2019 cost-effectiveness models to address some limitations we’ve identified over the years. We moved away from a fully quantitative model and transitioned to a more qualitative approach that—for each intervention type—compares the resources used and outcomes achieved across all the charities reviewed. Although this approach has limitations, it provided some insight into the cost-effectiveness of all reviewed charities regardless of the timescale or directness of their impact, allowing us to make comparisons that we were previously unable to make.

This year, we once again sent our organizational culture surveys to charities’ staff, and we decided to make participation an eligibility requirement for receiving a recommendation from ACE. We found that distributing our culture survey to each charity under review gave us a much fuller picture of the charity’s culture. We also found that distributing the same culture survey to every organization was essential because charities’ internal surveys vary widely in content, relevance, and quality.

Finally—towards the end of our evaluation process—we hired a Field Research Associate to improve our capacity to more deeply investigate empirical information. His primary role is to identify and verify the factual statements included in our research, including claims made by charities under evaluation.

The general timeline of our evaluation process was as follows:

Late June: selected charities to review
Early July: sent out invitations to participate, along with the charity evaluation handbook, and began gathering information and conducting research for comprehensive reviews
October: completed full drafts of the comprehensive reviews, solicited feedback from ACE’s board and Executive Director, and worked to incorporate it into the initial drafts
Late October: finalized our recommendation decisions
Early November: communicated recommendations to charities under review and sent them completed drafts of the reviews between November 1 and November 6, 2019
November: addressed charities feedback on our drafts and solicited charities’ approval to publish
Early December: published our recommendations on December 2, 2019

Four members of ACE’s research team completed most of the work on the reviews: Director of Research Toni Adleberg, Researchers Maria Salazar and Jamie Spurgeon, and EAA Fund Program Officer Marianne van der Werf. A team of volunteers (managed by ACE’s Managing Editor) helped summarize conversations with charities that had been transcribed using Otter Voice Notes. Another researcher, three board members, and ACE’s Executive Director provided feedback on the drafts of the reviews. The Managing Editor and Copy Editor worked to copy edit the reviews and all associated content. ACE’s communications team created the images that appear in our reviews, as well as published and announced the results of our evaluations.

**Fig. 1:** Flowchart depicting ACE’s 2019 evaluation process

Our Selection Process

We began our 2019 evaluation process by compiling an internal list of charities to consider evaluating. The list included:

Charities that requested to be evaluated
Charities that ACE staff and board members suggested evaluating
Charities that third parties asked us to evaluate
Charities we had considered or evaluated in the past that:
- Were selected as Top Charities in 2018 (We re-evaluate our Top Charities every year.)
- Were selected as Standout Charities in 2017 (We re-evaluate our Standout Charities every two years.)
- Had been close to the threshold for further investigation in the past but had been excluded for some reason
- We wanted to reconsider due to changes in their programming or our understanding of their activities
International charities that we had not previously evaluated and that we understood to have significant influence in their home countries

Following this process, we generated a list of 173 charities to consider evaluating in 2019.

Four members of ACE’s research team and the Managing Editor worked together to select which of the 173 charities under consideration would be reviewed. The first step was for each team member to use the charities’ websites, social media pages, and any other relevant sources of information (such as news stories or past reviews) to vote for the charities most fit to be reviewed. These initial votes were used to rank the charities from most to least likely to be selected for review. The team then met and discussed reasons for and against selecting specific charities until consensus was reached and 16 charities were selected. These charities were selected based on factors such as (i) how likely we thought each of the charities was to be recommended and (ii) how useful we thought the knowledge we would acquire and potentially publish from the comprehensive evaluation would be.

We contacted each selected charity to invite them to participate. At this point, four of the 16 charities declined to be reviewed.¹ We held an additional meeting and selected one more charity to invite to participate, but they also declined in favor of being evaluated next year instead. We ended up with a total of 12 charities to evaluate.

Comprehensive Evaluations

In total, we conducted comprehensive reviews for 12 charities in 2019. This included our four Top Charities from 2018, the four Standout Charities that were last evaluated in 2017, and four other charities that we hadn’t evaluated for at least the past three years.

Early on, we had a discussion about whether we wanted to write up exploratory evaluations for any charities this year. We decided that all charities under consideration were promising enough to evaluate comprehensively.

We conducted evaluations according to our general process for comprehensive reviews. Charities that agreed to participate were asked to provide information and documentation about their finances, accomplishments, and strategy by filling out a questionnaire and a spreadsheet of their top 3–5 program outcomes. We also scheduled conversations with their leadership. To assess workplace culture, we asked each charity to provide us with contact information for all of their staff members so that we could distribute our culture survey. After conducting interviews and gathering the documentation, we sent some follow-up questions to charities. In preparing and drafting the reviews, each member of the evaluation team researched and wrote the same one or two sections of each review.²

Managed by ACE’s Managing Editor, a team of volunteers summarized conversations with charities that had been transcribed using Otter Voice Notes. Three board members and our Executive Director provided feedback on initial drafts of the reviews. Our Managing Editor and Copy Editor then worked to copy edit the supplemental documents and comprehensive reviews. Finally, our communications team published and announced the results.

After editing our comprehensive reviews and making our recommendation decisions (described below), we sent the reviews to the corresponding charities for approval, along with our conversation summaries and other supporting documents we intended to publish. Charities had the opportunity to request edits, including requests to remove confidential information or to correct factual errors. Nonetheless, all reviews represent our own understanding and opinions, which are not necessarily those of the charity reviewed. This year, 11 of 12 charities for which we drafted comprehensive reviews agreed to have their reviews published.

Recommendation Decisions

After the majority of each comprehensive review was drafted, but before the reviews were entirely finished or sent to charities for approval, four members of the research team and ACE’s Executive Director met to discuss the selection of Top and Standout Charities. In preparation for this meeting, all participants indicated their individually prepared suggestions for Top and Standout Charities in a chart.³ There was substantial initial agreement on the status of some charities, but not all. We discussed the strengths and weaknesses of each charity and updated our individual opinions when necessary. At the conclusion of this meeting, we had reached unanimous agreement on the recommendation status of most charities.

The same four research team members and the Executive Director participated in several email threads to finalize our selection of 2019 recommended charities.⁴ A unanimous agreement was reached for most charities under review, and for the remainder, the Director of Research and Executive Director made final decisions. We will not provide more details about our decision process here; most of it involved the specific aspects of individual organizations that we describe in our reviews.

Additional Information

Of the charities that declined the opportunity to participate, most responded that they were too busy at the time or that they preferred to wait until 2020 because they had a recent change in leadership.
The one exception is our Director of Research, who was not at all involved in the evaluation process for one charity due to a conflict of interest.
Due to a conflict of interest, ACE’s Director of Research did not participate in discussions involving our evaluation of one charity.
Due to a conflict of interest, ACE’s Director of Research did not participate in discussions involving our evaluation of one charity.