Top

2016 Evaluation Process

This page discusses the specific process that led to our 2016 recommendation update; a more general description of our evaluation process is available elsewhere.

Overview

We periodically conduct evaluations of charities at various levels of detail and update our recommendations as a result of these evaluations. Prior to November 2016, our most recent recommendation update took place in December 2015.

During the first few months of 2016, we focused our attention on refining our evaluation process. We updated our criteria and conducted some of the foundational and intervention research that informs our reviews. Our evaluation process took place from June through November, as follows:

  • We conducted the “basic consideration” phase of our review cycle in June.
  • We began conducting exploratory reviews in late June. Some communications regarding exploratory reviews extended until November.
  • We began conducting comprehensive reviews in September. Drafts of our comprehensive reviews were completed in early November.
  • We finalized our recommendation decisions on November 4, 2016.
  • We published our recommendations on November 28, 2016.

Our research team did most of the work on this round of reviews, with help from our Executive Director throughout the process. Volunteers helped to transcribe conversations and an intern assisted with various tasks, including ensuring consistency between reviews and cost-effectiveness estimates.1 Our Executive Director, advocacy research Program Officer, and several Board Members provided feedback on a draft of the reviews. Finally, our communications team published the results.

Basic Consideration

We began our 2016 evaluation process by compiling an internal list of charities to consider evaluating. The list included:

  • Charities that requested to be evaluated.
  • Charities that members of the ACE staff and board suggested evaluating.
  • Charities that third parties asked us to evaluate.
  • Charities that we had considered or evaluated in the past that:
    • had been close to the threshold for some further investigation but had been excluded for some reason, or
    • we wanted to reconsider due to changes in their programming or our understanding of their activities.
  • International charities that we had not previously evaluated and that we understood to have significant influence in their home countries.

We also updated our Top Charity reviews to ensure that we had the latest understanding of their work, as well any Standout Charity reviews that had not been updated for more than a year.

At the end of this process, we had a list of 64 charities to consider evaluating in 2016, 28 of which were groups that we had considered or evaluated before, and 36 of which were new to us. Our internal master list of charities we’ve considered now includes 304 organizations and websites, up from the 268 organizations and sites we listed internally at the end of the previous review process.

We added all but one of the 36 new groups we considered this year to the 170 previously included on our published list of charities. We also added one group we had previously considered but not included on our published list because they lacked an English-language website. This brought the total number of charities on our published list to 206.2The group that we did not include on our published list works on an area outside the scope of the work we typically evaluate. The rest of the groups were added to the “Charities Considered” list or (based on later review) to our “Top Charities” or “Standout Charities” lists.

Exploratory Evaluations

Selecting Charities to Evaluate

We set a target of evaluating 15-28 charities this year, including exploratory evaluations, new comprehensive evaluations, and the reevaluation of our 2015 Top and Standout Charities. We reviewed our list of 64 charities to consider for evaluation in 2016, and we worked to identify the charities that seemed most valuable for us to evaluate.

In previous years, we chose to evaluate charities only if they had English language websites. However, this year, we decided to evaluate promising charities that did not have English language websites, provided that they had English-speaking employees who were able to provide us with information we needed.

We identified charities that were in one or more of the following groups as valuable for us to evaluate:

  • Charities that seemed to be promising candidates for recommendation, including:
    • Charities that we had flagged in previous rounds as strong candidates for an exploratory or comprehensive evaluation, but had not evaluated at that depth.
    • Charities doing unusual work or some type of animal advocacy related research that we thought might be promising.
    • Charities that seemed to be influential and plausibly cost-effective in countries outside the U.S.
    • Charities that we knew had made significant progress in their work compared to the last time we evaluated them.
  • Large or well-known charities that we are often asked about but have not yet evaluated.
  • Charities that work in areas other than farmed animal advocacy that are plausibly cost-effective.
  • Charities that seem interested in and receptive to constructive feedback.
  • New charities that might not have a long track record, but that we wanted to learn more about.

Each member of the ACE research team individually considered whether or not we should evaluate each of the 64 groups on our list. In June 2016, the research team had a meeting to make final decisions about which charities to evaluate. We agreed unanimously on 12 charities to evaluate and 18 charities to exclude from evaluation. We discussed the remaining charities until everyone agreed upon a total of 20 charities to evaluate, including the seven Top and Standout Charities due for reevaluation. We also chose two charities to place on reserve in case some of the 20 did not wish to be evaluated, and we chose to conduct phone calls with three charities that we had previously evaluated, in order to decide whether to update their reviews.

Conducting Evaluations

We conducted background research on each of our chosen charities, excluding those which we had already researched thoroughly in the past. We relied upon the charities’ websites and other publicly available information. We considered basic financial information such as assets, revenues, and expenditures for recent years. Nonprofits registered in the United States, and in some other countries, submit standardized financial information to the government, and in some countries this information becomes easily available to the public through the government or through third parties such as Guidestar.

Next, we contacted each charity to set up an interview with their leadership. Two groups declined to be evaluated, so we moved forward with the two charities we had on reserve. We also set up calls with the three charities that we had previously evaluated and were undecided about reevaluating, but we did not yet contact the seven Top and Standout Charities that we knew we would comprehensively reevaluate. Our Researcher and Research Associate conducted the phone calls for our exploratory evaluation process. We recorded the calls for internal use but chose not to publish call summaries, which would require staff or volunteer hours.

After conducting background research and phone calls with each charity but before drafting exploratory reviews, we selected 12 charities to proceed to the comprehensive review process (see below). We did not write exploratory reviews for these charities, the two that declined to be evaluated, or two charities for which we had previously written reviews and found little need to update them.

We wrote a total of nine exploratory reviews. We sent them to the corresponding charities for corrections and approval. Ultimately, eight charities allowed us to publish our exploratory reviews. Some reviews were altered after we communicated with the charity, but they still represent our own understanding and opinions, which are not necessarily those of the charity reviewed. One group asked us not to publish their review, so we listed them on our website, noting that they declined to be reviewed.

Comprehensive Evaluations

Selecting Charities to Evaluate

We set a target of conducting three to six new comprehensive reviews this year, in addition to the comprehensive reevaluation of our seven Top and Standout Charities from 2015.

To select new charities for comprehensive reviews, our research team had two meetings after researching the exploratory reviews but before writing them. In the first meeting, the team unanimously selected three new charities to comprehensively evaluate. In the second meeting, each team member came prepared with a ranked list of the remaining charities and we chose two more for comprehensive evaluation. Both were in the top three of each team member’s list.

In total, 12 charities were selected for comprehensive evaluation or reevaluation.

Conducting Evaluations

We proceeded according to our general process for comprehensive reviews. We conducted a second set of phone calls with the charities we selected for comprehensive reviews during the exploratory process, and we conducted one set of phone calls for the charities due for reevaluation.

We drafted comprehensive reviews and call summaries for each charity. This year, a board committee read an early draft of our reviews and provided feedback, both on our general evaluation process and on specific reviews. They also provided some initial thoughts about which charities we should recommend.

After editing our comprehensive reviews and making our recommendation decisions (described below), we sent the reviews to the corresponding charities for approval, along with our conversation summaries and other supporting documents we hoped to publish. This year, 11 out of 12 charities agreed to the publication of our comprehensive reviews. One group asked us not to publish their review, and we listed them in the “Charities Considered” section of our list of organizations, indicating that they declined to be reviewed.

Recommendations

After the sections of our comprehensive reviews corresponding to our seven criteria were drafted, but before the reviews were entirely finished, our research team and Executive Director held two meetings to select our 2016 Top and Standout Charities.

In the first meeting, each of us shared lists we had prepared individually of our suggestions for Top Charities, Standout Charities, not recommended charities, and borderline cases. There were some charities about which everyone agreed and others about which we had more diverging opinions. We will provide limited detail about our decision process here, since most of it involved the specific aspects of individual organizations that we describe in our reviews. After extensive discussion, we agreed about the status of eight charities. We continued to discuss the remaining four charities over the next several days, and then we held a second meeting to finalize our decisions.

Additional Information

The Process Leading to Our 2016 Recommendations (Blog Post)
Updated Recommendations: 2016 (Blog Post)
Archive: 2016 List of Considered Charities
Detailed Recommendation Process Description for December 2015 Recommendations
Detailed Recommendation Process Description for December 2014 Recommendations
Detailed Recommendation Process Description for May 2014 Recommendations

  1. Volunteers who worked on this round of evaluations and preferred to be credited by name were Sophie Kirkham and Christoph Sohn. 

  2. In previous years, other charities have been added to our internal master list but not to our published list of organizations. Reasons we excluded a charity from our published list have included: (i) the charity requested not to be listed on our website, (ii) we were unable to evaluate the charity because they did not have an English language website, (iii) the charity appeared to be no longer active, or (iv) the charity performed direct animal care and did not request to be considered. (In 2015, we began including groups on our published list that engage primarily in direct animal care, if they requested consideration.)