The Process Leading to Our 2017 Recommendations

This post provides a brief discussion of the specific process that led to our 2017 recommendation update. For a more detailed description of the process, please see our 2017 Evaluation Process page. For a general description of our evaluation process, please read about our General Evaluation Process.

Overview

Our formal evaluation process took place from May through November. The timing of some noteworthy events follows:

• We conducted the “basic consideration” phase of our review cycle in May.
• We began conducting exploratory reviews in June. Some communications regarding exploratory reviews extended into November.
• We began conducting comprehensive reviews in July. Drafts of our comprehensive reviews were completed in October.
• We finalized our recommendation decisions in late October and communicated them to the charities under review on October 25, 2017.
• We published our recommendations on November 27, 2017.

The process led to the publication of three new exploratory reviews, 11 new or updated comprehensive reviews, and an update to our recommendations.

Basic Consideration

We began our 2017 evaluation process by compiling an internal list of charities to consider evaluating. At the end of this process, we had an internal list of 629 charities to consider evaluating in 2017, 60 of which were groups we had considered or evaluated before, and 569 of which were new to us.¹

Exploratory Evaluations

We then reviewed our list of charities to consider for evaluation in 2017 and attempted to identify the charities that seemed most valuable for us to evaluate.² We initially selected a total of 23 charities for exploratory evaluation this year. We then contacted each of those selected charities to set up an interview with their leadership. In the time between this initial point of contact and when we began drafting exploratory reviews, 15 of the 23 charities selected for exploratory review either directly declined to be reviewed or didn’t respond to our emails in time for us to continue with the exploratory evaluation process.

We were surprised that such a high proportion of the charities selected for exploratory review declined to be reviewed at this stage. The most frequently reported reason by groups for declining to be reviewed at this stage (selected by five charities) was that the charity would like to wait until a later year to be evaluated. A greater number of charities selecting this reason may partly be due to a greater number of young charities being selected for exploratory review. Often, younger charities choose to wait to be reviewed in order to better establish their programs and staff and develop a longer track record. The seemingly high proportion of charities declining to be reviewed at this stage in our 2017 review process is something that we would like to consider further and better account for in future evaluation seasons.

After conducting background research and interviews³ with each charity, we selected four charities to proceed to the comprehensive evaluation process.⁴ We did not write exploratory reviews for those charities. We wrote four exploratory reviews for the remaining charities and sent them to the corresponding charities for corrections and approval. Ultimately, three charities allowed us to publish versions of our exploratory reviews.

Comprehensive Evaluations

In total, we selected 16 charities for comprehensive reviews this year. This included all three of our Top Charities from 2016, six of our Standout Charities from 2016, four charities that were initially selected for exploratory review in 2017, and three other charities that we had previously comprehensively evaluated.

We contacted each charity with whom we had not yet spoken to set up an interview with their leadership. At this point, two of the 16 charities declined to be reviewed. After completing some of the calls, we began drafting comprehensive reviews and call summaries for each of the remaining charities and sent follow-up questions to each of these charities. A board committee, our research scientist, and our Executive Director then read an early draft of our comprehensive reviews and provided feedback, both on our general evaluation process and on specific reviews. They also provided some initial thoughts about which charities should receive top or standout recommendations.

After editing our comprehensive reviews and making our recommendation decisions (described below), we sent the reviews to the corresponding charities for approval, along with our conversation summaries and other supporting documents we hoped to publish. All published comprehensive reviews were altered after we communicated with the corresponding charities, but they still represent our own understanding and opinions, which are not necessarily those of the charity reviewed. This year, three out of the 14 charities for which we drafted comprehensive reviews declined to have their reviews published.

Recommendations

After the majority of each comprehensive review was drafted, but before the reviews were entirely finished or sent to charities for approval, five members of our research team and our Executive Director held two meetings and participated in several email threads to select our 2017 Top and Standout Charities. In the first meeting, each of us shared lists we had prepared individually of our suggestions for Top Charities, Standout Charities, not recommended charities, and borderline cases. There was substantial agreement on the recommendation status of most charities and diverging opinions about the recommendation status of a few charities. After discussion, we agreed about the recommendation status of eleven charities. We continued to discuss the remaining three charities over the next several days, and then we held a second meeting to finalize our decisions. We will not provide more details about our decision process here, though most of it involved the specific aspects of individual organizations that we describe in our reviews.

We think that, overall, our Top Charities perform outstandingly well on our criteria. They have clear plans for effectively using additional funding. We think they have strong organizational structures and they’ve demonstrated an interest in and capacity for evaluating their own programs. This year, we thought many groups performed well on our criteria, including several groups that we had not evaluated before. We considered selecting more than three Top Charities, but after taking into account the amount of money we move annually and the amount of funding each charity could use effectively, we settled on three Top Charities that we thought were the very best.

Our Standout Charities are those we didn’t select for a top recommendation, but nonetheless wanted to call to the attention of our readers because we think they are quite promising—donations to them seem likely to have a relatively high expected value. We think some of these charities performed admirably on all our criteria, but not quite as well as our Top Charities. Others excel on only some of our criteria. For instance, they may work in a promising area, but one in which establishing a firm track record of impact is especially difficult. We hope that our range of Standout Charities is helpful to donors who have different perspectives than our own or are searching for an effective charity that works in a particular area.

This year, we introduced an additional designation within the Standout Charity category so as to be more transparent with regard to how we view each organization. We hope that this designation will allow our audience to make more informed donations. Specifically, we now classify Standout Charities as either “Standout: General Interest”⁵ or “Standout: Special Interest.”⁶ Some further information about these Standout Charity classifications, as well as the charities we currently recommend, is available in this recent blog post.

Important Considerations

A number of important questions arose in our discussions about which charities to recommend. We discussed the relative strengths and weaknesses of the charities we evaluated as well as Animal Charity Evaluators’ role in the animal advocacy movement.

Some of the questions we discussed are the same ones we tend to grapple with annually. For example:

• Is there an ideal number of Top and Standout Charities?
• How much money do we expect our recommendations to move to our Top Charities next year, and how does that affect the number of Top Charities we should select?
• How should we account for charities’ impact when it is difficult to estimate in terms of the number of farmed animal lives or farmed animal years averted?

Some other questions were more significant this year than previous years. For example:

• Compared to farmed animal advocacy in the U.S., how valuable is animal advocacy in other regions, such as neglected and populous South American, Eastern European, or Asian countries?
• How should we weigh the anonymous non-leadership calls when evaluating an organization’s culture?
• How should we respond to questions concerning charities that decline to be reviewed/published?

We think that answering any of these questions well is very difficult and laden with uncertainty. Still, we continually work to improve our understanding of effective animal advocacy.

For further information about our thinking, please watch our blog for upcoming posts on topics, including aspects of our 2017 evaluation process that surprised us and a chart comparing our current Top and Standout Charities.

1. Our list of charities now includes 326 organization and websites, up from the 205 organizations and sites we listed before the commencement of the 2017 review process. We choose not to include all of the charities we considered in this round on our published list because many of those charities were mainly humane societies or companion animal shelters located in France, Germany, Italy, or Spain and we thought there was limited value in adding those groups to our published list. We added 121 new groups to our list and updated our entries for a further 60 charities. 

2. We identified charities that were in one or more of the following groups as valuable for us to evaluate:

   • Charities that seemed to be promising candidates for recommendation, including:
     • Charities that we had flagged in previous rounds as strong candidates for an evaluation
     • Charities doing unusual work or some type of animal advocacy related research that we thought might be promising
     • Charities that seemed to be influential and plausibly cost-effective in countries outside the U.S.
     • Charities that we knew had made significant progress in their work compared to the last time we evaluated them
   • Large or well-known charities that we are often asked about but have not yet evaluated
   • Charities that work in areas other than farmed animal advocacy that are plausibly cost-effective
   • New charities that might not have a long track record, but that we wanted to learn more about

    

3. Most interviews happened via phone or video calls. In one case, we weren’t able to arrange a call with the charity and instead exchanged emails with them in order to collect the equivalent information. 

4. To select which charities that had originally been selected for exploratory review would be chosen for a comprehensive review, four members of our research team met after completing the research for the exploratory reviews but before writing the exploratory reviews. In that meeting, the four staff selected four charities to comprehensively evaluate. These charities were selected based on factors such as how likely we thought each of the charities was to be recommended, and how useful we thought the knowledge we would acquire and potentially publish from the comprehensive evaluation would be. Five staff members then discussed one charity further over email and then decided not to comprehensively review that charity. 

5. We consider these Standout Charities to be good choices for evidence-minded donors with limited time available to research their donations. We expect these charities would appeal to most donors who generally agree with our priorities and values. 

6. We think these Standout Charities are especially attractive to donors with certain specific interests. For example, some may appeal to donors interested in supporting longer-term work or more speculative projects. A Standout Charity would also fall in this category if we think donors may need to do further research in order to donate there effectively—for example, if donations should be restricted to a particular program.