The Process Leading to our 2015 Recommendations

This post describes some key features and issues of our recommendation process ending in December 2015. For more detail on our recommendation process in general, see this page.

What We Did

During 2015, we selected groups to review and conducted shallow and medium reviews in a manner similar to previous years. We also conducted our first deep review, intended both to evaluate an organization in greater detail and to find out whether doing so would provide us with useful information we wouldn’t otherwise have obtained. For more detail on the stages of our 2015 review process, see this page.

Basic Consideration

We considered organizations from several sources during this round. We had received requests to be considered from some organizations; we did consider all these groups for a review or recommendation. We also considered some groups that third parties had recommended that we evaluate. Additionally, we used a list of 316 animal advocacy organizations operating in China that another animal advocate had provided us, from which we selected those that seemed like possible candidates for our reviews. Finally, we reconsidered a number of groups that we had considered in the past. These were groups that in earlier rounds had been close to the threshold for some further consideration but had been excluded for time reasons, or that we wanted to reconsider for some particular reason having to do with changes in their programming or in our understanding of their activities. We also decided to update our top charity reviews to ensure that we had the latest understanding of their work.

Shallow Reviews

We conducted 30 shallow reviews during this round, including 4 that were updates to shallow reviews we had conducted in previous rounds. The groups we selected for these reviews were quite diverse, including groups we’d previously reviewed, groups working on farm animal issues, groups working on general animal protection with some focus on farm or wild animal issues, and groups running unusual programs we found promising, such as research and legal advocacy.

Because some shallow review organizations didn’t respond to our requests for financial information, had been reviewed in the past and didn’t appear to have undergone significant changes, or proceeded to the medium review phase, we ultimately drafted 16 shallow reviews. Of these, we’ve been able to publish 6. We think the proportion of shallow reviews we’re publishing is lower than it was last year in part because we cast such a wide net for organizations to review at this depth. Large general animal protection organizations, which made up a significant portion of the 10 groups of which we didn’t publish reviews, seem less likely to benefit substantially from allowing ACE to publish reviews of them. Additionally, our reviews are less positive when we review groups focused less directly on the programs that we think are most promising. Both of these issues will probably continue to arise in the future, as we find that we have already reviewed an increasing proportion of the most promising candidates for recommendations. We’re considering modifying our review process in response, perhaps by reviewing fewer charities overall, since we want to review groups that have a good chance of becoming top or standout charities and we prefer to be able to publish the reviews we write.

Medium Reviews

We conducted medium reviews of 6 new groups, selected from those we had conducted shallow reviews of as well as one other organization, and updated medium reviews for 2 of our 2014 top charities. So far we have published 7 of these new or updated reviews. The review process with the eighth group has suffered some communications delays, but we believe that we will be able to publish that review soon as well.

As with shallow reviews, our medium reviews involved a more diverse set of organizations than in past rounds. One outcome of this was that some reviews do not include a cost-effectiveness estimate. While we think that the cost-effectiveness of a charity’s work is an important consideration in deciding whether to support them, we have often struggled with readers taking our cost-effectiveness estimates more literally than we think they should be taken. All our estimates involve considerable uncertainty around the ultimate impacts of advocacy programs, but we think that providing them usually helps readers understand the specific ways in which different organizations seek to aid animals. Typically, we focus on short-term impact with these estimates, because they are easier to understand and because we are less uncomfortable when they are taken literally. However, some groups we reviewed this year have an impact on animals only or primarily in the medium or long term. In these cases, we chose not to include cost-effectiveness estimates because they would need to be interpreted too differently than the estimates in the majority of our reviews.

Deep Review

Early in the review process, we decided to conduct our trial deep review on The Humane League, and we pursued that review in parallel to the other stages of our review process. The Humane League was a good candidate for this review because they were one of our top charities in 2014, we had already worked with them on two medium reviews and some other projects, and we had found them open and helpful in past review processes. This led us to believe that the finished review would likely be useful to us and our readers, and that it would be a good test of the review process, since new information we discovered would not likely have arisen through additional repetitions of the medium review process.

During the course of the review, we conducted four site visits in three cities where THL works and spoke to a variety of THL staff, interns, volunteers, Board Members, donors, associates from other animal organizations, and teachers who invite THL to give humane education presentations. We plan to evaluate the usefulness of the deep review process this winter before deciding whether to carry out more deep reviews in 2016.

Top and Standout Recommendations

We selected new top and standout charities from among those we’d conducted deep or medium reviews on in an extensive conversation among our Executive Director, Director of Research, and Research Associate. Our top charities are the same as they were in December 2014, which we found interesting. We think this might suggest that additional work on improving our recommendations will have limited returns without the arrival of new evidence, such as studies on particular interventions. It may also suggest that our top charities last year were among the groups that would perform best according to our criteria even in an expanded set of groups considered. We plan to reflect more on this during the winter and potentially change our research priorities. We continued to talk over the next few days to make sure we were all comfortable with the decisions because our individual opinions had varied substantially and we had selected a larger number of standout organizations than some of us expected. For more detail about the specific organizations we selected, see this post.

We now have 9 standout organizations, 5 of which we added during this review process. While we had already decided not to set a cap on the number of standout organizations, this was a larger increase than in previous rounds. Additionally, some of the groups we added to the standout category have very different strengths than those which already occupied it. We think this is a good opportunity to clarify the difference in our thinking between our top charities and our standout charities.

Our top charities excel on all our criteria, taken as a whole. They have room to grow with additional donations; they work on programs with demonstrated effects for animals; and they have strong organizational structures and approaches to assessing their own programs. In order to provide a clear and memorable set of recommendations, we limit the number of top charities. If many groups perform well on all our criteria, we select only the best for our top recommendations.

Our standout charities are those we didn’t choose for a top recommendation, but want to call attention to for various reasons. Some performed well on all our criteria, but not quite as well as our top charities. Others excel in one way, while we would have concerns about recommending them for other reasons. For instance, they might work on very effective programs, but have trouble absorbing new funding. We don’t set a limit on the number of standout charities we recognize because we think each group in this category has important strengths regardless of how they compare to the others we have selected.

Additionally, offering a broad array of groups in this category may be helpful to donors who have slightly different perspectives from our own or different needs from the average donor. For instance, donors who plan to make unusually large donations may find that some of our standout groups can use their funding more effectively than they could use many small donations, and donors who are more confident than we are about advocates’ abilities to control the long-term future may find some of our standout charities especially promising. We encourage anyone who does donate to our standout charities because of our reviews to let us know so that we can track our own impact.

Some Big Questions

In the past, most of the groups in our medium review process had worked on fairly similar farm animal advocacy programs. This year, we considered some groups doing different types of projects, including research, legal advocacy focused on the long term, and the development of plant-based substitutes for animal products. As a result, our final recommendation decisions rested very clearly on our answers to some big questions about what programs to prioritize and where funding would be most useful.

We considered questions like:

• How should we account for the potential long-term impact of organizations? What should we think of the cost-effectiveness estimates for these impacts, which tend to be much higher than our estimates of near-term, direct impact?
• What is the difference in impact between spreading different animal advocacy ideas, e.g. the cruelty of animal agriculture, dietary change, antispeciesism, and wild animal suffering?
• If public sentiment becomes more friendly towards animals (e.g. it becomes generally accepted that we should get rid of animal agriculture), will legal and policy change inevitably follow?

As a team, we didn’t all agree on every point. However, we agreed that the questions were important, and we discussed them until we understood each other’s perspectives and could agree on a set of recommendations. We will be publishing a series of blog posts discussing the foundational questions we encountered during this round of recommendations, beginning later this week.