May 2014 Evaluation Process

ACE regularly conducts evaluations of charities at a variety of levels of detail; we update our recommendations periodically as a result of these evaluations. This page discusses the specific process that led to our May 2014 recommendation update; we discuss more general features of the process elsewhere. We also wrote a blog post about this process while we were working on it.

Overview

Prior to May 2014, our most recent recommendation update took place in 2012, while ACE was still the all-volunteer organization Effective Animal Activism. ACE staff had not been involved in selecting the previously recommended organizations and did not have access to complete records of the recommendation process from that time period. This recommendation update was the first that used our evaluation criteria and multi-step process for deciding on recommended and standout charities.

We conducted the basic consideration phase of our review cycle in January 2014.

We conducted shallow reviews starting in January. Some communications regarding shallow reviews extended until April.

We conducted medium reviews starting in February. All medium reviews were completed in May.

We published our updated recommendations on May 14th, 2014.

ACE’s Executive Director and Director of Research did most of the work on this round of reviews, with help from a volunteer in gathering information for shallow reviews, and from ACE’s Director of Communications in publishing the results.

Basic Consideration

We considered organizations from several sources during this round. We generated an initial list based on the list of organizations that Effective Animal Activism originally reviewed, adding some other organizations that we knew were working on behalf of farmed animals. We also reached out to several leaders in the farm animal advocacy community to ask for suggestions of organizations that they thought were particularly effective. This was our first review process and we felt we would have lots of challenges to deal with anyway, so we then restricted this list to organizations working in the United States. Because of our limited understanding of the specific challenges and advantages for organizations in other countries, delaying evaluating those organizations until another round helped to ensure that we would be judging fairly.

Our internal master list at the end of this process included 105 organizations and websites. We included 97 of these on our published list of organizations (excluding a few organizations for reasons discussed below). Ninety-four of these organizations were added to the “Charities Considered” section on the list, or (based on later review) to our top charities or standout charities lists. The three organizations we added to the “Charities Not Considered” section included ACE itself, VegFund, which has a major program that makes grants to other organizations we did or could consider, and Direct Action Everywhere, which we could not consider because of our donation conflict of interest policy.

The organizations that we had on our master list but did not add to our public list during this round included a few organizations which seemed to be defunct or no longer had functioning websites, as well as two organizations for which we conducted reviews. When we contacted these two organizations to obtain permission to publish reviews of them, they asked that no mention of them appear on our site, so while we included them in the total counts of organizations we considered and reviewed, we didn’t print their names on our site.

Shallow Reviews

We set a target of conducting around 30 shallow reviews during this round.

Selecting organizations to review

We compiled a list of possible organizations to review from the organizations considered in the previous round. We chose to review organizations that focused on farmed animals, directed a majority of their programming towards helping farmed animals or promoting plant-based diets, or were large enough that we might reasonably evaluate only the branch of their activities dealing with farmed animals. The organizations we selected varied widely in other ways, from local groups run entirely by volunteers to some of the largest animal advocacy groups in the United States, and from farm animal sanctuaries providing direct care to animals to an organization devoted to promoting the development of cultured meat.

At this point, we had 29 organizations on our list for shallow reviews.

Conducting reviews

We conducted reviews for each of these organizations based on their website and other publicly available information, in keeping with our usual process.

Before writing the shallow reviews, we selected 7 of the shallowly-reviewed organizations to proceed to the medium review process, and 6 agreed to participate in it (see below). We did not write shallow reviews for these organizations.

We wrote a total of 23 shallow reviews. We then sent these to the organizations involved for corrections and approval, both to ensure that our reviews would be factually accurate and to maintain good relationships within the animal advocacy community. Ultimately, 6 of these groups allowed us to publish versions of our reviews. Reviews which we altered after communicating with the organization involved still represent our own understanding and opinions, which are not necessarily those of the group reviewed. We also decided that one organization, VegFund, did not fall into our scope of consideration because we learned that granting funds to other organizations like those we would review is a major part of their activities. The other 16 groups either requested not to be reviewed or did not reply to our attempts to contact them: not all had seen the text of our review when they made that decision. We marked all of these groups as “declined to be reviewed”, except two groups which we left off our list entirely at their request, and did not publish their associated shallow reviews.

Medium Reviews

We set a target of conducting 6–8 medium reviews.

Selecting organizations to review

To select charities on which to conduct medium reviews, after researching the shallow reviews but before writing them, we had a meeting. Each of us (Allison and Jon) individually composed a private list of organizations for possible review. Then we discussed those organizations which appeared on one list but not both, to create a final list of organizations to approach for a deeper review. In general, organizations which we chose to continue reviewing:

• Attempted to help farmed animals through public outreach and/or legislative advocacy.
• Focused attention on the rights and welfare of farmed animals.
• Had at least some paid staff members.
• Maintained informative and up-to-date websites.

We contacted 7 groups about moving forward with medium reviews for them. Six groups agreed to proceed; one said they were not ready for an in-depth evaluation and asked us to proceed with a shallow review instead, which we did.

Conducting reviews

We conducted the reviews according to our general process for medium reviews.

We provided organizations with conversation summaries, reviews, and other documents as we decided which groups to recommend and what we wanted to publish. All organizations which decided to proceed with the medium review allowed us to publish reviews. Summaries of conversations we had with their staff were also made available separately.

Recommendations

After the research on the medium reviews was finished, but before the writing was finished, we decided which groups should be top recommendations and standout charities. These decisions were reached through several meetings over the course of about two weeks, during which we discussed both general recommendation criteria and factors to be aware of, and how well individual organizations performed in these areas. Since this was our first time arriving at recommendations, this process was slower and less formalized than we expect it will usually be. We accumulated and processed additional information while writing the reviews, which helped us reach decisions about groups we weren’t certain about at first. We’re limited in how much detail we can provide about this decision process, since most of it had to do with specific aspects of individual organizations. We knew how we planned to categorize each group before we reached out to any about publication of their review.

After this round, we announced two top charities and two standout organizations.

Additional Information

Recommendations Update: May 2014
Archive: May 2014 Considered Charities
Shallow Evaluations Process Update