Updates to Our Charity Evaluation Process

After two years and three rounds of evaluations using our charity evaluation process, we’ve decided it’s time to make some updates to our charity evaluation process, and to how we decide which ones to recommend. Here’s a brief overview of what we’ll be changing in 2016.

Revised Criteria

Our criteria are the backbone of our charity evaluation process, guiding our questions and considerations at every stage. We’ve made a few updates to better reflect the factors that actually drive the effectiveness of an organization, and to ensure that each one of our criteria is substantial enough to meaningfully address in reviews. You can visit our Evaluation Criteria page to see the new criteria, and we’ve summarized the changes below:

Criterion 4:

The organization possesses a robust and agile understanding of success and failure

We now consider innovation in the description of this criterion to emphasize that part of a clear understanding of success and failure is the ability of an organization to improve itself by trying various changes and preserving those which improve outcomes.

Criterion 6:

Old title: The organization has strong organizational leadership and structure
New title: The organization has strong leadership and long-term strategy.

This criterion still covers the leadership of the organization, but we have moved the structure of the organization, e.g. their training process for new staff, to Criterion 7. Instead, this criterion now includes the organization’s overall strategy and goals, which we evaluate both for logical coherence, e.g. do their goals seem attainable given their resources, and for compatibility with our goal of reducing as much animal suffering as possible. For example, an organization that promotes a plant-based diet for reasons of human health might perform well on all our other criteria, but we would have concerns that they might change course due to changing ideas of what diets are healthiest for humans, while we would think promoting a plant-based diet continued to make sense because of the effects on farmed animals.

Criterion 7:

Old title: The organization is transparent
New title: The organization has a healthy culture and sustainable structure

We’ve often found we didn’t have much to say about organizations’ transparency, and this criterion was much less important in our decision process than the others. We’ve combined some aspects of organizational structure from the older version of Criterion 6, transparency, and some characteristics about how the organizations relates to animal advocacy as a whole (for instance, empowering staff and volunteers to become better advocates) to make a criterion we think will be more useful and differ more between organizations.

New Review Levels and Terminology

We’re replacing our system of shallow, medium, and deep reviews with a system of only two review levels: exploratory and comprehensive. After conducting our first deep review last year, we found that we had learned more about The Humane League than we would have in a medium review, but not enough more to justify the time spent. In particular, we didn’t learn anything that made us think conducting further deep reviews would give us important information for recommendation decisions that we wouldn’t be able to obtain with a less intensive process.

We’ve also had problems with the shallow review process that we think would be reduced by contacting organizations earlier in the review process and basing parts of our reviews on conversations with staff members instead of solely on independent research. We’ll conduct phone calls with organizations before writing exploratory reviews of them. We likely won’t post summaries of these calls, so these reviews will still rely on publicly available and verifiable information, but this will help us understand organizations’ viewpoints and priorities.

Some charities we have reviewed at this level felt we were missing key aspects of their organizations by writing our review using publicly available information. We did submit each shallow review to the charity for them to provide feedback before publishing, at which point they can explain any key aspects we may have missed, but having a call with them before writing the review could make this process easier on the organization. These calls should also help us save time by increasing the likelihood that organizations will approve our reviews, and we expect more groups to be okay with the time commitment because we are now more established as an organization.

In our new system, exploratory reviews will take the place of the old shallow reviews, and comprehensive reviews will take the place of the old medium reviews. The new names are meant to be more descriptive than the old ones and to reflect the fact that we no longer have a third level of review. Additional changes are:

• We’ve added the ability to routinely spend existing income to the list of qualities we look for in deciding whether to conduct a comprehensive review. This was already something we considered at this stage.
• Comprehensive reviews of top and standout charities will have Criticism/FAQ sections similar to the one in our review of The Humane League. We thought this was the most valuable addition from our deep review process. We’ll change some of what we ask in conversations during the comprehensive review process to help us in writing this section of the reviews.
• When we report on room for more funding in comprehensive reviews, we’ll include information about how much additional funding the organization has concrete plans for, but not whether we think they would raise that funding without ACE’s help. We’ve found this particularly confusing to include in reviews for charities that ACE directed funding to during the previous year, and we think it has probably also been more confusing than helpful for our audience.

You can also read our full charity evaluation criteria, and further details about our review process.