Process for Evaluating Charities & Making Recommendations
ACE’s goal in evaluating animal charities is to find and promote the charities which work most efficiently to help animals. We want the charities we recommend to be very strong, both in terms of the programs they choose to implement and in their characteristics—such as the way they make decisions and how well they find, develop, and retain skilled employees and volunteers. We promote the strongest charities we have reviewed as our Top Charities, and we also recognize some excellent charities about which we are slightly less confident, or which are slightly less efficient, as Standout Charities. We do not rank or rate charities below these levels.
Our evaluation process consists of several rounds in which we consider progressively fewer charities in progressively greater detail. We give basic consideration to a very large number of organizations, conduct exploratory reviews of a smaller number, and then conduct comprehensive reviews of a small number of likely candidates for our recommendations. We experimented with conducting a “deep review” in 2015 of one of our Top Charities, The Humane League. As of 2016, we are not planning to conduct any more deep reviews, but we will incorporate some aspects of the deep review into our comprehensive reviews going forward.
After conducting these rounds of evaluation, we update our recommendations and the reviews published on our site. All our Top and Standout Charities have received comprehensive reviews, but not all charities for which we have written comprehensive reviews are in these categories.
This is the first round of our recommendation process. It is intended to generate a list of animal charities and their focus areas and to improve ACE’s overall understanding of the animal charity landscape. We need to have a sense of the types of animal charities in existence, and of how prevalent each is, in order to make strategic decisions about our research in general—as well as the evaluation process in particular.
Which organizations are considered?
Any organization can request to be considered in this round. We also maintain an internal list of charities to consider based on recommendations we have solicited from experts and the general public, other lists of animal charities maintained online, and charities we have encountered at conferences and through other means. We consider every organization that has requested consideration since the previous recommendation process, as well as most other charities on our list. We do not always consider organizations that provide direct care to animals—particularly companion animal shelters and rescues—because we find that these organizations are usually only able to help relatively few animals with a given amount of resources and because there are very many of these organizations, including thousands in the United States alone. We also do not consider charities if, from previous interactions with their leadership, we believe they would prefer not to be considered in our recommendation process.
What do we do to research each organization during this round?
We search for and visit each organization’s website to determine their general focus area and methods. We record basic information about the organization for future use. Our research in this round addresses the following questions:
- Is this a currently active animal advocacy charity with a presence on the internet? If a charity has no web presence, it will be difficult for us to learn enough about them to seriously consider recommending them, and our audience will have a hard time verifying our review and even donating to the charity. On the other hand, we sometimes find references or links to defunct charities, branches of other charities, and personal blogs or websites. While we occasionally review branches of larger charities (for example, the Farm Animal Protection Campaign of the HSUS) because donations can sometimes be meaningfully restricted to a particular program, we have no reason to review defunct charities or personal blogs.
- Is the organization within our scope of evaluation? ACE evaluates charities and nonprofits that work to benefit animals. We do not require charities to be registered in any particular country or under any particular framework; we would consider evaluating a for-profit company which had room for funding by individuals and strong evidence of cost-effectiveness in helping animals. We do not evaluate charities whose primary activity is making grants to other charities we could or do evaluate; this excludes some but not all foundations.
- What language or languages does the charity use on their website? ACE is based in the United States; we conduct the majority of our work in English. We look for charities to have a version of their website available in English; this signals that we will have an easier time learning about and communicating with the charity than if they operated only in another language. It also allows our audience to read about the charity in their own words. We may consider evaluating charities that appear very strong in other ways but do not have an English website, only if the charity and relevant ACE staff have a common language available. We do not currently have the resources to have entirely translated/interpreted interactions with any charity.
- What does the charity seek to accomplish, and what general tactics do they use toward this goal? These are big questions, and at this stage we only try to understand them in broad terms. For example, we expect to be able to distinguish among a charity which provides sanctuary for farmed animals, one which provides sanctuary for big cats, and one which advocates for farmed animals by lobbying legislators. We do not necessarily expect to be able to distinguish more specific divisions of resources, such as between a charity which has a major program advocating for lab animals and a side program advocating for farmed animals and a charity which has a major program advocating for farmed animals and a side program advocating for lab animals.
- Does the charity advocate or participate in violence or the destruction of property? We have not rigorously evaluated the effectiveness of these tactics, but as a matter of caution, ACE does not endorse violence or property destruction as a means to help animals. We make an exception for very small amounts of property damage in support of a larger project, such as when a lock is broken in order to take footage of a farming operation.
What do we publish as a result of this round?
We list charities we considered in this round on our website with icons denoting their focus areas and links to their websites. We do not list charities that did not have websites, websites that did not correspond to active animal advocacy charities, or charities that advocate violence. We also do not list charities whose websites are not available in English or another language we can read, and which we have decided not to investigate further for that reason, because it is not fair to say that we have compared these charities to the other charities we have reviewed. We err on the side of inclusion for charities which seem to be genuine but may be defunct.
If we have considered a charity at only this level and it appears to be within our scope of evaluation, it is listed as “Considered” on our complete list of charities.
This is the second round of our evaluation process. It is intended to identify those charities that engage in promising activities and may have room for more funding.
Which charities are considered?
At the start of the recommendation process, we set a target number of charities to investigate in this round, based on our available resources. We select charities for this round from those considered in the previous round and in previous recommendation cycles, using the following guides:
- Focus area. The area a charity focuses on has a high expected impact on their cost-efficiency with respect to ACE’s primary metrics of lives spared per dollar spent and suffering averted per dollar spent. Accordingly, this is usually the main factor we use in deciding whether or not to include a charity in our round of exploratory reviews. We prioritize reviewing charities working in areas where large numbers of animals are involved and which receive relatively little attention and funding. In practice, this means that we review many charities that focus on helping farmed animals, because there are so many farmed animals compared to other types of animals that animal charities typically focus on. We also see charities that focus on anti-speciesism generally as potentially very effective, though in practice such charities often have a large focus on helping farmed animals for reasons similar to our own. Finally, we are interested in reviewing charities that focus on reducing animal suffering in the wild, but have found few such charities (as opposed to charities focused on species or habitat preservation, which are plentiful but often focus on specific species at the expense of others, or on caring for individual animals in rehabilitation centers).
- Methods. The general methods a charity uses also have a significant impact on its cost-effectiveness. In particular, charities which rescue, rehabilitate, or provide sanctuary for individual animals are likely to spend much more per animal helped than charities that focus on creating widespread change through more leveraged means. Accordingly, we prioritize reviewing charities that focus on creating change through institutions such as governments and corporations, public outreach, and technological advancement. We are less likely to review charities that devote a large proportion of their resources to caring for individual animals, which is relatively costly compared to educational efforts—even when done as efficiently as possible.
- Ease or difficulty of evaluation. Because of our limited resources, we sometimes find charities that we would like to review based on their focus and methods, but that we are not yet able to devote sufficient resources to review fairly. This can be the case for charities which operate only in languages other than English, charities whose websites are particularly uninformative or appear outdated, or charities whose methods seem promising but unusual and would require significant additional time and research to understand. We have also used other factors under this general heading: for instance, for the round of evaluations which concluded in May 2014, we did not evaluate charities that did not work in the United States, because we did not have the resources to fairly evaluate such charities at that time. We do now evaluate some such charities. Similarly, we would expect to evaluate in the future some charities that we have so far delayed evaluating due to other factors that make evaluation difficult.
- General likelihood of receiving a recommendation. We also consider other factors that affect whether an charity is likely to receive a recommendation from us as a result of our review, if we know about them. For instance, because we do not condone violence as a means of helping animals, we are not likely to review a charity that advocates violence. If we have previously written an exploratory review for a charity, we are not likely to write another unless either some of our thinking on relevant issues has changed or we have reason to believe that the charity has changed in a relevant way. We are also less likely to review local charities than national or international charities; while a local charity could be highly effective and we may recommend them, we think that efficiencies of scale function to make national and international charities slightly more effective on average and therefore we prioritize reviewing them.
We do not have a formal process we always follow when selecting charities to include in a round of exploratory reviews. Typically we start from the list of charities that have been given basic consideration at any time during our history, and compose a shorter list using the criteria of focus area and method. We then restrict the list further using other criteria until we are in range of our target number. Typically, we apply slightly different criteria in each iteration of our process in order to evaluate as many new and promising charities as possible; for example, in one round (May 2014) we did not evaluate any charities working mostly outside the United States, but did evaluate local charities within the United States. In the next round (December 2014) we evaluated charities working in any country, but evaluated only national or international charities. We revisit each charity we have previously reviewed at least every third year, to account for changes in the charity over time.
What do we do to research each charity during this round?
Multiple ACE staff members visit each charity’s website for longer than during our basic consideration round. Our site visits are guided by a worksheet based on our charity evaluation criteria. Each person involved in the evaluation process is assigned at least one criterion to research for every charity in the exploratory review round, and visits the sections of the website they need to fill out the corresponding section of the worksheet, as well as to gain a general sense of the charity.
We also look up basic financial information regarding each charity, such as revenues, expenditures, and assets for recent years. This information is not always available on charities’ websites, but registered charities and nonprofits in some countries have to complete standardized forms which are available either through a government agency or through other online services. If we cannot find this information for a particular charity, we contact the charity to request it before writing an exploratory review, because we find that knowing the approximate size of a charity’s budget and reserves is crucial to our ability to write a fair report.
The exact worksheet we use changes from round to round as we learn more about the types of information we can reasonably expect to be available on charities’ websites and online in general. Here is a version we used in one round.
After collecting publicly available information about the charities we are reviewing, we contact a representative of each charity and ask to schedule a call to further inform our review. While not a required stage in the process, when possible these calls help us understand the charity’s outlook and priorities.
Once all information has been collected, staff members involved in the recommendation process read over each other’s notes and we discuss our overall opinions of the charities we have reviewed.
What do we publish as a result of this round?
For most charities, we write a brief review of the charity as a result of the exploratory review, consisting of a brief summary of their activities followed by a longer section describing what we found to be the most salient features of the charity, both positive and negative. We send this review to the charity for approval, often triggering small amounts of additional research as charity representatives identify mistakes we have made or provide additional information about their charity. We don’t publish supplementary materials along with our exploratory reviews, so we generally make updates to our reviews at this stage only if we believe they are reasonable based on publicly available information. The most common changes we make are changes in the emphasis of the summary of activities (for instance, if a charity engages in many activities and we have listed some of them, they may request we change which activities are listed) and wording changes which we do not believe substantially alter the meaning of our review.
If the charity concerned agrees, we publish the exploratory review (with any agreed-upon revisions) on our website and list the charity as “Reviewed.” If the charity does not reply to our contact after repeated attempts, requests that we not publish the review, or at any point stops communicating with ACE without having agreed to the review being published, we list the charity on our site as “Declined to be Reviewed/Published”, but do not publish the review. Occasionally, a charity specifically requests not to be listed on our site, and in that case we do not list them. We note at the top of our list of charities the number of such charities.
After researching the charities involved in this round, but before writing the reviews, we select a small number of charities to be part of the comprehensive review round. We do not write exploratory reviews for these charities unless they choose not to participate in the comprehensive review process.
This is the third round of our evaluation process. It is intended to allow careful and detailed reflection on the most promising charities so that we can select Top and Standout Charities.
Which charities are considered?
At the start of the recommendation process, we set a target number of charities to investigate in this round, based on our available resources. We always include the Top Charities from the previous review process. We don’t always include Standout Charities, since this could lead to difficulty finding time to evaluate new charities as the number of Standout Charities grows. We include other charities we have previously included in comprehensive review rounds, or that we researched during the exploratory review round, based on how likely we think we would be to recommend them. We revisit each charity we have previously reviewed at least every third year, to account for changes in the charity over time.
At this point we are dealing with a relatively small number of charities, and our decisions about which charities to include in the comprehensive review round are highly specific to the individual charities—many of which we already know a great deal about (particularly if they have previously participated in a comprehensive review). Some of the factors we consider are:
- How cost-effective do we expect the charity’s programs to be? Every charity is researched at the exploratory review level before being researched at the comprehensive review level, so we have some idea what their programs are and how efficiently they are conducted. We only include charities in the comprehensive review round if we think there is a substantial chance their programs have cost-effectiveness similar to those of our current Top Charities.
- Does the charity seem to collect and use appropriate evidence to guide its programs? Because animal advocacy and related fields have not been studied extensively, it is difficult to understand what works best. For this reason it is particularly valuable for advocates to carefully and thoughtfully collect data that allows them to assess which of their programs are most effective. While a focus on evidence and data collection is not a requirement, signs that a charity takes an active approach to monitoring its own programs make us more likely to include that charity in a round of comprehensive reviews.
- How much would we learn from conducting the review? We will learn more from reviewing a charity the first time than from reviewing a charity for the second time, unless the repeat charity has undergone serious changes recently or we have changed how we think about something relevant to their work. We will also learn more from reviewing a charity whose programs are very different from the programs of other charities we have included in comprehensive review rounds than from reviewing one whose programs are very similar to programs we have already investigated in detail. A review in which we learn a lot is better for us than one in which we learn less, and it is better for the charity concerned because there is a larger chance that we will discover new information that causes us to believe the charity in question is exceptionally effective and deserves a top recommendation.
We ask charities to participate actively in our comprehensive review process, so we also use these factors to select some backup charities in case one of the charities we want to include in this round does not wish to participate.
What do we do to research each charity during this round?
Once we have decided we would like to conduct a comprehensive review of a charity, we contact that charity and set up a time to talk with a high-level staff member of the charity, unless we already did so while conducting an exploratory review of the charity. During the conversation, we explain our review process in more detail and ask questions about the charity. We have several questions addressing our charity evaluation criteria which we ask each charity; we also ask questions tailored to the individual charity that we prepare in advance or that come up during the conversation.
From each charity, we request an approximate programmatic budget—detailing the programs funds are directed to, rather than accounting categories such as salaries or rent—as well as any other materials the charity has. This supplements their responses to our questions from the conversation.
We write summaries of our conversations, read the materials submitted to us and other materials published by or about the charity we are reviewing, and discuss each charity on several occasions. Sometimes we return to our contact with more questions or to request to have another conversation with someone else in the charity. We also conduct two confidential conversations with non-leadership staff in order to more fully understand the charity’s culture and work environment. While we write internal summaries of these conversations, we do not seek to publish them or show them to the charity we’re evaluating. Often we write most of our review of a charity before deciding whether to recognize it as a Top or Standout Charity or as neither.
For charities we have previously conducted comprehensive reviews for, the process is very similar to the process for new charities. The questions we ask in our conversations tend to be more closely tailored to the charity, and if the previous review was recent, the charity may not update certain information, such as their budget.
What do we publish as a result of this round?
We write a detailed review for each new charity that participates in our comprehensive review round, and update our previous review for charities we have written comprehensive reviews for already. The review includes a general summary section followed by sections explaining how well we think the charity fits each of our charity evaluation criteria. For Top and Standout Charities, we also include a section addressing criticisms or questions that we think are particularly important for understanding the charity’s impact. Because we use information in our comprehensive review round that may not already be publicly available, we also write summaries of the conversations we had with charity representatives and produce spreadsheet versions of our cost-effectiveness calculations. We send these, our review, and other supporting materials the charity provided to our contact at the charity for approval prior to publishing.
Because these reviews rely on information that may be confidential, we sometimes make substantive changes to our reviews as a result of feedback from the charity concerned, in order to protect private information. We also make changes similar to those we make on exploratory reviews—such as correcting errors or altering wording or emphasis—without affecting the substance of the review. Ultimately the conversation summary usually represents the charity’s views (because it focuses on the views of the person we spoke with), but our review represents our views.
If the charity concerned agrees, we publish the comprehensive review and approved supporting materials (with any agreed-upon revisions) on our website and list the charity as “Reviewed.” If the charity does not agree to the review being published, we list the charity on our site as “Declined to be Reviewed/Published” and don’t publish the review or supporting materials (unless the supporting materials are approved separately).
After the research on the comprehensive reviews is finished, but before the writing is finished, the research team solicits feedback from the board evaluations sub-committee. After that, and still before writing is finished, our executive director and the research staff each individually decide whether each of the charities for which we’ve conducted a comprehensive review should be a Top Charity, a Standout Charity, or neither—or if we are not certain. We compare lists and discuss each charity. In some cases we are very certain and agree with each other; in others we disagree, or some of us are not sure what status was appropriate for a particular charity.
After an extensive conversation on the subject, we reach a decision about each charity. We continue to talk over the next few days however, to make sure we are all comfortable with the decisions. We’re limited in how much detail we can provide about this decision process, since most of it has to do with specific aspects of individual charities. We know how we plan to categorize each charity before we reach out to any of them about publication of their review.
As of 2016, we have 3 Top Charities and 10 Standout charities. We don’t place a cap on the number of Standout Charities. We want the number of Top Charities to remain small, so that our recommendations provide a clear call to action. We don’t expect the number of Top Charities to grow much in the future.
We notify charities that they will be a Top Charity, Standout Charity, or that they will be neither, when we ask them to approve the materials we want to publish as a result of their participation in our round of comprehensive reviews. We ask Top Charities to work with us to set up a system for tracking donations directed to them through ACE’s review so that we can understand the impact of our recommendations. This includes either including a check box on their donate page where donors can acknowledge ACE’s influence on their donation, or tracking unique URLs resulting from ACE links to the charity’s website. We also publish shortened versions of our reviews for Top Charities on our site, linking to the longer versions. This allows visitors to our site to receive a brief, clear summary of why we have recommended a charity, and to choose whether to proceed to reading our full evaluation, so that the amount of information presented at one time is not overwhelming.