Responses to Common Critiques

As we continue steadily growing our organization, we are constantly trying to learn and improve. We find it useful to engage with critiques from others, and we share some of our responses and thoughts on feedback we’ve received below. We’ve selected critiques that we feel are most important, and most often presented to us.

• Our Current Leafleting Report and Using Older, Less Rigorous Data
• The Usefulness of Cost-effectiveness Calculations
• Recommendations of GFI and New Harvest
• Compensation for Evaluations
• Lack of Diversity in Our Recommendations
• Diverting Resources from Other Groups
• Regarding Animal Equality
• Conflicts of Interest

Our Current Leafleting Report and Using Older, Less Rigorous Data

Our current leafleting report was written in 2014 and does not accurately reflect our current views or more recent developments in research about animal advocacy. The change is substantial because ACE was a very young organization in 2014, and our team and views have evolved significantly since then, as has thinking about effective animal advocacy more broadly. Our confidence in the findings of some of the studies has declined, and we now think the report is too positive about leafleting in comparison to other animal advocacy interventions in general. The language we used in the report is also generally less cautious than the language we would use now.

The way we treat leafleting programs in current charity evaluations, while informed by what we learned in preparing the 2014 report, also takes into account newer evidence and other updates to our views. While we use cost-effectiveness numbers generated during this report, these form a small part of our recommendation process (see below), and our overall thinking has evolved considerably, from initially favoring interventions focused on individual diet change—such as leafleting—to a greater optimism for interventions that produce wider systemic or social change—such as corporate outreach and undercover investigations.

While we would love to have all our published content continually updated to reflect the latest state of our views and of research on animal advocacy in general, we have limited resources and often need to choose between updating old content and researching areas we haven’t previously written about. In 2016 we chose not to update our leafleting report because of these trade-offs. We feel that even though it is outdated, the current page still has some value, both for the information it contains and as a reflection of ACE’s past thinking. We do plan to update this page in 2017. We’re also beginning a project that will map dependencies between pages on our site, which we hope will help us better identify which research content it is most crucial to update in response to future changes in our thinking or in animal advocacy more generally.

We have plans to add a note at the top of pages that contain older content informing readers of that fact, and have made that change to the leafleting report.

The Usefulness of Cost-effectiveness Calculations

ACE’s cost-effectiveness estimates, both in our charity evaluations and for individual interventions, are one of the most discussed and most criticized aspects of our work. We publish estimates that account for short-term effects of programs–based on a combination of outcomes reported by charities, evidence collected through studies, and our informed estimates about relevant quantities. Because these estimates are incomplete and based on information of varying reliability, and because we believe other criteria that are less suitable for quantitative calculations are also important to evaluating cost-effectiveness, the estimates make up only a small component of our overall evaluations of charities and interventions. With particular regard to our estimates for charities, these differences are often smaller than our uncertainty.

We have certain reservations about our estimates, such as concerns about the strength of the evidence about animal advocacy in general and a concern that people may take certain estimates as being more definitive than we intend. However, we do also find these estimates to be a useful component of our attempts to compare the wide variety of programs that animal advocacy organizations engage in. We’ve written more about the benefits and challenges of providing cost-effectiveness calculations in a previous blog post. We did implement the major changes described there for our 2016 reviews, including providing estimates as ranges rather than single values to better indicate our uncertainty, and providing estimates of the duration of suffering prevented—which we felt made our estimates more meaningful as a comparison between interventions with varying types of effects on animals.

In some cases, criticism of our cost-effectiveness calculations reflects their usefulness in bringing clarity to a discussion. By making clear what effects we expect different interventions to have through actually assigning numbers to uncertain values, we allow people to understand our reasoning well enough to identify the specific points on which they disagree. If our evaluations were entirely qualitative in nature, it would be less likely for people who strongly disagree with us about the effectiveness of certain programs to pinpoint their disagreements, since our qualitative statements are more open to interpretation than our quantitative ones. Since our evaluative work (and, we think, any similar efforts at charity evaluation) involves significant assumptions and value judgements, it’s important for us to provide our reasoning transparently so that people can understand whether they feel sufficiently aligned with our thinking to want to take our recommendations.

For example, there is a general tendency to overestimate the effectiveness of interventions, probably both when a quantitative estimate is made and when one is not (though in the latter case it is difficult in retrospect to identify the implicit estimate that was made). People who want to correct for this more strongly than we do will have an easier time adjusting our quantitative estimates according to their own assessments than they will have adjusting our qualitative reasoning.

Another example is with estimates that are based on data which is not highly reliable, such as those for leafleting or online ads. This is because there is no available data which would be highly reliable. While of course we would prefer to have more reliable estimates, given the available information, the best we can do is to make clear what information we’re using so that others can adjust it if they feel it’s necessary. Furthermore, even these estimates provide a check of some kind on our immediate intuitions about the effectiveness of a particular program. Despite the inherent challenges of publishing cost-effectiveness calculations, not doing so would certainly limit the usefulness of our reviews, because of the additional clarity that quantitative communication can provide.

As mentioned above, we began addressing some of our concerns this year in our 2016 reviews. The primary way in which these adjustments manifested was in our decision to provide quantitative estimates as ranges rather than single values to better indicate our uncertainty. We think this is a major improvement in the way we communicate about cost-effectiveness, and we plan to gradually implement it for other cost-effectiveness calculations we provide.

Recommendations of GFI and New Harvest

We have written a separate post addressing questions on our recommendations of these charities.

Compensation for Evaluations

We have seen public assertions that ACE demands that charities pay for our evaluative services, and claims that ACE gives bad reviews to organizations that refuse to do this.

We have never, and will never, request or accept any form of compensation for our evaluations of animal charities. It is very important for us to remain independent, third-party evaluators (read more about this in our conflict of interest section below).

We allow charities to choose whether or not we publish their review. We feel this enables us to more accurately locate the most effective animal charities, as it encourages charities to be forthcoming and honest with us about their work and operations. We have never threatened a charity with a bad review, put any kind of pressure on a charity to allow publication, or demanded compensation for a better review.

Our mission is to find and promote the most effective ways to help animals. It is not our intention to identify lower quality organizations that likely would not receive a recommendation, as we do not think that would be useful in the pursuit of our mission. Highlighting organizations that we think are doing poor work could significantly inhibit our ability to locate the very best organizations, as the potential for a bad review could make it more likely that charities are unwilling to be open and transparent in their communication—thus limiting the amount of information we have available for our evaluation process. It is therefore actually in ACE’s best interest to not seek publication of “bad reviews” in the first place.

We occasionally make edits to our original reviews, as requested by the reviewee, in order to reach an agreeable place where the charity allows us to publish our review. This usually involves correcting small factual errors, revising wording, or clarifying our position on a program where we did not originally draft our position clearly. We never make significant edits in a way that compromises or risks miscommunicating our impression of the organization.

We feel that our current system of finding top animal charities makes sense, and that criticisms in this area likely result from confusion about our process. We will attempt to be more clear about our communications in this area in the future, but we do not intend to make any modifications to this part of our evaluation process, as we think it works as intended.

Lack of Diversity in Our Recommendations

It is true that the leadership of our recommended charities do not feature as much diversity as we would like, in terms of gender or race. We believe that this is an issue that affects ACE as well as the broader animal advocacy movement. By selecting charities from the broader movement, our recommendations will inevitably reflect some of these deficiencies. Ultimately, our judgment this year was that the current overall best charities unfortunately do not yet have the kind of diversity that we think we should all be aspiring to as a movement.

Of our current recommended Standout Charities, Animal Equality, Animals Australia, New Harvest, and Animal Ethics all have female leaders in their most prominent positions. The Nonhuman Rights Project had a woman in their top position when we reviewed them, but she has since moved on to teach law. Thus, of our 13 recommended charities, only five of them have or had women in top leadership positions. None of our Top Charities have female leaders.

We agree a lack of diversity is an overall problem in the movement, and intend to take a series of steps in the coming months to improve our own diversity. We will write about our progress on this front to bring more attention to this issue, and welcome suggestions on how we can do better.

We will not be able to solve this problem overnight, but we will make sincere efforts to address it so that we can both expand our movement and address patterns of racism, sexism, and other forms of oppression existing within it.

Diverting Resources from Other Groups

ACE’s mission extends far beyond simply making recommendations of Top and Standout Charities. In working to find and promote the most effective ways to help animals, we feel two of our most impactful areas involve bringing more attention to farmed animal advocacy, and providing advice and critiques that enable all animal advocacy groups to do better work. Our evaluations are one way in which we advance both of those areas; they focus on farmed animal advocacy charities, and they highlight the importance of self-assessment, transparency, and an overall drive to improve.

By only selecting some charities for recommendation, there are many other charities that are not recommended. This is in keeping with the entire concept of evaluations; inevitably, some will perform higher than others. We believe that conducting our evaluations annually to identify those groups that are doing particularly strong work will be more impactful in helping animals than simply continuing on with the status quo. When we change a recommendation it is because we believe that money would do more for animals by being directed to a different charity, and we want to inform our supporters, who are trying to help as many animals as possible, about our position.

We try to mitigate any damage we might do in this area by allowing charities to veto publication of our reviews. We also sometimes point donors to non-recommended charities that we still feel are doing good work, and that are focusing on initiatives that are important to those donors. Such conversations have resulted in over $100,000 in donations to charities that we have reviewed but that are not recommended as either a Top or Standout Charity.

It is also important to note that our recommendations sometimes bring in funding from those who did not previously give to animal charities at all; in those cases, we’re not shifting money around within animal advocacy but rather towards it. Our evaluations can give some donors more confidence in animal advocacy altogether and thus bring in new money to animal charities. We have surveyed our donors in past years in order to better understand the reasons why donors gave to our recommended charities, and so that we can better understand how to maximize our impact. Though the sample size of this previous survey was small, we again surveyed donors this year and have collected over five times as many responses. We look forward to analyzing the results of that survey, and will be writing about it on our blog in the near future.

We take great effort to provide resources to advocacy groups so that they can continue to improve regardless of their recommendation status, or even any participation in our process at all. This includes:

• Regularly making posts on our blog, including those from our annual advocacy advice month, which we archive on our site
• Regularly assisting organizations by providing answers to their inquiries about their own research projects
• Conducting interviews with advocacy leaders, academics, and industry professionals, to provide a public collection of wisdom from experienced advocates
• Providing resources on conducting surveys
• Offering grants to advocates interested in assessing the impact of their programs, and sharing that information with the broader advocacy movement

Regarding Animal Equality

Some people have expressed a concern that, in moving Animal Equality from our list of Top Charities to our list of Standout Charities, we will be taking money away from them.

It’s true that Animal Equality may receive less support from ACE donors after this change. However, we should consider the full picture in order to understand ACE’s impact overall.

Two years ago, before ACE recommended Animal Equality, they took in approximately $300,000 in revenue. This year they have taken in well over $2 million, over $1 million of which they attribute to ACE’s recommendation. In addition to these monetary gains, we directed many hundreds of donors to Animal Equality—something which helps them develop a stable base of supporters—all of whom they have had the opportunity to steward over the past two years. We’ve also spent time and money advertising Animal Equality, including through our blog posts, online advertisements, and printed materials.

By being moved to a Standout Charity, they will no longer receive most of the benefits of the previous paragraph. They will still be actively promoted as a Standout Charity, although of course we do not promote our Standout Charities quite as strongly or extensively as our Top Charities. Some of the donors who donated to Animal Equality because of ACE’s Top Charity recommendation will cease giving to them, and instead switch their donations to our current Top Charities. Animal Equality may also experience problems with meeting funding goals they set based on their most recent increase in revenue, of which a substantial portion resulted from ACE’s Top Charity recommendation.

While there are certainly some issues that might surface as a result of their change in status, we still feel that we’ve done much more good for Animal Equality than harm. We’ve also taken steps to mitigate any decrease in funding they may experience by writing a blog post telling our donors that we still think they should support them to some degree, and by instructing donors to that effect in personal conversations as well. We’ve also offered to evaluate them again next year after they’ve had a chance to resolve some of the issues we saw, which is a quicker timeline than normal for our typical reevaluation of Standout Charities. We extended this offer to them because they were especially close to our Top Charity recommendation, and given their strong track record in past years we wanted to see if time spent adjusting to their new level of income and larger staff size enables them to fix the problems that prevented us from recommending them as a Top Charity this year.

We do not intend to update our work in this area, though we do intend to continue producing more resources for all groups to use free of charge. For example, we just began work on creating a white paper project where we provide targeted advice for typical departments at animal advocacy charities.

Conflicts of Interest

We take conflicts of interest (COI) very seriously. It is a part of every discussion when considering who to hire as staff, interns, and Board Members. It is part of our decisions on whether or not to partner with an organization on a project. It is also a part of our evaluation process. We endeavor to identify every possible COI situation.

We have had a formal COI policy on our site for over a year now, and featured said policy in our by-laws for several years. In these documents, we discuss some of the considerations we think about when making decisions as an organization.

We also feature a disclosures page for both staff and Board Members of ACE, again something that has been on our site for the better part of a year. Staff are required to list any affiliations that they have with any considered charities, including of course our recommendations. The Board of Directors is required to list any affiliations they have with our recommended charities.

These issues extend to our charity evaluations, and our recommendations specifically. It’s true, for example, that GFI is a child organization of MFA, and that therefore there is a relationship between the two charities since MFA is already one of our Top Charities. It’s also true that Nick Cooney has had involvement with each of our current Top Charities at some point in time—he founded THL, he works at MFA, and he is a Board Member at GFI.

In this situation, we saw that these relationships existed before making our recommendations, something which should be apparent given the detail in our reviews. However, after internal discussions and many deliberations, which included contributions from staff and Board Members, we made the decision to recommend these three charities despite any reservations of perceived COI. We decided these were our top choices for our recommendations, and we didn’t feel that their relationships, current or past, created a significant concern which would cause us to change our opinions.

As for Nick Cooney’s involvement being an actual COI, and not a perceived COI, we do not feel that any of our staff members’ relationship with Nick Cooney was so strong as to concern us. It’s true that we regularly correspond with Nick, but not more often that we correspond with leadership from other animal advocacy groups. Nick hasn’t been involved with THL in a long time—he founded the organization, but the current Executive Director, David Coman-Hidy, joined the organization in 2010 and earned his current role in 2013—so we didn’t factor Nick’s founding of the charity into our review. Nick is on the board at GFI, but we didn’t speak to him during the review process concerning GFI. The only organization that we consulted Nick about was MFA, since that is where he is currently employed. Neither Nick’s prior involvement with THL nor his current status as a Board Member at GFI were factors in determining our recommendations of these two charities.

Another way that we attempt to avoid COI can be seen in our donation policy. We will not take in donations from key leadership at any charity we consider in our evaluations. This does not extend to every person who works at a charity, but specifies those who are in the most influential, decision-making positions. Our policy is not to accept donations from these individuals, but if they make a donation anyway, we contact the person and charity to let them know that if they want us to accept that donation, we will not be able to review that charity for a period of at least three years. We abide by this rule; we currently have one charity on that list as a charity that we cannot currently evaluate due to a prior donation, and we have turned down multiple donations this year in order to adhere to this policy.

Ultimately, we strive to be incredibly transparent—in fact, we know of no other animal advocacy organization that publishes more internal information than ACE—and to communicate our thought process every step of the way. We do this not only because we want people to trust our work, but because we want to encourage constructive critiques so that we can continue to improve as an organization. That is why we are writing this post. We want to communicate our thinking, explain our processes, address critiques, and be as honest, open, and forthcoming as possible.