How Should We Account for the Potential Long-term Impact of Organizations?

This is the second post of our Foundational Questions series. You can also learn more about this project and read the initial post Foundational Questions In Effective Animal Advocacy.

This year we evaluated two organizations, New Harvest (NH) and the Nonhuman Rights Project (NhRP), who focus primarily on medium- to long-term change, which we roughly define as impact that is more than five years away in expectation or involving a relatively large number of intermediate steps. NH promotes the development and adoption of cellular agriculture, such as cultured meat. The NhRP works to establish fundamental rights for at least some nonhuman animals. We added both groups to our Standout Charity list for their outstanding work.

Many other organizations we evaluate also consider long-term impact in their strategy, even those focused primarily on near-term outcomes like corporate policy change or individual dietary change. This long-term change often involves indirect outcomes associated with short-term change, such as the potential for a corporate policy change at one company to inspire other companies to make similar changes.

We conducted back-of-the-envelope cost-effectiveness estimates this year for NH and the NhRP. Unlike our other estimates so far, these evaluated medium and long-term expected impacts, such as the possibility that NH’s work will lead to a cellular agriculture company that creates a product that replaces a significant portion of animal products such as egg whites. They did not account for short-term impacts because we expected these to be relatively small and still highly uncertain, such as the possibility of the NhRP moving a few individuals to sanctuaries.

We did not include these estimates in our reports or recommendation decision-making. In this post, we expand on our reasoning behind that decision and consider the broader question of how to incorporate long-term considerations into our review process.

Why we didn’t include these estimates

Including these estimates without adding them for every organization would be unfair.

One of the most important reasons for this was that we felt it would be unfair to other organizations for whom we did not account for similar impacts. We want to be as consistent as possible in our reviews to avoid a bias towards any particular type of organization. We could publish quantitative estimates of the expected long-term impact of every organization we review at a medium level, but that would require a significant amount of our resources and, as described below, we’re not sure the estimates would be that useful.

Long-term estimates could be biased upward relative to short-term estimates, making them incommensurable.

These long-term estimates usually involve lower probabilities of larger outcomes, such as the likelihood that the actions of one organization become the deciding factor in the complete transition of society to animal-friendly food sources.

Given the prevalence of large numbers in animal advocacy, such as the number of animals affected by a new corporate policy, we think advocates and Researchers are comfortable assigning large values to potential impact. However, we believe they are usually uncomfortable assigning very low probabilities because it feels overconfident to say these sorts of outcomes are less than, say, 0.01% likely, even though that may very well be the case. This could lead to overestimating the expected value of long-term change.

Additionally, we have a huge amount of uncertainty in such estimates, to the point where we would feel the need to discount significantly based on the lack of robust evidence. We try to avoid such subjective discounting in our quantitative estimates, as explained below.

We’re not yet comfortable claiming any differences in expected long-term impact between the most promising animal charities, especially from a quantitative perspective.

Given the possibility of humanity to continue to expand—increasing the number of sentient beings in the universe—or at least maintain its size for a very long time, it seems that the expected value of the number of animals in the future far exceeds the current number of animals in the world. Therefore, if we believe we can have significant impact on long-term outcomes, the majority of expected impact from animal charities likely occurs through long-term change.

However, even if the majority of expected impact is in long-term change, we might still want to focus on short-term change because of the difficulty in differentiating between the long-term impact of different organizations. This is true even though we believe that animal advocacy interventions in general have very promising long-term impact.

At this time, we have such a poor understanding of the differences between long-term effects of various strategies within effective animal advocacy that we are unable to clearly differentiate between the expected long-term impacts of the most promising animal advocacy organizations.

As we conduct further research, we might improve our understanding, but our conclusions will probably be qualitative in nature so likely better captured through the qualitative portions of our reviews. For example, as we develop a better model of social change from examining historical social movements and other streams of evidence, we could find that certain tactics fit much better into that model than others. We could convert that qualitative conclusion to a quantitative one, but we think that would introduce unnecessary uncertainty and potential bias.

We worry that quantitative estimates are interpreted with more certainty and weight than we intend, and this seems especially concerning for long-term impact estimates.

When discussing the complex and nuanced conclusions of our reviews, it’s often appealing (for both us and our audience) to use bare cost-effectiveness estimates (e.g. “This charity spares this many animals per dollar!”) to present those conclusions in a quick and easily understood fashion. We worry that this presentation can make our estimates seem more conclusive than they actually are. This is something we frequently struggle with as an organization, even when presenting our conclusions on relatively well-understood interventions like corporate outreach.

We think this challenge would be even greater if we extended the scope of our quantitative estimates to include medium- and long-term outcomes because the level of nuance and qualification necessary to appreciate the quantitative results would be so great. This would also make it more difficult for us to use the estimates internally when weighing them against other considerations because we would need to keep track of those qualifications.

We also worry about assigning numbers to subjective judgment calls, and this would require us to do that even more.

Similarly to the previous consideration, we worry that when we assign numbers (or even ranges, although less so) to factors that have little definitive evidence, it is easily misread as more certain and objective than it actually is.

For example, we try to quantitatively compare the effectiveness of veg outreach, which seeks to reduce the total number of farmed animals suffering, to corporate reform outreach, which seeks to improve the living conditions of those animals. In order to do this, we need to assign a number for the relative impact of an improvement in living condition for an individual animal to sparing that animal entirely.

For cage-free reforms that cause egg-laying hens to live in a cage-free facility rather than battery cages, we currently believe that making this improvement for one individual is 10% as valuable as entirely sparing an animal from all the harm of animal agriculture (e.g. chronic health issues, worker mistreatment, slaughter). There is much uncertainty here. ACE staff members hold different views. Our views change over time. We have received feedback from people who disagree with the estimate in both directions. Overall we still think it’s useful to include such estimates, and think this feedback and disagreement is expected and appropriate.

We find it difficult to qualify this figure sufficiently in our reports, such that each reader appreciates how subjective and uncertain it is. Quantitatively estimating long-term change would involve many more such figures, which would make it even more challenging.

Note that we are strongly considering the possibility of using more ranges (e.g. 5-50%) in place of point estimates (e.g. 10%) in our cost-effectiveness estimates because we now have more research staff than we did previously. We think this will help, although the challenge will still exist.

Some tentative considerations for differentiating long-term impacts

We are constantly working to improve our review process, and it’s plausible that we could include quantitative estimates of long-term impact at some point. We hope to improve our understanding of long-term impact and use that to improve our overall evaluations of charities and interventions.

Veg outreach

A major uncertainty in this area is the long-term impact of veg outreach such as leafleting, online ads, and social media focused on individual dietary change. While this work is usually focused more on short-term change than some other activities such as advocating for the legal personhood of animals, it could still facilitate long-term social change in animal attitudes that leads society to make better decisions regarding nonhuman animals in the future. For example, a more veg-friendly society could be more likely to adopt cultured and plant-based meat as better technologies arrive.

One consideration for the long-term impact of veg outreach is that there are also plausible mechanisms by which it can be negative or approximately zero. For example, if veg outreach strengthens the association of animal advocacy with individual dietary change, this could make it more difficult to create large-scale change that involves viewing the suffering of animals as an issue that we all need to address. It could also increase the prevalence and perceived strength of counter-arguments to dietary change such as “It’s my right to choose what I eat,” or “Great that you’re vegan, but that’s a personal choice.”

Humane reforms

Similar concerns could apply to the long-term impact of humane reforms such as banning battery cages. It seems plausible that these reforms could make society more complacent with their consumption of animal products. Of course, it’s also possible that humane reforms increase the likelihood of future progress by setting precedent for policy changes that benefit farmed animals. We think, based on existing reforms and reactions, that the precedent-setting consideration outweighs the complacency consideration, meaning the expected long-term impacts are positive, but we are quite uncertain.

Short-term change as a proxy for long-term change

It seems plausible that focusing on short-term social change is one of the best strategies for creating long-term social change. Although this might not be true in the very-short-term, such as when an animal advocacy protest makes its audience uncomfortable, it would make sense if change on the scale of decades were well-correlated with change on the scale of centuries. This could mean that we can use short-term social outcomes, such as survey results or press coverage, as evidence for long-term social impact, which is an important part of all possible long-term impacts.

Things that could change our mind and areas of future research

We think more research into historical social movements, sociology, and other forms of macro-evidence could improve our understanding of long-term change. We are planning to continue conducting research into these areas, such as expanding our Research Library and continuing our Social Movements Project. Once we have collected more evidence, we hope to aggregate that evidence and make action-guiding conclusions about the most effective ways to help animals.