ACE Highlight: Our Use of Cost-Effectiveness Estimates

Introduction

ACE employs both qualitative and quantitative strategies in our work to identify highly effective ways to help animals. One way that we evaluate interventions or charities quantitatively is by assigning numerical values to their immediate costs and benefits. We then use those values to calculate a cost-effectiveness estimate (CEE) in terms of “lives spared per dollar” or “years of suffering averted per dollar.”¹ These estimates allow us to directly compare different interventions and charities, which helps us decide which ones to recommend.

On a new page on our website, we describe our process for developing CEEs. We also describe the role that they play in our intervention reports and charity evaluations. We explain some of the challenges we face in the process of creating CEEs, as well as some of the risks and benefits of making them public. This blog post provides a summary of those risks and benefits, as well as a statement on our current thinking regarding the appropriate use of CEEs. Read the full CEE page for more details on our process.

Some Risks and Benefits of Making and Publishing Cost-Effectiveness Estimates

Risks:

1. We worry that publishing our CEEs seems to imply that we place more weight on them than we actually do. We have consistently struggled to communicate to our audience the extent to which our CEEs are approximations, highly uncertain, and bias-prone. We worry that some readers may make donation decisions based solely on our CEEs, which we do not advise.
2. We also worry that publishing CEEs for interventions makes the evidence regarding the intervention appear stronger than it actually is. For instance, publishing a CEE for a leafleting program may seem to imply that there has already been sufficient research on the effects of leafleting on diet change. In fact, there is very limited evidence available about the effects of leafleting. We make our best estimate using the evidence that’s available, but we hope that research in that area will continue.
3. Some animal advocates may feel that it’s inappropriate to assign quantitative values to outcomes that affect animals’ lives. They may worry that focusing too much on numbers will distract us from what is really important: concern for each individual animal. It’s also possible that our focus on numbers could distract our readers from concern for each individual animal, which might result in fewer donations to our recommended charities.

Benefits:

1. Because we are committed to the principles of effective altruism, one of our primary goals is to identify the most effective ways to help animals given limited resources. Even when we take into account the risks and uncertainties described above, directly estimating cost-effectiveness is still one of the best ways we know of to identify highly cost-effective programs.
2. Cost-effectiveness estimates are sometimes useful for comparing different charities or interventions to one another. We develop CEEs using consistent methodology and data so that our CEEs for similar charities are meaningfully comparable.
3. Publishing our CEEs increases our transparency. Assigning numbers to uncertain values allows us to be clear about the effects we expect an intervention to have. It also allows our readers to identify specific points on which they may disagree.

Our Current Thinking

We want to be very clear that our cost-effectiveness estimates are approximations. They involve uncertain quantitative estimates, and they are subject to bias and error. They are not the only factor we consider when we evaluate interventions and charities, and our readers should interpret them carefully.

We do not advise our readers to accept our CEEs unquestioningly. We exercise our judgment in many areas of our modeling, and our judgment is not infallible. In fact, our judgment is not necessarily better than the judgment of our readers. We do have some advantages: we have multiple team members who are able to work together, and most of us think about effective animal advocacy full-time. Our research staff regularly practices calibration exercises to help us develop better 90% subjective confidence intervals.²

We believe there is value in publishing the results of our estimates so that others can use them as they see fit. Otherwise, any of our readers who are interested in comparing the cost-effectiveness of different charities might have to develop a model on their own, likely from scratch.

We do our best to communicate an appropriate level of confidence in our CEEs. In 2016, we began presenting our CEEs as ranges rather than point estimates, which we think better expresses our level of uncertainty. We hope that this post will also serve to clarify the tentative nature of our estimates. We encourage our readers to adjust our cost-effectiveness models however they feel is appropriate, and we hope the results will help inform their decisions.

Read the in-depth writeup of our cost-effectiveness estimates

1. For an example of how we calculate the cost-effectiveness of a program, you can read about how we evaluate social media impact. For examples of how we calculate the cost-effectiveness of charities, which are often engaged in many different programs, see our 2016 cost-effectiveness models for Mercy For Animals or The Humane League. 

2. A subjective confidence interval is a range of values that communicates judgments about an unknown quantity. We construct a 90% subjective confidence interval such that we feel 90% confident that the true cost-effectiveness value falls within the range.