Our Approach to Assessing Cost Effectiveness
Introduction
Cost Effectiveness is one of our four charity evaluation criteria. A charity’s recent cost effectiveness provides an insight into their overall impact on animals given the resources used to achieve their results and is a useful component in understanding how cost effective future donations to the charity might be. While the Programs criterion assesses the expected impact of a charity’s programs, the Cost Effectiveness criterion assesses the effectiveness of a charity’s approaches to implementing interventions, their recent achievements, and the costs associated with those achievements. A charity that performs well on this criterion likely utilizes their available resources in a cost-effective manner.
Our 2022 Cost-Effectiveness Assessment
Gathering Information
As part of our charity information request, we asked charities for a list and description of the main achievements of each of their programs.1 We also asked charities to report their expenditures for each program and estimate the percentage of program expenditures they spent on each key achievement. We estimated total program expenditures by taking the charity’s reported expenditures for each program and adding a proportion of the charity’s nonprogram expenditures weighted by the program’s size. This process allowed us to incorporate general organizational running costs into our consideration of cost effectiveness.
Selecting, Verifying, and Categorizing Achievements
We limited the number of achievements to factor into each charity’s cost-effectiveness assessment to up to five key achievements per program. When selecting which achievements to include in our assessment, we prioritized those we thought were most representative of their respective programs and that referred to completed work (rather than work in progress).
Within each charity’s three largest programs, we verified one key achievement. We aimed to find at least one source to verify the achievement using publicly available information, internal documents, media reports, and independent sources. When we could not find a verifying source, we asked the charities to provide evidence for the achievement. This year, we were able to fully verify most achievements that were selected for verification; in two cases, we were only able to partially verify them.2
For each key achievement, we identified the associated outcome and intervention type and assigned the respective intervention score.3 The intervention score represents the intervention’s priority level according to the ACE research team, which is scored on a 1–5 scale and also considered in the Programs criterion.
For example, if a charity reported achieving seven corporate cage-free commitments, that would correspond to the outcome improved welfare standards and the intervention type corporate outreach. Because corporate outreach has an intervention score of 3.5, we would assign the achievement an intervention score of 3.5.
Scoring Achievements
Based on a charity’s reported expenditure on each achievement, we computed how many such achievements the charity accomplished per $100,000. We standardized to achievements per $100,000 to allow for easier comparisons across achievements.
For example, if a charity spent $85,000 on achieving seven cage-free commitments, we estimated that the charity achieved 8.2 cage-free commitments per $100,000.
For each key achievement, we then assigned an achievement score ranging from 1–4, with lower scores indicating that the intervention was implemented as expected and higher scores indicating that the intervention was implemented exceptionally well. This scale was designed to help identify achievements that were particularly cost effective. Three researchers rated each achievement based on how cost effectively the intervention was implemented (number of achievements per $100k) and other contextual information, such as the species affected.4 The achievement score was computed as the average of the three researchers’ scores.
For example, when scoring corporate outreach achievements, we took into account not only how many commitments were achieved per $100k, but also the priority level of the animal group the commitments affect and the scope of the commitments (e.g., how many locations are affected).
Adjusting Intervention Scores
Next, we determined how to adjust intervention scores relative to achievement scores. We assumed that low-scoring achievements (achievement score = 1) implemented an intervention as expected, whereas higher-scoring achievements (achievement score > 1) implemented it better than expected. Therefore, we adjusted the intervention score up for those achievements scoring > 1.
To calculate the extent to increase the intervention score for each achievement, we first determined the maximum possible increase for all intervention categories. We did this by selecting one achievement with a low achievement score and another with a high achievement score for each intervention type. We then rated how much higher we estimated the scale (short term and long term) of animal suffering prevented to be for the high-scoring vs. the low-scoring achievement. By subtracting the lower score from the higher score, we determined the maximum score increase for that intervention type.5
For example, in the corporate outreach category, we compared 121,951 petition signatures as a low-scoring achievement to 41.5 cage-free commitments as a high-scoring achievement. On average, the three researchers rated the scale (of suffering prevented) of the low-scoring achievement as 1.51 and the high-scoring achievement as 3.31. The difference between these two values corresponds to a maximum score increase of 1.79 for the corporate outreach intervention category.
We scaled the adjustments to the intervention scores for each intervention type from +0 (achievement score = 1) to +“max increase” (achievement score = 4):6
- Achievement score = 1: +(0/3)*“max increase”
- Achievement score = 2: +(1/3)*“max increase”
- Achievement score = 3: +(2/3)*“max increase”
- Achievement score = 4: +(3/3)*“max increase”
Computing Program Scores
The raw program score is a weighted average of the adjusted intervention scores for each key achievement within a given program. The contribution of each achievement to the program score was based on the proportion of program funds spent on the achievement relative to the program expenditures spent on all selected achievements.
Computing Final Cost-Effectiveness Scores
The final cost-effectiveness score for a charity is the average of the raw program scores for the charity, weighted based on the program fraction of their total expenditures. This way, larger programs have a higher impact on the final score than smaller programs. The final score was assigned on a 1–5 scale and acts as a proxy for a charity’s estimated cost-effectiveness:7
- 1 = Very low cost effectiveness
- 2 = Low cost effectiveness
- 3 = Moderate cost effectiveness
- 4 = High cost effectiveness
- 5 = Very high cost effectiveness
Changes from Previous Years
From 2014–2018, we used quantitative cost-effectiveness models that compared charities’ outcomes to the expenditures for each of their programs and attempted to estimate the number of animals spared per dollar spent. After considering the limitations of such purely quantitative cost-effectiveness models, we transitioned in 2019 to a qualitative approach that analyzed the resources used and outputs achieved for each intervention type.
In 2021, we analyzed charities’ use of resources over the past 18 months and compared that tp the results of each of their main programs during that time. The goal was to understand if charities had successfully implemented their programs and whether their past successes were achieved at a reasonable cost. Our overall cost-effectiveness assessment was a binary assessment of whether there were any causes for concern.
In 2022, we sought to improve our cost-effectiveness assessment in three main ways. First, we wanted to distinguish more clearly between the Programs and Cost-Effectiveness criteria and reassess how they should best work together to contribute to a charity’s overall evaluation. We now assess the overall expected impact of a charity’s programs (considering animal groups, countries, outcomes, and interventions) in the Programs criterion. We then use the intervention score to further analyze a charity’s cost effectiveness.
Second, we aimed to provide a more fine-grained cost-effectiveness estimate rather than a binary assessment of whether we thought there was any cause for concern. To this end, we analyzed a charity’s key achievements in each of their programs and compared how cost-effectively different charities implemented the interventions they used. The goal was to assign each charity a cost-effectiveness score that could be used to rank charities relative to each other. In line with these changes, we updated our definition of cost effectiveness from “how well the charity has made use of its available resources” to “a charity’s overall impact on animals given the resources they used to achieve their results.”
Finally, we aimed for as much transparency about our process as possible so that donors who disagree with parts of our model can still make use of our intermediary estimates. To this end, we publish the complete list of achievements provided by charities and the cost-effectiveness model for each charity alongside their review.
Limitations and Planned Improvements
Assessing cost effectiveness by evaluating a charity’s key achievements comes with limitations. Key achievements are not fully representative of charities’ work; rather, they are a selection of achievements that we thought were important and most suitable for assessment using our current methods. This method will likely bias cost-effectiveness estimates upward to some extent, as it does not consider expenditure on less impactful achievements or work that did not result in an achievement. This may affect larger programs more, as their key achievements are more likely to account for a smaller proportion of the overall program costs and thus may be less representative of the program’s overall cost effectiveness. In future evaluations, we may take into account program expenditures that did not lead to any achievements.
Furthermore, because our intervention scores currently factor into both the Programs and Cost-Effectiveness criteria, interventions are given more weight than other factors (e.g., animal groups, countries, and outcomes). Our team took this imbalance into account during our 2022 evaluation process. In the future, we aim to carefully consider the goals of the Programs and Cost-Effectiveness criteria and try to avoid overlap as much as possible.
There are parts of our cost-effectiveness assessment that we are less confident in than others, such as our process of adjusting intervention scores based on achievement scores. If we continue to adjust intervention scores in future evaluation cycles, we will carefully consider and plan how to improve our method.
Finally, our 2022 cost-effectiveness assessment aims to strike a balance between our fully quantitative models (up until 2018) and our qualitative assessments (from 2019 onwards). This year, our quantitative scores form a proxy for actual cost effectiveness, rather than estimating the number of animals spared per dollar spent. For next year, we are considering moving further toward a quantitative approach that analyzes the level of suffering prevented given the resources used, while also quantifying our level of uncertainty in our estimates. Importantly, we will carefully review the reasoning for moving away from fully quantitative models in the past to avoid reinstating the same limitations.
We receive information from charities participating in the evaluation process midway through the year. To gather enough data to gain an accurate insight into the cost effectiveness of their work, we ask for the accomplishments they achieved in the previous year and the first six months of the review year (a total of the 18 months).
We followed our verification process from 2021, but we only verified claims from each charity’s three largest programs. We included a total of 35 claims in our 2022 verification process.
Intervention scores were assigned by taking the average of five researchers’ scores. Each researcher rated the scale (short term and long term), tractability, and neglectedness of each intervention type.
A small number of achievements were rated by only two researchers because we received the information from the charity after one team member went on leave. For each intervention category, the research team created a list of achievement factors to estimate how well a charity is implementing an intervention in terms of impact on animals and resources used. These served as guidance while scoring achievements. An example of an achievement factor for corporate outreach achievements is: “If implemented, how much will welfare improve?”
Intervention scores were determined based on an intervention’s scale (short term and long term), tractability, and neglectedness, with each researcher giving these factors a unique weighting. We used the average weights given to short- and long-term scale by the researchers to weight our ratings of low-scoring and high-scoring achievements when determining maximum increases. The tab “Assessing intervention score adjustments” in our 2022 cost-effectiveness assessment spreadsheets shows how maximum increases were calculated for each intervention type.
The formula in column P of the 2022 Cost-Effectiveness Assessment spreadsheet scales the maximum increases depending on the given achievement score and intervention type. For an example, see The Good Food Institute’s 2022 Cost-Effectiveness Assessment spreadsheet.
We use ratings of low, moderate, and high to help distinguish between the performance of charities that we review and make our numerical scores easier to interpret. These qualitative ratings are not reflective of a charity’s performance when compared to other charities that were not selected for review.
