Top

Animal Charity Evaluators

Helping People Help Animals

Header Right

Main navigation

Evaluation Criteria

Last Updated: April, 2017

Recommendation Criteria for Animal Advocacy Charities

Before we select which organizations to recommend, it is beneficial to transparently establish what criteria we will aim to rank organizations on and how we intend to assess that criteria. Below is a summary of the recommendation criteria we use to evaluate charities.

1. The charity has room for more funding and concrete plans for growth. It has plans that cannot be fully accomplished with the expected funding from other sources, and the barriers to accomplishing those plans are monetary, not due to time, a lack of qualified personnel, or other non-monetary issues. Receiving money from donors directed by ACE would be very likely to increase the charity’s total impact.

2. The charity engages in programs that seem likely to be highly impactful. Without considering the specific implementation, the interventions pursued by the charity are among those expected to be highly effective. Often this information will come partly from our intervention evaluation process. At the very least, the charity is working in a domain with high potential for efficiency (for instance, farm animal advocacy) or is working in another domain with methods that appear to be exceptionally cost-effective.

3. The charity operates cost-effectively, according to our best estimates. Considering the charity’s budget and its demonstrated successes, it appears to be cost-effective in the goal of reducing animal suffering, relative to other charities. If these calculations were produced with perfect information, they would be the only factor necessary to consider beyond room for more funding. However, because our calculations are necessarily estimates, corroborative evidence is required.

4. The charity possesses a strong track record of success. The charity has a record of successful achievement of incremental goals or of demonstrated progress towards larger goals. Note that this implies the charity has been in existence for some length of time. While very young charities may have strong potential to return large results for small initial amounts of funding, donating to charities without track records is inherently risky.

5. The charity identifies areas of success and failure and responds appropriately. The charity has both short and long term goals and can articulate signs that indicate whether they are moving towards or away from their goals. Regular self-assessments guide the charity’s program development. If applicable, the organization has responded appropriately in the past to signs that a program was not succeeding. Furthermore, they are able to innovate effectively: they regularly try small and large changes to programs, identify which changes improve outcomes, and act on that information.

6. The charity has strong leadership and a well-developed strategic vision. The current leaders of the charity seem competent and well-respected. The charity’s overall mission puts a strong emphasis on effectively reducing suffering, and the charity responds to new evidence with that goal in mind, revisiting their strategic plan regularly to ensure they stay aligned with that mission.

7. The charity has a healthy culture and a sustainable structure. The charity is stable and sustainable under ordinary conditions, and seems likely to survive the transition should some of the current leadership move on to other projects. The charity acts responsibly to stakeholders including staff, volunteers, donors, and others in the community. In particular, staff and volunteers develop and grow as advocates due to their relationship with the charity.

Criteria for Evaluating Potential Organizations

Before Animal Charity Evaluators (ACE) selects which organizations to recommend, it is beneficial to transparently establish what criteria we will aim to rank organizations on and how we intend to assess that criteria. Below, we outline and elaborate upon seven different criteria which we plan to use starting in 2016. The version of this document used in 2014 and 2015 is available in our Process Archive.

It’s worth noting that right now how to weigh each criterion relative to the others is subjective. While we’ve worked hard to standardize our opinions, we feel the best we can expect to do is make calls on a qualitative case-by-case basis as we start evaluating organizations, instead of with explicit quantitative weights. We have learned a lot from undertaking this process over the past two years. This document reflects that new information, and we expect our criteria to continue to change as ACE gains more experience evaluating organizations.

Criterion #1: The Charity Has Room for More Funding and Concrete Plans for Growth

The first criterion we need to look at is whether the organization has room for more funding and plans for growth. When it comes to deciding where to allocate donations, it doesn’t matter how effective an organization is if they have no use for additional donations.

Why might an organization not have room for more funding?

For an organization, there can be more bottlenecks to expanding their operations than just money. Some organizations may have a lot of money available to expand programs but not have the talent they need to logistically manage that expansion, or may be restricted by government legislation, or the inability to make partnerships with other groups, or any of a variety of different problems. For example, GiveWell once found that non-profit Smile Train had a bunch of money, but weren’t able to use it to hire needed developing-world doctors because the supply had been used up and there weren’t any more available to hire. More recently, GiveWell also found a similar problem with immunizations and micronutrient supplementations—there’s lots of money in this area, but there are logistical problems in creating the needed partnerships with the country governments, and these are problems that only time—not more money—can solve.

When evaluating the work of organizations focused on nonhuman animals, we need to make sure that even if we find something effective, that the intervention in question is bottlenecked specifically by money and that more money is what will make the intervention happen. Otherwise, donations might not be what the organization needs and it would be counter-productive to recommend them to donors.

How can we evaluate the organization for room for more funding?

The first way to evaluate room for more funding is to look at patterns in the financial data of the non-profit. In particular, we could check to make sure the amount of assets they hold is not unusually large relative to the size of the organization and see if the organization has been expanding in past years. It would be good to check this on both the level of the organization as a whole and the level of individual programs / interventions, if applicable.

The second method is to ask the non-profit directly. In GiveWell’s first year of research, they would ask as a part of their grant application process, “What would a significant increase in funding allow your organization to do that it couldn’t do otherwise?” Questions might get more specific about what an organization would do with certain levels of funding, for example, asking “what would your organization do with a one-time donation of $10K?” Though one must be careful as organizations may have a tendency to tell the donor what they think the donor wants to hear.

How can we evaluate other aspects of room for more funding?

It’s not enough to evaluate room for more funding from the organization itself, however. Instead, one must also understand the risk that another donor would donate to the organization if ACE were not to move money to it. For example, before GiveWell could finish considering non-profit GAVI for a recommendation, its $3.5B funding gap was completely filled by other donors.

Given that ACE is making donor recommendations this might not be an issue, but a large donor could potentially take out the room for more funding before ACE-directed donors could fill it. Thus, understanding the potential funding landscape for an organization by seeing what other funders in the space are doing is also important.

Criterion #2: The Charity Engages in Programs That Seem Likely to be Highly Impactful

How do we look for mission effectiveness?

It’s hard to assess mission effectiveness in a direct or easy way. What we intend to do is gather the relevant empirical evidence, expert testimony, and intuitions as to what interventions have high mission effectiveness and check to make sure the organization is working on employing these interventions.

However, there still is a strong need at moment to “keep an open mind” when it comes to interventions as the evidence base for nonhuman animal interventions is currently poor (for example, there is not yet any study with a control group, though several are now in development) and many potential interventions have not yet received any study. Thus, this criterion might be better used as a weak, heuristic approach to screen ineffective interventions out rather than screen effective interventions in.

What if a large organization has projects with both high and low mission effectiveness?

It’s important to note that when donating to an organization one is essentially donating to all of their projects and therefore is meeting them at their average mission effectiveness rather than their best. One might be tempted to think that this problem can be solved with restricted donations, however, this is generally not the case due to the problem of fungibility. Put simply, if an organization receives restricted funding to Effective Project #1, they can then use that donation to free up the unrestricted funding they would have put into Effective Project #1 and move that unrestricted funding to Ineffective Project #2. Thus, the donation did not accomplish any net increase to effective projects but rather resulted in a shift of funding, actually resulting instead in a net increase to ineffective projects.

Is pursuing a project with low mission effectiveness project grounds for disqualification?

This is a bit complicated. One might question an organization that is pursuing a project with low mission effectiveness as not really understanding success and failure (see Criterion #4). However, what does or does not have high mission effectiveness is not well-established and agreed upon, so perhaps the organization has a legitimate and justifiable difference of opinion.

Additionally, perhaps these ineffective projects are what draw in large amounts of donations from donors who are unconcerned with effectiveness, or perhaps these initiatives generate a lot of positive press for the organization, and therefore these projects have great indirect impact.

Therefore, it seems that pursuing a project with low mission effectiveness should be a negative, but not grounds for disqualification.

Criterion #3: The Charity Operates Cost-Effectively, According to Our Best Estimates

What goes into a back-of-the-envelope calculation?

The goal of the back-of-the-envelope calculation is to explicitly answer the question “How cost-effective is a donation to this organization / intervention, given what we know?” This is accomplished by creating an equation to get a figure of “years of nonhuman animal suffering averted per dollar”, which is a figure that appropriately takes into account the differences between nonhuman animals.

A lot of work would need to go into developing what this calculation would look like, how it would be adapted to different interventions, and how the information to complete the equation would be completed and updated. Because it’s so complex, we have devoted an additional section to it (see The Cost Effectiveness Calculation).

Why is a back-of-the-envelope calculation not enough?

In a perfect world with no uncertainty, a back-of-the-envelope calculation would be all that is needed to evaluate an organization. Indeed, one might be tempted to think that if we have a sufficient guess as to how many years of nonhuman animal suffering are averted, we have all we need to compare organizations and can just go with whichever one appears the most cost-effective. However, for reasons discussed by GiveWell in “Why We Can’t Take Expected Value Estimates Literally” and by Peter Hurford in “Why I’m Skeptical About Unproven Causes”, this is naïve.

Put simply, there is a lot of opportunity for bias in cost-effectiveness measures, and measures generally need to be “regressed back to the mean” to adjust for bias and uncertainty. Therefore, we need to look at other criteria in order to calibrate our estimate.

Another possible way of thinking about this is that we are creating many weak arguments in favor of a particular non-profit rather than one strong argument, which is the best we can do under the uncertainty that we will presumably be operating under.

How do we move beyond the back-of-the-envelope calculation?

First, we define the following terms:

management effectiveness: an organization has high management effectiveness if it is well-run, well-managed, and accomplishes its mission as best as possible with its given resources.
mission effectiveness: an organization has high mission effectiveness if it has picked a mission that is the most effective mission at reducing the suffering of nonhuman animals.

These concepts are distinct. We could have an organization that has high mission effectiveness yet low management effectiveness—when it comes to public health, think an anti-malaria organization that wastes 90% of its money and staff time on fundraisers with incredibly poor returns and only uses 10% on net distribution.

We also could have an organization that has low mission effectiveness yet high management effectiveness—when it comes to public health, think an organization that is focused on fighting breast cancer, yet is highly effective at getting NSF funding to the right researchers, attracting top talent to the research field, establishing best practices at cancer early detection, etc.

Generally, the optimally cost-effective organization for nonhuman animals (i.e., the organization we want to move money to) is going to be the one with the most management effectiveness (i.e., most able to efficiently carry out its interventions) among the set or organizations that have very high mission effectiveness with respect to our larger cause of reducing nonhuman animal suffering (i.e., those organizations that pick the optimal intervention or set of interventions for reducing nonhuman animal suffering).

The back-of-the-envelope calculation gets us some of the way to assessing mission and management effectiveness by taking these concepts into account, but looking at each of these two concepts more in depth from different angles can aid our assessment process.

Could it turn out that an organization has a mission effectiveness so high that their management effectiveness doesn’t matter?

Yes, though we currently consider this unlikely. While mission effectiveness might be dominant in calculating the impact from a donation, management effectiveness will matter a lot in complicated and more long-term ways as the organization seeks to scale up. Ultimately, it will always be better to donate to an organization that has high mission and management effectiveness.

Additionally, we have fairly wide confidence bars over estimates of the mission effectiveness at the moment, so it might be too risky to go “all in” on measures of mission effectiveness when issues of management effectiveness are raising red flags.

Criterion #4: The Charity Identifies Areas of Success and Failure and Responds Appropriately

What is a “robust understanding of success and failure”?

Now that we’ve assessed mission effectiveness, it’s time to turn our attention to management effectiveness. If an organization has high management effectiveness, it will very likely have what we call “a robust and agile understanding of success and failure”. This means the organization possesses detailed, thorough, and specific knowledge about what they want to achieve as an organization in the near and far term and what feedback would tell them whether they are moving toward or away from that mission and how they could move toward that mission better in a quick and iterative way. Put more simply, the organization can define “success”, knows what it looks like when they aren’t being successful, and acts in a way to gain quick feedback and update what they do. Furthermore, they are able to innovate effectively: they regularly try small and large changes to programs, identify which changes improve outcomes, and act on that information.

Why was this included as a metric?

Unfortunately, in the non-profit world, feedback mechanisms are scarce. Unlike the for-profit world where a declining bottom line is apparent, non-profits merely need to convince themselves and donors that they are successful, not actually be successful. It’s very easy for a non-profit to keep spinning its wheels and think it is advancing on its mission, especially if its mission is vague and utopic. Therefore, it is important that savvy non-profits include checks to ensure they don’t get caught up in these traps and have an understanding of where they are going. Even if a non-profit has evidence of their success, they should continue looking for ways to improve, because it is unlikely their programs are already perfect.

What would this look like? How would we find it in an organization?

To inquire into this metric, one should look at the website of the organization and the testimony of those working for the organization as to what mission they articulate and what plan they articulate for how to accomplish that mission.

Are their plans specific? Can they define outcomes that indicate success and failure as well as rationale for why those outcomes indicate success and failure? Do they care about self-assessment? Are they self-skeptical?
Is the organization actually measuring things? Are they measuring things with good methodologies? Are they measuring things that actually matter or are they just chasing certain metrics because they’re easy (i.e., measuring leaflets given instead of people converted)?

Moreover and most importantly, does the organization show a willingness to change in light of metrics pointing one way or another? If a leafleting organization found that online video ads were more effective or vice versa, would they make the switch? Has the organization made key changes in its methods before based on evidence collected? Are they actively trying new things and working to collect new information that could help improve their programs?

An organization that is robust and agile with regard to success and failure should be taking cues, as relevant, from agile program management or lean start-up management, whereby they seek to try out a particular approach or intervention, learn from that approach, and then update their methods before trying again, continuing to iterate their methods in response to testing and evidence. If an organization is hitting many of these points, then they probably have a robust understanding of success and failure.

Additionally, such an organization will have goals that are SMART — specific, measurable, achievable, relevant, and time-bound. Good organizations will have well-targeted goals that they can use to concretely assess where they are, what they want to achieve, and how they will get there. Having a time-bound is especially important as it keeps the organization accountable to their expectations of success.

Criterion #5: The Charity Possesses a Strong Track Record of Success

What is a “strong track record of success”?

An organization that has high management effectiveness, by definition, should be able to accomplish its mission. Therefore, if the organization has been around long enough, it won’t be enough for them to understand what makes them successful, but they should generally be expected to have actually been successful by now and have accomplished some steps along the way to accomplishing their mission. Even if an organization has lofty goals that will take many years to accomplish (e.g., animal liberation), it still should have tractable subgoals that it has made progress on.

What would this look like? How would we find it in an organization?

It’s easy to look at an organization’s website for this information, as organizations usually like to share their claims of success in pages or newsletters. If that doesn’t work, one can always reach out to the organizations directly and ask what they’ve done so far with their donations. The real test will be carefully vetting the organization’s claims against common sense and evidence.

Generally, organizations—even potentially effective ones like those who have been evaluated by Giving What We Can—tend to overstate their accomplishments and we need to make sure these accomplishments have evidence to back them up. An example of how to do this is to try to elaborate all the claims the organization makes in a particular document and then provide sources for as many of them as possible (for examples, see the Giving What We Can evaluations of climate change organizations or the Copenhagen Consensus Report).

What if we’re considering a new and unproven organization?

Supporting an organization in its infancy might be a powerful way for ACE to have impact, as even small donations from ACE could make the difference between whether an eager start-up non-profit (much like ACE once was) happens or doesn’t happen. However, doing so is taking a bit of a risk, since you won’t have this criterion to look at and this criterion is one of the most objective and reliable measures of management effectiveness. This is a trade-off between risk and potentially higher rates of return that ACE would have to consider when evaluating.

Criterion #6: The Charity Has Strong Leadership and a Well-Developed Strategic Vision

Why was this included as a metric?

An organization that has high management effectiveness will very likely have strong leadership as leaders’ values and skills determine its response to any changes in the environment or within the organization. Bad or dysfunctional leadership can spell disaster for the non-profits with even the highest mission effectiveness if not addressed. An organization with a strong long-term strategy for effectively reducing suffering will respond to new evidence and changing situations by continuing to pursue the programs that help animals most effectively. An organization who doesn’t share our own values of reducing suffering as effectively as possible, may happen to have programs which we think are very good for animals, but could shift their strategy significantly in response to new evidence. For example, a group which promotes plant-based diets because of the health benefits to humans would be more likely to decide to allocate resources to programs that didn’t affect non-human animals or even harmed them than a group which promotes plant-based diets out of concern for animal welfare or rights.

What would this look like? How would we find it in an organization?

This can be identified by looking to the leadership structure and organizational strategy, looking specifically at the reputation and competence of high-level staff, the organization’s strategic plan, and the way their activities fit into the overall animal advocacy movement and our understanding of social change.

Usually, high-level staff should seem personable and competent and be well known and liked among other related organizations. The organization’s goals should explicitly include helping animals and reducing suffering; ideally, these should be of primary concern in their own evaluations of their programs. The organization should have a sound strategic plan which is revisited regularly, and which includes both shorter and longer term goals.

The organization should also be working in an area which makes sense when considered in the context of the field of animal advocacy as a whole. Organizations working in particularly neglected areas, or performing especially well in areas vital to the animal advocacy community overall, do well in this consideration.

Criterion #7: The Charity Has a Healthy Culture and a Sustainable Structure

What is a healthy culture and sustainable structure?

An organization with a healthy and sustainable structure is one which is well-managed on an operational level. While this will look different depending on the unique needs of each organization, there are some characteristics of a healthy organization which we can look for specifically.

First, an organization should be stable and sustainable under ordinary conditions, and should seem likely to survive the transition should some of the current leadership move on to other projects. They should be able to cope with both typical conditions and the stresses of a leadership transition. The organization should not seem likely to split into factions, or to stop being able to raise the funds needed for its basic operations.

Second, a healthy organization acts responsibly to stakeholders including staff, volunteers, donors, and others in the community. In particular, staff and volunteers develop and grow as advocates due to their relationship with the organization. The organization is transparent with donors and the general public. Finally, the organization is involved in open dialogue with other advocacy groups and seeks out mutually beneficial collaborations to maximize their impact.

Why was this included as a metric?

An organization that has high management effectiveness will work well on an operational level. Healthy relationships among staff and well-developed methods and routines will allow projects to be carried out efficiently, without distractions caused by a lack of organization or recurrent small crises. A sustainable structure also enables the organization and its supporters to make long term plans. Finally, an organization with a healthy culture builds capacity in the animal advocacy movement as a whole, both internally through staff training and development and externally through supporting other organizations doing related work.

Some aspects of a healthy culture and sustainable structure are specifically important in enabling ACE to work with and recommend an organization. For example, we would not be able to conduct a thorough review of an organization which was not willing to be highly transparent with us, and we would not be able to publish the review unless they were willing to be at least moderately transparent with the general public. We would also have trouble recommending an organization which was unstable enough that it might dissolve or substantially change its structure over the course of the year following our recommendation.

What would this look like? How would we find it in an organization?

The organization should function smoothly under ordinary conditions. Most of the organization’s energy should appear to go towards their goals, rather than towards resolving conflict internally or with other advocacy groups. The organization should be able to demonstrate some sustained fundraising ability, without too many drastic budget fluctuations.

The organization should be able to survive upcoming transitions if the current leaders in the organization were to move onto different ventures and it would be good if the organization had a track-record of surviving similar transitions in the past. There should be strong internal documentation to ease staff transitions.

Leadership should onboard new staff and volunteers up to speed with good intentional training structure to build capacity of individual staff and volunteers. New staff and volunteers should efficiently learn how to operate key programs and multiple staff and/or volunteers should have the training and knowledge necessary to operate key programs. Ideally, staff and volunteers will have histories of taking on additional responsibilities within the organization, and staff who leave the organization will often go on to work with other advocacy groups in positions they would not have been able to fill before their experience with the organization being evaluated.

The organization should be open with people who request information, ideally even if they’re not intending to donate or move large sums of money. Better yet, the organization may publish reports online for all to see, not just to those who request it. It is a particularly good sign if the organization is not afraid of publishing their mistakes and negative things about themselves and puts substantial documentation online in the style of AMF, the style of GiveWell, or the style of Charity Science, with full budgets, external reviews, audit reports, documentation of future plans, and other pertinent information.

The organization should have friendly and respectful relationships with others in the field, even if they disagree on some matters. They should be able to provide examples of times they have helped other organizations that have approached them with questions or ideas for collaboration. Ideally, they should be proactive about seeking out collaborations where appropriate; they may publish material useful for other advocates, hold training workshops, or just have ongoing relationships with other groups that work in complementary ways.

The Cost-Effectiveness Calculation

Criterion #3 calls for a cost-effectiveness estimate of the non-profit. But how would this be estimated? What we’re looking for is an estimate of what outcomes would be produced per marginal dollar added to the funds of a particular organization. We can model this simply by looking to the following equation:

    \[CE = \frac{I}{C_u+(C_i*N_i)}\]

CE = Cost-Effectiveness, I = Impact, Cu = Upfront Costs,
Ci = per-item costs, Ni = number of items

This model states that cost-effectiveness equals impact divided by costs and then breaks cost down further (see Understanding Costs for more detail). Impact is not broken down further because it will vary depending on the context of the organization (see Understanding Impact).

Understanding Impact

There are many different approaches to helping animals, such as working at pet shelters, leafleting, protesting in Chipotle, lobbying Congress for changes in the law, and many more. However, estimates of cost-effectiveness would be nearly useless without some common metric to compare across these widely different approaches. While we think that the weakness of comparative methods and the high uncertainty in these calculations make cost-effectiveness estimates only somewhat reliable (see Criterion #2. for more), it’s good to attempt this.

What would a common metric look like?

Following prior work by Peter Hurford and Brian Tomasik, we suggest evaluating things in terms of “days of animal suffering averted” (DASA) (though see possible refinements in Understanding Impact.), which becomes DASA per dollar (DASA/$) when costs are taken into account (see Understanding Costs). This is superior to other metrics, such as “vegetarians created per dollar” (used by Jeff Kaufman) because it allows us to compare across interventions that don’t involve generating vegetarians (e.g., lobbying) and also allows us to handle counterfactual adjustment better.

Whenever we approach a cost-effectiveness estimate for an organization, the evaluator would generate the appropriate equation for that organization with DASA/$ as the evaluand, and then evaluate it.

II.1.b. What might the calculations look like?

Let’s first consider vegetarian advocacy (e.g., leafleting, facebook ads, etc.) as a possible example. We suggest this equation:

    \[DASA_{advocacy} = \Delta N * \frac{AS}{N} * \frac{T}{AS}\]

ΔN = number of people swayed by the advocacy campaign,
AS/N = average net animals saved per opinion change,
T/AS = average time spent in a factory farm averted per animal saved

Something like a protest in Chipotle could be similar, though modelled with potential upside/downside:

    \[DASA_{Chipotle} = (\Delta N * \frac{AS}{N} * \frac{T}{AS}) + (\sum^{R_c}p(R_c)*I_{R_c})\]

Same as previous, but…
ΣRC = summation of impact across all possible reactions of Chipotle,
p(RC) = probability of that particular reaction,
I(RC) = impact of that particular reaction

For a more complex but related intervention, we could consider investing money not in passing out leaflets, but in improving the design of the leaflets with the hope of convincing more people:

    \[DASA_{update} = (\Delta N_{new} - \Delta N_{old})*(\frac{AS}{N}*\frac{T}{AS})*TU_{new}\]

Same as previous, but…
ΔNnew = number of people swayed per leaflet of the new leaflet design,
ΔNold = number of people swayed per leaflet of the old leaflet design,
TUnew = total times the new leaflet will be used

And advocacy like lobbying could be simple, but entirely different:

    \[DASA_{lobbying} = A * \Delta W * T_L * CS\]

A = Number of animals affected by the change in legislation (per day),
ΔW = how much their welfare is changed (relative to not being factory farmed at all),
TL =length of the change in legislation (in days),
CS = counterfactual share in producing the change accredited to organization

II.1.c. How might we refine these calculations?

While the formulas above are a good start, and probably good enough given the inherent uncertainty, there still are some issues we will have to take into account when creating and using these equations.

Accounting for specific details
First, it could be important to be sensitive to specific details about each opportunity and include that in the calculations. For example, the impact of any intervention that seeks to have impact by creating vegetarians should adjust the impact of vegetarians for product elasticity in the supply and demand of meat (i.e., AS should be adjusted downward to account for elasticity). Similar details might exist in other relevant calculations and should be identified.

Counterfactuals
Second, it is important to adjust for a proper counterfactual of what would have happened had the campaign not taken place. For example, if the lobbying never happened, how long would it take for the law change to happen anyway? If this organization didn’t accomplish the law change, would another organization have done it?

Sure, leafleting might have created 100 relevant opinion changes in favor of vegetarianism, but is it possible that many of these shifts happened because of things other than the leaflets? Will welfare changes for animals become moot because of some large shift in the future, like a hypothetical widespread adoption of in vitro meat?

Here, we would want to do things like adjust ΔN downward using a control group when evaluating advocacy, use terms like “CS” when evaluating lobbying, and other changes.

Variations within species / per food
Another complication is that each animal does not suffer identically and is not eaten equally by people. For example, theorists have pointed out that hens that are raised for meat seem to be in a position to suffer disproportionately more than other animals and are also a disproportionate share of the American diet. Others have pointed out that cows, being more cognitively complex, might instead suffer more.

Some of this is evened out by focusing on days of suffering (as hens raised for eggs are kept longer than other animals), but we still might want to focus on a per-animal basis, as some animals suffer more intensely per day than others. Jason Kirschner has suggested that we adopt a base unit of “meat-raised hen suffering” and convert everything into that.

We could adapt our equation for the DASA for advocacy to instead be the “days of meat-raised hen suffering averted” (DMRHSA) and add data about each animal individually, like so:

    \[DMRHSA_{advocacy} = \Delta N * \sum^A(\frac{Spar_A}{\Delta N} * D_A * \frac{Suf_A}{D_A})\]

ΔN = number of opinion changes, ΣA = a summation across every relevant animal,
SparA/ΔN = the amount of that animal spared on average per opinion change,
DA = the number of days that animal suffers on average,
SufA = the amount of suffering that animal endures on average relative to a meat-raised hen

Here, having values for SparA, DA, and SufA be able to differ for each animal will allow us to be more accurate in accounting for inter-animal differences, though this could come at a cost (see ‘More complicated than it’s worth?’ below).

Time discounting
Yet another aspect we haven’t considered much is “time discounting”. Is an animal spared five years from now worth the same to us as an animal spared right now? If there is a relevant difference this could greatly affect how we compare different interventions as some interventions (e.g., lobbying) operate at significant delay.

Indirect conversions
Another large component of cost-effectiveness that is currently missing from equations is the large variety of indirect effects. One prominent such effect is indirect conversions—for example, someone converting to a meat-reducing or vegetarian lifestyle not because of the influence of the campaign directly, but because of the influence of someone influenced by the campaign. One could even imagine this being very drawn out—for example, someone inspired to become an activist and start their own campaign because of the influence of a previous campaign, or a campaign influencing someone who influences someone who influences someone who influences someone, etc. How do we account for this, assuming we should?

Other indirect effects
Some other indirect effects exist of different magnitudes. Jason Kirschner points to one example—potential human suffering due to ostracism from converting to vegetarianism or veganism. Though this effect is probably very minute compared to the impact on the nonhuman animals, it’s possible that larger indirect effects may exist that have not yet been identified.

A much bigger example is that of meme spreading. Some theorists like Brian Tomasik and Robert Wiblin have speculated that meme spreading and the influence of those memes (e.g., the chance they prevent wild animal suffering from being spread via space colonization or promote feelings of non-discrimination toward substrates and therefore reduce future computational suffering, etc.) might be the dominant form of impact from spreading concern for animal suffering rather than the animals spared from factory farming in the interim.

Another big example might be learning. By performing an intervention we learn a fair amount about how that intervention is undertaken and this information can then be used to improve the effectiveness of future interventions, thereby improving future impact. This makes each intervention valuable not just for its own sake, but also in how it improves future interventions.

We’re currently not sure how to add such a potentially large and speculative impacts to the existing calculations, but given that a large majority of the impact might come from these effects, this could be a concern for our quantification.

Differences in certainty
After accounting for these many effects, we might also want to correct for another difference between calculations—certainty. For example, some estimates might be more affected by biases than other methods (e.g., non-response bias) and some estimates might just have different levels of certainty than others. One idea is to adjust each equation for the probability that we find that equation correct so as to make better comparisons.

More complicated than it’s worth?
A final concern might be that all these additional adjustments to the equation make them more complicated than they’re worth. Previously, Peter Hurford compared a simple equation to a far more complex equation and found that the difference between the resulting estimates was rather trivial, only differing by a factor of three. This means that simple calculations might still get at the right answer while also being easier to use, avoiding the human error.

Every step we add to the calculation is another chance to introduce uncertainty and error. Furthermore, the more demanding we make our equations, the more demanding we have to make our measurements to collect the relevant data for these calculations, which puts pressure on our methodology to be less friendly (e.g., longer surveys or more complex surveys that weaken response accuracy, etc.). Therefore, we might want to consider keeping it simple unless absolutely necessary and/or having both a simple and complex estimate in our calculations. Moreover, this increases the rationale of having certain safeguards, like having multiple independent evaluators create and evaluate the equations and relying heavily on criteria other than the cost-effectiveness estimate.

Making it about the organization?

So we now have equations that can evaluate the impact of individual interventions. It seems likely that the impact of a marginal dollar to a particular organization will be equivalent to the impact of a marginal dollar to the particular intervention the organization is most likely to marginally fund, but it’s possible this might not always be the case.

One example is that we might have uncertainty about which intervention the marginal dollar would fund (typical in a large organization working on many projects) and therefore we could adopt an approach like this:

    \[DASA_{org} = \sum^I(p(I) * DASA_I)\]

ΣI = a summation across every possible intervention the org would fund,
p(I) = probability the marginal dollar will fund that particular intervention,
DASAI = DASA of that particular intervention

Another possible concern that exists on the organizational level is displacement in donations—by funding this organization, we may reduce other funding to that organization. This is a complex effect that is difficult to model, but that we may want to take into account.

There also may be other relevant ways that organizations differ from interventions in evaluation that I’m not aware of.

Understanding Costs

Understanding impact is important, but it’s only half the battle when it comes to cost-effectiveness—we also need to know how much it costs to buy that impact. Luckily, this should be a lot easier and have a lot less error than our calculations of impact.

To start, recall our cost-effectiveness model from earlier:

    \[CE = \frac{I}{C_u + (C_i*N_i)}\]

CE = Cost-Effectiveness, I = Impact, Cu = Upfront Costs,
Ci = per-item costs, Ni = number of items

We now know how to calculate I on a case-by-case basis for each organization. Now we need to calculate costs, which we have modelled as equalling the sum of up-front costs and the product of per-item costs and the number of items.

Understanding per-item costs

We’re actually going to start backwards and begin with per-item costs, because that’s where the bulk of the expenses will be as we’ve visualized it. Per-item costs are anything that is paid on a per-X basis, like salaries, rental fees, printing costs for leaflets, phone calls to Congress, hosting fees for a website, Facebook ads, etc.

We can understand it like this:

    \[TC_i = \sum^i(C_i*N_i)\]

TCi = total per-item costs,
Σi = summation of all items
Ci = cost of that item,
Ni = amount of that item

For example, imagine a leafleting campaign. Some per-item costs would be (1) the salaries of the employees and volunteers devoted to de leafleting, (2) the opportunity cost of each employee and volunteer, (3) and the costs of printing the leaflets, and (4) any relevant per-time overhead cost that contributes to the leafleting. One might think that if leafleting is 10% of the efforts of the organization, about 10% of the overhead cost should be attributed to leafleting, though this might not represent an accurate counterfactual about how the organization would scale back if it wasn’t leafleting.

So for leafleting, perhaps per-item costs would look like this:

    \[TC_{\text{leafleting per-item}} = \sum^E((S_E + OC_E) * T_E) + (C_L * N_L) + (OHC * S_{\text{leafleting}} * T)\]

ΣE = summation across all employees and volunteers,
SE = salary of that person, OCE = opportunity cost of that person,
TE = time spent by that person, CL = cost of a leaflet,
NL = number of leaflets printed, OHC = overhead cost of the org,
Sleafleting = share of overhead for leafleting
T = length of the leafleting campaign

Understanding up-front costs

Up-front costs are the costs of undertaking the campaign that happen just once for the entire campaign. They include costs like the the cost of developing and creating the campaign, the cost necessary to train employees to be skilled enough to undertake the campaign, and the general overhead of running the organization, but do not include things like salaries or opportunity costs (as these are per unit of time) and do not include things like printing costs (as these are per item printed).

This leads us to an equation like this:

    \[C_u = C_d + C_t\]

Cu = upfront cost,
Cd = development cost,
Ct = training cost

II.3. Putting It All Together

Remember again the formula for cost-effectiveness:

    \[CE = \frac{I}{C_u + (C_i * N_i)}\]

CE = Cost-Effectiveness, I = Impact, Cu = Upfront Costs,
Ci = per-item costs, Ni = number of items

We now know enough to put everything together. Again, we think it’s most constructive to consider an example. Here’s a potential complete calculation for the days of animal suffering averted by leafleting:

    \[DASA/\$_{leafleting} = \frac{\Delta N * \frac{AS}{N} * \frac{T}{AS}}{\sum^E((S_E + OC_E) * T_E) + (C_L * N_L) + (OHC * S_{\text{leafleting}} * T) + C_d + C_t}\]

ΔN = number of opinion changes, AS/N = average net animals saved per opinion change, T/AS = average time spent in a factory farm averted per animal saved, ΣE = summation across all employees and volunteers, SE = salary of that person, OCE = opportunity cost of that person, TE = time spent by that person, CL = cost of a leaflet, NL = number of leaflets printed, OHC = overhead cost of the org, Sleafleting = share of overhead for leafleting, T = length of the leafleting campaign, Cd = up-front development cost of campaign, Ct = up-front training cost of campaign