How to Conduct Studies That Will Encourage More Effective Altruists to Care about Animals

Note: This post has been edited to fix an incorrect attribution of a leafleting study to Humane League Labs. While THL helped gather data for the study, it was actually run by Farm Sanctuary.

Over the next 3 years, $1 million dollars will be committed to empirical studies within animal advocacy. This, combined with the Open Philanthropy Project’s announcement that they’ve hired a Factory Farming Program Officer who may also be interested in funding studies, means that empirical studies relating to animal advocacy are going to receive more money than ever before. These studies could fill clear gaps in animal advocacy by providing rigorous evaluation of the effects caused by different interventions. However, before we become too optimistic about how advocacy may be improved in light of these studies, it’s worth acknowledging that some past research relating to animal advocacy has been far from perfect. To help future studies avoid the same mistakes, this post outlines possible ways to increase the likelihood of acquiring useful results when conducting an empirical study in animal advocacy.

Randomization of control and experimental groups

Some groups have attempted to produce research studies to help animals, but the usefulness of such research is limited when there are significant methodological flaws. For instance, in a 2012 study which focused on the effects leafleting had on diet choice conducted by Farm Sanctuary, participants were compared to the “average American meat-eater,” which fails to account for the possibility that college students may simply eat less meat. To determine the effect of an intervention, the experiment group has to be compared to a randomized control group so that statistically significant differences between the groups can be attributed to the intervention.

Minimize spillover effects between groups within the experiment

Researchers should be careful of possible spillover effects from the experimental group which could contaminate the control group and make it harder to detect the effect of the intervention. For instance, people that convert to veganism in the experiment group may easily influence and contaminate people they know in the control group into changing their diet. This can be prevented by an appropriate study design which may have several schools participate instead of several classes within a school. That way the control and experiment group will have limited interactions.

Public Pre-commitment to a Strong Analysis Plan

Prior to data collection and analysis, pre-committing to what will be analyzed and a method of analysis is essential. Otherwise humans have an incredible ability to ignore what they actually see in favor of (a) what they expect to see and (b) what they want to see.

For more on what should be included in a pre-analysis plan please see this. Without this, pre-commitment studies are more likely to find misleading results as Researchers can knowingly (or unknowingly) engage in cherry picking unrepresentative data. The use of a pre-analysis plan helps prevent this and other widespread problems within empirical studies like p-hacking for statistically significant results. This may help to avoid issues like the possible problems with ACE’s Leafleting outreach study in 2013.

Transparency

To further guard against questionable statistical analysis of the data, the raw data of the study should be made public. That way, many can examine the data, reducing the chance of drawing misleading conclusions. Animal advocacy studies so far have done much better on this front than is typical of academic science; an example is the full data set of The Humane League’s second leafleting study being publicly available.

A focus on important metrics

Some important metrics to identify the best methods to help animals are animal product consumption and both perceptions of animals and reducing animal product consumption. A study that measures all these metrics is preferable because it will have increased statistical power which in turn makes its results more likely to be useful for animal advocates. It’s worth mentioning that key features of the pre-analysis plan should be to ensure the use of multiple measures doesn’t result in fishing for statistical significance and that results should bear in mind the various possible combinations of all findings. It would also be preferable that the same metrics are used across empirical studies so that comparisons between different interventions can more easily inform advocates.

Adequate Sample Size

To be able to detect very small effects on these important metrics an adequate sample size is required to be representative of the general population and will limit the influence of any outliers in the data set. Furthermore, an adequate sample size will mean that if the study doesn’t detect an effect, then it’s likely that the true effect is below some threshold, and this provides a helpful estimate of the intervention effect size. For an example of relevant sample size calculations please see the sample size section of the Methodology for Studying the Influence of Facebook Ads on Meat Reduction.

Be Aware of Selection Bias

Two powerful biases routinely occur in empirical studies relating to animal advocacy. The first is a bias in who participates in the study and the second is a bias in the answers of those who participate in the study.

Selection bias often comes in two forms. In its first form it causes a bias in who initially participates. An example of how this may occur is if people know that a survey is about topic ‘x’, those who are unfamiliar with or uncomfortable with topic ‘x’ may not want to be included in the survey. Or if people know that someone is going to offer them a leaflet on something that they don’t care about, then they’re less likely to take that leaflet.

Another type of selection bias occurs when following up with participants. If only 20% of participants are followed up with, they could very easily be unrepresentative of the entire sample if inappropriate follow up methods are used. Often the low proportion of participants who are followed up with in these types of studies makes them particularly problematic. A great study would aim to have high response rates, ideally in the 80% range.

Be Aware of Social Desirability Bias

The second bias is a social desirability bias. This occurs when respondents believe that a particular response is in some way preferred, or more socially desirable, and then become more likely to give that response. A clear example of this would be if the surveyor was known to have passed out certain flyers in the first phase of a leafleting study, or was wearing a vegan t-shirt while giving out the survey.

To recap, things that may increase the likelihood of acquiring useful results when conducting an empirical study in animal advocacy are:

• Randomization of the control and experimental group
• A study design limiting spillover between the experimental and the control group
• A public pre-commitment to an analysis plan
• Transparency with the original data
• A focus on important metrics
• An adequate sample size
• Methods to account for selection bias
• Methods to account for social desirability bias

Using these techniques could take us from relying on personal experience and intuition to having beliefs based upon high quality scientific evidence. As such, well done empirical studies are one of the most powerful approaches the animal advocacy movement can use to determine the means by which they can achieve their ends, but groups need to be careful to perform and analyze studies with great care to avoid drawing false conclusions.