Guest post by Harish Sethu
Societal change is complex; an intricate web of economic, cultural and other factors influence individual behaviors and institutional practices. Our instincts, our common sense, and our self-assured rationalizations, however well-meaning, are an overvalued tool to cut through this complexity and discover the approaches most likely to bring about the changes we seek. Well-gathered data and credibly conducted scientific research, on the other hand, offer a surer path to effective advocacy. Without a doubt, animal organizations must value and promote a culture where advocacy is informed by well-grounded research, but should they be conducting their own research?
Research which informs the work of an animal advocacy organization can be produced by either the organization itself or by an outside entity such as an academic institution or an independent research outfit. There are certain benefits to an animal organization conducting its own research and there are also some downsides, but as I will outline below, given high standards and adequate transparency, there is little reason to confine research to any particular class of institutions.
The Tricky Problem of Bias
A frequently-raised downside to an animal organization conducting its own research is the hint of bias that can cloud perceptions about its studies, especially when the results or the claims validate the approaches used by the organization. That an animal organization would deliberately manipulate data or results to validate its perspective in the movement is an unjustifiably cynical position to hold. However, it is definitely true that even the most honest among us are susceptible to unconscious biases which can dictate the questions we ask, the methodologies we use, and the inferences we draw from the results. It takes work just to recognize and be mindful of our own biases; eliminating them entirely is an even higher bar to clear.
However, there are at least three things an organization can institutionalize to reduce the risk of bias. First, an animal organization can implement a separation in the reporting lines between its activism units and its research units as a means to reduce the influence of the former on the latter. This would be akin to how newspapers may establish a separate and independent reporting line for their advertising units to reduce the incentive to report news favorable to advertisers.
Second, the research unit of the organization can follow a practice of pre-declaring with sufficient specificity, for each study, high standards for making inferences from observed results—when these standards are observed, there is less room for bias to creep in.
Third, establishing a high degree of transparency in research methods and analysis, such as by releasing all the data and the code used through the entire lifecycle of the research study, can substantially enhance trust in a study’s claims.
In this context, it is vital to recognize that contracting out the research to an academic at a university is not guaranteed to lower either the perception or the possibility of bias in the results. Academics in many departments are under constant pressure to secure external grant funding—there is substantive evidence that the conclusions of an academic publication are highly correlated with what is favorable to the source of the funding. Besides, academics are also humans—they are just as likely to come with the baggage of ideological biases, rigid worldviews, and self-interest in validating them.
The nature of bias is so innately human that it is an inescapable reality in all of our endeavors. It is not institutional affiliation that marks a lack of bias; rather it is more the character of the individuals running the study, the purity of their motivation, and their willingness to confront their biases with well-gathered data. A deep, genuine, and abiding commitment to the cause of animal advocacy, more likely to be found in animal activists than in dispassionate academics, can even serve as a stronger source of inspiration to fight bias and discover the real truth about what will best save animals from suffering.
Quality of Research
Well-conducted scientific research comprises a lot more than the gathering of data, entering them into a spreadsheet and then having formulas or a piece of statistical software spell out conclusions. Scientific research is exceptionally harder to conduct than it appears, and even experienced Researchers routinely fail to interpret statistical results correctly. Animal organizations usually do not have people on their staff who are well-trained in hypothesis testing or in the statistical nuances that make it treacherous to dabble in data science. This is sometimes used as an argument for why animal organizations should not be conducting research and instead leave it to the professionals at academic institutions.
But the lack of in-house expertise is likely a short-lived problem at current animal organizations. Research in animal advocacy, especially farm animal advocacy, is still a young field without a body of credible knowledge base and with few established conventions or recognized experts. As the movement grows, it will surely attract experienced professionals moved by the cause who will want to find meaningful work at animal organizations. All fields of research, from the time they are born to the time they become mature, progressively improve in the quality of research they produce, and our early efforts are no indication of the quality that the movement is capable of delivering.
It is sometimes argued that academics at universities are motivated to produce higher quality research because of the peer-review system and the strong pressure to publish their work. Academic peer-review serves well to filter out most of the obviously inferior work, but it is no panacea—it remains an imperfect process in which reported studies are rarely required to have been pre-registered, there is little or no data transparency, and reviewers almost never double-check calculations or verify the code used to arrive at the results. It is an under-appreciated fact that even deliberate fraud makes it through the peer-review process just fine. In fact, poor statistical methodologies and false interpretations are more commonplace in peer-reviewed literature than is generally recognized. It is simplistic to assume that there is some formula, whether based on institutional affiliation or publication venue, that guarantees quality.
The most reputable academic journals and conferences do constitute an exception, but their criteria for valuing the worthiness of a publication are rarely the same as what animal organizations might value. The novelty of the methodology, the originality of the questions addressed, and positive results and counter-intuitive surprises are valued over the potential of the research findings to help animals. What is intellectually interesting does not always overlap neatly with what is useful or actionable for animal activists. Without the pressure to meet expectations of journal editors, in-house research at an animal organization is better positioned to focus its research on offering insights most pertinent to the organization’s mission.
Relevance and Authentic Replication
Animal organizations directly engage in confronting animal use and abuse through various means. They may work on persuading corporations to adopt stronger animal welfare policies, they may bring about legislative reforms, they may build a community of activists, or they may reach out to the public with messaging to promote compassion and advance animal rights. Routine interactions with people outside the movement regarding our use of animals is an extraordinary teacher of what works, what does not, and where we know too little to say much. This proximity to actual advocacy lends animal organizations valuable insight into the right questions to ask of research and the right setting for experimental observations. Related to this is the crucial advantage enjoyed by animal organizations when they conduct their own research—their ability to replicate advocacy as conducted in the field.
Context matters in ways substantial enough that advocacy approaches used in experimental settings may yield entirely different results from those in actual scenarios. It is not straightforward, sometimes not even feasible, for an outside entity or a university academic to authentically replicate an intervention used by an organization in the movement. It can require extensive cooperation with the animal organization, especially so in the early stages involving experiment design and data gathering. The team effort required to engineer such replication is more easily envisioned and promised than delivered. An animal organization, with a cohesive team of Researchers and activists, has a higher chance of success at producing results that can be credibly applied to its advocacy. In-house research, coupled with authentic replication, also contributes to increasing the likelihood that recommendations emerging from a study are adopted.
Good research can be expensive. Small investments in research which cannot generate actionable recommendations or which we already know cannot lead to conclusive suggestions on activism strategies can be an unwise use of scarce resources. However, a research study which seeks validity in real-life scenarios and is capable of producing results strongly suggestive of specific courses of action for activists can be logistically hard to conduct and unaffordable for small organizations. This is a legitimate concern and is among the most compelling reasons for an animal organization to forgo conducting its own research. Ideally, in the movement, larger organizations must assume the burden of such research. But the rapid rise of the effective altruism movement and the growing interest in research on animal advocacy offers hope that even small organizations can find the support they need for good research.
High-quality credible scientific research can be conducted at any institution, including animal organizations, academic institutions or other independent non-profits. It is folly to suggest that only certain types of institutions or organizations can or should conduct research in animal advocacy. If animal organizations foster a culture of transparent research, reduce structural incentives for bias, and organize into a vibrant community that engages in constructive criticism and constantly strives for higher standards, we can be well-justified in conducting our own research and well-positioned to keep it relevant to the cause of animals. Yet, it is also true that an organization does not have to conduct its own research to be effective. As long as someone is conducting the research, an animal organization can be effective by simply paying attention to good research no matter its origin and no matter what ideological bubbles its recommendations happen to puncture.
Note from ACE: Funding is available for those who wish to conduct research on the best ways to help animals through the Animal Advocacy Research Fund.
Caroline Fiennes says
Very pleased to see discussion of this issue of the choice of research producer matters.
I’ve written & spoken about this a lot.
There are two parts to my view. First, it’s useful to distinguish various different types of ‘research’ because the issue of who’s best to produce them varies. For gathering monitoring-type data, the NGO/nonprofit itself it probably best, e.g,. how many people receive the emails, sign up, go to demonstrations, cost of campaigns, no. beneficiaries (where relevant). But ‘evaluation’ (i.e., which the UK National Audit Office nicely defines as “as serious attempt to establish causation”) is often better produced by people who are independent and expert in research techniques. This is because, as you allude, NGOs often lack research skills & money to do it well – as well as have inadequate sample size and poor incentives.
There is evidence from experiment (~1500-org RCT) of bias in the ‘evaluation’ questions that NGOs even ask: they’re (unsurprisingly) more interested in evals likely to show their work favourably than ones which might challenge it.
I like your statement that “That an animal organization would deliberately manipulate data or results to validate its perspective in the movement is unjustifiably cynical.” If you read Ben Goldacre’s Bad Pharma, you’ll see plenty of evidence of that in other sectors…
Pieces I’ve written on the topic of who should investigate the effectiveness of nonprofits’ work include:
http://www.giving-evidence.com/oops – from Alliance Magazine
ww.giving-evidence.com/M&E – from Third Sector Magazine
https://giving-evidence.com/useful-measurement/ – from Civil Society Magazine
http://www.giving-evidence.com/no-evaluation/ – from Stanford Social Innovation Review