ACE Highlight: Survey Guidelines

ACE is proud to release our newest project: Survey Guidelines. We developed this in collaboration with the Faunalytics, and our hope is that this resource will be a useful tool for all individuals and organizations seeking to measure their impact.

Animal advocacy lacks solid research on what is most effective. A compounding problem is that the small number of studies measuring the impact of interventions (tactics to help animals) that have been conducted often use different scales and metrics to track progress. From something as simple as a demographic scale to more complex measurements like reduction in meat consumption, the results are rarely comparable.

Example of incomparable scales:
Study A: Reduced consumption by food frequency questionnaire (3-6 times per day, 3-6 times per week etc)
Study B: Reduced consumption by qualitative scale (significantly reduced, slightly reduced etc)

Several other common problems occur when designing surveys to measure meat consumption. Well-intentioned organizations sometimes create a bias through the manner in which they present questions. An example of this is when images or material that would immediately identify the surveyor as being affiliated with an animal advocacy group are presented early on in the survey process; because of response bias, knowing that an animal advocacy group is conducting the study will cause participants to answer in a favorable way. Relatedly, social desirability bias, or wanting your results to be viewed favorably by others, can also affect responses.

Other organizations assume that asking a respondent to qualify their diet as “vegetarian” or “vegan” are sufficient to report on their actual diet, whereas research has shown that these self-identifications are often inaccurate. Many respondents report being veg, but subsequent food frequency questionnaires show that they eat chicken or fish regularly.

These scale discrepancies, bias issues, and problems with self-identification are the motivation behind ACE’s latest project. Working in close collaboration with Faunalytics and a team of volunteers, and considering questions already used by major animal advocacy groups in their own research efforts, we cataloged and refined questions across various categories. The result is Survey Guidelines – Questions. While an organization’s research project might ask additional specific questions related to their study, we highly recommend using at least some of these questions, as they will ultimately allow easy comparison of results across multiple studies.

We have also recently published a series of pages as a part of this Survey Guidelines project that are meant to help with other considerations. For example, intervention studies should ideally use a control group and randomly assign participants, a method known as a randomized-control trial, or RCT. This method allows you to truly track the impact of a specific variable, and is often referred to as the gold standard of studies. This and many other helpful tips can be found in our General Advice webpage.

Additionally, the Survey Guidelines project provides specific food frequency questionnaires that we encourage groups to use when measuring dietary patterns, and we even provide a downloadable Word document that you can use for your own efforts. We also give advice on using a social desirability scale, and discuss selecting a scale that is right for your specific project.

It’s not easy to design a perfect study, and we certainly don’t think that it is necessary to have everything in an ideal condition in order to conduct research. Simply making efforts to measure impact can prove incredibly beneficial for a multitude of reasons, including informing advocates on the general effectiveness of various techniques and serving as education for future research projects. For example, we learned a lot from previous leafleting studies before we conducted our own leafleting study, and we hope that others will in turn learn from our efforts. However, time and resources are limited, and it behooves all of us to make as much effort as possible to design high quality studies.