How We Evaluate Social Media Impact

Before we began our 2016 charity evaluations, we researched the ways that animal charities use social media and developed a plan to quantitatively estimate the social media impact of each charity. We developed a two-step process for estimating social media impact. First, we estimate the impact that a social media page has in creating new vegetarians, vegans, and meat-reducers. Second, we estimate the impact that a social media page has in supporting current vegetarians, vegans, and meat reducers.

We built a calculator that produces estimates of the number of lives spared and years of suffering averted when we input certain social media metrics such as fan reach, non-fan reach, and video data. In this blog post, we explain the reasoning behind our calculator.

Survey of Animal Charities’ Social Media Use

To better understand how animal charities use social media, we conducted a brief survey of the social media pages of our 2015 Top Charities, Standout Charities, and a random sample of the other charities we have considered.¹

We noticed that Facebook is, by far, the most widely used social media platform. In fact, every animal charity that we investigated has a Facebook page. The second most widely used platform is Twitter. Other commonly used platforms include Instagram, Youtube, and Vimeo. We observed that several charities are beginning to use Snapchat.

We decided to rely only on Facebook data when evaluating charities’ social media impact for two reasons:

• Facebook is likely responsible for the bulk of each charity’s social media impact. Many more people follow animal charities on Facebook than on any other platform.
• Many charities post the same content on their Facebook pages and on other platforms like Twitter. We don’t know the extent to which a given charity’s Facebook fans and Twitter followers overlap, but it seems plausible that a charity’s Twitter followers are more or less a subset of their Facebook fans. If that is the case, Twitter posts may expose people to the same content twice, rather than exposing new people to the content.

We also observed that some charities post a variety of content while others focus on one specific kind of content. For example, some charities use social media specifically to propagate farmed animal rescue stories or to announce their own research findings. There is reason to think that certain types of messages are more effective than others at convincing people to go veg,² so we decided to consider each charity’s specific messaging strategies when evaluating the impact that their Facebook pages have as an outreach tool.

Finally, we noticed many different ways in which social media pages might help to prevent recidivism. There is reason to think that using multiple techniques is more effective than using one technique to support a behavior, so we took the number of techniques used by each charity into account when evaluating the impact that their Facebook pages have as support.

The Impact of Facebook as an Outreach Tool

To assess the impact of a social media page on reducing the consumption of animal products, we need to estimate (i) the chance the page will convince a given non-vegetarian to become a vegetarian³ and (ii) the number of non-vegetarians the page will reach.

The Chance a Page Will Convince a Non-Vegetarian to Reduce their Meat Consumption

It is difficult to estimate the rate at which social media convinces people to give up meat because, to our knowledge, it has never been studied directly. The most pertinent available data may come from studies of online ads. Though we don’t know the precise success rate of online ads, we estimate that for every 1000 clicks on Facebook ads, about seven new vegetarians⁴ are created.

We use the limited information we have about online ads to estimate the behavior changes Facebook pages inspire through their video posts. Of course, most Facebook videos have very different content than the ad videos and they may not create behavior change as effectively. A clip of a hen watching National Geographic, for instance, is not likely to be as effective as the undercover investigation video used in many ads. To calculate the impact of a particular Facebook page, we decided to count the number of views of videos that have (i) a strong “go veg” message, (ii) a weaker “go veg” message, and (iii) a very weak or no “go veg” message. We use this count to determine a discount factor which we apply to the estimated success rate of the videos on a page.⁵

We applied a second discount factor to the estimated effects of videos shown in countries with lower levels of meat consumption than the U.S. For instance, in Germany, per capita meat consumption is roughly 75% of its level in the United States.⁶ If a charity’s Facebook page appears to be mainly serving a German-speaking audience, we multiply the expected effects of those videos by 0.75. If a charity has one page that serves a German-speaking audience and another page with about a third of the video views that serves an English-speaking audience, we multiply the expected effects of that charity’s videos by 0.833.⁷

There are at least five other important factors to consider when comparing online ad and social media impact:

1. Online ads load in new pages that are free of distractions, whereas Facebook videos often appear in newsfeeds amid other content.
2. Facebook pages include other content, besides videos, that may convince people to go veg.
3. People may be more influenced by posts their friends have engaged with than by videos labeled as ads.
4. Facebook users probably view repeats of ad videos fewer times than they view multiple videos from the same veg page, and each would be counted as a “unique” view.
5. The non-vegetarians who view a video on a veg Facebook page might be more pro-veg and/or more exposed to more pro-veg content than the viewers of the online ads. (If they do not follow a veg page themselves, they are likely exposed to the video because they have at least one friend who follows a veg page.)

On balance, these factors suggest that a view of a video posted on Facebook may be slightly less effective than the same view of a video used as an online ad. So in addition to assigning discount factors to each Facebook video based on its content and the expected meat consumption of its audience, we applied a discount factor of 0.8 to all Facebook videos compared to ad videos.⁸

The Number of Non-Vegetarians a Page will Reach

Because the success rate of online ad videos is formulated as the number of new vegetarians divided by the number of people who actively click on an ad to watch a video, we are interested in the number of people who actively watch a video on Facebook, not the number who simply scroll past one in their newsfeeds. Facebook provides a measure of the number of people who watch at least 95% of a video on a Facebook page, which we think indicates a level of engagement comparable to viewers’ engagement with ads.

Because the audience of online ad videos likely reflects the general population and may be almost entirely non-veg, we are interested in the number of non-vegetarians who watch 95% of a video on a Facebook page. To approximate the number of Facebook video viewers who are non-veg, we used the following estimates:

• The percentage of viewers who are fans of the page and not vegetarian: 77.5%
• The percentage of viewers who are not fans of the page and not vegetarian: 93.3%⁹
• The number of fans who watch videos on the page relative to the number of non-fans. This required the following estimates:
• The number of the page’s fans who are exposed to the page’s videos relative to the number of non-fans who are exposed to the page’s videos: (We calculate this estimate for each charity separately based on their Facebook “post reach” metrics.)
• The likelihood that a non-fan will watch a video relative to the likelihood that a fan will watch a video: 42.9%

We based our estimates on information about the prevalence of vegetarianism, Facebook’s metrics indicating fan and non-fan page “reach,” and some data about fan and non-fan engagement with ads. Each member of our research team made estimates independently and we discussed them to determine which estimate to use, often taking the average of our independent estimates. You can view our estimates and some of our reasoning on the sixth page of our calculator.

The Impact of Facebook as Support

In addition to creating new vegetarians, online social networks can potentially do a lot of good by preventing recidivism. A survey conducted by Faunalytics found that 10% of adults in the U.S. identify as former vegetarians and vegans while just 2% identify as current vegetarians and vegans. These numbers suggest that the majority of self-identified vegetarians return to eating meat at some point in their lives.

Facebook pages may help prevent recidivism in the following ways:

• Providing education, news, and information
• Posting strategies for maintaining a veg diet, including meal ideas and recipes
• Renewing people’s motivation to eat veg
• Publicizing vegan services, restaurants, and events
• Promoting discussion among users
• Allowing users to ask questions and providing advice
• Fostering a sense of group belonging among veg followers

The idea that these factors may reduce recidivism has some empirical support via surveys of current and former vegetarians. Many former vegetarians cite health factors as the reason they returned to eating meat. Providing Facebook users with health and nutritional information may lessen their health concerns. Research also suggests that social support is important for maintaining a vegetarian diet.

There have also been studies of the effect of online social networks that support behaviors related to health (e.g. diet, exercise, or smoking cessation). These studies often find that online interventions have a small positive effect on behavior.

To assess the impact that a Facebook page has by reducing recidivism, we considered the number of vegetarians who engaged with the page over a period of 28 days and we estimated the average monthly rate of recidivism and the chance that the Facebook page would prevent someone from returning to eating meat.

Upon considering research on the reasons for vegetarian recidivism and the effectiveness of online behavioral interventions related to health, we estimate that a Facebook page with all seven of the support features might help about 15% of the people who might otherwise abandon their diets to stay vegetarian. Pages with fewer support features may be less effective, so we apply discount factors to those pages.

One difficulty with quantifying the impact of veg retention via social media is that veg Facebook users likely follow more than one veg page. If using Facebook prevents someone from going back to eating meat, we would not attribute the resulting impact to one page, but rather to all the pages the Facebook user follows. We recognize this as an area of uncertainty and hope to update it as we gather more data about the people who follow veg pages, but currently tentatively estimate that most pages have fans who follow an average of two other pages. We therefore apply a discount factor of 0.333 to typical pages. For pages with more than 1 million fans and for local groups, we estimate that half of their fans may not follow any other pages, but the other half may follow an average of one other page. We apply a discount factor of 0.667 to such groups.

Other Considerations and Avenues for Future Research

Aside from creating and retaining vegetarians and vegans, there are less direct ways that social media may be used to help animals. For example, animal charities can use social media to solicit donations, post petitions, encourage activism, promote veg products, and generate conversations. We already take some of these activities into consideration when we do our evaluations. For example, if a charity uses social media to gather signatures on a petition, we consider the resulting impact when we assess the effectiveness of the charity’s campaigns which were supported by that petition. We understand that some of these activities may have important effects for animals which we are not currently able to quantify, and we try to account for them in the non-quantitative parts of the evaluation process.

Since there has been very little research to date on the impact of animal-related social media pages, we relied on data about social media in general and about online behavioral interventions from other fields in order to develop our social media calculator. Out of necessity, we also made certain judgments as a team in order to develop some of the estimates used in the calculator. The aim of this post is to be completely transparent about our methodology. We realize that some of our readers may disagree with some areas of our reasoning and/or some of the judgments that we made. Those who don’t fully agree with the way that we evaluate social media impact should keep that in mind when reading the “cost-effectiveness” sections of our charity reviews. We do believe there is room for disagreement about various aspects of our cost-effectiveness estimates, but we remain confident in our Top Charity recommendations. Our quantitative estimates comprise only one of the seven criteria that we consider when making our recommendations.

As the online presence of animal charities continues to grow, we hope that further research will allow us to refine our evaluations of social media impact. We especially hope to find answers to the following questions:

• What are the demographics of the people who follow veg pages on social media? How many are vegetarian or vegan?
• How effective is non-video content on social media at convincing people to go veg?
• How accurate is our estimate of the effectiveness of online ad videos?
• How does a video on Facebook really compare to viewing a video in an online ad?
• How many veg pages does a typical fan of veg pages follow?
• What is the rate of recidivism among vegetarians who engage with veg pages on social media?
• What effect does social media have on other important outcomes, such as inspiring people to become activists?

As we discover new information pertinent to the impact of social media, we will continually work to improve our evaluation process, including our social media calculator.

1. We sampled every fourteenth charity on our list of charities that was published in 2015. (Note that, while the linked list of charities includes the same organizations we sampled from, they are listed in a different order than the published version from 2015.)  

2. Examples of studies on veg messaging include: Humane League Labs’ “Is Animal Cruelty or Abolitionist Messaging More Effective?” and Mercy For Animals’ “How to Dominate Facebook” and “Making Effective Videos.” We don’t necessarily endorse all of the conclusions drawn in these studies. 

3. Note that we are simplifying here by only using changes from non-vegetarian to vegetarian. A large part of Facebook impact probably comes from other diet changes, such as non-vegetarian to reducetarian, but we are counting those in terms of vegetarian equivalents. 

4. By “vegetarian,” we refer to typical vegetarians as well as vegetarian-equivalents, e.g. two people who reduce their meat consumption by 50%. 

5. We did not apply a discount factor to videos with a strong “go veg” message, we applied a discount factor of 0.1 to videos with a weak “go veg” message, and we applied a discount factor of 0 to videos with a very weak or no “go veg” message. More details are available on the second page of our calculator. We checked that each member of the research team arrived at roughly the same discount factor for a given set of videos, but just one research team member did all of the scoring during the review process, to ensure consistency. 

6. See here. Note that the most recent year for which a German-U.S. comparison is available is 2011. 

7. 0.75 x 0.667 + 1 x 0.333 = 0.833 

8. Each member of our research team estimated this discount factor independently and we discussed them to determine which factor to use. You can view our estimates and some of our reasoning on the sixth page of our calculator. 

9. Note: we expect fans of veg Facebook pages, and others in their networks, to be more likely to be vegetarian than the general population.