How Should We Evaluate Social Media Impact?

This is the fifth post of our Foundational Questions series. Learn more about this project by reading the initial post Foundational Questions in Effective Animal Advocacy.

Summary

Several of the charities we reviewed in 2015 reported accomplishments based on their social media presence. This isn’t surprising; social media is a large part of animal advocacy and the world we live in. ACE itself includes social media goals in our yearly plans. As an increasing number of organizations point to their social media accounts and associated outcomes as evidence of their accomplishments, we need to understand which indicators of social media success are most meaningful, and how their associated impacts compare to other accomplishments.

The reporting of social media outcomes was more common and prominent in 2015 than in 2014. This is partly due to the ongoing development of social media as a tool for advocates. For example, during most of 2014, Facebook did not support uploading videos directly to posts, so organizations who shared videos through Facebook did so by sharing links or embedding of videos hosted on other sites (especially YouTube). Now uploading videos directly to Facebook is common, and Facebook automatically begins playing those videos in people’s newsfeeds. Some organizations we reviewed in 2015 believed that social media made up a significant part of their impact, including Animal Equality, Animal Ethics, and Mercy For Animals.

We aren’t aware of any concrete evidence on the impacts that animal advocacy through social media has on animals, aside from work that has been done to gauge the impact of online ads.¹ We think that social media outreach is similar in many ways to educational outreach, such as leafleting or pay-per-view events, but these similarities are limited. Social media outreach also comes with a variety of automatically tracked outcomes, such as views, likes, and followers.

All of this means that while there are many ways to estimate the impact of social media outreach, there’s no one way which is obviously the most useful. In the past, we’ve used shares of content as the outcome of interest, comparing them in 2014 to leaflets distributed and in 2015 to online ad clicks, although with some discount factors to account for differences in audience and content. In 2016 we plan to decide how to account for social media outreach before beginning our charity evaluations, so that we have more time to spend on the question and can request information about specific outcomes from charities during our reviews.

In the rest of this post, we elaborate on the metrics available, their pros and cons for understanding advocacy impact, the comparison to educational outreach, and further areas of research.

Social Media Platforms Provide Many Metrics for Evaluating Outcomes

While social media sites can’t provide any direct information about the ultimate impact of an organization’s online outreach on animals, they provide a great deal of information about the number of people who see and engage with content shared on the platform. This information fits into a few basic categories:

Audience size: The number of followers a page or account has. For example, the number of Twitter followers an organization has, the number of likes on an organization’s Facebook page, or the number of subscribers to its YouTube channel. These help determine the number of people who will likely be exposed to each post the organization makes; groups with more followers and subscribers will have a larger initial reach with each post. However, not every follower or subscriber will see each post, because people aren’t constantly active on social media and because some networks have content recommendation algorithms that don’t show every post to every follower.

Views/Impressions: The number of people who have actually seen a particular post or watched a particular video. Social media platforms may also provide the number of unique viewers, the number of views by people who aren’t followers or subscribers, or the number of people who watched a certain amount of a video, such as the first 30 seconds or the entire video. Automatic systems may also provide simple aggregations across multiple posts, such as the number of views an organization’s content received during the month of March. If an organization wants a more nuanced or specialized indicator, such as the number of views their veg*n recipes have received versus the number of views their pictures of baby farmed animals have received, they usually must compute it themselves from the data provided.

Engagement: The number of times a piece of content has been shared, liked, retweeted, clicked on, commented on, or otherwise responded to in ways the social network can recognize. Many of these behaviors also cause the content to be seen by more people. This is the intention of a user who shares or retweets content, but content recommendation algorithms also respond to other engagement by further disseminating the piece, showing it to friends or followers of the people who liked or commented on it. And when a user clicks a link within a post, they’re taken to another page the organization wanted them to see. They also indicate that someone has not just seen the content, but read it or thought about it at least a little.

Choosing Good Metrics to Focus On

The information social media sites provide about reach and impact is limited to what happens on the sites and is fairly universal. They can’t track how people who view posts behave later offline, or directly measure whether viewing posts changes how they think about animals. They also have limited ability to differentiate between the various types of posts an organization might make. We would expect a post containing video footage from an undercover investigation to have a different effect from a text post about a new vegan ice cream being produced, but typically organizations post many kinds of content and collect statistics that don’t separate them.

This presents a challenge for evaluation: there’s a lot of available data, but it isn’t clear how to convert this data to outcomes for animals. Furthermore, different types of content may vary in how the outcomes measured by online systems relate to impacts for animals. Cute animal pictures might get a lot of likes and shares while causing little change, while petitions against cruelty could cause more change and get fewer likes. As evaluators, we need to select metrics that are as close as possible to recording meaningful outcomes, while remaining aware of the distance between the outcomes measured on social media and what we actually care about – the impact on animals.

In 2015, we chose to use the number of shares an organization’s social media posts had received. We liked this metric because it applies to all types of posts (videos, photos, links, etc.) and because it is easy for organizations to provide.

It also indicates strong engagement with the post, which makes it easier for us to understand the user experience. We’re not social media user experience experts, so we don’t know what percentage of post views include the user stopping to interact with the content, rather than just scrolling past looking for something interesting. It’s easier to trust our intuitions about metrics that show deeper engagement.

Finally, shares are a major way that an organization’s content gets shown to people who aren’t already followers, getting the message out to people who might not yet be concerned about farmed animals or animal advocacy in general. Providing social encouragement and information to those already engaged is valuable, but if that were all an organization was accomplishing through its social media efforts, we’d think it was more like fundraising and less like other educational efforts.

We also considered some other possible metrics. One is 30-second video views, since we think these are also fairly strong engagement, and videos seem like especially powerful ways to send a message. We could also use views or other metrics separated into follower and non-follower groupings, instead of trying to use shares to understand how widely posts spread. We decided both of these metrics would be more complicated to use than shares and not worth implementing in 2015, but might use them in the future. Finally, we could ask for metrics differentiated by the content of the post they refer to, to try to account differently for views of very different content, like undercover farm footage and cute animals playing. However, this last plan would require significant work in categorizing posts, which might make it too time-intensive to be worthwhile.

Comparing Social Media Outreach to Other Educational Outreach

Since we don’t have much direct information about the impacts of social media outreach on animals, we use its similarity to other forms of educational outreach to estimate its effectiveness. This allows us to use studies conducted on those methods of outreach to inform our expectations for social media outreach, making them more realistic.

In 2015, we chose to compare social media shares to online ad clicks. This has several advantages:

• both take place online, so viewers are in similar settings,
• both involve some active engagement on the part of the viewer,
• the video shown after someone clicks on an ad is similar to what we’d expect would be the most impactful social media content (videos explicitly opposing animal farming), and
• we had some quantitative data about how people respond after clicking on the ads, used in our online ad calculator.

However, it also has some disadvantages:

• not all social media content is similar to what’s shown with online ads, and it’s probably not all equally effective,
• clicking an online ad takes the viewer to a screen with fewer distractions than their Facebook or Twitter news feed,
• the audience for an organization’s social media content is probably more saturated with animal advocacy content than the audience for online ads,
• the audience for an organization’s social media content is probably already more veg*n and more involved with animal advocacy than the audience for online ads, and
• sharing content is a different kind of engagement than clicking on ads, and they could be differently related to other behaviors.

For our reviews, we estimated the effects of social media shares by multiplying the impact of online ad clicks by discount factors corresponding to the reasons online ads might be more effective than social media outreach. We multiplied by 10% to account for the difference in content, 50% to account for the difference in audiences, and 50% again to account for the increased distractions of viewing content directly in a news feed. This led to an estimate that 0.07 animals were spared per share on social media (mostly Facebook).

Further Areas for Research

We’re planning to spend some time researching social media outreach this spring before deciding what metrics to use on our evaluations for 2016. We hope to get a thorough understanding of the types of metrics organizations are likely to have readily available, so that we can choose the most appropriate for our purposes. We will also consider ways to account for the varied content types that organizations share. Finally, we’ll continue to look for direct information on the effects of social media outreach that would let us use a simpler method of determining its impact.

1. While online ads programs often run on social media platforms like Facebook, we aren’t considering them as social media outreach. The major difference between online ads programs and social media outreach is that social media outreach uses the social relationship between an organization, its followers, and their followers or friends as the mechanism by which content is spread. Online ads programs generally don’t interact with these social networks, instead targeting ads according to demographic information about users held by the platform.