Analyzing the Hidden Face of Food | Animal Charity Evaluators

When we published our report on online ads, which is now outdated and archived, the best source we found then on the effectiveness of online veg outreach ads was a study conducted by The Humane League in 2011 (The Hidden Face of Food). The study consisted of some analytics data from ads run by The Humane League, together with a survey distributed to people who had visited the webpage and video (that the ad led to) and either ordered a Vegetarian Starter Kit or liked the page on Facebook.

The code that follows is the R code used in our analysis. If you would like a copy of the data used to replicate or extend our analysis, please contact us.

Of 60 respondents who ordered a Vegetarian Starter Kit:

30 stopped eating red meat,
22 stopped eating chicken,
19 stopped eating fish,
14 stopped eating eggs, and
10 stopped eating dairy.

Of 44 respondents who liked the page:

13 stopped eating red meat,
13 stopped eating chicken,
15 stopped eating fish,
9 stopped eating eggs, and
4 stopped eating dairy.

Other respondents reported reducing their consumption of these foods, and many reported that others had also been influenced by their dietary change. But we don’t know how to quantify the impact of those actions, and we think they are more subject to mis-reporting than the figures cited above, which are already subject to response and social desirability biases. (Respondents knew who was giving the survey and what it was about, so they may have been more likely to report if they thought what they had to say would sound good, and they may have exaggerated any changes to sound better.)

We calculate the proportion of each sample that stopped eating each type of animal product.

x y

x/60

## [1] 0.5000 0.3667 0.3167 0.2333 0.1667

y/44

## [1] 0.29545 0.29545 0.34091 0.20455 0.09091

The study team also says that about 31% of respondents who ordered a guide went vegetarian, while 26% who liked the page did. These results are a bit difficult to interpret, especially in light of the leafleting survey for which The Humane League published the raw data, which had many more respondents cutting out single categories of food than going entirely vegetarian. For the group who ordered a Starter Kit, they suggest that everyone who cut out fish also cut out chicken and red meat, or that some respondents already did not eat certain kinds of meat, which raises questions about what part of the effects measured were due to the video. For the group who liked the page, the situation is even stranger:

11/44

## [1] 0.25

12/44

## [1] 0.2727

No whole number of the 44 respondents could reasonably approximate to 26%. In the absence of criteria for identifying fractional vegetarians, we consider it most likely that the 26% is a typo for 29% and the study team chose to consider how many new vegetarians could be “patched together” from the changes reported. In this view, one person who cut out red meat, plus one person who cut out chicken, plus one person who cut out fish is a new vegetarian, regardless of whether the three people were in fact the same person or three different respondents.

We’ll work the same way, considering for each group the minimum of the three numbers for stopping meat consumption, the number of respondents who stopped egg consumption, and the number of respondents who stopped dairy consumption.

We’ll model each behavior change according to a binomial distribution, assuming each respondent in a given group had the same probability of giving up meat, eggs, or dairy as each other respondent. For each group and type of product, we’ll construct a Jeffreys 95% credible interval for the proportion of people who stopped eating the product. This Bayesian interval, based on the non-informative Jeffreys prior, also has the good frequentist property of being equally likely to lie above the true value of the parameter as below it.

qbeta(c(0.025, 0.975), 19.5, 41.5)

## [1] 0.2097 0.4408

This suggests that between 21 and 44% of the people who requested Vegetarian Starter Kits went vegetarian.

qbeta(c(0.025, 0.975), 14.5, 46.5)

## [1] 0.1405 0.3512

Between 14 and 35% of them cut out eggs.

qbeta(c(0.025, 0.975), 10.5, 50.5)

## [1] 0.08904 0.27552

Between 9 and 28% of them cut out dairy.

qbeta(c(0.025, 0.975), 13.5, 31.5)

## [1] 0.1768 0.4400

This suggests that between 18 and 44% of the people who liked the page went vegetarian.

qbeta(c(0.025, 0.975), 9.5, 35.5)

## [1] 0.1063 0.3402

Between 11 and 34% of them cut out eggs.

qbeta(c(0.025, 0.975), 4.5, 40.5)

## [1] 0.03147 0.20180

Between 3 and 20% of them cut out dairy.

We must assemble these estimates into an estimate of how many of the people who clicked an ad (and therefore watched some portion of the video) became vegetarian or stopped eating eggs or dairy. Since we believe our estimates discussed so far are high (due to response bias and social desirability bias) and since we do not have any data about how people who did not like the page or request a VSK behaved, we will assume that none of these people changed their diet. This estimate is almost certainly biased low, but there is no other estimate that we could make with any understanding of its properties.

The study reported that 1.5% of viewers ordered a Vegetarian Starter Kit and 7% liked the page. The uncertainty on these numbers should be very low compared to the uncertainty on the survey results, since they are determined from the full population of viewers during the study period. Thus there is no problem of sample selection, only possible variance between the time period of the study and the time period to which we attempt to apply the results.

Assuming that the population who liked the page and the population who ordered a VSK are distinct, our low, middle, and high end guesses are:

c(0.21, 0.14, 0.09) * 0.015 + c(0.18, 0.11, 0.03) * 0.07

## [1] 0.01575 0.00980 0.00345

On the low end, about 1.6% of viewers cut out meat, 1.0% cut out eggs, and 0.3% cut out dairy.

c(0.32, 0.23, 0.17) * 0.015 + c(0.29, 0.2, 0.09) * 0.07

## [1] 0.02510 0.01745 0.00885

At our best guess, about 2.5% of viewers cut out meat, 1.7% cut out eggs, and 0.9% cut out dairy.

c(0.44, 0.35, 0.28) * 0.015 + c(0.44, 0.34, 0.2) * 0.07

## [1] 0.03740 0.02905 0.01820

On the high end, about 3.7% of viewers cut out meat, 2.9% cut out eggs, and 1.8% cut out dairy.

However, there is nothing about the study design that indicates that the population of people who ordered VSKs and the population who liked the page did not overlap. In fact, this seems unlikely in practice; those who were most influenced by the video would likely have both ordered a guide and liked the page. Therefore, we must consider the conservative estimate where the populations overlapped completely. To simplify matters, we will assume that the respondents who liked the page were from the subpopulation who liked the page and did not order a VSK. This subpopulation, by hypothesis, is 5.5% of the total viewers.

Then our low, middle, and high end guesses are:

c(0.21, 0.14, 0.09) * 0.015 + c(0.18, 0.11, 0.03) * 0.055

## [1] 0.01305 0.00815 0.00300

On the low end, about 1.3% of viewers cut out meat, 0.8% cut out eggs, and 0.3% cut out dairy.

c(0.32, 0.23, 0.17) * 0.015 + c(0.29, 0.2, 0.09) * 0.055

## [1] 0.02075 0.01445 0.00750

At our best guess, about 2.1% of viewers cut out meat, 1.4% cut out eggs, and 0.7% cut out dairy.

c(0.44, 0.35, 0.28) * 0.015 + c(0.44, 0.34, 0.2) * 0.055

## [1] 0.03080 0.02395 0.01520

On the high end, about 3.1% of viewers cut out meat, 2.4% cut out eggs, and 1.5% cut out dairy.

Accounting for our uncertainty as to how much the two populations overlapped, we can estimate that between 1.3 and 3.7% of viewers went vegetarian, between 0.8 and 2.9% cut out eggs, and between 0.3 and 1.8% cut out dairy. Our best point estimates are probably that about 2.1% of viewers cut out meat, 1.4% cut out eggs, and 0.7% cut out dairy.

Resources

The Humane League. (2011). Hiddenfaceoffood.com Facebook ads survey.