Analysis of THL Labs Leafleting Study | Animal Charity Evaluators

This is the Animal Charity Evaluators statistical analysis of the publicly available data from a study performed by Humane League Labs. For more context, see ACE’s narrative analysis of this and a similar study. This document is intended to be read as a supplement to ACE’s analysis.

The code that follows is the R code used in our analysis. If you would like a copy of the data used to replicate or extend our analysis, please contact us.

We used data provided by HLL and formatted by Rob Wiblin to further analyze the results of the HLL study.

Rob names(Rob)

## [1] "ID" "Gender"
## [3] "Male" "Age"
## [5] "Age13to17" "Age18to22"
## [7] "Age23to29" "Age30plus"
## [9] "Red1" "Poultry1"
## [11] "Fish1" "Eggs1"
## [13] "Leaflet" "ReceivedBefore"
## [15] "Red2" "Poultry2"
## [17] "Fish2" "Eggs2"
## [19] "IntentChangeRed" "IntentChangePoultry"
## [21] "IntentChangeFish" "IntentChangeEggs"
## [23] "IntentChangeAverage" "IntentChangeAveragePFE"
## [25] "IntentChangeAveragePE" "Red3"
## [27] "Poultry3" "Fish3"
## [29] "Eggs3" "ChangeRed"
## [31] "ChangePoultry" "ChangeFish"
## [33] "ChangeEggs" "ChangeAverage"
## [35] "ChangeAveragePFE" "ChangeAveragePE"
## [37] "AnyLeaflet" "Why"
## [39] "All" "Cruelty"

First, we checked the impact of receiving any leaflet on our three variables of interest (change in meals eaten with red meat, change in meals eaten with poultry, and change in meals eaten with all animal products (averaged)).

summary(lm(ChangeRed ~ AnyLeaflet, data = Rob))

##
## Call:
## lm(formula = ChangeRed ~ AnyLeaflet, data = Rob)
##
## Residuals:
## Min 1Q Median 3Q Max
## -14.426 -1.426 0.574 1.574 13.574
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) -1.326 0.489 -2.71 0.0068 **
## AnyLeaflet 0.752 0.508 1.48 0.1389
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 3.31 on 621 degrees of freedom
## Multiple R-squared: 0.00352, Adjusted R-squared: 0.00192
## F-statistic: 2.2 on 1 and 621 DF, p-value: 0.139

summary(lm(ChangePoultry ~ AnyLeaflet, data = Rob)) # marginally significant

##
## Call:
## lm(formula = ChangePoultry ~ AnyLeaflet, data = Rob)
##
## Residuals:
## Min 1Q Median 3Q Max
## -12.548 -1.548 0.452 2.452 14.452
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) -1.435 0.524 -2.74 0.0063 **
## AnyLeaflet 0.982 0.544 1.80 0.0716 .
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 3.55 on 621 degrees of freedom
## Multiple R-squared: 0.00522, Adjusted R-squared: 0.00362
## F-statistic: 3.26 on 1 and 621 DF, p-value: 0.0716

summary(lm(ChangeAverage ~ AnyLeaflet, data = Rob))

##
## Call:
## lm(formula = ChangeAverage ~ AnyLeaflet, data = Rob)
##
## Residuals:
## Min 1Q Median 3Q Max
## -36.18 -4.18 0.82 4.82 36.83
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) -3.83 1.21 -3.16 0.0017 **
## AnyLeaflet 2.00 1.26 1.59 0.1121
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 8.22 on 621 degrees of freedom
## Multiple R-squared: 0.00406, Adjusted R-squared: 0.00246
## F-statistic: 2.53 on 1 and 621 DF, p-value: 0.112

There was one marginally significant effect, that respondents in the control group reduced poultry consumption more than respondents who received leaflets did.

We checked the impact of whether the leaflet was focused on why to reduce animal product consumption or how to reduce

summary(lm(ChangeRed ~ Why, data = Rob))

##
## Call:
## lm(formula = ChangeRed ~ Why, data = Rob)
##
## Residuals:
## Min 1Q Median 3Q Max
## -14.406 -1.338 0.594 1.594 13.594
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) -0.6615 0.1842 -3.59 0.00035 ***
## Why 0.0676 0.2663 0.25 0.79977
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 3.32 on 621 degrees of freedom
## Multiple R-squared: 0.000104, Adjusted R-squared: -0.00151
## F-statistic: 0.0644 on 1 and 621 DF, p-value: 0.8

summary(lm(ChangePoultry ~ Why, data = Rob))

##
## Call:
## lm(formula = ChangePoultry ~ Why, data = Rob)
##
## Residuals:
## Min 1Q Median 3Q Max
## -12.634 -1.634 0.366 2.366 14.671
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) -0.671 0.197 -3.40 0.00072 ***
## Why 0.305 0.285 1.07 0.28568
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 3.56 on 621 degrees of freedom
## Multiple R-squared: 0.00184, Adjusted R-squared: 0.000228
## F-statistic: 1.14 on 1 and 621 DF, p-value: 0.286

summary(lm(ChangeAverage ~ Why, data = Rob))

##
## Call:
## lm(formula = ChangeAverage ~ Why, data = Rob)
##
## Residuals:
## Min 1Q Median 3Q Max
## -36.33 -3.76 0.67 4.67 35.24
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) -2.240 0.457 -4.90 1.2e-06 ***
## Why 0.566 0.660 0.86 0.39
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 8.23 on 621 degrees of freedom
## Multiple R-squared: 0.00118, Adjusted R-squared: -0.000429
## F-statistic: 0.733 on 1 and 621 DF, p-value: 0.392

There were no statistically significant effects.

We checked the impact of whether the leaflet discussed all farmed animals or focused on chickens.

summary(lm(ChangeRed ~ All, data = Rob))

##
## Call:
## lm(formula = ChangeRed ~ All, data = Rob)
##
## Residuals:
## Min 1Q Median 3Q Max
## -14.398 -1.335 0.602 1.602 13.602
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) -0.6017 0.1765 -3.41 0.00069 ***
## All -0.0637 0.2686 -0.24 0.81250
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 3.32 on 621 degrees of freedom
## Multiple R-squared: 9.07e-05, Adjusted R-squared: -0.00152
## F-statistic: 0.0563 on 1 and 621 DF, p-value: 0.813

summary(lm(ChangePoultry ~ All, data = Rob))

##
## Call:
## lm(formula = ChangePoultry ~ All, data = Rob)
##
## Residuals:
## Min 1Q Median 3Q Max
## -12.30 -1.61 0.39 2.39 14.70
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) -0.390 0.189 -2.06 0.04 *
## All -0.313 0.288 -1.09 0.28
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 3.56 on 621 degrees of freedom
## Multiple R-squared: 0.0019, Adjusted R-squared: 0.00029
## F-statistic: 1.18 on 1 and 621 DF, p-value: 0.278

summary(lm(ChangeAverage ~ All, data = Rob))

##
## Call:
## lm(formula = ChangeAverage ~ All, data = Rob)
##
## Residuals:
## Min 1Q Median 3Q Max
## -35.95 -3.95 0.91 4.91 34.91
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) -1.910 0.438 -4.36 1.5e-05 ***
## All -0.139 0.666 -0.21 0.84
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 8.24 on 621 degrees of freedom
## Multiple R-squared: 6.98e-05, Adjusted R-squared: -0.00154
## F-statistic: 0.0433 on 1 and 621 DF, p-value: 0.835

There were no statistically significant effects.

We checked the impact of whether the leaflet focused on cruelty reasons for reducing consumption or health reasons.

summary(lm(ChangeRed ~ Cruelty, data = Rob))

##
## Call:
## lm(formula = ChangeRed ~ Cruelty, data = Rob)
##
## Residuals:
## Min 1Q Median 3Q Max
## -14.357 -1.357 0.619 1.619 13.643
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) -0.6186 0.1765 -3.51 0.00049 ***
## Cruelty -0.0245 0.2686 -0.09 0.92741
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 3.32 on 621 degrees of freedom
## Multiple R-squared: 1.34e-05, Adjusted R-squared: -0.0016
## F-statistic: 0.00831 on 1 and 621 DF, p-value: 0.927

summary(lm(ChangePoultry ~ Cruelty, data = Rob))

##
## Call:
## lm(formula = ChangePoultry ~ Cruelty, data = Rob)
##
## Residuals:
## Min 1Q Median 3Q Max
## -12.469 -1.483 0.517 2.517 14.531
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) -0.5311 0.1893 -2.81 0.0052 **
## Cruelty 0.0143 0.2881 0.05 0.9603
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 3.56 on 621 degrees of freedom
## Multiple R-squared: 3.99e-06, Adjusted R-squared: -0.00161
## F-statistic: 0.00248 on 1 and 621 DF, p-value: 0.96

summary(lm(ChangeAverage ~ Cruelty, data = Rob))

##
## Call:
## lm(formula = ChangeAverage ~ Cruelty, data = Rob)
##
## Residuals:
## Min 1Q Median 3Q Max
## -35.98 -3.98 0.91 4.91 35.02
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) -2.017 0.438 -4.61 5e-06 ***
## Cruelty 0.110 0.666 0.16 0.87
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 8.24 on 621 degrees of freedom
## Multiple R-squared: 4.38e-05, Adjusted R-squared: -0.00157
## F-statistic: 0.0272 on 1 and 621 DF, p-value: 0.869

There were no significant effects.

We checked whether the intent to reduce consumption was associated with an actual reduction in consumption.

summary(lm(ChangeRed ~ IntentChangeRed, data = Rob))

##
## Call:
## lm(formula = ChangeRed ~ IntentChangeRed, data = Rob)
##
## Residuals:
## Min 1Q Median 3Q Max
## -14.585 -1.131 0.415 1.415 13.960
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) -0.4146 0.1292 -3.21 0.0014 **
## IntentChangeRed 0.5457 0.0669 8.15 2e-15 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 3.16 on 621 degrees of freedom
## Multiple R-squared: 0.0967, Adjusted R-squared: 0.0952
## F-statistic: 66.4 on 1 and 621 DF, p-value: 1.99e-15

summary(lm(ChangePoultry ~ IntentChangePoultry, data = Rob))

## Call:
## lm(formula = ChangePoultry ~ IntentChangePoultry, data = Rob)
##
## Residuals:
## Min 1Q Median 3Q Max
## -12.788 -1.788 0.212 2.011 14.212
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) -0.2115 0.1471 -1.44 0.15
## IntentChangePoultry 0.4017 0.0639 6.29 6e-10 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 3.45 on 621 degrees of freedom
## Multiple R-squared: 0.0599, Adjusted R-squared: 0.0584
## F-statistic: 39.6 on 1 and 621 DF, p-value: 5.99e-10

summary(lm(ChangeAverage ~ IntentChangeAverage, data = Rob))

##
## Call:
## lm(formula = ChangeAverage ~ IntentChangeAverage, data = Rob)
##
## Residuals:
## Min 1Q Median 3Q Max
## -37.59 -4.13 0.78 4.41 34.87
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) -1.8679 0.3294 -5.67 2.2e-08 ***
## IntentChangeAverage 0.1818 0.0603 3.02 0.0027 **
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 8.18 on 621 degrees of freedom
## Multiple R-squared: 0.0144, Adjusted R-squared: 0.0129
## F-statistic: 9.1 on 1 and 621 DF, p-value: 0.00266

The association was statistically significant, with respondents reducing consumption on average by a respectable percentage of their intended reduction target. But the correlation was weak, explaining in the best case (shown below) about 9% of the overall variation in reduction.

plot(Rob$IntentChangeRed, Rob$ChangeRed)