Conversation with Josh Kalla
Josh Kalla is a PhD student in political science at University of California, Berkeley. He researches the effectiveness of online ads in changing the behavior and attitudes of voters.
The following is a summary of a conversation that took place on March 11, 2016 as part of our investigation of online ads. Josh spoke with Jacy Reese, ACE Research Associate.
Josh’s Research on Online Ads
Josh ran two randomized controlled trials (RCTs), during the 2012 and 2013 election cycles, on the effect of online ads on voter turnout through Rock the Vote. The 2012 study randomly assigned over 700,000 voters to receive Facebook ads or to be in a control group that did not receive any sort of targeted online outreach from Rock the Vote. The study design was capable of detecting a minimum effect size of 0.3% with 80% power. The results showed turnout of 56.485% (treatment) and 56.490% (control), demonstrating no significant effect of the online ads on voter turnout.
Some post-hoc exploratory analysis of the data suggested that among the subset of people who had registered through a previous Facebook ad, run earlier in the election cycle, the later treatment ad was effective in mobilizing votes. It is easy to see why this might be, since those people have demonstrated that they notice Facebook ads and are willing to click on them. However, Josh and his colleagues were wary of false positives, cautious of this result given its post-hoc nature.
In 2013 they replicated the experiment, again with Rock the Vote, but with a much smaller sample size of approximately 9,000 because 2013 was an off-cycle election. They found the same results as in the 2012 study: no significant difference in turnout among voters presented with online ads. This time, that null result held even among subjects who had previously clicked Rock the Vote Facebook ads, confirming that their 2012 “fishing expedition” was just that, and not a robust finding. Nonetheless, this underscores the research utility of online ads because they allow experiments to be replicated relatively cheaply.
Comparing Online Ads to Other Interventions
Taking the literature as a whole, it seems that any effects online ads have are fairly weak on a per person basis. For example, David Broockman published a study in 2012, which Josh regards highly. Its results are generally disappointing for activists interested in using Facebook ads, showing, for example, that while you may be able to increase a candidate’s name recognition, you probably can’t persuade many people to like a candidate or vote a certain way through online ads. Another experiment by Neil Malhotra and colleagues (2012) looked at email communication and found that personal contact from an official, like the county election administrator, increased turnout where emails from Rock the Vote did not. Of course, even if these interventions have a very small impact on a per-person basis, they are so cheap that they could still prove cost-effective.
Why Don’t Ads Have Much of an Impact?
Appealing to psychological research, Josh explains that the harder you can get someone to think about what you’re talking to them about, the easier it is to change their attitude or behavior. The problem with online ads is that they’re easy to ignore. If you skim the headline of the ad, it may remind you to do something you were already meaning to do – go and vote – but it’s not leaving much of a lasting imprint on you. A review by Green and Gerber of the hundreds of experiments that have been done on all different tactics, including email, post, door-to-door canvassing, and phone calls, shows that the more personal the communication the larger the effect size. Canvassing is most effective, then volunteer phone calls, then paid phone calls, then physical mail and email. Josh is currently involved in replicating a canvasing study, and is seeing results consistent with that finding.
How Effective Are Alternatives?
Based on Green and Gerber’s review, on average you generate about 1 vote for every 15 people you talk to when canvassing. For volunteer phone calling, you generate about 1 vote per 35 contacts. For paid phone calling, 1 vote per 125 contacts. Robo-calling yields 1 vote per 900 individuals called. For email communication, there are no detectable effects except when sent by an individual official.
Although impersonal interventions like online ads are far less effective than alternatives like canvassing, they are very easy and cheap, and they scale well; but scale is only valuable if there is some effect. In calculating cost-effectiveness you have to make some calls about what overhead to include in the calculation, but Green and Gerber conclude, with their assumptions, that one vote costs about $31 via door-to-door canvassing, or $63 via paid phone calls, or $91 via direct mail. Since there has been no evidence that online ads are effective for political causes, their cost-effectiveness is unclear.
The Impossibility of Determining Cost-effectiveness
In the context of marketing, Randall Lewis and David Reiley (2014) showed a definitive effect of Yahoo ads on sales through an RCT involving 1.6 million customers. Lewis also wrote on the impossibility of ruling out the cost-effectiveness of online ads. Because some online ads are so cheap, they could be cost-effective with an effect size so small that there aren’t enough people available on the planet to run an experiment with the necessary statistical power. For this reason, Josh is skeptical of the result of any one experiment, especially with online ads, and emphasises the need for lots of replication.
References
Malhotra, Neil, Melissa R. Michelson and Ali Adam Valenzuela. 2012. “Emails from Official Sources Can Increase Turnout.” Quarterly Journal of Political Science 7, 3: 321-332
Green, D. P., & Gerber, A. S. (2008). Get out the vote: How to increase voter turnout. Brookings Institution Press.
Lewis, R. A., & Reiley, D. H. (2014). Online ads and offline sales: measuring the effect of retail advertising via a controlled experiment on Yahoo!. Quantitative Marketing and Economics, 12(3), 235-266.
