A/B Testing

As part of my internship with Animal Charity Evaluators I recently helped to complete a preliminary round of A/B testing on their website. In what follows I give an overview of A/B testing, briefly delve into how to do it on a website, describe the eye-opening results and also express personal views about why I think that it’s such an important thing to do.

In essence, A/B testing compares the effects of two methods. What makes it a particularly good technique to do so is it randomly assigns participants to the control or the experiment group. As a result of this random assignment statistically significant differences found in the two groups can be attributed to the differences caused by the experimental intervention rather than variation amongst participants.

A helpful way of thinking about A/B testing is to note its similarity to the randomized control trials that are pervasive throughout the scientific literature. In fact, A/B testing is just adapted business language for the randomized control trial experimental method. Consider that for a moment: the golden standard of scientific evidence which has taken our approach in medicine from blood sucking leeches to eradicating polio and continues to drive advances in knowledge can be applied by the Animal Rights movement to things such as website optimization and perhaps even informing intervention selection.

I mention informing intervention selection because in the past Animal Charity Evaluators used a randomized control trial to explore the effect of leafleting and I eagerly await the results of Mercy For Animals’ randomized control trial exploring the effect of online advertising to be released. If this powerful scientific method is properly harnessed and applied to website optimization and intervention technique selection then the effects for many animals would be substantial.

Applying A/B testing to websites

Before going further let me stress that I found that it easy to initiate A/B testing in Google Analytics. In total it took me only a few hours and I feel that if I was to do this process again then I could do it in less than an hour. The main reasons for this were that Google Analytics has an easily understandable built in interface for A/B testing and there are also myriad easily accessible helpful guides.

When using A/B testing to improve a web page’s performance you must construct a variation page that is to be tested against the original page. Differences between the variation page and the original page that could be tested in an experiment include but are not limited to changing or adding:

• Headlines and headers
• Images and icons
• Text
• Calls to action
• Page layout

It is important that the variations being tested be bold and easily noticeable so that the results are conclusive. There is little value in testing some small aspect of the web page as it will be unlikely to result in statistically significant results and the effect size of such a variation would likely be small.

The relevant data that can be used to judge the performance of the pages being tested and determine which is in fact the better page include but are not limited to differences in:

• Bounce rates
• Conversion rates
• Average session duration
• Pages per session

The best metric or metrics to be used in determining web page performance will depend on the specifics of the page being tested. For instance, the best performance metric on a donation page may be donation conversion rate whereas the best performance metric on a page describing research may be the average session duration of visitors.

It is extremely important to pre-commit to what you are measuring in your A/B test and what your reaction to certain experimental results will be. Otherwise humans have an incredible ability to ignore what they actually see, in favor of (a) what they expect to see and (b) what they want to see. If there is no pre-commitment then be acutely aware of confirmation bias and the associated phenomenons of publication bias and p-hacking for statistically significant results.

Lessons learned

An aspect that I found fascinating about the A/B testing process was how marvellously wrong I was. Prior to the experiment I confidently predicted that the variation page, to which we had added an image and re-ordered the text on the page, would easily outperform the original page and I publicly expressed this fact in pre-test meetings. The experiment not only refuted this hypothesis but the data showed that on the metrics that we assigned the most importance the original page was ~30% better. This was a large effect that occurred at a statistically significant level.

Therein lies the true beauty of A/B testing. Even though I was confident that the changes made would be a significant improvement, by actually putting this belief to the test I found it to be false. Instead of causing a decrease in website quality, because of A/B testing I was able to maintain the existing quality. A/B testing means that instead of relying largely on personal experience coupled with questionable application of rules of thumb or intuition we can actually test different options and update our opinions based upon the relevant data. As such A/B testing is a powerful approach that can be used to determine the means by which animal advocates can most effectively achieve their ends. For that reason the Animal Rights movement can only benefit by informing key facets of its approach with A/B testing.

Kieran Greig has been a Communications Intern at ACE since May 2015, where he has implemented strong analytics and reporting practices to measure the impact our on online communications. He is also an employee at Charity Science.