October 14, 2016

How I Improved Home Page Engagement by 20.6% with a Simple A/B Test


The client is a Brisbane-based manufacturer of high-quality outdoor umbrellas. The website’s home page currently explicitly states their product’s monetary value by including pricing in the headline. Although this immediately eliminates those who are unable to afford their products, we believe it could be hindering engagement and de-motivating visitors to explore the website (an important step in converting visitors).

Description and Hypothesis

Users wanted to view product images and know specifications, although weren’t clicking the primary home page calls-to-action (based on Google Analytics data). We want to know whether mentioning pricing as the first experience with a brand is a positive one, and whether not mentioning pricing forces users to explore and develop their own judgement of value. Additionally, we want to amplify the effects of our changes by developing a shorter headline that we know users are going to read. The combined effects of these changes will result in improved engagement.


Split Test Comparisons

A/B Split Test Results

Although all tracked goals displayed improvements, we were unable to reach a statistically significant result for all goals due to the sample size. We were however able to attain a statistically significant result for overall engagement on the page, displaying a 20.6% improvement.

Control = 49.67% Conversion Rate

Variation = 59.9% Conversion Rate (+20.6%, 98% sig.)Screenshot of Optimisely

Crazy Egg Scroll Map Results

Results from the Crazy Egg Scroll Map revealed that users who were exposed to the Variation headline scrolled further down the page. Approximately 65% of users exposed to the new headline reached the primary product categories, compared to approximately 35% in the Control.Scroll Depth Analysis

Next Steps

Having a home page headline that states pricing is like trying to sell tickets to a concert without saying which band is playing. Why focus on the price when the band is enough of a draw card? Although the test was successful in improving engagement, we now need to optimise the user’s experience site-wide to ensure visitors are converting.

/> />
/> />