It’s now been just under a year since we switched from Google Content Experiments to the Optimizely platform for AB testing. In that period we’ve launched almost 300 different experiments, across numerous territories and platforms. The majority of those have been A/B, or MVT, tests with a clear focus on conversion rate optimisation. However we have also utilised Optimizely beyond that, making it a staple of our tech workflow. Below are a few examples:
Drip-fed roll outs
Over the past year we’ve launched some exciting features, which integrate heavily with 3rd party API’s. Using Optimizely, we redirected a percentage of users to the pages featuring the 3rd party content. This allowed us to monitor impact on user behaviour, as well as allowing us to quickly turn off the integration in the event of any bugs.
We started the experiment to 10% of users, eventually ramping up to 100%. Once comfortable that the integration worked correctly, we removed the boolean conditions around the content and served it to all users without the use of Optimizely.
Landing page creation
We initially begun AB testing dynamic keyword insertion, and later customising landing pages to match the keyword bidding on through AdWords. By granting access to all in the PPC marketing team, we used Optimizely to customise the landing page based upon query string. An example is below, where we customised the landing page for Spa keywords. This Optimizely experiment was then run at 100% of traffic, to a custom audience targeted on query string.
Secret Escapes runs on a number of servers, which means that even a simple hotfix can take a number of minutes to be deployed. On occasion urgent bugs have been spotted, and in the period of deploying a permanent fix, we have used Optimizely to hide/edit the content.
Once example was with the ad serving platform, which suffered a failure – which resulted in empty blocks of white content on the Secret Escapes site. We reacted quickly to hide these through Optimizely, and used CSS to hide the boxes for 100% traffic until the platform had stabilised. The downtime was only 45 minutes, and would likely have been fixed by the time a hotfix had been deployed, so by using Optimizely we were able to quickly react to ensure there was no reduction to user experience.
Minimum Viable Products
Like most startups in 2015, we generally follow the MVP concept. We look to test and validate our hypotheses, as quickly as possible. Generally speaking, any code which needs to go into the application requires a vast amount testing, in-depth code reviews and if it loses, then all of this needs un-doing. By using Jquery and using Optimizely to serve this, we are able to build, control and track our MVPs significantly quicker. We have used free tools such as Google Forms in conjunction with this to build a complete, polished MVP.
In one example we wanted to validate the idea of whether uses would want to be contacted via phone to complete their booking. We used Jquery to serve a dummy button, which opened a lightbox containing a styled Google Form (worth reading this site for some of the ways you can ‘hack’ Google sheets) which when submitted populated a spreadsheet and sent off confirmation emails to users, hotels and our in-house team. By doing this solely on the front end, we were able to quickly validate the popularity of such a feature.
Once validated, we were able to take our learnings and build a more robust experience for our users inside the SE web application.
Whilst Optimizely is definitely strongest at AB testing, it’s worth considering these things and giving them a go – if they help speed up your Tech workflow.