6 steps to get the most out of AB or Multivariate testing
With the risk of teaching grandmothers to suck eggs, I’m going to spend a bit of time today talking about A/B testing and multivariate testing. Last time I wrote about this was five years ago when I wrote about how you could do A/B testing to improve engagement on sites. In five years lots has changed: I have grey hairs and A/B testing is a lot easier to do than it was back then. You have free tools and you have cheap tools that are readily available to you that you just didn’t have back then. Now is the time to make use of them.
However the processes in those last five years haven’t changed.
1. Identify the Need
First step on any part of your project should be to look at the data on what needs improving. This means doing the part of the process that looks at the analytics. Working out where the problems are arising based on your data.
That means looking at conversion funnels (in Google Analytics or Omniture or whatever) and working out where the pain points are. By going from your macro analysis to your micro analysis you should be able to find places where changes would improve things.
Of course (and typically) many times identifying the need is based around Hippos. This is where someone high up in the organisation says “I think this would work better, let’s put it on.” Or where there are disagreements between stakeholders on the best way to put something forward.
In these cases you can definitely step in and get your users to decide which version works better.
2. Decide what sort of test you want to do
Effectively you can do three different sorts of tests and you need to choose at around about this point which one is going to be the best option for you.
A/B/X test. These tests involve you putting forward one or more versions of a part of a page to test to see which one works. This might be a button, it might be a banner or it might be three quarters of the page.
Multivariate test. These tests involve you putting forward one or more versions of more than one part of a page to test to see which combination works best. This can be buttons, banners, headlines, copy, etc.
Split URL test. These tests are somewhat similar to an A/B/X test, but usually you have radically different pages, rather than just a different element within a page.
3. Design decisions
Doing A/B or Multivariate testing isn’t an excuse to cut back on design work.
You should do all the usual things that you would normally do when designing a page (or a journey). That means doing wireframes, looking at branding, making sure the colours fit with each other, the text is consistent you’ve thought about scent and your persuasion architecture.
What you should definitely do, if you have the budget available to you, is to test your designs with real users. I remember being at a conference and hearing someone (whose face escapes me at the moment) who suggested that at worst you should give it to your mum to have a look at.
4. Choose your conversion point(s)
Arguably you should do this before you start designing, because you would use it in your design decisions, but having come up with your designs then you need to work out what it is you want your users to do.
This may come back to your conversion funnel, but it is an important stage in the process. Too often the you see the conversion metric being click through to the next page, when in reality you want people to continue through to a sales page or sign up to a newsletter or whatever. If your metric is click through rate then the best solutions will be “Free Money” in large letters.
Having come up with your key conversion points you might want to rank them so that you can decide if one of them is increased and another is decreased. Or you could apply values to each one so that when you add them up you come up with a good statistical model. This is always easier when there is a real monetary value associated with the conversion, but it doesn’t mean that you can’t give arbitrary values (eg 5 for a Newsletter sign up and 10 for a responding to a classified).
5. Choose Your Tool
There are literally hundreds and you could pay anything from nothing to a fortune.
Choosing a tool depends on a whole host of things, including which Analytics tool you use (you want the integration to work), how complex the changes to the site are, how many tests you think you are going to do and how many people you want to be included in the test.
As a good starting point you have two cheap tools:
Visual Website Optimiser – this tool is what we usually use as the entry point for a site with smaller volumes of visitors. You can do all the testing types as above, you can do it to certain user types and you can have more than one conversion point set up. We’ve also set it up to send which version is being shown into Omniture as well as Google Analytics (and you can probably do it with WebTrends and all the other analytics tools too).
Maxymiser or Autonomy (nee Optimost) – these are more than just tools. They tend to be managed services that help with all of the processes above. They’re good for companies that know they need to do something, but don’t have the analysis or design skills.
Test and Target or WebTrends Optimize – these are the enterprise level versions of the tool that integrate seamlessly. You use these if you are spending money with those companies already, have large numbers of page views. They can of course both do targeting as well as testing as a primary function of the tool. These are designed for companies who want to do optimisation on a long term plan, rather than a one off project.
The last step is for you to just start testing! Get enough people through your pages that you can statistically say which gives the best conversion rates. Then iterate and do it all over again.