Integrating Voice of The Customer with Analytics data

After the last post where I got all Philosophical on you, I'm going back to my roots this week.  Partly because I spent half of last week sitting in front of the computer with a bit of writers block.  Possibly accentuated by the fact that I had a cold ("Ahhh, Alec" is what you're meant to say).  Plus I've had to rewrite this opening paragraph a couple of times every time I started afresh.  So what I've decided to do this week is talk a little a bit about what we call 'voice of the customer'.  Web Analytics is great at telling you what it is that users do on the site.  You can monitor what they search for, what pages they look at, where they came from, etc.  If you want to know why they do it though, you have to go to the next step available and ask them.

These two things don't have to be entirely independent though.  In fact, they should feed into each other in a way that is seamless.  Let's start at the beginning and see how you should use your analytics to feed into your surveys (then we'll do the more important bit of doing it the other way around).

I wrote about this (a bit) when I was talking about the EU cookies law, because someone on Slashdot was commenting that you can get your site design right solely by using this sort of methodology.  Really I want to push a bit of a point here.  Your Analytics systems should be telling you about places where you have need for improvement.  Whether that be because of a high bounce rate, or a high drop off.  To fix the problems you have two choices - you can try a continuous improvement process with lots of A/B testing.

Or you can go down a slightly different route and start asking people what they think about the process.  There are two ways of doing this - one is to hire some people and sit down with them whilst they do it.  This will get you limited sample data.  The other method is to put up a survey whilst they are there (or whilst they are bailing out) and ask them then.  These online surveys have been around for years and now many of them are free (and very easy to implement).

Online voting is a step away from asking your users what they don't like about a bit of your site - Intelligent Measurement

This very easy method of tying up the data that you are producing from your systems can give you a huge return.  Whilst doing A/B testing is useful, you need to have some ideas of the things that you are going to change.  Asking the users why they bailed out may show you that the reason wasn't necessarily because of the  wording/pictures/order, but maybe more to do with the length of the survey, etc, etc.  Really you just want to give yourself more ideas to test with.

Voice of the Customer into Analytics

The other thing that you can do with you voice of the customer is link it back in the other way.  And why not. Firstly, you need to think about where you are going to put the data and what you are going to do with it.  For example - if you want to know why a customer is bailing out of a particular transaction at a certain point, then you are going to want to have the survey as an exit survey and you are going to want to know why they left (so that you can do the stuff talked about above).

I'd always strongly suggest working out the reason for the survey first, without just plonking it on the site.  However I am well aware that many people will use the survey as a reasonable measure of user satisfaction over time, by running the same request over and over.  This is particularly true for non profit organisations where user satisfaction maybe one of the KPIs that they monitor.

What you can do though, is collect the data in your web analytics system.  Particularly with tools like Omniture where you have the ability to do lots of customisation of the parameters that you collect and insert data where appropriate.

So you have two options in Omniture.  Either you can create a whole, completely blank, new report suite and dump the data in there.  Remember in Omniture you can set up multiple correlations between your collected variables, allowing you to do some drilling down into the responses that you may not have thought was possible using the tool that the survey provider has available.  Gary Angel has quite a comprehensive method of collecting this data on his blog (so I won't repeat).

The second method that you can use, as Gary also mentions is that you can collect the data in your live report suite.  There are upsides and downsides of this - firstly you are going to be collecting page views that aren't really people viewing your site, secondly you may not have all the s_props available to be able to do this (especially if you have lots of questions).

So good I had to use the gag again

This is why I quite like the method that you can use of uploading the information through SAINT.  Just in case I didn't point it out the first time, you have full correlation ability within your custom traffic variables if you use SAINT.  This is amazingly powerful because it means that you don't have to go through the process of setting up the correlations and you don't have to worry about using up hundreds of custom traffic variables.

The downside of this approach is that you have to be able to get all the data out of your survey systems and link it back to a primary key.  This may be the step that is most tricky.  In theory it is just a spreadsheet with the primary key down the left (ie the bit that identifies each respondent) and the questions along the top with their answers in the middle.  You can then upload that back into your system using SAINT as mentioned above.  In practice, getting this data may be more difficult than just uploading it in the first place.

I'd also recommend using not just uploading the information into an s_prop, but also uploading it into one of your custom conversion variables.  To recap on your custom conversion variables - these work essentially like a series of marketing campaigns.  You input your values and then that sticks with you throughout your visit allowing any custom event (or a sale of your product) to be associated with your original variable.  These are often set up to monitor people who log in or view promotions on the site.  The downside of these variables is that you can't breakdown more than one at a time, but that's probably ok for what we are going to do.  You can, of course, use the same SAINT treatment as above on your variables if you only want to use one of them rather than one for each question.

What do I do with the data when it is in SiteCatalyst?

Well, without wanting to teach a grandma to suck eggs, there are quite a few things that you could do which could give you an immediate advantage.  I'm going to assume that your survey is a random one that is shown to the user at some point in their journey and that you are asking them questions about satisfaction and ease of use of the site (as you probably would do):

• Take you custom traffic variable data and plot your satisfaction against whether a user completes one of your events (eg they used a tool or you sold them something).  Are those that are more satisfied using your tools/buying stuff?  Why aren't the one who are unsatisfied using tools/buying stuff?  How can we show them the benefits of using tools/buying stuff?
• Breakdown your total tools used/stuff sold by not just whether they were satisfied, but where they came from.  Are you mis-selling them something on another website?  Can you contact that website and change the message?  Are the search terms they are typing in showing a different levels of satisfaction/ease of use?  Does it depend on the landing page?  Can we change the landing page?
• Breakdown each of your different types of tools/things sold by how satisfied they were.  Do certain journeys make users more satisfied?  Can you replicate those journeys on the ones that made them less satisfied?

Suddenly we are now going from a stage of knowing how satisfied a user is with the site to know what it was that was making them satisfied/unsatisfied.  Plus we have all that data in the custom traffic variable that we can play with in the normal way too.

I will add one last thing and it is very important.  User satisfaction and the use of a tool (for example) may be correlated, but this is different from causation.  A user may be satisfied because they used a tool or they may have used the tool because they were satisfied.

XKCD - making the world a bit more comic orientated

What is a Web Analyst

"Oh Alec!" I hear you saying, "Are you feeling melancholy and wondering what life is all about?"  No more than normal is the answer that you'll all be happy to hear.  It is a new year, but there is no new whencanistop out there, I am afraid.  Obviously speaking about my alter ego in the third person could be slightly new, but I'm going for the personal interpersonal touch.  If you know what I mean.  And I'm sure you don't , because I've lost it already and we're only on the first post of 2010 and already I'm drivelling in the opening paragraph.  Oh wait - no, that is normal behaviour.

Actually what has really been going on with me is a little discussion that I eavesdropped on at the end of last year (that sounds so long ago, whilst actually only being last week!) that I didn't want to comment on until I had read some of the stuff that has been going on.  So lets start at the beginning.  Or the end, as the case may be.  It started with me picking up a 'conversation' that Stéphane Hamel and Eric Peterson had been having on Twitter around 'models' (in the Business sense, not the entertainment sense :)).  To cut a long story short, Stéphane has created a model (which you can download, read yourself and then pass back comment to Stéphane as I have) which describes how 'mature' or advanced a company is with its Web Analytics.

Kate Moss (not the right sort of model in this case) - Wikipedia

The model describes through six different 'pillars' how mature your organisation is on a scale of 0 to 5 (0 being not at all and 5 being brilliant - to paraphrase).

This also led me into reading Joseph Carrabis' "The unfulfilled Promise of Online Analytics" Part 1 and Part 2.  They are lengthy reads - I'd recommend that you plan your dinner for a bit in the middle.  Yes, I was researching this during my spare time and no I should probably not have been doing.  Part 1, in case you are wondering ponders what the problem is and part 2 ponders what to do about it, using some of the methodology that is picked up by the Maturity model suggested by Stéphane (see there is a point to this).

Anyway, all the pondering led me to an epiphany and I stopped asking What is Web Analytics? and started asking: What is a Web Analyst?  This should be something that is simple to answer for whencanistop given that it is in my job title.  I even proclaim to be one in that little blurb at the top of the screen.


What is Web Analytics?


Stéphane proclaims to be able to define Web Analytics at the start of his Maturity model as:

“The extensive use of quantitative and qualitative data (primarily, but not limited to online data), statistical analysis, explanatory (e.g. multivariate testing) and predictive models (e.g. behavioural targeting), business process analysis and fact-based management to drive a continuous improvement of online activities; resulting in higher ROI.”

Which is, as he quite rightly points out, a bit of an increase on what Wikipedia says about it (although the Wikipedia article looks a bit like it has been hijacked by someone from Nielson) .  Really what we're talking about is collecting data about what people do on the site (by using a Web Analytics tool or by asking them in a survey) and then using it to improve the performance of the site.

What is a Web Analyst


So a Web Analyst (like yours truly) does stuff with that information.  What sort of stuff?  Well we take all that data from the tools, the surveys and any other source that we can find and we turn it into actionable insight (interestingly this last word doesn't appear anywhere in the Web Analytics page on Wikipedia).

A web analyst (the one on the right, the one on the left is a coatis)

So we take all that information and turn it into insight that the Business people can make their website better.  Is that all we do?  Well, no not at all.  When those Business people implement it, we then tell them afterwards whether it has worked or not. In theory, you then use a continual improvement programme of measuring and providing insight and changing.


However actionable insight relies on a number of things.  Any insight can be actionable.  One of the insights I'd probably have made about this blog very early on is that I should have put it on Wordpress.  So I should transfer it to Wordpress, copy all the content across, redirect all the links, etc, etc.  But that isn't going to happen, because the effort and time it would take me to do it isn't worth my while.  The benefits are there, but the pay back from it would probably take a long time to come into fruition.  That means that not only do I have to come up with insights that are actionable, but the benefits have to outweigh the cost quickly.

Not only do I have to work out the benefits, but I have to work out what the cost is to work out if the ROI is there.  How do I do that?  I need to have a vague working knowledge of the systems that we are going to need to change.  If we're making a change to a site, then we need to work out the Architecture of the IT, how competent the developers are (if there are any), whether it will have knock on effects, will there need to be lots of testing, etc, etc.  This means I have to have a quite detailed knowledge of the IT systems (or at least access to someone who does).

Now, I've worked out that the change I'm suggesting has benefit, I've worked out when it has benefit, how much and how long it will take to do, then there is the next step.  I have to persuade the Business that it is more important than any of the other projects that they are doing.  Not least those pesky ones that nobody knows how long it will take, how much it will cost and what the benefits will be.  When you're working with numbers, you can be very precise about the benefit.  When you're suggesting that you turn your logo into a dancing hamster, you can make it up as you go along, because nobody will be able to argue with you through numbers.

So we've got to know:

• The information to provide the insight
• The IT infrastructure and how to alter it (plus all those processes that always go off in IT)
• The Business framework (how you can get funding)
• The current projects so that you can push your project

Basically, it turns out you have to be a Business Analyst as well as a Data Analyst.  No wonder people are getting disenchanted.  Web Analytics may or may not be hard.  Being a Web Analyst is hard.

ps - would you have ever thought you'd see a picture of me and Kate Moss on the same blog post?