Build a Measurement Framework before you do any Analytics to avoid failure

I'm repeatedly amazed by how often people come to me to tell me about their struggles in analytics and it stems from failing to do one basic step right at the beginning. I say 'basic', where of course I should say that this is hard. If it was easy, everyone would be doing it and not starting with this blog to give them ideas on how to do it.

The first step you should do when starting with any analytics is a Measurement Framework.

A measurement framework is something that allows you to work through from your business objectives to show the metrics that you should be showing to your boss to show how well the website is doing.

If you've chosen your metrics before you've gone through this process, the metrics that you use to persuade your management that your website is doing well will either be misleading, or they will lack the conviction that the metrics that older parts of your business use. Either way you will end up undermining your insight and recommendations. That means more work for you and less money for your boss.

I tend to use a bastardised version of the measurement framework that Avinash uses (although most of the rest of my organisation don't - they make it up as they go along depending on what the client wants, likes to see or what day of the week it is). I like this because I think it is a very visual way of describing the framework. I think my favourite phrase as the moment is that you can hang this up on the wall in your office and point to it any time anyone asks why you are using that metric (I've yet to meet anyone who has ever hung one of these up on the wall).

Business Objectives

Working out your business objectives is the hardest part of the process.

Avinash says they should be 'DUMB' (Doable, understandable, manageable, beneficial) - which is the very start of the process.

This is a strategy question - you should spend time and effort working out what it is that you are trying to do with your business before working out whether your metrics are showing it or not.

Working out your business objectives is something that companies spend a lot of money hiring consultancies (including mine) to do for them. If you want to do this on your own, you will find it very beneficial, but you do a measurement framework without it (a framework from a consultancy will cost you more, but will get more buy in).

I quite often do this with clients who don't want to (or don't have the funds to) run proper strategies. This isn't an easy task. You need to ask the right questions and infer an awful lot. You may make an unpopular choice that people disagree with - but stick to your guns.

Good Luck.

Website Goal

The goals of the website (or whatever digital media you have) are how you enact those business objectives. Some organisations will call them critical success factors, some will just not name them at all. Generally they are content and marketing describing what you want your users to do.

These goals may be qualified or unqualified. They may be things that you know will meet your business objectives, they may be things that you want to prove later cause your business objectives to be true (through surveys or econometrics or 'big data') or they may be things that you just assume will meet your business objectives.

I have had several arguments with people in my office about whether you will ever have more than one goal per objective. I think you can possibly end up with very specific objectives, where objectives overlap where you are unlikely to have more than one goal per objective.

Your goals are the sorts of things that you should be able to spot relatively quickly from an understanding of the website (or digital activity).

KPIs

Your Key Performance Indicators (KPIs) are the things that show whether the users are doing the things that you set out in your goals. Your goals, by now, should be close enough that picking your KPIs is a relatively simple job.

KPIs are all metrics. Not all metrics are KPIs. Bounce rate shouldn't be a KPI for your website. Bounce rate is something that describes whether you are going to meet your KPIs or not.

Visits to the site may be a KPI, but I would suggest that if one of your goals is just to get people to the website then you haven't got your business objectives quite right.

KPIs are the name of the metric that you are using to show your objectives. You should be able to say X number of visits (or people or whatever) did my KPI Y.

Targets

Targets are the things that say whether the value that you get for KPI Y is good or not. Knowing that you got 200 downloads is all well and good, but if your target is 500 then that represents a poor return, whereas if your target is 50 then it represents a good return.

Creating targets, however, isn't as easy as it sounds. It may be relatively simple as looking at the profit that you would get from doing one of Y, but that only works if you've got the most obvious sales line possible and only one product.

What you can do is look at historical performance to help you build targets. You can look at your conversion rates based on the spend that you are running. You can put a finger in the air and then change it after three months when you realise that you aren't going to make what you wanted.

Just like your work life, you should create targets that are SMART, particularly focussing on the attainable and time bound bits.

Segments

When it really comes down to it, what you really want to know is what you can change to make it so that you can hit your targets for your KPIs.

This is where your segments come in. These are the things that you report that breakdown your KPIs into smaller groups. For example Marketing A drove z of KPI Y, Marketing B drove w of KPI Y.

If you can identify these things up front, then it makes your job as an analyst much easier, because you know where you can start your analysis. It means you can ignore stupid things like what browser the person was using or what version of JavaScript they have on their computer.

Web Analytics Tools

Of course I am biased, but one of the great things about the Measurement Framework is that I can use the thing to set up my analytics system. You make sure that your goals, or custom events or custom metrics (or whatever your tool calls custom metrics) are set to be your KPIs. You make sure your segments are set up as custom dimensions, custom conversion variables, custom variables (or whatever your tool calls them).

Now when someone says I want so and so report from the tool, you can ask them why and what they are going to do with it. If it is something simple then you can say "Yeah, the tool does it out of the box - here's how you find it." If it is something complex then you can challenge the person as to why they really want this.

Different Industries and Business Models

Every industry thinks that they do this well or badly. The truth is that the number of times I've seen this done well I could count on the fingers of one hand regardless of industry and business model.

Those in industries where you sell something then this seems like it is an easier job, because you can talk about sales. But that isn't the only point of your business and the hard sell can sometimes compromise things like persuading people to come back in the future, leaving comments about products, talking to their friends about the company, etc.

Those in industries where they don't sell anything find this very difficult because of the lack of direct link between website activity and sales. But that sometimes mean that they have a much tighter control over what they are trying to do on their site.

What WebTrends should do for Analytics version 12

I've been in my current job for about a year now and I've been working quite a lot with WebTrends (as well as Google Analytics and all sorts), but this week I've been working on SiteCatalyst with a client again. It reminded me of a post that I wrote back at the end of 2010 on what SiteCatalyst should do for version 15, so I thought that I would write one for WebTrends as well.

WebTrends is currently on version 10.6, with the last major release (version 10) being in April 2011. Version 10 is a superficial improvement in the user interface over version 9 ('the admin interface'). This lack of update in the last couple of years to their flagship product (if it even is any more) means that WebTrends is falling behind SiteCatalyst (see who would win in a fight between SiteCatalyst and WebTrends) and Google Analytics. Therefore the point of this post is things that WebtTrends could do better to encourage people to use their Analytics tools again.

1. Live Segmentation

WebTrends can do segmentation and it can do it really easily. You can apply filters to individual reports in the backend of the system and you can apply whole filters to entire profiles.

This is very powerful.

But it is misused. At the moment it is used to create reports that can be used on a regular basis. This is reporting.

Segments should be used to allow analysts to create analysis for insights and recommendations. This is one of the pillars of the analytics maturity model for organisations. At the base level you are producing reports, at the more advanced levels you are doing analysis. If you have to know in advance of your users doing things that you need the reports then your not doing analysis, you are doing reporting.

The trouble WebTrends has is that the people using Analytics tools have changed in the last 3 years and WebTrends hasn't. You need to be one step ahead of your audience.

Google Analytics does it, SiteCatalyst version 15 does it, WebTrends should do it.

2. Standard Custom Measures in any report

Setting up measures in WebTrends is one of the most annoying things ever.

I want to be able to code on the page WT.metric1=true and metric1 will appear in any report that I want it to.

I don't want to have to go into custom measures and create a new one. 

I don't want to have to add it to every report in the back end. 

I just want it to be an option that appears in all reports when a user wants it to. It may not make sense with the visits and the page view based measures, but I'll let the people looking at the reports work that one out.

Currently the barrier to adding measures to reports means that I have to spend too much time up front working out which measures work in which report. I have to set up measures that are visit based or page view based depending on the situation and if I have too many of them then it ist5 annoying to have to create two reports. Just let the users add them as they want.

3. Second dimensions on the fly

SiteCatalyst does it, Google Analytics does it, even educated fleas does it.

You have two reports. One has, say, the keywords that you are bidding on in your search engine. You have another report that shows the search terms that you pick up from the referrer field when a user clicks through. 

Want to break one down by the other? 

Tough, unless you set it up in advance.

However, it turns out that you want one dimension broken down by five different ones in various different ways. That shouldn't need five individual reports, it should need one report that you can add a second dimension to on the fly.

Google Analytics does this in a way by allowing you to break each report down by a subset of the other reports, depending on which one it is. This would seem like the safest solution for WebTrends, but WebTrends has lots of custom reports so a SiteCatalyst style breakdown might work better in practice.

4. REST user interface

Rest is a great idea. But it feels like someone has implemented it in a real half arsed way.

It's great that you can just download data from reports by using the API so that you can create your own interface. You can put the data in your third party systems much more efficiently and you can put data into Excel however you want.

What Rest really needs is a GUI for people like me who don't want to have to bother with coding stuff on pages. What Rest needs is a Report Builder.

Why does Rest need to do this? Because Rest has been lost in a past that no longer exists in most companies where the analytics tool is looked after by the IT team. The Analytics tool should be looked after by a Marketing team (or the end users of the reports, who are usually Marketing). Because of this the way of getting data into excel should be in the most simple fashion possible.

5. Improved Dashboards

Dashboards in WebTrends are frankly quite poor and probably unusable.

I wouldn't recommend any of my clients use them and instead export the data themselves into Excel to manipulate there. I think it would be rare that you wouldn't advise even the basic user to manipulate some of the data in the tool itself.

This is probably a symptom, slightly, of some of the other shortcomings of the tool rather than a symptom of the dashboards themselves.

I could probably write a whole blog post on things that could be improved in the dashboards, but starting with:

• Simple filters of the reports
• Adding more than one measure to reportlets
• Having more than five lines in the reportlets
• Sharing the dashboards with other users
• Adding in custom stuff (like a box for normal text to explain stuff)

6. Targets

I've spent the last twelve months telling clients that they need to set KPIs that they know reflect their business objectives to build confidence in what they are doing.

Having set KPIs they need to give targets to those KPIs so that they can tell whether they are performing well or not. 

The favourite question is "What does good look like?" Good looks like hitting your targets for your KPIs. How do you know whether you've hit them or not? You have uploaded them into your analytics tool and the reports that you get in your newly created dashboards show how you are performing against them.

What to do with the demographics of your twitter followers with demographicspro

When I got an email in my inbox from Schmap the other day about a new tool they had released I thought I was going to leave it sitting in my junk box. In reality I sent it around the office to everyone telling them about it and asking if they had any clients who might want to use it. We probably won't do, because of the nature of most of our clients (they work in regulated markets with few twitter accounts). But you guys could be interested in it, if I tell you what it is and how it works.

What is this new tool?

Demographicspro is the name of a new tool and was launched last week. The tool costs money, but not very much - if you've got a tiny account like mine then it could cost about $20 for a one off analysis, in fact if you've got anything up to 10,000 followers it will cost as little as $80 for a one off analysis. You can, of course, buy a plan for regular reporting to show how your demographics change over time.

How does it work?

The company says it uses algorithms to crunch proprietary data across four general focus areas -- network, consumption, language and physical appearance -- though it doesn't say where it comes from. At any rate, it's better than what Twitter offers brands today, and it spits all of this information out into your choice of an Adobe PDF report or Microsoft Excel spreadsheet.

says Andrew Nusca over at ZDNet. Translating this into English it means that they use meta data about tweets (if you say where you are or what tool you are using, for example), information in the tweets themselves and any information entered into the profile of the twitterer.

My guess is that there are a number of standard things that if you put in a tweet relates you to groups of demographic information - use of hashtags, use of language in tweets, etc, etc.

What data do you get?

I'll give you some information about my Twitter followers (@whencanistop, in case you didn't know!), just to get you started.

Here you can see some information about my followers that shows their gender, marital status, age and income. In the middle it shows what percentage of my followers fall into each of the demographics and on the right hand side it shows the entire twitterati.

The most important bit of information on the chart is that little black dot on the right hand graph. It shows you how you compare to the entire world. The further right the dot is (in the dark red) the higher the representation that group presents of your followers compared to the average. anything that is on the far left in the yellow is much less represented.

My followers are more likely to be male (this is hardly surprising in this industry), more likely to be married and/or have children than average, in their late 30s and high earners. If that describes you and you are a follower, well done.

To give you a quick flavour of some of the other bits of information you'll get, above we have some information on the location of the followers, what the interests of followers (77% are interested in technology - I think that someone could do with breaking that large group down a bit in future developments) and what occupation the followers have:

Is it any surprise that so many are into 'Sales/Marketing' (does 'Sales' as a job even exist any more? And if it does, these people are surely really different from Marketing people that it isn't worth grouping them). It is nice to know that I'm also connected to a lot of Senior Managers - that should make it easier to sell my services over Twitter (although I do use my twitter solely for professional purposes, so maybe it isn't so much of a surprise).

We also have information about which people your followers are likely to follow too (Pete Cashmore appears at the top of the list - he must be very popular around my groups!). We also have information on the sorts of Brands that my users are interested in:

These bits of information are probably found from various bits of data about check ins, tweets and generally linking demographics of people to the demographics of the sorts of brands they like. Here it is more important than ever to look at the dot to see where you sit compared to the global audience. KFC appears high on my list, but my users are far less likely to like KFC than the whole world. Me tweeting "@KFC, isn't it great! #chickenforlunch" probably wouldn't go down so well.

Finally we have some information about how long these people have been on twitter, how many followers they have, how many they follow and their daily activity. Knowing how often my followers tweet gives me some insight into how often I should be tweeting too (my first tweet was just over 5 years ago and I've tweeted almost 1,500 times, so once a day makes sense).

How can a Brand use this data?

I see there being five ways to use this data:

1. Tone - you can use information about your followers to dictate the tone of your tweets. You may think you know about them, but you may not necessarily have the level of detail that allows you to decide what tone of voice you are going to use.
2. Compare to peers and competitors - use the information that you gain about your brand and compare to peers and competitors by running the analysis on them too. You can gain insight from comparing your tactics to theirs and the results of the followers.
3. Work out what brands you might want to partner with - knowing other brands that your users are interested in may allow you to build strategic partnerships with 'upstream' or 'downstream' brands - things that are related to your product, but aren't direct competitors (eg if you sell socks then you might want to partner with a shoe company).
4. Use the general interests of your users to create personal tweets - if you know what users are interested in then you'll know whether to make tweets about events (eg my followers are universally uninterested in music, tv or film - so I know not to tweet about these things, but tweeting about new technology or social media is encouraged).
5. Frequency and timing of tweets - knowing where your users are and how often they tweet will let you know when you should be tweeting for maximum visibility and how often to get the users attention

Remember though that your overall twitter strategy should integrate with your overall strategy and should drive real world outcomes. You get data from this tool and you need to analyse it to get insight that you can use to change stuff. You then need to use your standard measures to show the impact of these changes. You can start using this data as part of your metrics, of course, and run this analysis on a regular basis.

6 steps to get the most out of AB or Multivariate testing

With the risk of teaching grandmothers to suck eggs, I'm going to spend a bit of time today talking about A/B testing and multivariate testing. Last time I wrote about this was five years ago when I wrote about how you could do A/B testing to improve engagement on sites. In five years lots has changed: I have grey hairs and A/B testing is a lot easier to do than it was back then. You have free tools and you have cheap tools that are readily available to you that you just didn't have back then. Now is the time to make use of them.

However the processes in those last five years haven't changed.

1. Identify the Need

First step on any part of your project should be to look at the data on what needs improving. This means doing the part of the process that looks at the analytics. Working out where the problems are arising based on your data.

That means looking at conversion funnels (in Google Analytics or Omniture or whatever) and working out where the pain points are. By going from your macro analysis to your micro analysis you should be able to find places where changes would improve things.

Of course (and typically) many times identifying the need is based around Hippos. This is where someone high up in the organisation says "I think this would work better, let's put it on." Or where there are disagreements between stakeholders on the best way to put something forward.

In these cases you can definitely step in and get your users to decide which version works better.

2. Decide what sort of test you want to do

Effectively you can do three different sorts of tests and you need to choose at around about this point which one is going to be the best option for you.

A/B/X test. These tests involve you putting forward one or more versions of a part of a page to test to see which one works. This might be a button, it might be a banner or it might be three quarters of the page.

Multivariate test. These tests involve you putting forward one or more versions of more than one part of a page to test to see which combination works best. This can be buttons, banners, headlines, copy, etc.

Split URL test. These tests are somewhat similar to an A/B/X test, but usually you have radically different pages, rather than just a different element within a page.

3. Design decisions

Doing A/B or Multivariate testing isn't an excuse to cut back on design work.

You should do all the usual things that you would normally do when designing a page (or a journey). That means doing wireframes, looking at branding, making sure the colours fit with each other, the text is consistent you've thought about scent and your persuasion architecture.

What you should definitely do, if you have the budget available to you, is to test your designs with real users. I remember being at a conference and hearing someone (whose face escapes me at the moment) who suggested that at worst you should give it to your mum to have a look at.

4. Choose your conversion point(s)

Arguably you should do this before you start designing, because you would use it in your design decisions, but having come up with your designs then you need to work out what it is you want your users to do.

This may come back to your conversion funnel, but it is an important stage in the process. Too often the you see the conversion metric being click through to the next page, when in reality you want people to continue through to a sales page or sign up to a newsletter or whatever. If your metric is click through rate then the best solutions will be "Free Money" in large letters.

Having come up with your key conversion points you might want to rank them so that you can decide if one of them is increased and another is decreased. Or you could apply values to each one so that when you add them up you come up with a good statistical model. This is always easier when there is a real monetary value associated with the conversion, but it doesn't mean that you can't give arbitrary values (eg 5 for a Newsletter sign up and 10 for a responding to a classified).

5. Choose Your Tool

There are literally hundreds and you could pay anything from nothing to a fortune.

Choosing a tool depends on a whole host of things, including which Analytics tool you use (you want the integration to work), how complex the changes to the site are, how many tests you think you are going to do and how many people you want to be included in the test.

As a good starting point you have two cheap tools:

Google Analytics Experiments (nee Google Website Optimiser) - this is a great little tool if you have a simple test and only one conversion point. You put the two pages live and the JavaScript on the original page and off you go. Great for simple sites, but anything with more than one conversion is going to be scuppered.

Visual Website Optimiser - this tool is what we usually use as the entry point for a site with smaller volumes of visitors. You can do all the testing types as above, you can do it to certain user types and you can have more than one conversion point set up. We've also set it up to send which version is being shown into Omniture as well as Google Analytics (and you can probably do it with WebTrends and all the other analytics tools too).

Maxymiser or Autonomy (nee Optimost) - these are more than just tools. They tend to be managed services that help with all of the processes above. They're good for companies that know they need to do something, but don't have the analysis or design skills.

Test and Target or WebTrends Optimize - these are the enterprise level versions of the tool that integrate seamlessly. You use these if you are spending money with those companies already, have large numbers of page views. They can of course both do targeting as well as testing as a primary function of the tool. These are designed for companies who want to do optimisation on a long term plan, rather than a one off project.

6. Test!

The last step is for you to just start testing! Get enough people through your pages that you can statistically say which gives the best conversion rates. Then iterate and do it all over again.

What should managers do with their Analytics function?

Whilst at Measure Camp a couple of weeks ago I attended one session entitled 'Does anyone outside this room care about Web Analytics?' At it, I posited that many people's dissatisfaction with organisations was just a misinterpretation of a maturity model put forward by Stephane Hamel: they were actually just seeing the impact of deficiencies in other areas and that managers were getting the brunt of it. I was, however, rebuffed on this somewhat. Whilst discussing this in the office a couple of days later I realised the reason why. Managers would not be doing their job properly if they were the most advanced of all the disciplines.

So here we have the big guide to how you should manage analytics if it is your responsibility. Previously when talking about the maturity model, I did a post on the sorts of things that I thought you should know if you were a Web Analyst.

This time I'm not going to assume you know anything about analytics, but I'm going to assume you can work out where you are as an organisation by putting into place the model. But before we do that, here are the basics of the maturity model:

The model takes six facets of analytics that would lead an organisation to be able to optimise in the most efficient manner. You then work out how mature you are in each area, on a scale of 0 - 5, and it leads you into a little graph like the one above.

If you are equal in all areas then you are optimising in an efficient manner, but you could be getting better value for your money by being more mature. If one of your arms is more mature than the other then you will be wasting money by not being able to use it to its full potential.

The classic examples are: buying a tool that is all singing and all dancing whilst not having anyone in place to look at the data; or not having the processes in place to enact the recommendations that your tool and experts recommend because you need to wait 12 months for the next development cycle.

So what are the six facets?

1. Are you optimising with respect to your business objectives? Whilst this seems like a yes/no question, rather than a 0 - 5 scale, but really it isn't. You can go from nothing to being simple channel objectives, or overall organisational objectives.
2. Do you have the processes in place to make the recommendations that the analytics suggests a reality? This is a a scale from a large waterfall process through Agile to multivariate testing on the fly (and it doesn't just have to refer to the technical side, but also to your marketing or organisational processes)
3. Do you have tools in place to give you the right levels of data? We quite often call this 'Data and Tools' because you can have a great tool that gives you crap data. The scale here goes from nothing through basic tools, basic reports through to fully integrated report sets for all levels of the organisation.
4. Do you have people on the ground to do the analytics? Avinash always recommends the 90/10 rule (for every £10 you spend on technology, you should spend £90 on people doing stuff with the data). The scale runs from nobody, through part time work, through multidisciplinary teams to experienced and empowered individuals.
5. What are the things that you do analytics on? This can start as something as simple as a single tactic (eg email) through to the whole web presence and finally all the way up into multichannel.
6. Finally, where does the responsibility for this sits in a management team? Does it sit with a project manager or does it go all the way up to the top of the organisation.

You as a manager should work out where you are in each of these sections and then evaluate your own presence too.

You should always try and be even in all areas and you as a manager's job is to ensure that you bring each of the other areas up to the appropriate level.

But equally as a manager you should know that you shouldn't waste your time fighting battles that you're not going to win and transformation frequently is one of those. Therefore as a management team you should ensure that you are equal lowest of all of those.

I'll repeat that - to be a good manager, if you have hold of analytics you should be equal worst in this list.

This is the reason that so many analytics people end up complaining about management. It isn't so much that management are holding things back, it's that they know if it wasn't them, then something else would be the road block in an almost identical fashion.

So you're a manager and you want to work out where everything is on that scale. How do you do it?

1. It would be remiss of me, as the head of a function at a consultancy that I didn't say 'Hire a consultancy'. But I think this is a route that only people who are very serious about implementing recommendations that a consultancy make. These changes are likely to be organisational and they are likely to impact your job. The price won't be cheap. What you will get is a company that will 'interview' various members of your team and those around you to find out what you do on a day to day basis. They will look at the implementation of your tools. They'll look at your reports. They'll look at the decisions that you make from those reports. Then they'll workshop what you should do to bring yourself up to a level playing field. It will have credibility because all teams will have been involved in setting up a series of actions and a timeline of when you want them to be in place.
2. You can do the above in an internal manner. You'll get a biased view and you may lose credibility of the results because they'll seem to be sponsored by one part of the organisation.
3. You can do a simplified version of the above. I quite often do this with teams that I work with - we'll sit in a room with a couple of people who work with a client, I'll go through the levels and we'll work out where we think an organisation is. Then we'll come up with a couple of simple ideas that will push the organisation up in an area we think they're deficient in. It stops us just repeatedly offering the same service over and over where it might not be applicable.
4. You can listen to snake oil salesmen whose job it is to sell you their product (I jest, but quite often you'll find companies try and push a product, particularly tools, when the reality is that isn't the reason that you aren't taking advantage of what you currently have).

Is the ball in your court? Yes, because you're the one in charge of analytics. Don't expect it just to happen.