Why were the Polls about the 2015 General Election Wrong?

Never one to miss an opportunity to jump on a current affairs bandwagon, some of you may have noticed that there was this general election thing that happened in the UK recently where we voted for a load of people. Some people were quite angry because it turned out that the people who were predicting who was going to win got it a bit wrong.

Many of the pollsters are doing some soul searching trying to establish how they got it wrong. This article aims to help them and explain to you why they might have got it wrong.

If you are wondering why you should care, then this next paragraph is for you. The pollsters who do political polling also do surveys for all sorts of different companies to help them with their strategies and many other things. If you haven’t actively used one in your company, you’ll almost certainly be using data produced by them to help inform your decisions through word of mouth. It’s important that you have trust in them and it is important that you understand their errors so that you don’t repeat them when you do your own surveys.

1. Sampling

You can’t ask all the people in a poll. That’s the point of the election. So instead what you do is take a subset of them, ask them what they are going to do and then use that as a representation of what would happen if you did ask them all.

A large part of doing polling and surveying is trying to make your sample random and representative. Inevitably it will be difficult as those who have strong opinions are particularly willing to impart them if you do a ‘come to me’ approach and a ‘go to them’ approach might mean that you end up lumping those who are ambivalent with those who would actually do whatever it is you are asking them about.

To solve the problem of inevitable bias in the sampling pollsters use weighting (more of that later) so that their sample is representative of reality.

There are three broad methods of sampling at the moment, leading to different issues with the way that the questions are asked (more on that later as well):

1. Internet panels: As used by YouGov (and many others) the panel works by inviting people to sign up to the online panel, then sending a questionnaire invite (usually via email) to a representative sub-set to complete a series of questions. Not everyone will fill one in, so you can’t be sure of an absolute representative sample, so you will still need to weight. The upside is that by doing it via internet you have a great history of the person’s past responses, the downside is that you exclude everyone who isn’t on the internet (or doesn’t like actively participating)
2. Telephone surveys: As used by Com Res (and many others), the pollster takes a random set of telephone numbers from the phone book (plus some mobiles), randomises a couple of the digits at the end and then ring the person up. Not everyone answers, of course and depending on when you call you might end up speaking to a certain demographic more often than another demographic. The upside is that this is much more random (you going to them) and the downside is you obviously miss a growing number of people who don’t have a phone
3. Face to face surveys: There aren’t really any that do this any more for political polling, but you might find one for another type of survey. They are more common these days when taking a small number of people as part of more in depth interview. These were largely phased out after the 1992 election when it was decided that it was impossible to avoid bias in the questioning at this point

2. Weighting

Having accepted that we aren’t going to get a representative sample, the data is subsequently weighted so that it does represent the total audience.

This is usually done in political polling by using demographic data (age, sex, location, who they voted for last time, etc) and then matching it against published data sets to weight up demographics under-represented in the sample and weight down over-represented demographics.

The ‘shy Tory’ that you’ve heard about was largely a result of this weighting. In 1992, as Martin Kellner explains, the weightings were based off census data from 1981 and were largely out of date, resulting in areas more likely to vote Conservative being under-represented in the final weighting.

In 2015 this should no longer be problem. There is a whole plethora of data that is regularly updated that can tell you down to a postcode level the exact levels of demographics.

The second way that political polls are weighted is based on a question that is asked of ‘likelihood to vote’. This is an attempt to weed out those who say they are going to vote one way or another, but are largely apathetic. If you look at the datasets from a poll then you’ll see the question where people are asked on a scale of 0 – 10 how likely they are to vote (eg YouGov’s last survey (pdf warning)), these are then weighted differently by different types of pollster (here is a great summary of that).

This is where it gets tricky for the pollsters – in the example from YouGov they had a turnout of 76% (people voting 10/10 to vote). This is much higher than the actual turnout, however the pollsters tend to weight each party equally here (a 10/10 20 year old male from London to vote Labour is treated the same as a 10/10 70 year old Yorkshire woman to vote Conservative) – whether that is right or not is open to conjecture.

3. Question

The question that you ask is vitally important. Who would have thought it was so easy to ask the same question in different ways? It turns out it is. Here is the question that YouGov asks to get their voting intention weighting and the question that Lord Ashcroft uses:

• The general election will be held this week. On a scale of 0 (certain NOT to vote) to 10 (absolutely certain to vote), how likely are you to vote in the general election?
• Many people say that if there was a new general election they probably wouldn’t vote at all, while others say they definitely would vote. Please say how likely you would be to vote if there was another general election?

If that question wasn’t difficult enough, the question on who you are going to vote for is even more difficult to do and more prone to differences:

• The general election is this Thursday, 7th May, which party will you vote for?
• At the general election on Thursday, which party will you vote for?

Who would have thought it? Just asking the same question in a different manner can result in different results. You can see the effect of this by looking at Ashcroft’s marginal polls (here is Croydon Central (pdf warning)). In this poll Ashcroft asks his general intention question and then subsequently asks them to think about their own constituency. The results are different:

If there was a general election tomorrow, which party would you vote for?

Conservative 31%
Labour 36%
Liberal Democrat 4%
UKIP 23%
Green 4%
Other 2%

Thinking specifically about your own PARLIAMENTARY constituency at the next General Election and the candidates who are likely to stand FOR ELECTION TO WESTMINSTER there, which party’s candidate do you think you will vote for in your own constituency?

Conservative 33%
Labour 39%
Liberal Democrat 4%
UKIP 19%
Green 4%
Other 1%

The same people literally gave two different sets of answers in the same poll to what is effectively the same question. How important is the question? Very!

4. Answer

If you thought composing the question was difficult, imagine what it is like composing the answers that the respondents can give. It’s easier for a general election because there are a finite list of parties, or at least so you’d think.

Towards the end of 2014 the pollsters started prompting for UKIP and in 2015 started prompting for the Green party as well in some cases. Traditionally the answers were just Labour, Conservatives and Liberal Democrats. Was this a good idea? The argument against prompting for smaller parties is always that this will cause their vote share to be overestimated as people will see them as an option, but then change their mind come the actual day (see the question above). The argument for it is the opposite: the ballot paper lists them all in alphabetical orders, so people will see them on the ballot paper and may be more likely to vote for them. Reality is probably somewhere in between (and many pollsters decided that it didn’t make a huge difference prompting or not).

Then there is the question of which order you list them in, whether this makes a difference online (it does), whether it makes a difference in telephone calls (it almost certainly does) and whether there is a difference between the two (there will be). Tradition is that you randomise the order in each survey, but does that have the effect of homogenising the answers to mask actual differences.

Then how do you list the parties? On the ballot paper it has the person’s name and party. But unless you ask them where they live first and have a handy list then this makes it difficult. Plus you then have to make sure you ask enough people in each seat to not end up with a bias based on which constituencies you have (there are 650 mini general elections, not one big one). If you don’t name the individual, will a person remember on the day that there is this one person that they’ve heard of and will vote for them?

5. Intention

There are no right answers to all of the above (which is why there are so many companies who do it with such differing results). Having taken all of that into account, there is a subsequent process of working out intention, that many different pollsters will largely base on the responses users give.

One argument, the ‘shy Tory’ one, is that we should overweight responses from certain demographics because either they are more likely to lie (“I might not vote” but then do) or exaggerate (“I’ll definitely vote” then don’t). The trouble with this approach is that we don’t have an awful lot of evidence to back this up. How do you weight it? Saying you should weight it, without knowing by how much is difficult.

My personal opinion (based on not a lot) is that the errors in the polls were probably caused not by a particular party being more likely to vote when they said they were, but by demographics being more (over 65s) or less (under 35s) likely to vote than they said they were going to. This then translates into different results for parties, because different age groups are more likely to vote than others.

That said, how do you ‘correct’ for that? One suggestion is that the ‘how likely to vote are you?’ question is causing the problems and that may need to change. How? Well that is the million dollar question and the person who answers it will be paid a lot of money by political parties and strategists alike.