Four Lessons We Can Learn From Polling Mistakes
In the aftermath of Trump's shock election victory marketers should be concerned. As worrying as the man's policies are I find myself thinking more about the fact that the polls were wrong yet again. That's three swing and misses in a row: 2015 general election, Brexit and now Trump. Considering our industry is still heavily reliant on consumer-facing market research should we not be really fretting that detailed polling has failed to predict three (more or less) 50:50 decisions in a row?
Asking people what they think and how they will behave is always a risk. It's certainly worth doing, don't get me wrong, but to put everything on consumer say so is always a gamble and here's an overly cynical view as to why:
You'll ask the wrong people
They'll lie to you
They don't know how they will behave
You won't listen to them anyway
The last three major political votes that will affect the U.K. have all been predicted wrongly
Of course this is intentionally inflammatory but all of these are valid observations on the limitations of consumer-based market research. Let's look at them one by one.
1. Asking the wrong people
Following the disastrous poll performance of 2015 - where instead of a predicted hung parliament we ended up with a huge Conservative majority - the British Polling Council ordered an independent report into the failing. The number one reason giving was unrepresentative sampling. If you ask the wrong people you'll get the wrong answer. For marketers the trap is clear but how careful are we really when selecting our representative sample? I'm talking less about demographic targeting and more mindset and lifestyle. Are the people prepared to answer your questions truly the time-poor millennials you're trying to target? Are the Londoners living within a few miles of the focus group offices really the same as the millions you're trying to reach around the country. A NatCen report into the same 2015 fiasco explained the result as "Labour voters were easier to contact in the two-to-three day window most pollsters had to conduct their surveys than Conservative ones, who took longer to reach". It's as simple as that. Too often we're lazy about who we ask and how many people we ask. Small mistakes in representative sampling are magnified when those samples are multiplied. There is a very real danger that the wrong people will give the wrong answer.
2. They'll lie to you
Unrepresentative sampling may explain the 2015 election result but surely the same mistakes weren't repeated just a year later with the referendum vote - one which was infinitely simpler. Here another explanation is given - the shy tory factor. The theory is simple. Respondents are embarrassed to have voted in a particular way - usually for more right wing policies or candidates - and will therefore hide those from the pollster. This effect has been used to explain the US election result as well. There are doubters of this theory. YouGov in particular have pointed out that the majority of their polling is now done online without face to face contact. But what is undeniable is that trait desirability is a major issue. What a consumer tells you may be what they think but it could also be to try and please you or conform with others. This is particularly true in focus group type situations. John Zaller in his book The Nature and Origins of Mass Opinion argues that surveys do not reveal deep underlying preferences of the public, but rather are "top of the head" replies by people trying to come up with answers to survey questions." We need to understand that how people respond to questions is polluted by social drivers and a desire to conform and be accepted.
3. They don't know how they will behave
How consumers think they will behave and how they actually behave can be very different. If you ask the rational mind how it will behave you will only find out how the rational mind thinks it will behave. What you are missing is the emotional part of your brain or as it's most commonly known system one thinking. We know how people behave is dictated more by emotion than rationality. Yet when we interview people we only seem interested in their rational answers. John Kearon has been leading the charge to try and convince the industry to see the limitations of only looking at system two thinking. For the pollster the danger is the voter who rationally justifies their choice but makes a last minute switch due to emotional reasons. For a marketer the risk is the consumer who doesn't truly understand why they make decisions at the point of purchase. They may believe it's because of price, convenience or taste but in truth it's other emotional drivers that are affecting them. This has led to brilliant creative work that has never seen the light of day because it has 'failed' research. It has also led to some of the most disastrous product launches in history - including McDonald's salads and 'New Coke'.
'New Coke' outperformed the old version consistently in blind taste tests, but the emotional side of consumer brains was outraged at the change and after three months 'Old Coke' was back on shelves
I conducted an experiment with a client last year. We wanted to know how people would react to a certain celebrity partnership they were considering. We asked consumers how many of them would be more likely to share content if it had that celebrity in it. The majority said they would. But we also ran a direct AB test to a similar audience with both options, with celebrity and without. The content without had a higher engagement, like and share rate. No one lied to us on this occasion, it's just how consumers thought they would behave rationally was very different to how they actually behaved emotionally.
4. You won't listen to them anyway
One of the biggest dangers of going into any investigation is if you already have an answer in your mind. You'll be amazed at your ability to find evidence to support that answer. Former pollster Mona Chalabi sites the arrogance of polling organisations as one of the main reasons for them getting it wrong "they couldn’t resist the temptation to try to guess human behaviour. They were so convinced that they knew the answer, they ignored the results that could have suggested otherwise". Matthew Syed in his book Black Box thinking talks about detectives that when faced with irrefutable evidence to disprove their theory go to extreme and extraordinary lengths to protect their original conviction. In one instance of rape, when the suspect's DNA did not match that found in the victim, the detective argued that the victim must have had consensual sex before she was raped and the rapist used a condom. This theory would be more plausible if the victim was not seven years old. Post-rationalisation on this level is rare but the point is valid - if we are using consumers to justify a theory we already have in our mind we will find a way to prove it whether that's through leading questions or analysis of the data.
Traditional pollsters and media were blindly convinced that Clinton would succeed
What's the alternative?
Consumer research isn't going anywhere and is invaluable when conducted well, but it shouldn't be the only thing we rely on. We now have the opportunity to find out how consumers actually behave not just how they say they will. In today's programmatic world we have the ability to set up fast and relatively economical tests. These tests are no more expensive than surveys or focus groups but they will have a much more accurate audience correlation. Because people won't know they are in a survey they will respond accurately. Because they are actually behaving not saying how they would behave you capture emotional and rational thought processes. Most importantly you can trust the result and allow yourself to be proved wrong - as I was in my case.
