Back in June’s inaugural newsletter we brought you analysis of the hard numbers following the May federal election. Earlier this month EKAS were on hand to hear from three eminent polling experts for their own post-mortem.
Moderated by AMSRS conference chair Rebecca Huntley, the first question to the panel was whether the pollsters had really erred that badly.
Murray Goot, Emeritus Professor at Macquarie University, was particularly forthright on the subject: “The polls are always bad when they get the result wrong.
“The last time they got the result this wrong was in 1980. In 1980, when all the polls fell on the wrong side, the predicted results showed Labor ahead by as much as 9%, and the Liberals ultimately won.
“If we aggregated the polls at this election to create a poll of polls and treated them as one, the combined effort would have been out by 5.2% points in estimating the Coalition’s winning margin in primary vote. That’s five times the normal margin of error on an 8,000 person sample.”
But the polls have a much more chequered record than people might think. Goot explained: “In six elections since 1993, at least one poll got the first preference vote for Labor or the Coalition wrong by 4% or more.”
“So, not such a good performance,” Huntley deadpanned to general mirth.
Getting it right
“The key thing in polling is you’ve got to pick the winner,” Goot stressed. “The second thing is how close you come in the distribution of the numbers, but that’s less important than picking the winner.”
Amanda Hicks, Partner at KPMG Acuity, said it was certainly a big shock to the Australian public how far wrong the polls had got it at this election.
“When we do polling, there’s always a slight under-estimation of the Liberal vote. But I think the number of quiet Australians is the bigger picture,” she said.
“There are a few other external factors at play too. The populace were showing signs of being both disengaged with the process and distrustful of politicians. And at that point weaknesses in polling are brought to the fore. People simply aren’t engaged to the end, or are pre-polling long in advance, as we saw at this election.”
Chris Lonergan, Managing Director of Lonergan Research, was in agreement: “There is so much polling going on. Our response rate as pollsters, collectively, is very low. It’s no surprise people were saying ‘bugger off mate’ and Liberal voters tend to say that more.”
The quiet Australians
Lonergan’s comments brought the ‘Shy Tory’ theory to the fore – the tendency of people who vote conservative to be less willing to admit it to pollsters.
“I’m not sold on the Shy Tory view myself. In the US election in 2015 and UK Brexit referendum in 2016 they didn’t find reluctant Tories,” Goot said.
Lonergan countered: “Then how do we explain that we can ask people five minutes after they vote in Australia and still get the wrong numbers?
“The real question, I think, is why are we missing them? They’re likely disengaged with the process, as Amanda said.
“For the exact opposite reason, Greens voters are often overestimated. We’ve had groups of Greens voters lining up to take an exit poll and we’re saying, ‘Guys, this isn’t how it works’,” Lonergan added.
Is the issue over-estimating the Labor vote instead of just underestimating the Liberal vote, Huntley theorised. After all, surely more data comes from the hyper-engaged, who tend to lean left.
“It’s certainly a combination of the two, Lonergan said, adding “if you look at robos [robotic telephone polling], you don’t know if it’s a business or household you’re getting. It’s a factor we take into consideration when we do our weighting.”
Harris advanced the theory of the ‘Inaccessible Tory’, covering those who pre-voted or flat out avoided polling altogether.
“It’s certainly not cool to be a Liberal voter; people are unlikely to tell you, or avoid being asked,” Lonergan concurred.
“It does pose a very big question for pollsters, Goot said: “Imagine you included a hypothetical survey instrument on engagement. You’d expect, with a Shy Tory view, the most disengaged would be Liberal voters.
“But then, they still got the polling wrong in Queensland and it’s the weakest Labor state. If it’s not ok to be a Liberal there, where is it?”
Media influence – are they marking their own homework?
Outside of the voters themselves, Huntley pivoted to the issue of the media’s relationship with polling companies – and whether it was a healthy one.
There is a parasitic relationship, Goot observed. “The media gets its sense of what’s happening via the polls, so the media is compromised. They pay for and own the polls they publish and are largely uncritical of the polling.
“It deserves more criticism than it gets. The polls haven’t been credible for a long time. There’s no way you can run poll after poll and find the results haven’t moved, it just doesn’t happen.
“There was a suspicion that Labor was going to win, so they felt the need to reflect this in the polls.”
Other polling concerns
The ancillary issues of polling that often get overlooked were also debated by the panelists. Chief among them was the much maligned ‘don’t know’ option within surveys.
Goot pointed out the ‘don’t knows’ were “a big part of the 1948 US election debacle”, which saw incumbent President Harry Truman win an unexpected second term against highly favoured Republican Thomas Dewey.
He added: “It’s still relevant. When you ask a question you’ll get 10% to 15% who say they don’t know. Our categories don’t allow us to distinguish those. But in the end, there’s no reason you can assume they will split like everyone else.
“I think we should ask, why not push more of these don’t knows (something like 3/1) to the conservative side?”
For the don’t knows, Longeran proposed to “give them one option the second time around, and they’ll either say what they think or end the survey.”
Budget was another vexing issue for the pollsters.
Lonergan took up the charge, saying: “Every pollster knows that robos aren’t as good as face to face polling. But unless budgets rise, there’s no option but to robopoll. And that’s always going to produce lesser quality data.
“Personally, I think we should make robo polling illegal. People hate it, and it’s driving down the cost. I’d rather see it banned, despite the fact we make money out of it.”
Goot was also in favour of bigger budgets devoted by the media for polling. “What they could also do is be more open about methodology. It’s scandalous how little they show. And it would be refreshing if they expressed less certainty about what they don’t know.”
Harris had the last word: “As we reel-in more and more data, we need to look into how we get more sophisticated. Overall, we need to look at how we get better at predictive analytics.”
