Why the Herald Independence Poll Today (2.1.17) is Unreliable

Birmingham-based, BMG Research surveyed 1002 Scots between December 9th and 13th. I’m going to take a wee but intelligent leap here and say that these results come from cold calls, mostly in a Birmingham accent (?), to telephone land lines. I’m basing that assumption confidently on the time scale being too short for face-to-face interviews or questionnaire returns. The former is highly unlikely anyway because it costs too much and the latter tends to get a very poor response rate. Finally, I’m basing it on information from employees and ex-employees on a website where they have commented on their experiences of working for BMG. If I’m wrong about this, all that follows in pointless. First see these reservations, from YouGov, about telephone interviews from the opinion polls which got the EU Referendum so wrong:

‘There’s a big difference between the online and telephone polls on the EU referendum – with online polls showing the sides neck-and neck and telephone polls showing about a 15% gap in favour of ‘remain’. Why? It’s striking that both methodologies right across the different polling companies give about the same number to the ‘leave’ campaign, around 40%. The difference is in the ‘remain’ number, which is around 52% from the telephone polls but only 40% for online polls.’

So, commonly, telephone surveys generate conservative, negative or status quo returns. Respondents are more likely to say no to a question about a big change of some kind. I don’t know what effect an English accent would have.

Also, BMG haven’t published the response rate. The more people who refused to answer the questions the less reliable the results.

In another YouGov report we read:

‘Now however we can reveal a real, significant and evidence-based difference between the two methodologies that explains why they are divergent and why it is online that appears to be calling it correctly.’

See this online survey report from the, far from sympathetic to Scottish Independence, Scotsman newspaper in June 2016:

‘Nearly six out of 10 Scots say they’d vote Yes in a second independence referendum. In a clear reflection of the growing backlash north of the Border to Thursday’s Brexit result, a ScotPulse online survey of 1,600 Scottish adults on Friday (24 June) showed that 59% of Scots now back leaving the UK.’

Further, not everyone has a landline to be called on. Roughly 20%, especially younger and economically disadvantaged citizens do not have one so cannot be surveyed. As the Herald report points out, the young and the less-well-off are more likely to prefer independence.

Here’s an even more interesting thought, from the USA admittedly:

‘There now may be something unusual about people who are willing to answer the phone to talk with strangers, and we should be sceptical about generalizing from the results of these surveys. It is possible that the new habit of non-phone-answering is evenly distributed throughout the population (thus reducing this as a sampling confound), but this seems unlikely.’

Now, are NO voters more unusual than Yes voters?

https://www.indeed.co.uk/cmp/Bmg-Research/reviews

https://yougov.co.uk/news/2016/02/23/commentary-what-explains-difference-between-phone-/https://yougov.co.uk/news/2016/02/23/commentary-what-explains-difference-between-phone-/

https://yougov.co.uk/news/2016/05/20/revealed-evidence-greater-skews-phone-polls/

https://www.statista.com/statistics/386778/share-of-calls-enabled-landlines-in-uk-hoseholds/

http://www.scotsman.com/news/poll-puts-support-for-scottish-independence-at-59-1-4163338

http://www.counterpunch.org/2004/09/20/what-if-the-problem-with-phone-polls-is-that-they-are-phone-polls/

Advertisements

10 thoughts on “Why the Herald Independence Poll Today (2.1.17) is Unreliable

  1. Clydebuilt January 2, 2017 / 1:30 pm

    Happy New Year John …… And readers…..

    Checked your blog yesterday, your posts dated 31st Dec and 1st Jan didn’t show up….. Strange old world the Internet.

    Like

    • Clydebuilt January 2, 2017 / 7:14 pm

      Yeah ….noticed that…..

      Like

  2. Stuart January 2, 2017 / 6:21 pm

    Dear Me, the Professors reputation for detecting ‘Bias’ is just laughable, as outlined here;

    https://ahdinnaeken.wordpress.com/2014/01/21/moan-mcvulpine-tv-bosses-wise-up-or-face-the-wrath-of-the-nationalist-inquisition/

    DR JOHNSON Robertson is a respected academic, particularly in Nationalist circles, with a long history of saying what the Nats like to hear.

    His work includes detecting bias in everything the media does.

    Now the Reader in Media Bias and his researchers have turned their expertise to the Scottish emancipation campaign.

    Dr Robertson’s undisclosed ‘team’ at the List D University of the West of Scotland spent a year, looking for instances of bias in the BBC and STV.

    Guess what? They found them. Quelle surprise!

    It’s a not so well known maxim that if you go looking for bias in a medium, you will find it.

    You’ll even be able to make your grievance credible by supporting it with selected evidence.

    Anything to say by ay of reply ‘Professor’?

    Like

    • johnrobertson834 January 2, 2017 / 9:15 pm

      Hi Stu
      That piece is nearly three years old! I honestly didn’t go looking for bias. I used grounded theory to avoid that. Calling UWS a List D University is a bit snobby even nasty, isn’t it? Where did you go? What degree, what classification?
      John

      Like

  3. Corrado Mella January 2, 2017 / 10:23 pm

    I have a landline, foisted on me by my broadband and TV supplier, that is not even connected to a phone, but one mobile for each member of the family.

    All my phone numbers are VoIP and ex-directory.

    I work in telecoms, and can say with absolute certainty that the ABCDE and age bias on polls conducted through calls to landlines is extremely lopsided.

    If that subset (middle class, over 50) see Independence at 45% Yes, it’s DOUBLE than what was in 2014.

    EXTREMELY good.

    Like

  4. Alex Beveridge January 3, 2017 / 9:56 am

    Many thanks, once again, for your dissection of these “polls”, John. I have never been asked for my opinion in any shape or form, but as you are aware, the media will use any stick to beat independence movement, and their propaganda will become more and more hysterical as we approach Indyref2.
    Again, as those of us on this side of the fence are all too aware, some of the electorate only see the headlines, not bothering to read the complete article, which of course may, or may not, reveal the true nature of the article.
    While the power, and influence, of the M.S.M, led by their cheerleader the B.B.C may be diminishing, they still represent our greatest obstacle to gaining our independence, and it will take one heck of an effort to defeat them come independence day.

    Like

  5. clipper January 3, 2017 / 2:18 pm

    Stuart Jan 2nd 6.21

    How typically unionist, snide, smarmy and stuck up the arse of your own ignorance.

    Liked by 1 person

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s