Talking-up Scotland

Scottish Tory parliamentary question today:

The ‘research’ report used by BBC Scotland News, on 25^th September 2018, to generate the headline:

‘Over a third of children with autism have been unlawfully excluded from school in the past two years’

is unsound on the basis of both a predisposing bias and a fatally flawed methodology leading to the headline being essentially a lie.

Artlessly, the researchers signal their predisposition to condemn Scottish schools catering for autistic children, by naming their report ‘Not included, not engaged, not involved’ and then peppering the text with more than forty melodramatic and sometimes redundant references to ‘unlawful’ exclusions.

Off course, if the methodology used had been able to scientifically demonstrate the existence of practice in schools warranting these accusations, then the criticism would have only been of lack of taste. However, the methodology as designed and implemented is utterly inadequate for that purpose.

The researchers tell us (p14) that they surveyed only parents and carers of children. 1434 chose to respond with some forms incomplete but still included. We are told that this represents around 10% of the pupils with ASD in Scotland. If the sample was random then it might be valid but given that the respondents were self-selecting, we cannot tell whether they meaningfully represent the total population or whether, as is more probable, they represent only those who perceive problems. For all we can tell, a strong majority of parents and carers may be satisfied with practice in schools.

The authors seem to acknowledge this but then go on to write their findings as if they had forgotten it. Worse still, the level of non-completion of parts of the survey further reduces the sample to a very small number of responses. For example: ‘[Only] 37% of the 478, [only], who answered this question (p19) indicated that their child had been excluded without formal recording’. That’s only around 160 pupils out of their sample of 1 434 and out of a possible total population of 10 000. How on earth do you get from those data to the researchers’ own conclusions? Even worse, the headline by BBC Scotland saying: ‘over a third of children with autism have been unlawfully excluded’, becomes frankly dishonest?

Furthermore, to only interview parents and carers and to fail to access school documents reporting absence or to interview head teachers, is to thoroughly weaken the research – triangulation anyone? We only know, from this research, what a perhaps unrepresentative sample of parents THINK schools are doing with regard to recording informal exclusions. We do not actually know what even a sample of schools IS doing in terms of recording.

Finally, the researchers make the astonishing assumption that percentages should be calculated from those responding to any question and not from the total sample. For example (p23):

When asked whether their child had been placed on a reduced timetable on more than one occasion, 63% (n=248) of parents told us that they had.

This sentence is utterly inaccurate, bananas, and should use the full sample to give the percentage – 17.2% (only)! This practice is repeated throughout to suggest that problems are more common than they are – astonishing incompetence or naked bias?

This research was commissioned, paid for, by Children in Scotland, the National Autistic Society Scotland and Scottish Autism. As you might expect, the researchers have clearly known from the outset what was expected of them.

The ‘researchers’, notengaged.com have no published track record as researchers. Does their name suggest that they, like their funders, are campaigners and not really researchers?

https://www.notengaged.com/download/SA-Out-Of-School-Report.pdf

I wrote to BBC Scotland to complain and they apologised.

Here’s the BBC response:

Dear Professor Robertson

Reference CAS-5101273-HLBVZ3

Thank you for your correspondence. Your comments were passed to the Editor of Reporting Scotland, who has asked that I forward her response as follows:

“Thank you for being in touch about Reporting Scotland on the morning of 24th September. You have raised concerns which I share about a report on a survey of parents and carers of autistic children. In retrospect we should not have carried this story in the way that we did and I apologise for that. I am taking steps to try to ensure that we do not do so again with similar stories. I believe that this was an honest mistake and that there was certainly no intention to mislead our audience, but that does not detract from the fact that we got it wrong. I am grateful to you for taking the time and the trouble to explain the reasons for your concern.”

No doubt Mr Swinney will want to be a bit kinder than I have been, but bad research is to be condemned.