Politics Scotland on 6th March headlined with: ‘Holyrood hears allegations children with additional needs are being unlawfully excluded from lessons!’
The piece was then introduced with: ‘It’s a follow-up from work done on 2017 which concluded there was a lack of support for children.’
These allegations are based on deeply flawed research, which I demolished in 2018, and which BBC Scotland have already apologised for using:
In retrospect we should not have carried this story in the way that we did and I apologise for that. I am taking steps to try to ensure that we do not do so again with similar stories.’
The report has many flaws, see below, but based on extracting one key element, the objective misuse of statistics to exaggerate, I was able to write this, in October 2018:
Here’s my original complaint:
On Tuesday 24th September 2018 at 06:26am and repeated five times by 09.00am, BBC Scotland headlined: ‘A report commissioned by three charities suggests that over a third of children with autism have been unlawfully excluded from school in the past two years.’
https://www.notengaged.com/download/SA-Out-Of-School-Report.pdf
This is an incorrect reading of the report suggesting 37% of all children with autism, with the effect of seriously exaggerating their evidence. On page 19, the report says: ‘We described unlawful exclusions to respondents as when a child has been sent home from school or asked not to attend, without being formally excluded (e.g. school asking parents to pick up their child early). 37% (n=478) of parents who responded to this question told us that their child had been excluded in this way.’ Note that 37% of the 478 parents who responded (176) indicated ‘unlawful’ exclusion. The full sample was 1 434 and that was thought to be around 10% of the total population of autistic pupils in Scotland (page 14). So, more accurately and thus more informatively for the TV audience, the headline should have said: ‘A report commissioned by three charities suggests that over a third of the 478 children with autism whose parents responded have been unlawfully excluded from school in the past two years.’ More helpfully, the percentage should have been changed to 12% of those who took part in the research and the size of the sample, 10% of the total population, should have featured in the report. More helpfully still, the deeply flawed research should not have been reported. See this for a full assessment of it:
Here’s the BBC response:
Thank you for your correspondence. Your comments were passed to the Editor of Reporting Scotland, who has asked that I forward her response as follows:
“Thank you for being in touch about Reporting Scotland on the morning of 24th September. You have raised concerns which I share about a report on a survey of parents and carers of autistic children. In retrospect we should not have carried this story in the way that we did and I apologise for that. I am taking steps to try to ensure that we do not do so again with similar stories. I believe that this was an honest mistake and that there was certainly no intention to mislead our audience, but that does not detract from the fact that we got it wrong. I am grateful to you for taking the time and the trouble to explain the reasons for your concern.”
Here’s the full criticism of the report:
The ‘research’ report used by BBC Scotland News, this morning, to generate the headline ‘Over a third of children with autism have been unlawfully excluded from school in the past two years’ is unsound on the basis of both a predisposing bias and a fatally flawed methodology leading to the headline being essentially a lie.
Artlessly, the researchers signal their predisposition to condemn Scottish schools catering for autistic children, by naming their report ‘Not included, not engaged, not involved’ and then peppering the text with more than forty melodramatic and sometimes redundant references to ‘unlawful’ exclusions.
Off course, if the methodology used had been able to scientifically demonstrate the existence of practice in schools warranting these accusations, then the criticism would have only been of lack of taste. However, the methodology as designed and implemented is utterly inadequate for that purpose.
The researchers tell us (p14) that they surveyed only parents and carers of children. 1434 chose to respond with some forms incomplete but still included. We are told that this represents around 10% of the pupils with ASD in Scotland. If the sample was random then it might be valid but given that the respondents were self-selecting, we cannot tell whether they meaningfully represent the total population or whether, as is more probable, they represent only those who perceive problems. For all we can tell, a strong majority of parents and carers may be satisfied with practice in schools. The authors seem to acknowledge this but then go on to write their findings as if they had forgotten it. Worse still, the level of non-completion of parts of the survey further reduces the sample to a very small number of responses. For example: ‘[Only] 37% of the 478, [only], who answered this question (p19) indicated that their child had been excluded without formal recording’. That’s only around 160 pupils out of their sample of 1 434 and out of a possible total population of 10 000. How on earth do you get from those data to the researchers’ own conclusions? Even worse, the headline by BBC Scotland saying: ‘over a third of children with autism have been unlawfully excluded’, becomes frankly dishonest?
Furthermore, to only interview parents and carers and to fail to access school documents reporting absence or to interview head teachers, is to thoroughly weaken the research – triangulation anyone? We only know, from this research, what a perhaps unrepresentative sample of parents THINK schools are doing with regard to recording informal exclusions. We do not actually know what even a sample of schools IS doing in terms of recording.
Finally, the researchers make the astonishing assumption that percentages should be calculated from those responding to any question and not from the total sample. For example (p23):
When asked whether their child had been placed on a reduced timetable on more than one occasion, 63% (n=248) of parents told us that they had.
This sentence is utterly inaccurate, bananas, and should use the full sample to give the percentage – 17.2% (only)! This practice is repeated throughout to suggest that problems are more common than they are – astonishing incompetence or naked bias?
This research was commissioned, paid for, by Children in Scotland, the National Autistic Society Scotland and Scottish Autism. As you might expect, the researchers have clearly known from the outset what was expected of them. The ‘researchers’, notengaged.com have no published track record as researchers. Does their name suggest that they, like their funders, are campaigners and not really researchers?
https://www.notengaged.com/download/SA-Out-Of-School-Report.pdf
Talking-up Scotland 