The ‘research’ report used by BBC Scotland News, this morning, to generate the headline ‘Over a third of children with autism have been unlawfully excluded from school in the past two years’ is unsound on the basis of both a predisposing bias and a fatally flawed methodology leading to the headline being essentially a lie.
Artlessly, the researchers signal their predisposition to condemn Scottish schools catering for autistic children, by naming their report ‘Not included, not engaged, not involved’ and then peppering the text with more than forty melodramatic and sometimes redundant references to ‘unlawful’ exclusions.
Off course, if the methodology used had been able to scientifically demonstrate the existence of practice in schools warranting these accusations, then the criticism would have only been of lack of taste. However, the methodology as designed and implemented is utterly inadequate for that purpose.
The researchers tell us (p14) that they surveyed only parents and carers of children. 1434 chose to respond with some forms incomplete but still included. We are told that this represents around 10% of the pupils with ASD in Scotland. If the sample was random then it might be valid but given that the respondents were self-selecting, we cannot tell whether they meaningfully represent the total population or whether, as is more probable, they represent only those who perceive problems. For all we can tell, a strong majority of parents and carers may be satisfied with practice in schools. The authors seem to acknowledge this but then go on to write their findings as if they had forgotten it. Worse still, the level of non-completion of parts of the survey further reduces the sample to a very small number of responses. For example: ‘[Only] 37% of the 478, [only], who answered this question (p19) indicated that their child had been excluded without formal recording’. That’s only around 160 pupils out of their sample of 1 434 and out of a possible total population of 10 000. How on earth do you get from those data to the researchers’ own conclusions? Even worse, the headline by BBC Scotland saying: ‘over a third of children with autism have been unlawfully excluded’, becomes frankly dishonest?
Furthermore, to only interview parents and carers and to fail to access school documents reporting absence or to interview head teachers, is to thoroughly weaken the research – triangulation anyone? We only know, from this research, what a perhaps unrepresentative sample of parents THINK schools are doing with regard to recording informal exclusions. We do not actually know what even a sample of schools IS doing in terms of recording.
Finally, the researchers make the astonishing assumption that percentages should be calculated from those responding to any question and not from the total sample. For example (p23):
When asked whether their child had been placed on a reduced timetable on more than one occasion, 63% (n=248) of parents told us that they had.
This sentence is utterly inaccurate, bananas, and should use the full sample to give the percentage – 17.2% (only)! This practice is repeated throughout to suggest that problems are more common than they are – astonishing incompetence or naked bias?
This research was commissioned, paid for, by Children in Scotland, the National Autistic Society Scotland and Scottish Autism. As you might expect, the researchers have clearly known from the outset what was expected of them. The ‘researchers’, notengaged.com have no published track record as researchers. Does their name suggest that they, like their funders, are campaigners and not really researchers?