The ‘research’ report used by BBC Scotland News, this morning, to generate the headline ‘Over a third of children with autism have been unlawfully excluded from school in the past two years’ is unsound on the basis of both a predisposing bias and a fatally flawed methodology leading to the headline being essentially a lie.
Artlessly, the researchers signal their predisposition to condemn Scottish schools catering for autistic children, by naming their report ‘Not included, not engaged, not involved’ and then peppering the text with more than forty melodramatic and sometimes redundant references to ‘unlawful’ exclusions.
Off course, if the methodology used had been able to scientifically demonstrate the existence of practice in schools warranting these accusations, then the criticism would have only been of lack of taste. However, the methodology as designed and implemented is utterly inadequate for that purpose.
The researchers tell us (p14) that they surveyed only parents and carers of children. 1434 chose to respond with some forms incomplete but still included. We are told that this represents around 10% of the pupils with ASD in Scotland. If the sample was random then it might be valid but given that the respondents were self-selecting, we cannot tell whether they meaningfully represent the total population or whether, as is more probable, they represent only those who perceive problems. For all we can tell, a strong majority of parents and carers may be satisfied with practice in schools. The authors seem to acknowledge this but then go on to write their findings as if they had forgotten it. Worse still, the level of non-completion of parts of the survey further reduces the sample to a very small number of responses. For example: ‘[Only] 37% of the 478, [only], who answered this question (p19) indicated that their child had been excluded without formal recording’. That’s only around 160 pupils out of their sample of 1 434 and out of a possible total population of 10 000. How on earth do you get from those data to the researchers’ own conclusions? Even worse, the headline by BBC Scotland saying: ‘over a third of children with autism have been unlawfully excluded’, becomes frankly dishonest?
Furthermore, to only interview parents and carers and to fail to access school documents reporting absence or to interview head teachers, is to thoroughly weaken the research – triangulation anyone? We only know, from this research, what a perhaps unrepresentative sample of parents THINK schools are doing with regard to recording informal exclusions. We do not actually know what even a sample of schools IS doing in terms of recording.
Finally, the researchers make the astonishing assumption that percentages should be calculated from those responding to any question and not from the total sample. For example (p23):
When asked whether their child had been placed on a reduced timetable on more than one occasion, 63% (n=248) of parents told us that they had.
This sentence is utterly inaccurate, bananas, and should use the full sample to give the percentage – 17.2% (only)! This practice is repeated throughout to suggest that problems are more common than they are – astonishing incompetence or naked bias?
This research was commissioned, paid for, by Children in Scotland, the National Autistic Society Scotland and Scottish Autism. As you might expect, the researchers have clearly known from the outset what was expected of them. The ‘researchers’, notengaged.com have no published track record as researchers. Does their name suggest that they, like their funders, are campaigners and not really researchers?
https://www.notengaged.com/download/SA-Out-Of-School-Report.pdf
Talking-up Scotland 
Sadly sounds like a very shoddy piece of work – then compounded by beeb inadequate reporting skills. How to stich up a story and generate tabloid headlines in a couple of simple steps.
LikeLiked by 1 person
Pretty basic errors in reporting then Prof. You’d expect any journalist worth their salt to review the information and conclude pretty quickly that this info would best be kept out of reach of even a bargepole for fear of tainting the respect for journalistic standards beyond redemption.
LikeLiked by 1 person
On reading the report by Notengaged, the following section stood out for me:
“Understanding the findings
Results are of a self-selecting sample, and those parents that took part chose to do so, knowing that the
survey was asking for experiences of children who had missed time from school.
We are not claiming that the results are representative of the experiences of all autistic children in Scotland.
The results also do not offer a statistically accurate picture of the nature and extent of different types
of absence across Scotland. What they do offer is important insight into the experiences of the families
who responded.
We did not define “support” to those completing the survey. Due to the breadth of different supports
available – with the huge range of personnel, technologies, strategies, communication aids,
visual supports and so on that can be provided to children, it is impossible to provide an exhaustive list.
We were interested to know what parents would recognise as support and left this open for this reason.
All questions in the survey were optional (except for the demographic information), and not every
respondent answered every question. The percentages given in this report are calculated from the
number of responses to each individual question. Corrections to the spelling and grammar in quotes
have been made but no content has been changed.”
The question arises – What the hell was someone at the BBC thinking when they decided to broadcast a damaging mainstream report based on work which itself admitted, big-time, that it was flawed and should not be taken as having any relevance to the generality of special education for autism?
The barrel of anti Scottish material must have been scraped now to a fine polish – or could there be a less charitable explanation?.
LikeLiked by 1 person
And how did the authors of the report get from those reservations to the claims they made?
LikeLike
It’s a pity that the negative experiences of some children and families are presented in this way because the controversy over the report’s failings to offer a representative sample then overshadow any challenges in schools. Exclusion and disengagement are problems for some children (with or without autism) but highlghting them in this way without any degree of balance makes it less likely that any existing positive support can be identified and used to promote good practice
In some ways this report has been presented in the same way as the controversy over P1 assessments. As far as I can see some very vocal campaign groups are making a lot of noise without any attempt to consider the scale of opposition or support. These groups have a loyal following and some big name supporters but make no attempt to quantify the number of families indifferent to or supportive of the assessments
As highlighted in William’s reply above this report states”The results also do not offer a statistically accurate picture of the nature and extent of different types of absence across Scotland. What they do offer is important insight into the experiences of the families who responded” but then goes on to include figures and percentages which are open to misinterpretation by a media which seems incapable of doing any analysis or research for themselves
Thanks John for taking the time to research this story, we are very lucky to have you and other key people willing to look beneath the surface of what is presented to us as news
LikeLiked by 1 person
My twisted pleasure!
LikeLike