Article for Category: ‘Evidence-based practice’

Similar but different: differences in comprehension diagnosis on the Neale Analysis of Reading Ability and the York Assessment of Reading for Comprehension

May 04th, 2016

Psychometric tests are behavioural tests. We use measures of behaviour (which we can see directly) to infer something about a latent variable (something that can’t be seen or measured directly). Take tests of reading ability as an example. Reading occurs in the brain and is therefore a latent variable. We can’t see or measure it directly. The consequence is that different tests will provide different results depending on a number of factors, including the skills/items that the test samples and the normative population. 

Danielle ColenbranderLyndsey Nickels and Saskia Kohnen have just published a study that investigated differences in the content and scores obtained from two commonly used reading tests: The NARA and the YARC. The whole can be found in the Journal of Research in Reading. 

The Abstract

Identifying reading comprehension difficulties is challenging. There are many comprehension tests to choose from, and a child’s diagnosis can be influenced by various factors such as a test’s format and content and the choice of diagnostic criteria. We investigate these issues with reference to the Neale Analysis of Reading Ability (NARA) and the York Assessment of Reading for Comprehension (YARC).

Methods

Ninety-five children were assessed on both tests. Test characteristics were compared using Principal Components and Regression analyses as well as an analysis of passage content.

Results

NARA comprehension scores were more dependent on decoding skills than YARC scores, but children answered more comprehension questions on the NARA and passages spanned a wider range of difficulty. Consequently, 15–34% of children received different diagnoses across tests, depending on diagnostic criteria.

Conclusion

Knowledge of the strengths and weaknesses of comprehension tests is essential when attempting to diagnose reading comprehension difficulties.

 

For help with dyslexia, ADHD, autism spectrum disorders and other developmental and learning disorders in the Gold Coast and Tweed regions contact the Understanding Minds Clinic.

Like us on Facebook for updates on dyslexia related matters, information on other developmental disorders like autism spectrum disorders, Asperger’s and ADHD, and general mental health info.

Comments Off

Kevin Wheldall on why Australia sucks at reading

January 21st, 2013

Kevin Wheldall is an Emeritus Professor of Macquarie University. He is a Director of the reading intervention MultiLit and has a list of awards as long as my Dad’s arm.

Here is Kevin’s opinion on Why Australia sucks at reading.

Don’t forget Jennifer Buckingham’s op-ed piece on the same topic that I blogged about here.

 

 

Comments Off

To repeat or not to repeat: That is the question

January 10th, 2013

This article was published in the Learning Difficulties Australia Bulletin, Volume 39, No 4, December 2007.

Towards the end of each school year, teachers and parents can find themselves faced with a vexing question: should my child repeat his/her school grade?  Some may be driven to this question on the basis of social immaturity while others may be driven by failures to achieve the academic standards set for each grade level. Although the prevalence of grade retention in Australia is far lower than in countries such as the USA, anecdotal evidence suggests that its use as an intervention method continues. This brief article will summarise the available evidence to assist teachers and parents in making a difficult decision.

Who is retained?

The characteristics of students who are retained in a grade are wide and varied and there is very little Australian-based literature. However, it is safe to say that those who are retained tend to be:

  •     Male
  •    Experience academic failure or delay
  •    Have poor classroom conduct
  •    Display emotional immaturity
  •    Perceived as being less competent by both parents and teachers.

Does grade retention improve student outcomes?

Academic achievement

There have been scores of studies conducted since the 1970’s on the issue of grade retention. Many of them, however, suffer from significant methodological and statistical flaws. One should be careful therefore in relying too much upon the data from a single study; particularly if one is not familiar with sound research and statistical methodology. Fortunately a number of reviews and meta-analyses have been conducted which obviate the need for interpretation of individual studies (e.g., Holmes, 1989; Holmes & Matthews, 1984; Jackson, 1975; Jimerson, 2001).

On balance, these reviews have indicated that grade retention either has a negative impact on academic achievement (relative to equivalent promoted peers) or that the effect is null. That is, using retention as an intervention tool has little effect on academic achievement. When positive effects on academic achievement are reported they tend to diminish over time. Indeed, any benefits on achievement are lost when the retained children and their equivalent promoted peers face new material (e.g., Jimerson, Carlson, Rotert, Egeland & Sroufe, 1997).

Mental health

Two studies have reported that older primary school children view grade retention as being in the top three stressful life events: along with losing a parent and going blind (e.g., Anderson, Jimerson & Whipple, 2005). Young children view retention as a punishment and experience sadness, fear and anger when not promoted. In the short-term retained children can face social isolation. For example, there is some evidence showing that peers choose younger same age peers with whom to play rather than the older retained child. In the longer-term retained students tend to experience poorer social adjustment and emotional health, including lower self-esteem and perceived competence, than equivalent promoted peers (e.g. Jimerson et al., 1997).

Student behaviour

The presence of behaviour problems is a predictor of grade retention. Yet the evidence suggests that retention in a grade actually exacerbates the problem (e.g., Jimerson et al., 1997). In male students, grade retention can has long lasting adverse effects on inattentiveness, oppositional behaviour and aggressiveness (Pagani et al., 2001). A similar ‘spike’ in disruptive behaviour is typically seen in female students; however, unlike their male counterparts females display these behaviours for just a short period.

Does timing of retention affect outcomes?

Some authors have argued that age and maturity are significant factors in early school success and that perhaps holding children back will lead to better academic outcomes. Despite the intuitive appeal of holding back a student seen as immature, the evidence does not support the practice. While retention in later grades may be more harmful than when conducted in early grades, the effect is relative and does not mean that early retention is useful or effective.

The alternatives

The most often quoted alternative to grade retention is grade (or social) promotion, where the student is promoted along with his or her grade-peers. While some studies have reported small benefits for promoted students over retained peers, both groups perform more poorly than control students (those without any learning, emotional or behavioural difficulties; Silbergitt, Jimerson, Burns, Appleton & James, 2006). In other words, in the best possible case the promoted student will do slightly better than the retained student. However, both will continue to experience significant difficulties within the areas of function identified as being impaired. Grade promotion on its own then is hardly an alternative to retention.

What is required is grade promotion coupled with intensive intervention methods designed to specifically target the identified weaknesses. Even before this occurs, schools can reduce the possibility of retention through a process of early identification of children ‘at-risk’ (e.g., of reading or learning difficulties). Theoretically-driven and evidence- based early intervention programs (e.g., Direct Instruction programs for word-reading and oral comprehension skills, social skills programs and teacher training in behaviour change) can prevent the failure that leads to the dreaded question of to repeat or not to repeat.

** UPDATE

When I wrote this article in 2007 I suggested that “What is required is grade promotion coupled with intensive intervention methods designed to specifically target the identified weaknesses.”

In fact, there is no evidence for this statement. To settle the question of grade retention forever we would have to conduct a study with four equivalent groups. Group 1 is retained with no additional intervention. Group 2 is retained with the “intensive intervention” I suggested. Group 3 is promoted with no additional intervention. Group 4 is promoted and given “intensive intervention”.

Of course, this study will never pass an ethics committee and therefore will never be conducted. We are therefore stuck with making decisions on less than perfect evidence. On balance, the probabilities still favour grade promotion + intensive intervention.

 

References

 Anderson, G.E., Jimerson, S.R., & Whipple, A.D. (2005). Students’ ratings of stressful experiences at home and school: Loss of apparent and grade retention as superlative stressors. Journal of Applied School Psychology, 21(1), 1-20.

Holmes, C.T. (1989). Grade-level retention effects: A meta-analysis of research studies. In L.A. Shepard & M.L. Smith (Eds.). Flunking grades: Research and policies on retention (pp. 16-33). London: The Falmer Press.

Holmes, C.T. & Matthews, K.M. (1984). The effects of nonpromotion on elementary and junior high school pupils.: A meta-analysis. Reviews of Educational Research, 54, 225-236.

Jimerson, S.R. (2001). Meta-analysis of grade-retention: Implications for practice in the 21st  century. School Psychology Review, 30, 420-438.

Jimerson, S. R. Carlson, E., Rotert, M., Egeland, B., & Sroufe, L.A. (1997). A prospective, longitudinal study of the correlates and consequences of early grade retention. Journal of School Psychology, 35, 3-25.

Pagani, L., Tremblay, R.E., Vitaro, F., Boulerice, B., & McDuff, P. (2001). Effects of grade retention on academic performance and behavioural development. Development and Psychopathology, 13, 297-315.

Silbergitt, B. Jimerson, S.R., Burns, M.K., Appleton, J.J. (2006). Does the timing of Grade retention make a difference? Examining the effects of early versus later retention. School Psychology Review, 35(1).

 

 

 

 

Comments Off

If you want to sell it, put a brain on it

December 19th, 2012

It feels like 90% of the products being hawked as treatments for learning and developmental disorders like dyslexia, language impairments, autism, Asperger’s and ADHD make claims about the brain. Claims include:

  • “based on brain science”.
  • “designed by neuroscientists”
  • “brain based”
  • “neuroplasticity”
  • “sharpen memory and attention with brain games and tools”
  • “change your brain”.

There’s a good reason why companies invoke brain science in selling products. Research has shown that consumers are more likely to rate a claim as credible if it is accompanied by a picture of a brain image (McCabe & Castel, 2008). This seems to happen even if the claim is complete nonsense. Is it any wonder that programs and products (see here, here and here for examples) invoke the brain in the marketing process? It seems human lose their powers of reasoning when presented with a brain.

The neuroscientist Molly Crockett has given a recent TED talk titled “Beware neuro-bunk” in which she cautions against placing too much stock in claims ‘based on neuroscience’.

Crockett cautions us that there’s always more to the story than the brain images. She says “if someone tries to sell you something with a brain on it, ask to see the evidence. Ask for the part of the story that’s not being told.”

Here are other cautionary tales on the same subject:

Psychology Today

The New Statesman

The New Yorker

Journal of Cognitive Neuroscience

The Conversation (Max Coltheart)

The Conversation (Anne Castles & Genevieve McArthur)

Dorothy Bishop

Happy holidays.

Comments Off

Coloured overlays and glasses as a treatment for dyslexia

September 09th, 2012

I probably have a thing variously called Irlen-Meares syndrome, scotopic sensitivity syndrome, visual stress or visual discomfort. I use the term visual discomfort in deference to my old PhD supervisor Liz Conlon (my PhD is old not Liz) who is a leader in the field although she hardly ever gets cited. Here are some of her papers you bums! (Conlon et al, 1999; Conlon, Sanders, & Wright, 2009; Conlon & Humphreys, 2001; Conlon et al., 1998; Conlon & Sanders, 2011).

What is visual discomfort?

No one really knows what visual discomfort is. My own view, somewhat consistent with the literature, is that it is an abnormal sensory sensitivity to stimuli of high contrast, and/or low spatial and/or high temporal frequencies. Black text on a white page is an example of a high contrast stimulus. Single spaced text with small font size is an example of a stimulus of low spatial frequency. One of the reasons universities require assignments to be typed in 12-point font, double spaced is that double spaced text is more comfortable to read than single spaced. High temporal frequency stimuli are characterised by rapid flashing. Think of the rapid flicker emitted by strobe lights or a fluorescent bulb. These stimuli lead to excessive neuronal firing that can lead to perceptual distortions or, in my experience, simply make the stimuli uncomfortable to be around.

My own visual discomfort manifests as light sensitivity, too much time in harsh sunlight sans sunglasses results in eye strain and a headache (although this may be psychosomatic as I spent many years as a kid surfing and playing cricket sans sunglasses with no ill effects). I dislike fluorescent lights, which unfortunately light our clinic offices. One office has a bulb that runs directly along my left eye line as I sit in the therapy chair. After a heavy day of consulting I can actually feel a “buzzing” and some days a bad headache in the part of my head that seems to match where the light runs. Again, this may be neurotic but needless to say I attempt to avoid this room. I had trouble with the old CRT computer screens at university; LCD screens were still rare back then. Flashing lights drive me nuts. Laser shows, strobe lights at concerts, and my 3-year old son’s flashing Batman toothbrush (a light on the brush flashes, not Batman) all make me a little more grumpy than usual.

Visual discomfort, reading and dyslexia

There is a theory that visual discomfort causes print to become distorted, which affects word reading and comprehension in turn. Visual discomfort is also claimed to affect reading efficiency such that sufferers can only read for short periods and are prone to reading related headaches.

Visual discomfort has been reported to be more prevalent in dyslexic populations. However, the relationship between dyslexia and visual discomfort remains controversial. Visual complaints are made by many healthy people and visual discomfort also exists in skilled readers (I’m an example). That visual discomfort exists in skilled readers makes a nonsense of claims that it is a form of “visual dyslexia”. Dyslexia and visual discomfort are separate conditions.

Visual discomfort, dyslexia and coloured overlays/lenses

Coloured overlays or lenses are a common treatment for visual discomfort (Allen, Gilchrist and Hollis, 2008; Wilkins, 19952003; Wilkins, Huang and Cao, 2004). Coloured overlays are thin transparent coloured films that are placed over a page of text. They are designed to colour the page without affecting clarity of the text.

The evidence for whether coloured overlays improve reading is mixed. A lot of the existing data published in “scientific” journals are plagued by methodological concerns, including no controls on other therapies/intervention or poorly matched intervention groups. Of the better studies Singleton and Trotter (2005) used undergraduate students with (n =  10) and without (n =  10) dyslexia. Each group had 5 students with high visual discomfort (HVD) scores and 5 with low visual discomfort (LVD) scores. All participants read faster using their chosen overlay. The dyslexics with HVD scores made significant gains in reading speed with an overlay while the other groups made non-significant change (gains of 3-4%).  Singleton and Henderson (2007) showed children (6-14 yoa) made greater improvement in reading rate with coloured overlays relative to reading-age matched controls. In contrast, Ritchie, Della Sala and McIntosh (2011) reported on 61 children (7-12 yoa) with reading difficulties (77% were diagnosed by an Irlen diagnostician as having the visual discomfort). There was limited evidence that individually prescribed Irlen coloured overlays had any immediate benefit for reading rate.

A recent study from the lab of respected reading scientist, Maggie Snowling, investigated the effect of coloured overlays in a well-designed experiment. They took 26 controls and 16 people with dyslexia, all undergraduate students, matched for IQ. Both were tested on two reading tests. The Wilkins Rate of Reading Test (WRRT) measures the impact of overlays on reading. The WRRT requires speeded oral reading of a passage of text comprising 15 high-frequency words (familiar to children from 7 years) that are repeated in random order, ensuring that no word can be guessed from the context. The test was administered with and without the chosen overlay placed over the text to test for an immediate benefit in reading rate. Reading rate was calculated as the number of words read correctly per minute (wpm) not including errors, omitted words and omitted lines. They also used two passages adapted from passages in the secondary school edition of the York Assessment of Reading for Comprehension (YARC). Passage 1 consisted of 311 words, and Passage 2 consisted of 302 words. Five  comprehension questions followed each passage.

Both groups read more words per minute in the Overlay versus No Overlay condition. The group with dyslexia showed marginally greater gains relative to controls. However, these data need to interpreted with a healthy dose of salt because the dyslexics were slower readers to begin with and therefore had more room to improve.

When reading real text (YARC), there was an effect for Group on passage reading time. Unsurprisingly, the dyslexic group was slower than controls in both Overlay and No Overlay conditions. But there was no effect for Overlay (the overlay made no difference to reading rate for either group) or a Group by Overlay interaction (there was no relative advantage for the dyslexic group in the Overlay condition v No Overlay). Reading comprehension scores did not change in either group as a result of using an overlay. These data are consistent with those reported by Ritchie, Della Sala and McIntosh (2011) in children. They suggest that coloured overlays are not as effective as claimed for improving reading accuracy or fluency.

 

 

Comments Off