Maths is a morning subject. Really?

Across my Twitter feed this morning was the bold claim that "Teaching maths in the morning improves results - TES", which stems from a press release of the Royal Economic Society which can be found here:

The TES article gives a brief summary of the conference paper, ending with the tantalizing quote "These results...present a low-cost intervention which can lead to an increase in student performance without substantial investments in school inputs." Which would seem to be the prize for all schools - an intervention that works without much (if any) additional cost. The suggestion being that just by restructuring the day results will go up.
A quick Google uncovered the actual paper itself, which can be downloaded here

The paper itself in the abstract opens with "The findings indicate significantly lower maths scores during afternoon classes by 0.082 and higher test scores for history classes by 0.069 standard deviations, respectively" - that seems tiny to me. Cohen tells us that if means differ by one standard deviation, we have d=1, by half a standard deviation we have d=0.5 and so on. In this case they differ by 0.069 (d=0.069) and 0.082 (d=0.082). Cohen suggested that d=0.2 be considered "small", 0.5 represents "medium" and 0.8 a "large" effect size. That means that if two groups effect sizes don't differ by 0.2 (or more) standard deviations the difference is trivial, even if it is statistically signifincat (which the author points out is the case)

Buried in the paper itself is the line "(for history) Statistically significant afternoon effect is only observed between the 40th and the 70th quantile of the distribution with declining magnitude for higher quantiles" - so the reported figure 0.069 and the claim of significance does not apply to all the data - the author has cherry picked the numbers looking for something significant to report.

So, my first problem with this headline is that the data in the paper isn't as conclusive as the headline would appear to suggest.

I also find it troublesome that this is a conference paper, reported via the press release. This (unrepresented) conference paper hasn't been peer reviewed in the manner of a journal paper - that is a fundamental issue here. We cant be sure that this paper and the findings have been through the academic mill and actually stand up to rigorous analysis.

Finally, and this is often a criticism of academic papers - if you actually download and read the paper, I challenge you to actually understand the majority of the narrative and charts presented. Papers like this are not written for the people whom they proport to be aimed at (teachers, education decision makers). They are written for the small, closed community of academics who both have the time and experience to actually decipher them. As a teacher, member of SLT or school governor the paper is rich with data but the nuances of (over) analysis make interpretation very challenging.

However, there are some nuggets in the body of the paper though - the author finds that lower ability mathematicians are more negatively effected by afternoon lessons (no surprise there), that learners with higher absences do less well (shock), that unexplained absences are higher in the afternoon than the morning (again no surprise).

This demonstrates a few things:
(1) When the media promotes a story, please link to the actual paper
(2) That teachers need to be research literate and actually look for the papers mentioned in the news briefings
(3) That academics need to write papers keeping in mind who they are attempting to influence
(4) That just because it's in the paper, doesn't mean that we need to act upon it.

comments powered by Disqus