What the actual #*$#?

Sometimes you see something that makes you question the very fabric of space time - or at the very least, the entire canon of maths, science and educational knowledge.

Look:

Reproduced from a school website - I can genuinely say that I'm totally flummoxed over what the graph is actually showing -- and in this case, what it is meant to be showing to the "non informed" (ie parent) reader.
Overall, I suppose the graph shows the limits of progress that we expect this learner to be making - framed within the boundary of aspirational to minimum progress. OK - I buy that. It also shows that this learner over the past three reporting cycles has improved from "minimum" to "aspirational" - and that at present, they are tracking at above aspirational progress.
But I have no idea what progress this learner is making; where they started and where they are at present. What does the Level 4b relate to? Where's the y-axis scale? What do the grid lines show? On what is this data based?
Buried in the commentary on the website it says that the "three straight lines represent what 3, 4 and 5 levels of progress would look like over the 5 years" - so (for the orange line) I could divide the 6 scale divisions into the 3 levels of progress and conclude that each scale division is half a level? Except for the 4 levels, its 9 divisions and for 5 levels its 10 divisions. Mmmmm
And exactly where does the "Level 4b" come in? Is that the students level at the "Starting Point"? If so, why does the y-axis start half way up the page - the implication is that the axis starts at Level 0?
All this on what is held up as an example of supporting documentation on a students end of term report.

Setting aside the educational / pedagogical reasons behind why I don't think this approach is sound and focusing just on the data representation for a moment:

Issues
Non existent y-axis / scale
A bigger issue than face value. For sure, you can't tell what the data is showing - only trends (which might be the point, by design) - but even then it's flawed. If I track forward along the orange line and see what this student is likely to get by the end of Year 11 - as there's no y-axis scale, I can't use the lines to predict, so they are meaningless. I can't have a conversation with this learner that goes "if you carry on as you are, you are likely to get XYZ at the end of Year 11". Pointless.

White space
More than half the chart is empty. I guess this is a "standard" chart and the y-axis is scaled so it fits for all learners. So what! As a parent, I want the best visualisation for my child, not some "one size fits no one" approach. Most of the x-axis is whitespace - this is because the axis is labelled with the upcoming assessments that this student has not had yet. Again - so what? Other than to make a "one size" chart that will do for all years, for all learners. Sloppy and lazy. Plus useless (see the previous point) - plus, for a bonus, it means I can't really see what's going on as the trend lines obscure the data.

Apples and Oranges & Granularity
The "starting point" is apparently externally marked "SAT" style examinations from Year 6. The remainder are internal tests, some teacher assessed AFL style and some more summative "exams". Surly we would expect this student to make different progress on different styles of testing? So, firstly, this chart is comparing differnt things and secondly, the granular detail of such comparisons can be misleading.

Some Excel-fu later: (not sure if I've got the "levels" right - but the shape of the graph is consistent)

Now at least I can have a conversation (with data / levels) about the progress that the student is making and the chart isn't mostly white space.

Call to Action
  1. Don't just turn out the standard charts that your MIS system produces - students are individuals
  2. Think about what the chart actually shows, compared to what you know it does -- field test with real parents
  3. Learn to use Excel


comments powered by Disqus