Everythin wrong with school data visualisation in one place...

A grand title, but when I received this via email, I genuinely didn't beleive it until I checked out the web site it came from: (here reproduced as 100% from the school site)
Under the title: "How much progress do pupils make between 11 and 16?" is this chart:

As an educationalist
Firstly, lets imagine we are a teacher, slt or other "educationalist" that deals with data like this on a daily basis - do you understand this plot? Assuming you know how to interpret CVA scores and have seen charts like this before, in this case, it make absolutely no sense. None, nada, zilch, zip. Given that the date on this chart is 2008 and that the image was uploaded in May 2011, the chart has existed for 7 years and has been visible on-line at least 4 years - why has no one pointed out to the school that the chart is meanigless?

What's missing is an indication of the CVA for this school - a big fat "X" marks the spot - nothing, missing. The chart just shows the CVA for all schools - we can't pick out the school (and the CVA is not listed on the web site).

Had the chart looked like this, with the school listed as X or Y - we could have made a judgement on the relative performance of the school based on the CVA (X being within the top 10% and Y within to bottom 10%):

As a parent
Ignoring the fact that the chat is missing the school data, so it is of no use, and assuming the chart looked like my revision above, with a big fat "X" - would we expect a parent to be able to interpret and draw conclusions from such a chart? I think not.

Dangerous assumption - Parents know what CVA is and can interpret these charts
Fair point and the school acknowledges this by trying to explain it:

"The chart shows our school’s contextual value added (CVA) score relative to that of other secondary schools" - good start, but not really explaining it. And the chart doesn't show this as you forgot to actually plot it.

"CVA is a statistical means of assessing the effectiveness of a school, by measuring pupils’ progress using their test and examination results." - still not great, starts to sound a bit "edubabble"

"The confidence interval shows the range within which we can be confident the score (calculated on the results of only one year group) represents the overall effectiveness of a school." - Nope, "edubabble" and you've not actually plotted this either.

"The percentile rank shows the percentage of schools with a score equal to or higher than ours." - Nope - (I assume you mean the x-axis) but you've not labelled it so how would a parent know what you're on about. More importantly, the x-axis on a CVA chart doesn't show this. It shows a distribution of CVA scores and shows where a school compares to the national distribution - not other schools relative to your school.

Interestingly, the 2011 Ofsted inspection graded this school as "outstanding" - maybe, but not at communicating to parents -- but I guess we don't value that.
Is the chart needed at all?
Assuming the audience of the website and the data is parents, I'm not sure that a CVA chart is helpful at all - especially when broken. Assuming a CVA of 1038, would this be more meaningful for the audience?

Call to action
  • Make sure your data makes sense
  • Proof read your web site
  • Update it more often
  • Ask your PTA to comment on the "sense" of your charts
  • Consider if a chart is needed at all - less is more

comments powered by Disqus