The words ‘statistics’ and ‘measures’ often strike boredom into the hearts of even the nerdiest. Especially when we’re talking about ‘measuring impact’ or any similar corporate jargon phrase.
But this week’s news about the uncertainty behind reports of ‘falling NHS productivity’ just shows how crucial and controversial these measures can be. You might think that before you make a huge change to the health system, you would want to know how it’s performing. The Lancet has shone a spotlight on the fact that it’s very, very difficult to measure this, and it’s not actually clear whether NHS productivity is going up or down.
This isn’t a trivial thing. Measures might be a bit soul-destroying at times, but they are literally the raw material of science, so ‘evidence-based’ policymakers should always be looking very hard at the measures they’re using.
The story behind the Lancet article isn’t that ‘actually productivity is rising’. The problem is that we don’t know. As the article explains, there were so many uncertainties in the data and how it was interpreted that the actual message became ‘more research is needed’.
But that soundbite doesn’t make a very interesting headline, and it’s no use to politicians trying to argue their case (whatever that might be). As the Lancet article suggests, ‘uncertainty among health services researchers as to how best to measure improvement in quality engendered scepticism about any method, which, in turn, led to the dismissal of the issue as unimportant.’
Quite obviously, the fact that we don’t have a good way to measure how well the NHS is doing is NOT unimportant. So it’s a good thing that this problem is being highlighted. But regardless of what happens with the proposed changes to the NHS, there’s an even bigger question – how do we know when our healthcare system is doing well?
I don’t have an answer, except my old friend ‘more research is needed’. It might not be sexy or cool or even controversial, but more research IS needed. The quest to improve research and improve ‘measures’ is the unsung hero of evidence-based anything – policy, science, life as we know it. Otherwise we’re just flailing our arms in a sea of unknown unknowns, or the slightly less threatening known unknowns.
I hope that message comes out of this latest controversy. As the Lancet article concludes, ‘accurate and competing estimates of health-care productivity are needed… that translate improvements in quality into generic measures’. Researching research is not a waste of time – it’s the only way to make sure we have the best evidence to make decisions, whatever those decisions end up being.