Gaming the System – the unintended consequences of measuring the wrong things

“There is a general rule in social and economic life” wrote W. Brian Arthur, an economist at the Santa Fe Institute, “given any system, people will find a way to exploit it. Or to say this more succinctly: All systems will be gamed”.^⁠

Last week’s news bears this out. The NHS in England has withdrawn its four-hour A&E waiting time target, in part because it led to the wrong behaviour from staff. The target was for 95% of patients to be treated, discharged, admitted or transferred within four hours. A noble goal in theory. But as the clock ticked ever closer to the metric’s red-line, A&E staff changed their treatment sequencing away from those who’d been triaged as having the most urgent need, towards those who would impact the metric.

It’s just one example of the way metrics, measures and monitors create behaviour that conforms to the target but runs counter to the policy’s original intention. Like a call pick-up measure in a bank contact-centre. Someone, somewhere, designs the system so that your call is immediately picked up, thus meeting their call answer target. But you then have to endure endless options and life-sapping elevator music before you get to speak to a real person. It makes their numbers look good, but as a customer experience, it’s the pits. Surely this wasn’t their intention?

People don’t engage in such behaviour because of an inherent human tendency towards selfishness or stupidity. It’s the policy, rather than the people, that’s at fault. Many performance metrics are designed around strict but simple evaluation criteria that doesn’t take into account the complexity of the environment they are working in. And when the system is rigged in a way that seems unfair or unworkable, but where there are consequences for not meeting the metric, you’ve created the perfect conditions for people to game the system.

In his book, The Tyranny of Metrics, Jerry Muller outlines the plethora of ways people do this. In some cases, like the examples above, it’s gamed by designing the system so that it technically meets the metric, but in doing so completely misses the point of the metric’s intention. It meets the letter, but not the spirit of what was intended.

Another way that the system is gamed is to improve the number by omission or distortion of the data. Independent schools in the UK, for example, often withdraw students from final exams if they perform poorly in their prelim exams so that the numbers they are judged on (actual exam results) are rigged towards high performers. Creaming the results is another gaming tactic. That is finding simpler targets or choosing clients with less challenges, such as the police tackling simple and minor crime with a good clean up rate rather than complex, longer-term cases.

Many leaders working in a complex system have very little sway on major structural decisions around the metrics, measures and monitors of their work, so the choice is look bad, or fudge the numbers. “The most characteristic feature of metric fixation” says Muller, “is the aspiration to replace judgement based on experience with standardised measurement”. In the case of the four-hour A&E waiting target, this means blending all patients together in an analogous whole. A sprained ankle is subject to the same target as a cardiac arrest.

How do we avoid it

There are many things that we can do to change this, but there is one that is immediately applicable. Shift the focus from measuring performance to guiding improvement. It’s a subtle but important pivot. If you use data to guide performance, then the data becomes a useful tool in helping teams determine where to make improvements. In order to achieve this, you need to transfer the ownership of the data to the team who generate it. If a team or division own their own data, in real time, for the purpose of guiding improvement, they are more likely to work productively with that data. If it’s owned by the senior leadership, by HR, or by ministers and regulators, then people will continue to game it.

Of course, not all measurement, metrics and monitors are defective. But like most flaws in leadership logic, it isn’t the use of approaches that are wholly bad that leads us to solve the wrong problems. In fact, it is precisely because in some settings these approaches are so successful that makes them so enticing. The danger is that we blindly adopt them without thinking about their appropriateness for the challenges we’re facing. We apply the wrong logic, followed by the wrong tool and then we stand back, scratch our head and wonder why we aren’t getting the right results. It will be interesting to see what Ministers will replace the A&E four-hour target with.

References

Muller, J. 2018 “The Tyranny of Metrics” Princeton University Press

Arthur, W. B. 2014 “All Systems will be Gamed: Exploitive Behavior in Ecomonic and Social Systems. Santa Fe Institute Working Paper.