Subversive metrics (surrogation)

Don't let metrics undermine your business by Harris and Taylor is a thought-provoking piece in the wonderful Harvard Business Review.

It concerns a tough old problem, that of metrics themselves becoming the focus of attention within the organization rather than the objects of measurement and, more importantly still, the business activities for which the metrics are intended to support improvement.

"Every day, across almost every organization, strategy is being hijacked by numbers ... It turns out that the tendency to mentally replace strategy with metrics — called surrogation — is quite pervasive. And it can destroy company value."

According to Wikipedia, Charles Goodheart advanced the idea in 1975, although I suspect people have been manipulating metrics and duping each other pretty much since the dawn of measurement. 

My eyes were opened to the issue by Hauser and Katz in Metrics: you are what you measure! Krag Brotby and I wrote about that in PRAGMATIC Security Metrics. 

Surrogation is surprisingly common in practice, for example "Thank you for your business. Please give five stars feedback after this transaction" is vaguely coercive, more so when appended with something along the lines of "My bonus depends on high scores" or "Visit our Facebook page to enter our prize draw".

Government officials and politicians do it all the time - it's a job requirement to know how to appear to be doing good things for the nation or society, regardless of reality. [The game cuts both ways: 'the opposition' is naturally expected to criticise the government's performance by challenging the results and/or the measures, perhaps simply casting doubt on their veracity.]

VW was famously caught doing it by having their engine management systems detect the conditions indicating that emissions testing was being performed, enabling the emission controls to ace those tests then disabling them to improve other aspects of performance (such as fuel economy) after the emissions tests were done. Sneaky - and a risky strategy as VW discovered to its cost and shame. I would be astonished to discover that VW was the only, or indeed the worst culprit though.

If a process or system is measured by a metric, and if the metric governs bonuses or other benefits for those performing the process, then they have an incentive to optimize the process/system and/or the metric: both routes lead to reward. Creative if unethical thinkers can often find ways to drive up apparent performance without necessarily improving actual performance, and if the bonuses or benefits are substantial, the pressure to do so can be strong.

One unethical way to optimize a metric is to manipulate the measurement process, for example selectively discounting, blocking or simply ignoring bad values, creating a bias such that the metric no longer truly represents the process being measured - an integrity failure. Comparative metrics such as benchmarks can be optimized by decreasing the actual or apparent (measured) performance of peers or other comparators: that may not align with business objectives and would generally be considered unethical. Subjective metrics can be manipulated by coercion of the people doing the measurement, at any stage of the process (data collection, analysis, reporting/presentation and consumption ... perhaps even way back at the metrics specification and design phase, or during 'refinements' of an existing metric).

The same thing applies, by the way, if those 'bonuses or benefits for good performance' are in fact penalties or disincentives for poor performance. Manipulating the measurement, analysis and reporting activities to conceal actual performance issues may be easier than addressing underlying problems in whatever is being measured, especially if the measurement aspects are poorly designed and lack adequate controls ... 

The risk of someone gaming, subverting or hacking the measurement processes and systems is, of course, an information risk, one that ought to be identified, evaluated and treated just like any other. The classical risk management approach involves:

• Considering the probability of occurrence (threats exploiting vulnerabilities) and the impacts or consequences of incidents with an obvious emphasis on critical or key metrics, plus any that lead directly to cash or convertible assets, such as the stock options commonly used as performance incentives for executives;
• Deciding what to do about the risks;
• Doing it, generally by implementing suitable measurement process controls such as monitoring and managing the processes/systems to pick up on and address any issues in practice, including obvious or more subtle signs of manipulation/gaming/coercion - a step in the risk management process that (in my experience) is woefully neglected when it comes to metrics. Metrics aren't fire-and-forget weapons.

That's enough for today. I'll return to explore the management and other controls around metrics at some future point. 

Meanwhile, please vote for this blog.
My bonus depends on it. 

Not really. It's a joke.

I don't get a bonus.

I'm lucky to get paid.