Security metrics pissing contest

A lengthy white paper ably if inadvertently demonstrates the value of the PRAGMATIC approach. "Using Security Metrics to Drive Action" includes a page or three of advice from a bunch of mostly big-company CISOs concerning the security metrics they use to communicate security program effectiveness to business executives and the board.

According to the report, Tenable asked 33 'IT security experts' the following question: "Your CEO calls and asks, “Just how secure are we?” What strategies and metrics do you use to answer that question?"

Unfortunately, there is little consensus among the 33 contributors, with stark discrepancies between them in some cases. They don't even discuss metrics and measurement strategies in the same terms. Most wax lyrical on their favorite (pet) metrics, although some seem confused about the term, referring vaguely to areas of concern rather than actual metrics. Some say more about how to present metrics than what metrics to present. Some promote technical metrics, while others say they prefer business metrics ... even if they then go on to discuss tech metrics! Some are happy to talk about the metrics they use within their teams but only reluctantly address CEO-level reporting mentioned in the question. 

In short, 33 CISOs equates to more than 33 opinions. Good luck making sense of that jumble of diverse and often contradictory advice. It's hard even to identify themes or clusters of agreement.

What's missing is a common conceptual framework enabling a productive discussion about the pros and cons of various metrics. Before recommending metrics, we first need a shared understanding about what they are and their purpose/s, plus the quality criteria for assessing security metrics. Without that, how can anyone sensibly choose between all the metrics on offer? How can we identify the points of agreement and gaps in the measurement blueprint if every contributor has their own unique perception of what the blueprint is? We're left with witchcraft and prejudice, pet metrics and biased personal opinions - a "my metrics are better than yours" pissing contest.

If you have the time and interest, the whitepaper is still worth reading and contemplating but please take all that sage advice with a large pinch of salt. Be very wary of adopting any security metrics until you figure out your own requirements. Before picking the answers, what are the questions?

Find out more at www.SecurityMetametrics.com and read our book!