Expressing security metrics in business terms
On the Security Leadership Solutions Executive Council Faculty Advisor blog, Kathleen Kotwica raised a good point about ineffective security metrics:
"The issue many security practitioners incur is a) not measuring at all or b) measuring things by simply counting them (e.g., workplace violence incidents or lost laptops), rather than demonstrating the value Security brings to the business. By way of example, convey savings to the company by your program's reduction of workplace violence issues. That is, the cost of managing an event and lost employee time; or cost savings by reducing any potential acts because of your background due diligence program."
Although the blog concerns physical security metrics, Kathleen's point applies equally to the broader class of information security metrics. And the issue is not simply about counting things, but about presenting simple counts, or indeed other statistics, without giving them relevance and meaning to the business.
Suppose, for instance, I include the number "27" in a management report. The bare number, by itself, is meaningless, begging the question "27 what?" The 'what' is the unit, for example "27 security vulnerabilities." That may be a valid data point but it barely qualifies as information without additional context, and it's a long way from providing worthwhile knowledge and insight that will help run the business.
One way to enrich a metric is to indicate the trend, for example "This reporting period, the number of security vulnerabilities fell from 29 to 27" ... mmm, better but still rather obscure. A reduction in the number of vulnerabilities sounds like it is probably a good thing, but the reduction from 29 to 27 is hardly a dramatic fall ... unless it has been stubbornly stuck at 29 for ages, or had been increasing over prior periods. Plotting (graphing) the number over time is the simplest way to show trends, while we can get creative with the axes, annotation and colors to add yet more information, so long as it actually helps. [The gaudy figures we use to illustrate this blog are deliberately intended to catch your attention and fire up your imagination: our style may look out of place in most internal corporate reports, but why does everything have to be so drab and lifeless? Banish monochrome dullness!]
Annotations can be text pointing out interesting, relevant features directly printed on the graph, or brightly colored numbered/lettered blobs on the graph linking to a separate panel of text, or better still a presenter physically pointing to items of interest and discussing them. Textual descriptions are usually required for any specialist terms on the graph, not least the title or caption.
The wording of the annotations/text is very important, yet few statistics or metrics books go into this aspect. Compare, for instance "Vulnerability Reduction Project comes into effect" with "Vulnerabilities fall at last!" The former is informative but flat, while the latter makes a more emphatic statement, which may or may not be appropriate. These are extremely trivial examples, but I'm sure you can think of real-life situations where suitable or unsuitable text would make a huge difference to the way the same metric is perceived. Look through any security survey for examples of how to point out and expand on the numbers. It takes skill to do this well, without misleading or boring the pants off the audience and none of us gets it 100% right - not least because 'the audience' is often a wide range of people with widely differing perspectives and information needs. Some want just the facts, others need to be led through the analytic thinking process, perhaps as far as pointing out the key conclusions for them.
Getting back to Kathleen's point, expanding on the business implications of the numbers is generally a sensible move for management audiences, which means pre-considering the obvious response "So what?" Why should anyone care that the number of vulnerabilities is 29? What does that figure say to those making decisions about security risks and controls? If you can't answer such questions convincingly in a way that makes sense and motivates the audience to do something appropriate with the information you are providing, then you should not be surprised if the security metric has no appreciable effect on your universe. And don't be upset if I suggest that the metric is a waste of valuable space.
Consider, also, turning this entire discussion on its head. Start by figuring out the issues that need to be addressed, and the questions that doing so will raise, then generate the metrics accordingly to address the questions. Make the entire metrics selection/design process business-led from the outset, rather than attempting to justify your metrics in business terms after the fact. That way, security metrics have a clear purpose, so the context and relevance are easy to determine - in fact, if highly PRAGMATIC metrics may not need to be 'explained in business terms' in an explicit fashion, since to a large extent they will be self-evident, obvious even. The Actionability and Meaningfulness of a metric are specifically assessed for this very purpose, while Relevance and the remaining PRAGMATIC criteria play their parts too.
Consider, also, turning this entire discussion on its head. Start by figuring out the issues that need to be addressed, and the questions that doing so will raise, then generate the metrics accordingly to address the questions. Make the entire metrics selection/design process business-led from the outset, rather than attempting to justify your metrics in business terms after the fact. That way, security metrics have a clear purpose, so the context and relevance are easy to determine - in fact, if highly PRAGMATIC metrics may not need to be 'explained in business terms' in an explicit fashion, since to a large extent they will be self-evident, obvious even. The Actionability and Meaningfulness of a metric are specifically assessed for this very purpose, while Relevance and the remaining PRAGMATIC criteria play their parts too.