On being cast adrift in a sea of metrics
With a spot of brainstorming and Googling around, it's not hard at all to come up with hundreds of candidate security metrics, often in fact entire families of potential metrics based on any starting point such as the 150 metrics in our book (we'll show you how that works with our next 'example metric of the week', here on the blog). There are loads of information-security-related things that could be measured, and loads of ways to measure them. This is a point we discussed in chapter 3, describing many potential sources of metrics inspiration.
If you don't perceive a vast ocean
of possible security metrics before you,
you're either lacking in experience
or you need to look harder!
or you need to look harder!
Having come up with a big bunch of possible security metrics, the PRAGMATIC method is a great way to filter out the few that are actually worth putting into production. Metrics with relatively low PRAGMATIC scores naturally gravitate to the bottom of your list while the high-achievers gently rise to the top. Instead of feeling overwhelmed with a confusing mass of possibilities, your job is simply to cream off the floaters, perhaps revisiting a few more that show promise but don't quite make the grade.
Aside from quietly contemplating your shortlist, metrics workshops* work well in many organizations, bringing the people who will generate and use the metrics together to consider and discuss their objectives, requirements and constraints, and to pore over a set of candidate metrics. Another suggestion is to run trials or pilot studies, trying out a few metrics and comparing them side-by-side for a few months to discover which ones work best in practice. Don't forget to ask your audiences what they make of the metrics, which ones they prefer, and why.
The GQM (Goal -> Question -> Metric) approach is yet another way to figure out what to measure. GQM doesn't necessarily lead to particular metrics, but it emphasizes the business needs and priorities first, using those to focus attention on particular questions or issues of concern in the management of information security risks. The strategic perspective is well worthwhile, at least in suggesting what kinds of security metrics are needed and why i.e. the areas or aspects that ought to be controlled and hence measured.
Furthermore, the manner in which your metrics are analyzed and presented is another opportunity for creative expression: the graphs and images that illustrate this blog are deliberately bright and arguably a bit weird in order to catch your eye. The more formal corporate reporting situation may be different although we would advise against using monochrome line/bar/pie charts unless you have to, for some reason. Security metrics needn't be as dry, dull and boring as a badly-delivered statistics lecture. Try a splash of color at the very least. Let your passion for the subject shine through. You never know, it might just rub off on the audience ...
* If you'd like me or Krag to lead your metrics workshop, do drop us a line. Actually using the PRAGMATIC method for real is an obvious next step once you have read the book. As well as sharing our passion, knowledge and experience in this field with you and your management, we'd welcome the chance to bring you quickly up to speed on PRAGMATIC as well as helping you address your security metrics issues.