PRAGMATIC Security Metric of the Quarter #3
PRAGMATIC Security Metric of the Third Quarter
These are the example information security metrics we have discussed and scored over the past three months, ranked in descending order of their PRAGMATIC scores:
| Example metric | P | R | A | G | M | A | T | I | C | Score |
Metametrics | 96 | 91 | 99 | 92 | 88 | 94 | 89 | 79 | 95 | 91% |
Access alert message rate | 87 | 88 | 94 | 93 | 93 | 94 | 97 | 89 | 79 | 90% |
| Asset management maturity | 90 | 95 | 70 | 80 | 90 | 85 | 90 | 85 | 90 | 86% |
| Compliance maturity | 90 | 95 | 70 | 80 | 90 | 85 | 90 | 85 | 90 | 86% |
| Physical security maturity | 90 | 95 | 70 | 80 | 90 | 85 | 90 | 85 | 90 | 86% |
Thud factor | 82 | 80 | 60 | 60 | 70 | 45 | 85 | 86 | 84 | 72% |
| Business continuity spend | 75 | 92 | 20 | 82 | 95 | 70 | 70 | 70 | 70 | 72% |
| Benford's law | 84 | 30 | 53 | 95 | 11 | 98 | 62 | 98 | 23 | 62% |
| Controls coverage | 87 | 89 | 65 | 40 | 74 | 35 | 46 | 40 | 30 | 56% |
| Homogeneity | 67 | 70 | 40 | 59 | 67 | 50 | 33 | 65 | 45 | 55% |
| Access control matrix status | 70 | 50 | 60 | 60 | 88 | 25 | 40 | 20 | 40 | 50% |
Unaccounted software licenses | 1 | 1 | 90 | 84 | 1 | 70 | 50 | 81 | 30 | 45% |
Unauthorized/invalid access count | 61 | 78 | 33 | 16 | 33 | 0 | 44 | 35 | 33 | 37% |
On that basis, we are happy to announce that the Information Security Metric of the Quarter is <cue drum roll> metametrics meaning a systematic measure of the quality (utility, value, fitness for purpose etc.) of the organization's information security metrics.
Clearly we have in mind here the PRAGMATIC approach, but in fact other/variant approaches are possible, so long as there is a sensible, rational way of assessing the metrics that makes sense to management. The key point of the metric, and the reason it scores so highly, is that measuring the quality of the organization's security metrics is an important step enabling management to improve the suite of metrics in a systematic manner, rather than the much more common ad hoc approach. Better information security metrics, in turn, allows management to get a firm grip on the organization's information security arrangements, bring them under control, and improve them systematically too. There is in fact a positive feedback loop at play, since better, more reliable and suitable information security arrangements generate better, more reliable and suitable data concerning information security risks and controls, in other words better metrics.
Clearly we have in mind here the PRAGMATIC approach, but in fact other/variant approaches are possible, so long as there is a sensible, rational way of assessing the metrics that makes sense to management. The key point of the metric, and the reason it scores so highly, is that measuring the quality of the organization's security metrics is an important step enabling management to improve the suite of metrics in a systematic manner, rather than the much more common ad hoc approach. Better information security metrics, in turn, allows management to get a firm grip on the organization's information security arrangements, bring them under control, and improve them systematically too. There is in fact a positive feedback loop at play, since better, more reliable and suitable information security arrangements generate better, more reliable and suitable data concerning information security risks and controls, in other words better metrics.
That said, we fully accept our own obvious bias in this matter. Having invented and written about the PRAGMATIC approach, we inevitably see metametrics through rose-tinted glasses. Things may not be quite so rosy from your perspective, and that's fair enough. But when you have a moment to yourself, take another look at the 13 metrics on the summary table above, plus those covered in the previous 2 quarters (browse back through the blog, or visit the Security Metric of the Quarter #1 and Security Metric of the Quarter #2), and draw your own conclusions. You probably disagree with us on the scoring of some of the metrics (even in the hypothetical context of an imaginary company). But, overall, do you accept that this is a reasonably straightforward, sensible way to consider, compare and contrast metrics? Would you agree that, on the whole, the metrics that score well on the PRAGMATIC scale are better than those that score badly? Is this discussion about the pros and cons of security metrics, using metametrics, something that you might use back at the ranch?
Bottom line: if we have persuaded you that the PRAGMATIC approach has merit, perhaps even that it might be a valuable addition to your arsenal of security management techniques, read the book for the full nine yards. This blog is just a taster of what's to come.
Gary & Krag
Gary & Krag