PRAGMATIC Security Metric of the Quarter #7

PRAGMATIC Information Security Metric of the Seventh Quarter

According to the overall PRAGMATIC scores assigned by ACME's managers, the latest metric discussed was the top choice in the three months just past, but it was a close-run thing:

Example metric	P	R	A	G	M	A	T	I	C	Score
Information security incident management maturity	90	95	70	80	90	85	90	85	90	86%
Information security ascendancy	97	87	15	94	86	90	99	97	99	85%
Quality of system security	83	88	83	73	90	68	80	82	10	73%
Integrity of the information asset inventory	82	66	83	78	80	43	50	66	70	69%
Proportion of systems security-certified	72	79	73	89	68	32	22	89	88	68%
Number of different controls	71	75	72	75	88	30	50	65	43	63%
Controls consistency	78	83	67	60	71	33	27	31	27	53%
Value of information assets owned by each Information Asset Owner	48	64	78	57	79	38	50	22	26	51%
Number of information security events and incidents	70	60	0	50	72	35	35	70	50	49%
% of business units using proven identification & authentication	69	73	72	32	36	4	56	2	50	44%
Distance between employee and visitor parking	1	0	6	93	2	93	66	45	66	41%
Employee turn vs account churn	30	30	11	36	44	36	62	57	20	36%
Non-financial impacts of information security incidents	60	65	0	20	60	6	30	20	17	31%

"Maturity of the organization's information security incident management activities" seems to us to be an excellent proxy or indicator for the organization's overall approach to information security. The maturity scoring process we have described makes this a valuable metric, not just in terms of the final maturity rating but also the additional information that emerges when comparing current practices against accepted good practices.

Just as interesting are the metrics languishing at the bottom of the league table. For example, "Non-financial impacts of incidents" may appear, at first glance, to hold considerable promise as a security metric but the PRAGMATIC score clearly indicates ACME management's severe misgivings once they explored the metric in more detail.

Instead of simply selecting metrics on the basis of their the overall PRAGMATIC scores, management could instead select high-rating metrics for any one of the individual PRAGMATIC criteria, or any combination thereof - for example, 'information security ascendancy' is rated the most predictive and cost-effective security metric of this little lot.

In researching and developing the PRAGMATIC method for the book, we explored the possibility of weighting the PRAGMATIC ratings in order to place more or less emphasis on the criteria. There may be situations where that is a sensible approach but, in the end, we decided that the overall PRAGMATIC score was the most valuable and straightforward metametric.

PRAGMATIC Security Metric of the Quarter #6

The league table for another 3-month's information security metrics shows a very close race for the top slot:

Metric	P	R	A	G	M	A	T	I	C	Score
Power consumed by the computer suite versus air conditioning capacity	81	69	89	92	80	99	98	90	98	88%
Security governance maturity	95	97	70	78	91	89	90	85	90	87%
Business continuity plan maintenance status	75	75	90	73	84	76	80	77	93	80%
Number or proportion of security policies addressing viable risks	65	76	91	73	83	77	70	61	78	75%
Quality of security policies	80	85	40	66	72	75	80	80	80	73%
Number of controls meeting defined control criteria or objectives	88	86	88	65	78	60	26	90	70	72%
Corporation's economic situation	72	80	10	80	80	80	61	80	79	69%
% of highly privileged or trusted users or functions	86	80	51	40	65	39	55	95	60	63%
Time lag between incident and detection	80	70	72	30	75	50	50	65	65	62%
Number of unapproved or unlicensed software installations identified on corporate IT equipment	58	55	82	73	86	47	64	66	17	61%
Extent to which QA is incorporated in information security processes	75	70	66	61	80	50	35	36	50	58%
% of incidents for which root causes have been diagnosed and addressed	85	85	67	40	77	40	48	16	40	55%
Psychometrics	40	24	0	79	15	55	10	42	5	30%

[Click any metric to visit the original blog piece that explained the rationale for ACME's scoring.]

Hopefully by now you are starting to make out themes or patterns in the metrics that score highly on the PRAGMATIC scale.

Having so far discussed and scored more than half of the example metrics from the book, plus a bunch more metrics from other sources, there's a fair chance we have covered some of the security metrics that your organization currently uses. How did they do? Do the PRAGMATIC scores and the discussion broadly reflect your experience with those metrics?  

We would be amazed if your metrics rate exactly the same as ACME's but if any of your scores are markedly higher or lower, that itself is interesting (and we'd love to hear why - feel free to comment on the blog or email us directly). The most likely explanation is that you are interpreting and using the metric in a way that suits your organization's particular information security management needs, whereas ACME's situation is different. Alternatively, it could be that you are applying the PRAGMATIC criteria differently to ACME (and us!). To be honest, it doesn't matter much either way: arguably the most important benefit of PRAGMATIC is that is prompts a structured analysis, and hopefully a rational and fruitful discussion of the pros and cons of various security metrics. 

PRAGMATIC Security Metric of the Quarter #5

Example Information Security Metric of the Fifth Quarter

The PRAGMATIC scores for another 3-month's worth of information security metrics examples are as follows:

Example metric	P	R	A	G	M	A	T	I	C	Score
Information access control maturity	90	95	70	80	90	80	90	85	90	86%
Security policy management maturity	90	95	70	80	88	85	90	82	88	85%
Number of important operations with documented & tested security procedures	95	96	91	85	95	84	62	90	60	84%
Information security budget variance	70	90	85	77	80	77	80	90	95	83%
% of information assets not [correctly] classified	75	75	97	85	90	80	80	80	80	82%
Policy coverage of frameworks such as ISO/IEC 27002	70	75	90	69	85	76	72	65	85	76%
% of policy statements unambiguously linked to control objectives	92	91	64	60	85	65	45	75	75	72%
Rate of change of emergency change requests	64	71	69	73	78	70	70	69	83	72%
Total liability value of untreated/residual risks	88	98	59	33	96	33	77	38	10	59%
Entropy of encrypted content	78	66	23	78	3	93	74	79	34	59%
Embarrassment factor	26	38	20	50	63	72	40	87	87	54%
% of security policies that satisfy documentation standards	66	47	79	45	74	38	44	50	35	53%
Patching policy compliance	66	52	55	77	19	36	11	8	5	37%

Top of the heap are two maturity metrics scoring 85% and 86%, with a further 3 metrics also scoring in the 80's.

While it is tempting to recommend these and other high-scoring metrics to you, dear reader, please bear in mind that they were scored in the context of a fictional manufacturing company, Acme Enterprises Inc.  The scores reflect the perceptions, prejudices, opinions and needs of Acme's managers, given their current situation.  Things are undoubtedly different for you.  We don't know what's really important to you, your managers and colleagues, about information security.  We have no idea which aspects are of particular concern, right now, nor what might be coming up over the next year or three.  Hence we encourage you to think critically about the way we describe the metrics, and preferably re-score them.   

Furthermore, PRAGMATIC scores alone are not necessarily a sound basis on which to select or reject metrics.  It's not that simple, unfortunately, despite what you may think given the way we bang on and on about PRAGMATIC!  The scores are intended to guide the development of an information security measurement system, a well-thought-out suite of metrics plus the associated processes for measuring and using them.  Considering and scoring each security metric in isolation does not build the big picture view necessary to measure information security as a coherent and integral part of the organization's management practices.

The book describes PRAGMATIC scoring as the heart of a comprehensive method, an overall approach to information security metrics.  The method starts by figuring out your metrics audiences and their measurement requirements, building a picture of what you are hoping to achieve.   Knowing why certain security metrics might or might not be appropriate for your organization is arguably even more important than knowing which specific metrics to choose ... but, that said, the act of gathering, contemplating, assessing and scoring possible metrics turns out to be a productive way both to determine and to fulfil the needs.  It's a deliberately pragmatic approach, a structured method that achieves a worthwhile outcome more effectively and efficiently than any other approach, as far as we know anyway.  Perhaps you know different?

PRAGMATIC Security Metric of the Quarter #3

PRAGMATIC Security Metric of the Third Quarter

These are the example information security metrics we have discussed and scored over the past three months, ranked in descending order of their PRAGMATIC scores: 

Example metric	P	R	A	G	M	A	T	I	C	Score
Metametrics	96	91	99	92	88	94	89	79	95	91%
Access alert message rate	87	88	94	93	93	94	97	89	79	90%
Asset management maturity	90	95	70	80	90	85	90	85	90	86%
Compliance maturity	90	95	70	80	90	85	90	85	90	86%
Physical security maturity	90	95	70	80	90	85	90	85	90	86%
Thud factor	82	80	60	60	70	45	85	86	84	72%
Business continuity spend	75	92	20	82	95	70	70	70	70	72%
Benford's law	84	30	53	95	11	98	62	98	23	62%
Controls coverage	87	89	65	40	74	35	46	40	30	56%
Homogeneity	67	70	40	59	67	50	33	65	45	55%
Access control matrix status	70	50	60	60	88	25	40	20	40	50%
Unaccounted software licenses	1	1	90	84	1	70	50	81	30	45%
Unauthorized/invalid access count	61	78	33	16	33	0	44	35	33	37%

On that basis, we are happy to announce that the Information Security Metric of the Quarter is <cue drum roll> metametrics meaning a systematic measure of the quality (utility, value, fitness for purpose etc.) of the organization's information security metrics.  

Clearly we have in mind here the PRAGMATIC approach, but in fact other/variant approaches are possible, so long as there is a sensible, rational way of assessing the metrics that makes sense to management.  The key point of the metric, and the reason it scores so highly, is that measuring the quality of the organization's security metrics is an important step enabling management to improve the suite of metrics in a systematic manner, rather than the much more common ad hoc approach.  Better information security metrics, in turn, allows management to get a firm grip on the organization's information security arrangements, bring them under control, and improve them systematically too.  There is in fact a positive feedback loop at play, since better, more reliable and suitable information security arrangements generate better, more reliable and suitable data concerning information security risks and controls, in other words better metrics. 

That said, we fully accept our own obvious bias in this matter.  Having invented and written about the PRAGMATIC approach, we inevitably see metametrics through rose-tinted glasses.  Things may not be quite so rosy from your perspective, and that's fair enough.  But when you have a moment to yourself, take another look at the 13 metrics on the summary table above, plus those covered in the previous 2 quarters (browse back through the blog, or visit the Security Metric of the Quarter #1 and Security Metric of the Quarter #2), and draw your own conclusions.  You probably disagree with us on the scoring of some of the metrics (even in the hypothetical context of an imaginary company).  But, overall, do you accept that this is a reasonably straightforward, sensible way to consider, compare and contrast metrics?  Would you agree that, on the whole, the metrics that score well on the PRAGMATIC scale are better than those that score badly?  Is this discussion about the pros and cons of security metrics, using metametrics, something that you might use back at the ranch?

Bottom line: if we have persuaded you that the PRAGMATIC approach has merit, perhaps even that it might be a valuable addition to your arsenal of security management techniques, read the book for the full nine yards.  This blog is just a taster of what's to come.

Gary & Krag