SMotW #90: % of business units with proven I&A

Security Metric of the Week #90: proportion of business units using proven identification and authentication mechanisms

This metric hinges on the meaning of "proven". Proof is a relative term. What level of proof is appropriate? It's a matter of assurance, trust and risk.

ACME managers implicitly assumed* that the metric would be self-measured and reported by business units. Given a central mandate from HQ to implement specific controls, business units are obviously under pressure to confirm that the required controls are in place ... even if they actually are not. Aside from the risk of business units simply reporting whatever HQ expects to hear, there is also a distinct possibility that the business units might have misunderstood the requirement, and failed to implement the control effectively (perhaps mis-configuring their security systems).

That brings us to the matter of the nature and extent of control implementation. If a business unit has the required identification and authentication (I&A) mechanism in place for some but not all of their systems, how should they report this? What if they have made a genuine effort to implement it on most systems, but the few that remain are particularly important ones? What if the identification part is working as per the spec but the authentication isn't, perhaps using a different mechanism for valid business or technical reasons? There are several variables here, making it tough to answer honestly a typically naive checklist question such as "Are your IT systems using the proven I&A mechanisms required in the corporate security standards (Y/N)?"

On that basis, the managers gave this metric a PRAGMATIC score of just 44%, held back by abysmal ratings for Genuineness and Independence (see page 207 in PRAGMATIC Security Metrics). 

The metric is not necessarily dead in the water, though, since it would be possible to address their main concerns through some form of independent assessment and reporting of the I&A mechanisms. Certifying IT systems is something rarely seen outside large military and governmental organizations, who have the governance structures in place to:

1. Define security requirements including technical controls such as specified I&A mechanisms, methods, software etc.;
2. Mandate those requirements on the various business units;
3. Implement the controls locally, often with central support (e.g. technical support plus standards, procedures and guidelines);
4. Accredit certification functions who are competent to test and certify business units' compliance with the security requirements;
5. Test and certify the business units, and re-test and re-certify them periodically;
6. Deal with any noncompliance.

That little lot would generally be viewed as an expensive luxury for most organizations (impacting the metric's Cost-effectiveness rating), although the global spread of ISO/IEC 27001 certification is gradually assembling most of those pieces, and making more organizations familiar with the concept of accredited certification.

Meanwhile, ACME quietly parked this metric in the "too hard for now" bucket, pressing ahead with the higher-scoring metrics still on their shortlist.

* PS Unless someone present happens to notice and point out assumptions like this, they tend to remain unspoken, and are a frequent cause of misunderstandings. At some stage (perhaps after a PRAGMATIC workshop has shortlisted a reasonably small number of metrics thought worth pursuing), the metrics ought to be specified in sufficient detail to dispel such doubts. Several security metrics standards and websites give examples of the forms typically used to specify metrics, although most appear obsessed with the statistics, often neglecting valuable information such as the reasoning behind and justification for the metrics, the intended audiences and so forth. I'm sure "How should we specify security metrics" would spawn an interesting thread on the Security Metametrics group on Linkedin ...