A PRAGMATIC security/privacy compliance metric

In the course of considering how to measure an organization's compliance with security and privacy related obligations, the PRAGMATIC method has proven itself a valuable way to structure the analysis.  Today I want to discuss how taking the PRAGMATIC approach led me to design a better compliance metric by addressing the weaknesses in one of the candidate metrics.

I started by brainstorming possible ways to measure security/privacy compliance activities, focusing on the key factors or parameters that are most likely to be of interest to management for decision making purposes.  With a bit of Googling and creative thinking in odd spare moments over the course of a few days, I came up with a little collection of about 8 candidate compliance metrics:

• The rate of occurrence of security/privacy-related compliance incidents, possibly just a simple timeline or trend, but ideally with some analysis of  the nature and significance of the incidents;
  
  
• A 'compliance status' metric derived through reviews, audits or assessments across the organization;
  
  
• Compliance process maturity using a maturity scale; 
  
  
• 'Compliance burden'.  Management would presumably be quite keen to know how much compliance is really costing the organization, and could use this information to focus on areas where the costs are excessive;
  
  
• Plus 4 other metrics I won't bother outlining right now, plus an further undetermined number of minor variants. 

In exploring the 'compliance burden' metric idea, it occurred to me that although it is technically possible for management to attempt to measure the time, effort and money spent on all security/privacy-related compliance-related activities such as compliance reviews/audits, disciplinary action, legal and other enforcement actions, it would be difficult and costly to measure all aspects accurately.  There is also the issue of 'double-accounting', in other words categorizing costs under multiple accounting headings and so artificially inflating the total.

However, simply recording, tracking and periodically reporting security/privacy-related enforcement actions (i.e. penalties imposed, disciplinary actions taken, successful prosecutions etc.) would significantly reduce the Cost (and complexity) of the metric, and at the same time makes it more Accurate, Meaningful and Relevant.  Focusing on enforcement improves the metric's Independence too since enforcement actions are almost invariably formally recorded somewhere, making it much harder for someone to falsify or ignore them - which a manager might well be tempted to do if, say, the metric reflects badly on his/her department.

The icing on the cake is that the metric remains highly Actionable: it is patently obvious that a department with a bad record of enforcement (e.g. a string of costly noncompliance penalties) needs to up its game, significantly improving its compliance efforts to reduce the threat of  further enforcement actions.  Since most enforcement actions either have direct costs (fines and legal bills), or the costs can be quite easily calculated or at least estimated, the metric could be expressed in dollars, resulting in the usual galvanizing effect on management.  

Creative managers might even be prompted to initiate enforcement actions against third parties who fail to comply with the organization's security/privacy requirements imposed through contractual clauses, nondisclosure agreements etc., since successful actions might offset enforcement actions against the organization and so improve the metric in their areas of responsibility.

This, then, is an example of an indicator: measuring enforcement actions, specifically, does not account for the full costs of compliance but looks to be a reasonable analog.  Over time, I anticipate management improving compliance activities to bring negative enforcement costs down and positive enforcement actions up to acceptable levels - the metric should gradually level off and act as a natural restraint against excessive, overly-aggressive and counterproductive compliance actions.

That's it for now.  I won't elaborate further on using the PRAGMATIC scores to rank the candidate metrics or to guide the design and selection of the best variants of the 8 metrics I started with, but if you have specific questions, please comment on this blog or raise it on the SecurityMetametrics forum.