PRAGMATIC Security Metrics sample chapter

The publisher has released chapter 2 as a sample of the book.   Chapter 2 concerns the reasons or purposes for using security metrics.

Over on CISSPforum, we've just been chatting about the validity of security certifications such as SSAE16 (formerly SAS70), PCI-DSS and ISO/IEC 27001.  These schemes use formal compliance audits to determine whether organizations deserve to be certified.  Formal compliance audits are based around highly specific and explicit requirements, leaving the auditor and auditee little leeway to "interpret" the obligations.  It is implied that such "interpretation" would be a bad thing, although there is a strong argument to the contrary.

Unfortunately for this approach, information security refuses to be put into a neatly labelled box.  It simply is not possible to specify security control requirements in sufficient detail for the classic compliance audits, in a generic or universal one-size-fits-all specification.  There are far too many variables.  A control that might be entirely suitable for one organization might be over the top for another, hopelessly inadequate for a third, and totally inappropriate for a fourth.  Yes, context matters.  Nobody has succeeded, thus far, in writing a formal security controls specification that takes full account of all the different circumstances in which it needs to be applied, at least not without resorting to "recommendations" and "implementation guidance" such as in ISO/IEC 27002.  Aside from anything else, a comprehensive requirements specification would be so complex and wide-ranging as to be virtually unmaintainable, which is exactly the situation with '27002 right now.

Some industry sectors have done their best to create generic requirements specifications tailored to organizations in those industries.  SSAE16, PCI-DSS and ISO/IEC TR 27015 are examples from the financial services industry.  It is generally acknowledged that these are not very effective security standards, since the controls that are commonly applicable enough to be formally defined as requirements are a subset of the controls each organization actually needs to be considered secure.  They are the lowest common denominators of security.  Adopting the required controls falls well short of actually being secure.

Worse still, compliance certification has become a game, a costly* diversion of scarce resources from the vital business of addressing information security risks.  The evolution of PCI-DSS demonstrates this, with successive releases being more and more carefully word-crafted to make it harder for organizations to wheedle out of their security compliance obligations "on a technicality".  That PCI-DSS allowed organizations to continue using WEP and WPA long after these were known to be pathetic controls illustrates the problem.

Let's look at this from a different perspective.  SSAE16, PCI-DSS and ISO27k are being used to assess  organizations' security by auditing their compliance with generic security specifications.  They are security metrics.  Therefore, they can be evaluated and improved using the PRAGMATIC method.  What's more, there are many other potential metrics, some of which might prove even more valuable for the purposes of external stakeholders - primarily for assurance reasons but also for accountability etc. (see section 2.4 in chapter 2!).  I feel it's time to break away from the myopic formal-specification-compliance-audit-and-certification mindset, with decent security metrics for external as well as internal stakeholders in the organization's information security status being the key that unlocks much more creative and ultimately more effective approaches.  

What do you think?

* Costly for organizations being certified, costly for their customers, employees and other parties who lose out due to inadequate information security, but highly profitable for the small army of certification bodies, certification auditors and training companies gathered around the trough.