Hannover/Tripwire metrics part 4

Today we're looking at the next two security metrics from the Hannover Research/Tripwire CISO Pulse/Insight Survey.  

The fourth most popular metric, "Platform compliance scores", isn't giving much away with just those three words to go on.  What platforms?  Compliant with what?  And how are they scored?   The fact that the survey was sponsored by Tripwire is a massive clue.  The surveyors and respondents were presumably thinking on the basis of using a technical tool to assess various IT system configuration parameters against something - possibly comprehensive corporate technical security standards, more likely recommendations from the tool and operating system vendors.  Such tools typically generate relatively simple metrics: counts of systems checked, non-compliance issues found and so forth.  They often classify issues by severity, although how they do so is uncertain ...  This all sounds fantastic, but how does the metric stack up in PRAGMATIC terms?  It concerns technical vulnerabilities, which are somewhat Predictive of technical security incidents, which are directly Relevant to IT security and partly relevant to information security.  The numbers are probably Genuine unless someone manipulates the assessment criteria and standards, which is possible since the people reporting the metric are the same people with the technical knowledge and access (i.e. the metric has limited Independence).  The metric is Meaningful to those same people, particularly if there is a more detailed breakdown by severity, but less meaningful to general management.  Accuracy is an issue since some vulnerabilities cannot be readily measured by a tool, while others may be found and reported even if they are not exposed to potential threats, or if exploitation causes no materials impact (i.e. the metric does not take in the bigger risk picture).  Gathering and reporting the metric with an automated tool should be Timely, although slower human involvement is beneficial to interpret the raw numbers, present them and use them.  Overall, the metric is quite Cost-effective as an IT or technical security measure.

"Baseline defenses coverage" was the fifth most popular metric.  Again. it's hard to be sure what the survey's authors and respondents understood by those three words.  Presumably the metric is getting at a technical security baseline standard defining a suite of technical security configuration parameters etc. to provide a basic level of platform security.  Coverage could refer to the proportion of applicable systems that have fully implemented the baseline, and/or the proportion of security issues that are duly covered in the baseline standard.  This ambiguity immediately affects the metric's Meaning, as well as its Genuineness and Accuracy.  This is another narrow, technical metric that may have some Predictive value and Relevance to the IT security professionals looking after system and network security.  Like the previous one, it is Actionable in the sense that low compliance levels and/or coverage obviously implies the need to increase compliance and/or coverage, but the only some of the necessary details of what actually needs to be done are buried in the raw data.

By now, we're forming the distinct impression that the metrics listed in the survey are specifically technical measures that any decent vulnerability scanner produces - no surprises there given Tripwire's involvement. 

More to follow: if you missed them, see the introduction and parts one, two, three and five of this series.