SMotW #9: Vulnerability index

Security Metric of the Week #9: Vulnerability index

Well-known vulnerabilities in commercial software are commonly identified by patch-checking tools such as PSI from Secunia and Microsoft Update.  PSI generates a convenient system security score - a simple percentage related (in some way, determined by Secunia) to the patch status of the system.  Microsoft Update generates a count of the number of missing patches, categorized by the severity of the security vulnerabilities the patches (supposedly) fix, as determined by Microsoft.  Whether you are managing a single PC or a network of thousands, tools and metrics such as these are a helpful way to focus attention on the systems that need patching.

However, there is a lot more to security vulnerabilities than simply patching commercial software e.g.:

  • Finding and patching security vulnerabilities in private/non-commercial/obscure software, including programs such as spreadsheets, macros and batch files written by amateurs, and all manner of Java utilities and apps that are not addressed by PSI etc.;
  • Finding and patching currently unknown software security vulnerabilities through various forms of testing (software security testing, clear box/black box testing, penetration testing, fuzzing, static/dynamic source code analysis hacking, software audits);
  • Finding and fixing fundamental security design flaws in software, hardware and processes (e.g. missing policies and inadequate security awareness activities fail to address commonplace vulnerabilities to social engineering).

Furthermore, the importance or significance of different vulnerabilities varies markedly e.g.:
  • The risk represented by a missing security patch on a system exposed on the Internet is probably quite different to that on an isolated internal system tucked away behind multiple layers of defense (some people refer to this factor as 'exposure');  
  • Some vulnerabilities are trivially simple to exploit, whereas others can only be exploited under very specific circumstances, often with a lot of effort and sometimes a lot of luck;
  • Some vulnerabilities are of little concern in terms of their consequences, whereas others are extremely problematic since they allow vital controls to be totally disabled, undermined or negated.  
In short, the number of vulnerabilities does not necessarily reflect the amount of risk, even if we somehow take account of their severity in the metric.  Risk also depends on the threats and impacts of incidents.

We scored the metric thus:
P
R
A
G
M
A
T
I
C
Score
74
85
71
74
60
32
46
33
19
55%




The Accuracy, Integrity, Timeliness and Cost aspects all suffer if we intend to use the metric to manage information security as a whole, as opposed to simply helping us manage software security patching.  That said, a patching-type vulnerability metric may still be valuable at the operational level for those managing the IT infrastructure.