SMotW #45: extent of security testing

Security Metric of the Week #45: extent to which information security is incorporated in software QA

Well-managed IT development projects incorporate information security at all applicable stages of the systems lifecycle, from initial outline specification and business case, through design, development, testing and release, on through operational use, management and maintenance of the system, right through to its retirement/replacement at the end of its life.  It would be possible to measure that in order to generate some sort of security index for all systems, using the index to drive-up security integration and quality, but doing so would be a tall order for most organizations.  Perhaps we should talk about that another time.

This week's example security metric is far simpler with a much tighter scope, measuring information security activities only during the "software Quality Assurance" (testing) phases of a development.  

The "extent to which information security is incorporated" is rather vague wording but we assume the metric would be specified more explicitly by the organization.  For instance, someone could examine all the development methods being used to identify all the QA stages (plural), then the steps or activities where information security could or should be involved.  The next step would be to survey all the currently-active development projects, checking available evidence to confirm whether information security is or is not involved as it could or should be.

The individual checks may involve crude decisions at each point ("Is information security involved, or not?", a binary option) or more sophisticated assessments of the nature of the involvement, for instance using a predefined Likert scale (e.g. "Not involved at all (score 1), slightly involved (2), involved (3), highly involved (4) or fully involved (5)".  Furthermore, the importance of information security's involvement at each step could be assessed in a similar way ("Is information security involvement necessary or optional at this stage?" or "How important is the involvement at this stage: (1) not important at all; (2) quite important; (3) important; (4) very important; (5) absolutely critical?").  Finally, the individual projects will vary in the relevance or necessity for information security involvement, depending on the associated information security risks relating to factors such as compliance obligations, technical complexity, business security demands, privacy aspects etc., and again these could be measured.

Notice that more sophisticated, information-rich versions of this metric come at a higher cost in terms of the need to specify the metric's Likert scales or other scoring parameters, to measure the projects against them, to analyze and report them, and of course to interpret and use them.  It takes time and effort to do all that.  Notice also that there is subjectivity throughout - things have to be interpreted in the specific context of each development project.  Even the definition of what constitutes a "development project" for this metric is subject to interpretation (e.g. does it only involve projects run in-house by IT, or does it include those run by IT outsourcers and cloud suppliers, and by business people on their desktops, tablets, smartphones and maybe BYOD equipment?).

Using the PRAGMATIC method, ACME Enterprises Inc scored this metric thus:

P	R	A	G	M	A	T	I	C	Score
85	80	67	62	70	50	35	35	50	59%

The assessors were evidently quite impressed with the metric's Relevance, Predictability and Meaninfulness for information security, but concerned about its subjectivity.  The Cost criterion is neutral on the basis that the metric could be quite simple, quick and cheap at least at first, but may evolve into something more sophisticated later if it turns out that the additional information would be valued (implying that ACME management are actively and consciously managing a suite of security metrics).