The ultimate question

In the field of security metrics, much has been written about measuring all manner of detailed parameters that are deemed relevant and important, but the kinds of big-picture strategic questions that matter most to senior management are seldom addressed.  

Take for example the disarming, devilishly simplistic question "Are we more or less secure than we were a year ago?"  

Imagine being asked precisely that question by, say, the CEO or the minister.  How would you actually respond?  What if it turned out the question was not merely an off-the-cuff comment but a deadly serious request for information posed on behalf of a management team struggling to make sense of next year's infosec budget requests in relation to all the other proposals on the top table?  

Go ahead, picture yourself squirming in the hot seat.  What's going through your mind?  Where do you even start to address such a naive question?

For some of us, our knee-jerk reaction is to spew forth a confusing muddle of half-baked assertions and mumbo-jumbo.  We trot out a stream of primarily technical measures, some of which are so narrowly defined as to be of dubious value even to those professionals responsible for managing information security and other risks.  In many areas, we fall back on highly subjective measures that smack of "The answer is 42: what was the question again?"  Faced with a tsunami of dubious numbers, the CEO is left with the overriding impression that he's just been told precisely how fast we are going, and yet we still don't know for sure that we are headed the right way.  That's no way to direct the organization.

Alternatively, we may resort to a purely defensive approach, claiming that the question is unreasonable because infosec is 'too difficult' or 'too complex' to measure, and the situation is 'highly dynamic' to boot.  What may initially appear a perfectly reasonable and honest response from our rational perspective may be counterproductive when viewed in strategic business terms.  With mounting outrage, the CEO may well respond along the lines of "Are you seriously telling me that we don't know whether we are more or less secure than last year because information security is 'special'?  So how come we can measure production, finances, quality and human resources, but we can't measure infosec?"  Oh oh, now we're really in trouble!

Sorry to disappoint you if you are expecting me to answer the CEO's question for you: I won't but perhaps I can help you figure out how to address it yourself.  In fact, that's what I've just been doing.  A useful way to develop worthwhile security metrics is to pose yourself a bunch of rhetorical management questions in order to tease out the underlying concerns.  What are the key security issues facing your organization?  What are the big business drivers this year?  Which aspects matter most to your management: compliance, cost-effectiveness/efficiency, risk, accountability, assurance, adequacy or something else entirely?  

Unless you are a senior manager, or somehow become aware of those earnest discussions in the boardroom, you can only guess at what might be playing on the CEO's mind in respect of information security.  You might therefore get yourself ready to address not one but a whole bunch of potential big-picture questions.  The point is to identify the common themes, and to spot the information (and hence the underlying data) that would be needed to formulate meaningful answers.

In the book, we discuss using the GQM approach in its implied sequence: first determine the Goals, then the associated Questions, and finally the Metrics.  What I'm suggesting here could be termed the Q(GM) approach: first pose those rhetorical Questions, then figure out the Goals, objectives or issues that might have led to them, as well as the Metrics, information and data that would be necessary for the answers.  

Alternatively, do nothing, just sit back and wait for that fateful day when, finally, someone important demands to know "Are we secure enough?" or "How secure are we?" or "Do we really need to spend so much on security? ... or "What has Information Security ever done for us?"