Metrics case study on Boeing

The Security Executive Council has published an interesting case study concerning the review and selection of metrics relating to physical and information risks at Boeing.  [Access to the article is free but requires us to register our interest.]

The case study mentions using SMART criteria and a few other factors to select metrics but doesn't go into details, unfortunately.  Nevertheless, the analytical approach is worth reading and contemplating.

If we were to conduct such an assignment for a client today, we would utilize a combination of tools and techniques across six distinct phases:

1. Background information gathering concerning Boeing's business situation, information risks, and existing metrics, using standard analytical or audit methods, clarifying the as-is situation and building a picture of what needs to change, and why. This phase would typically culminate in a report and a presentation/discussion with management.
   
   
2. GQM (Goal-Question-Metric) assessment eloquently described by Lance Hayden in IT Security Metrics. This is a more structured and systematic version of the approach outlined in the case study. A workshop approach would be useful, probably several in fact to delve into various aspects with the relevant business people and experts. The output would be a matrix or tree-root diagram illustrating the goals, questions and metrics.
   
   
3. PRAGMATIC assessment and ranking of the metrics generated in phase 2 using the approach documented in our book. The output would be a management report containing a prioritized list of metrics ranked according to their PRAGMATIC scores, leading to a further presentation/discussion with management and, hopefully, agreement on a shortlist of the most promising metrics, those actually worth pursuing. This and the previous phase would take a creative approach, thinking about what needs to be measured, why, how, when etc., using both GQM and PRAGMATIC to firm-up the metrics that best fit the requirements  and focus groups to finalize the metrics (both existing metrics that are worth retaining possibly with some changes, and novel metrics being introduced).
   
   
4. Planning and preparing for the implementation phase, perhaps including pilot studies (possibly involving actual pilots). 
   
   
5. Implementation: making the changes needed to collect, analyse, report and most of all use the metrics.  This might well involve retiring or recasting some of the client's existing metrics that haven't earned their keep, in a way that teases out the last dregs of value from the data gathered previously.
   
   
6. Ongoing metrics management and maintenance: using information from the GQM and PRAGMATIC steps to monitor and if appropriate refine or replace the metrics, ensuring for instance that they are proving valuable to the business (i.e. they should be cost-effective - one of the PRAGMATIC criteria conspicuously absent from SMART).  

In parallel with that sequence would be conventional project management activities - planning, resourcing & team building, motivation, tracking, reporting and assignment risk management.

Get in touch if you'd like some assistance to review and update your metrics: we'd love to help!