Awareness paper on authentication and phishing metrics

We've just republished a management-level security awareness paper on metrics relating to user authentication and phishing.

The introduction asks "How do we tell whether our authentication controls are effective?" and "What does 'effective' even mean in this context?" - two decent questions that could be addressed through suitable metrics.

Questions like these are central to the GQM (goal-question-metric) method (see IT Security Metrics by Lance Hayden), and not just literally in terms of their position in the handy acronym. They link the organization's goals or objectives relating to information security, to the information security metrics that are worth measuring.

In your particular circumstances, the effectiveness of authentication controls might or might not be of sufficient concern to warrant generating the associated metrics. Other aspects might take precedence, for example the amount invested in authentication controls, and the ongoing operating and maintenance costs of those controls. It's usually not too hard to think up a whole raft of aspects, parameters or concerns relating to the topic area, but focusing on the things that are likely to matter most to the organization (business priorities) is a good way to keep the list within reasonable bounds. Once you know what they are, the next step is to figure out the questions arising e.g. "Are we spending appropriately (neither too much nor too little) on authentication?"

From there, it's simply a matter of deciding what data would help address the questions, and those are your metrics!  Job done!  Errr, well, no, not quite: if you have several goals/areas of concern and numerous questions arising, each requiring multiple metrics to generate the answers, there is a distinct risk of being overwhelmed with possibilities. It is infeasible and in fact counterproductive to attempt to measure everything. Less is more! This is where the PRAGMATIC method comes into play as a way to whittle down the long list to a shortlist of metrics showing the most promise. The GQM approach also suggests filtering out the metrics that don't address the questions very well, and trimming down on metrics addressing questions that are only marginally related to the organization's business goals. Both approaches have their merits.