Metrics lifecycle management


This week, I'm thinking about management activities throughout the metrics lifecycle.

Most metrics have a finite lifetime. They are conceived, used, hopefully reviewed and maybe changed, and eventually dropped or replaced by something better. 

Presumably weak/bad metrics don't live as long as strong/good ones - at least that's a testable hypothesis provided we have a way to measure and compare the quality of different metrics (oh look, here's one!).

Ideally every stage of a metric's existence is proactively managed i.e.:
  • New metrics should arise through a systematic, structured process involving analysis, elaboration and creative thinking on how to satisfy a defined measurement need: that comes first. Often, though, the process is more mysterious. Someone somehow decides that a particular metric will be somewhat useful for an unstated, ill-defined and barely understood purpose;
  • Potential metrics should be evaluated, refined, and perhaps piloted before going ahead with their implementation. There are often many different ways to measure something, with loads of variations in how they are analyzed and presented, hence it takes time and effort to rationalize metrics down to a workable shortlist leading to final selection. This step should take into account the way that new or changed metrics will complement and support or replace others, taking a 'measurement system' view. Usually, however, this step is either skipped entirely or superficial. In my jaundiced opinion, this is the second most egregious failure in metrics management, after the previous lack of specification;
  • Various automated and manual measurement activities operate routinely during the working life of a metric. These ought to be specified, designed, documented, monitored, controlled and directed (in other words managed) in the conventional manner but rarely are. No big deal in the case of run-of-the-mill metrics which are simple, self-evident and of little consequence, but potentially a major issue (an information risk, no less) for "key" metrics supporting vital decisions with significant implications for the organization;
  • The value of a metric should be monitored and periodically reviewed and evaluated in terms of its utility, cost-effectiveness etc. That in turn may lead to adjustments, perhaps fine-tuning the metric or else a more substantial change such as supplementing or dropping it. More often (in my experience) nobody takes much interest in a metric until/unless something patently fails. I have yet to come across any organization undertaking 'preventive maintenance' on its information risk and security metrics, or for that matter any metrics whatsoever - at least, not explicitly and openly. 
  • If a metric is to be dropped (retired, stopped), that decision should be made by relevant management (the metric's owner/s especially), taking account of the effect on management information and any decision-making that previously relied upon it ... which implies knowing what those effects are likely to be. In practice, many metrics circulate without anyone being clear about who owns or uses them, how and what for. It's a mess.
Come on, this is hardly rocket surgery. Information risk and security metrics are relatively recent additions to the metrics portfolio so it's not even a novel issue, and yet I feel like I'm breaking new ground here. Oh oh.

I should probably research fields such as finance and engineering with mature metrics, for clues about good metrics management practices that may be valuable for the information risk and security field.

Popular posts from this blog

Pragmatic ISMS implementation guide (FREE!)

Image
Early this morning ( very  early!) I remotely attended an ISO/IEC JTC 1/SC 27/WG 1 editing meeting in London discussing the planned revision of ISO/IEC 27003:2017 . Overall, the meeting was very productive in that we got through a  long  list of expert comments on the preliminary draft standard, debated the objectives of the project and the standard and reached consensus on most points. In summary: 27003 is to be revised to align with the current 2022 releases of ISO/IEC 27001 , 27002 and 27005 : These changes are  mostly  minor aside from the new section 6.3 on ISMS changes.
Read more

Two dozen information risks that ISO forgot

Image
Selecting the wrong controls - controls that are inappropriate, ineffective, too costly, impracticable, fragile, unnecessary, counterproductive or whatever, often as a result of blind faith in fads and fashions of the day and FOMO e.g. MFA, AI, cyber Failing to select the right controls - controls that are ideal for the particular situation, both now and in perpetuity, for whatever reason - mostly ignorance and prejudice Selecting and implementing controls at the wrong time or in the wrong way (where 'wrong' includes ineffective, inappropriate, sub-optimal e.g. bolting on controls rather than designing and building them in) Inept and inaccurate identification, analysis and quantification of risk, including reliance on p oor quality (incomplete, inaccurate, out of date, misleading, unreliable ...) information about actual risks, particularly subtle and emerging risks plus those involving deliberate concealment and misdirection e.g. fraud, misinformation, disinformation, propagan
Read more

Philosophical phriday - compliance risk

Image
According to a vendor's promotional video interview I saw recently, the 'cybersecurity compliance burden' has allegedly become so significant that [customer] organisations are eagerly buying [their] software tools and services to help them manage and fulfil their obligations. The vendor's argument goes that, instead of accumulating a ragtag bunch of policies and other controls relating to user Identification and Authentication (I&A), for instance, it makes sense to:  Identify all the cybersecurity-related laws, regulations and standards that apply to the organisation; Examine them for any security control requirements relating to, say, I&A; Rationalise the I&A controls down to the smallest set that satisfies all the requirements - the lowest common denominator; Design, implement, use, manage and maintain those I&A controls; Have the I&A controls checked or audited to gain assurance that the compliance requirements are met.  OK so far? Sounds reasonab
Read more

ISMS internal audit priorities

Image
A thread on the ISO27k Forum sparked my imagination over coffee this morning. Hope had previously asked for assistance with an ISO/IEC 27001:2022 audit plan.  Bhushan offered a lengthy and generally sound response explaining how to use a spreadsheet with tabs to plan and record the audit work performed on 100% of the main body clauses and 50% of the 93 Annex A controls, day-by-day. That's OK ... except it wasn't entirely clear that he was interpreting and elaborating on the standard's actual requirements. ISO/IEC 27001 does not explicitly require, for example, that (as Bhushan stated) "ALL the management system clauses from 4 to 10 AND their sub-clauses need to be listed and audited" in an ISMS internal audit, although evidently he interprets it in that way. In clause 9.2.1, the standard states a requirement for internal audits to provide information on whether the ISMS conforms to the organization’s own requirements for the ISMS plus the requirements of the stan
Read more

Reading between the lines of ISO27001 [L O N G]

Image
ISO/IEC 27001 is a succinct, formally-worded standard for two key reasons: It is deliberately generic, being applicable to all manner of organisations regardless of difference in location/s, size, industry, maturity, structure, information risk and security status ... and so on. In effect, it specifies the lowest common denominator - the things that ALL organisations should be doing to manage their information security controls, as a minimum. The hurdle is set low enough that every organisation ought to find value in designing, implementing and operating an I nformation S ecurity M anagement S ystem as laid out in the standard. It is a certifiable standard, explicitly specifying the characteristics that every certified organisation's ISMS is expected to have. Again, it is a minimal specification with no concept of typical, average or maximum security: that is entirely down to the organisations themselves to determine, following the information risk management processes minimally de
Read more

Passionate dispassion

Image
Someone who is actively involved in, or is managing, an activity is patently not independent of it. They may well make a conscious, rational and determined effort to be objective, dispassionately reviewing evidence etc ., but their subconscious/emotional biases/prejudices and beliefs/value-systems will inevitably influence what they do. With the best will in the world, they will struggle to challenge and assess their past decisions and activities, especially if they were "certain" or "determined" or genuinely believed they were "doing the right thing". Furthermore, it is very hard for anyone to review the things they did not do, decisions they did not make or options they did not even consider. Mostly, they remain out of sight or out of the question.
Read more

45 ISO Management Systems Standards

Image
The ISO website  currently lists 45 published M anagement S ystems S tandards: 1. ISO 7101:2023 Healthcare organization management — Management systems for quality in healthcare organizations — Requirements 2. ISO 9001:2015 Quality management systems — Requirements 3. ISO 10012:2003 Measurement management systems — Requirements for measurement processes and measuring equipment 4. ISO 13485:2016 Medical devices — Quality management systems — Requirements for regulatory purposes 5. ISO 14001:2015 Environmental management systems — Requirements with guidance for use 6. ISO 14298:2021 Graphic technology — Management of security printing processes
Read more

Philosophical phriday - a noncompliance ramble

Image
In a previous philosophical phriday post , I moaned about vendors of security compliance support/management tools and services over-promising and under-delivering - an admittedly biased, even cynical opinion piece about the compliance imperative . A recent article in Corporate Compliance Insights notes that "CISOs are not just defenders against cyber threats but also champions of compliance and operational resilience". Hmmm, are CISOs 'compliance champs', really? Today, I'm discussing alternatives to being compliance-driven. How else can organisations drive their information risk, security and related concerns in a positive direction?
Read more

Adaptive SME security Crowdstrike special

Image
As if on cue, along comes a golden opportunity to consider what the Adaptive SME security  approach has to say regarding the Crowdstrike incident: That's not 20/20 hindsight but foresight: I've picked out the most relevant rows from the security controls table published in the guide 24 hours before the incident.  Although Crowdstrike primarily supplies much larger enterprises than SMEs, the incident could equally have afflicted other security software, or indeed operating systems such as Windows and assorted cloud apps commonly used by SMEs. Regardless of the details, it is a wake-up call, an opportunity to consider and respond to the information risks ... and to adapt , accordingly.
Read more