Micro vs. macro metrics
Whereas "micro metrics" focus-in on detailed parts, components or elements of something, "macro metrics" pan out to give a broad perspective on the entirety. Both types of metric have their uses.
Micro metrics support low-level operational management decisions. Time-sheets, for example, are micro metrics recording the time spent on various activities, generating reports that break down the hours or days spent on different tasks during the period. This information can be used to account for or reallocate resources within a team or department or identify. Normally, though, its true purpose is to remind employees that they are being paid for the hours they work, or as a basis on which to charge clients.
Macro metrics, in contrast, support strategic big-picture management decisions. They enable management to "see how things are going", make course-corrections and change speed where appropriate. The metric "security maturity", for example, has implications for senior managers that are lost on lower levels of the organization. I have a soft spot for maturity metrics: they score strongly on the PRAGMATIC criteria, enabling us to measure complex, subjective issues in a reasonably objective and straightforward fashion.
The sausage-machine metrics churned out automatically by firewalls, enterprise antivirus systems, vulnerability scanners and so forth are almost entirely micro metrics, intensely focused on very specific and usually technical details. There are vast oceans of security-related data. Lack of data is not a problem with micro metrics - quite the opposite.
The sausage-machine metrics churned out automatically by firewalls, enterprise antivirus systems, vulnerability scanners and so forth are almost entirely micro metrics, intensely focused on very specific and usually technical details. There are vast oceans of security-related data. Lack of data is not a problem with micro metrics - quite the opposite.
Some security professionals are 'boiling the ocean' using big data analytics tools in an attempt to glean useful information from micro metrics but a key problem remains. When they poke around in the condensate, they don't really know what they're looking for. The tendency is to get completely lost in the sea of data, constantly distracted by shiny things and obsessing about the data or the analysis ... rather than the information, knowledge, insight and wisdom that they probably should have gone looking for in the first place.
It's like someone stumbling around aimlessly in the dark, hoping to bump into a torch!
Just as bad, when a respected/trusted metrics "expert" discovers a nugget and announces to the world "Hey look, something shiny!", many onlookers trust the finder and assume therefore that the metric must be Good, without necessarily considering whether it even makes sense to their organization, its business situation, its state of maturity, its risks and challenges and so forth ... hence they are distracted once more. As if that's not enough, when others chime in with "Hey look, I've polished it! It's even shinier!", the distractions multiply.
The bottom-up approach is predicated on and perpetuates the myth of Universal Security Metrics - a set of metrics that are somehow inherently good, generally applicable and would be considered good practice. "So, what should we be measuring in security?" is a very common naive question. Occasionally we see various well-meaning people (yes, including me) extolling the virtues of specific metrics, our pet metrics (maturity metrics in my case). We wax lyrical about the beauty of our pet metrics, holding them up to the light to point out how much thy gleam and glint.
What we almost never do is explain, in any real detail, how our pet metrics help organizations achieve their objectives. We may describe how the metrics are useful for security management, or how they address risk or compliance or whatever, but we almost invariably run out of steam well before discussing how they drive the organization towards achieving its business objectives, except for a bit of vague hand-waving, cloud-like.
By their very nature, it is even harder to see how micro metrics relate to the organization's business objectives. They are deep down in the weeds. Macro metrics may be up at the forest canopy level but even they are generally concerned with a specific area of concern - information security in my case - rather than with the business.
I guess that's why I like the Goal-Question-Metric approach so much. Being explicit about the organization's goals, its business and other high-level objectives (e.g. ethical or social responsibility and environmental protection), leads naturally into designing macro metrics with a clear business focus or purpose.
By their very nature, it is even harder to see how micro metrics relate to the organization's business objectives. They are deep down in the weeds. Macro metrics may be up at the forest canopy level but even they are generally concerned with a specific area of concern - information security in my case - rather than with the business.
I guess that's why I like the Goal-Question-Metric approach so much. Being explicit about the organization's goals, its business and other high-level objectives (e.g. ethical or social responsibility and environmental protection), leads naturally into designing macro metrics with a clear business focus or purpose.