What are KPIs?

Krag and I have discussed this question from time to time and, although we are broadly aligned in our thinking, we haven't yet totally resolved our differences ... which makes the exchanges fun.

With that in mind, I always wonder what someone really means when they talk about KPIs. To some, Key Performance Indicator has a very specific and particular meaning, although I suspect if we assembled a dozen such people in a room to discuss it, we'd soon end up realizing that we have more than a dozen different interpretations! 

To others (including me, as it happens), KPI is a generic, blanket term for a class or type of metric that satisfies the criteria implied by the term: 

• Key implies that the metric itself is especially important, crucial or vital even, given that there are many many different ways to measure and assess things but most of them are of limited value. Picking out the few things that truly matter is a core issue in metrics. 'Spam volume' is an example of a metric that is both narrow and shallow, whereas 'email risk level' is a much broader, deeper and richer metric, and is far more likely to be considered key (even if we happen to be talking specifically about metrics relating to spam filtering). However, the criticality and value of metrics does depend on the contexts or situations being measured and the perspectives and information needs of their users. It is conceivable that 'spam volume' may be considered a KPI for the anti-spam controls, but that's a narrow perspective. Key may also refer to the performance, in the sense that the KPI is an indicator concerning an important issue; 
• Performance is a distinctly ambiguous word, implying concern for the process and/or its outcome. Are we measuring key activities (typically in order to assess and improve the efficiency of the process) or its outcomes (typically to assess and improve the effectiveness of the process), neither or both? I have seen KPI used in several different senses, although usually it is not totally clear (perhaps not even to the person discussing it!) which one is meant; 
• Indicator generally means simply a metric or measure but it may imply imprecision or approximation. A typical car's fuel gage, for instance, gives a fairly vague indication of the amount of fuel in the tank, whereas the equivalent metric on an aircraft tends to be much more precise, accurate and reliable, for obvious reasons. The car's fuel gage may not tell you how many litres remain but if it heads into the red zone, you know you need to find a filling station soon. Indicator often also implies a forward-looking or predictive rather than purely historical measure. 'Trends' are common examples, used to manage various aspects of information security where precision is nice but not vital (e.g. supporting security investment or resourcing decisions). 

As far as I'm concerned, then, a KPI is generally a predictive metric concerning some critical outcome of an important process. In the information security context, KPIs are most likely to measure core security processes such as risk assessment, and key controls such as authentication and access control. Efficiency is secondary to effectiveness, in that security failures resulting from ineffective controls can lead to serious and potentially very damaging incidents, whereas inefficient controls are merely a bit wasteful (that's actually a strong bias, one worth challenging in situations where security becomes onerous for information users and administrators, perhaps so onerous that they bypass or disable the controls sending us back to square one!).