Metrics episode 1

Choosing/designing, using and improving metrics can be modeled as a rational process:

A. The starting point is to determine or clarify the ultimate/strategic goals for the area being measured (e.g. information risk and security management), plus any interim/tactical objectives, preferably in business terms. These may already be documented in the form of, for example, the rationale in a business case proposing an ISO/IEC 27001-style Information Security Management System, the mission statement for the Information Risk and Security department/function, and/or the organisation’s information risk management strategies.

B. The information risk and security goals and objectives will often beg questions or imply success/fail criteria. For example, the objective “To comply with applicable legal, regulatory and contractual obligations concerning information security and privacy” begs questions about the nature and number of those obligations, the compliance status, the costs and benefits of compliance (including the risks associated with partial or noncompliance), enablers and barriers to compliance etc. Generally speaking, the questions or criteria relate in some manner to strategic, management or operational decisions, although the relationship is not always direct and obvious. While it may be tempting to try to address all points at once, identifying a smaller number of key questions, criteria or issues may be a more effective approach.

C. The questions and criteria imply the need for information, in other words they can be viewed as the requirements for a suite of information security metrics. Based on the requirements, new metrics can be selected (e.g. from published or private collections of information security metrics, suggested by professional colleagues or discovered by research) or designed from scratch, or existing metrics may be selected and if necessary modified, to generate the requisite information. Selecting or designing suitable metrics involves balancing and comparing various parameters such as their predictive value, relevance to information security, accuracy and net value (benefits less costs).

D. Having selected a number of information security metrics, the next logical step is to start gathering relevant data. ‘Instrumentation’ refers to the process of obtaining data from the organisation’s IT systems, processes and activities, for instance configuring the logging and reporting facilities built-in to automated systems to send data to collection points for analysis and reporting. For manual procedures, instrumentation involves ensuring that relevant information is recorded routinely.

E. Measurement data may be “pushed” to a collection point, or it may be “pulled” from the sources. Either way, the flows may be periodic and regular, ad hoc (on-demand or sporadic) or some combination, depending on the particular metric. The collection point must collate and store the information in a suitable structure. It must also protect the information to ensure its confidentiality, integrity and availability, as with any information asset.

F. Analysis may be as simple as determining a binary condition (e.g. the presence or absence of a conformity certificate) but is normally a matter of assessing the degree or extent of various parameters (often several in combination) relative to criteria. Statistical methods are commonly applied. Identifying beneficial or adverse trends is another form of analysis, implying historical analysis. The parameters, analyses, criteria, may have been explicitly pre-defined as part of step C, but are often left to the discretion of the analysts, working in conjunction with the intended audiences for the metrics, allowing for dynamic changes according to the measurement values, emergent properties and the changing business and security contexts.

G. Reporting or presenting metrics is best viewed as an interactive process in which information is exchanged, considered, interpreted and challenged, generating the motivation to address any issues identified, along with details such as the direction and amount of adjustment needed.

H. The decisions supported by information security metrics are many and varied, as are the actions arising. Decisions relating to the effectiveness and efficiency of the ISMS as a whole, for example, may lead to systematic improvements or changes of approach. 

I. From time to time, the organisation’s information security metrics, along with the associated analysis, reporting and actions arising, should be reviewed to determine whether they are suitable and sufficient. These reviews can happen at several levels, ranging from someone reconsidering the specification of a single security metric up to an organisation-wide review or audit of the information security measurement system as a whole. Deficiencies may lead to fine-tuning of the parameters (such as altering the reporting period for regularly-reported metrics) or changes to the mix of metrics (such as retiring metrics that are no longer of value, and perhaps replacing them with better alternatives).