SMotW #21: unclassified assets

Security Metric of the Week #21: proportion of information assets not marked with the correct classification

There are three key assumptions underlying this week's Security Metric of the Week:

1. The meaning of "information asset" is clear to all involved;  
2. There are suitable policies and procedures in place concerning how to risk-assess and classify information assets correctly;
3. The metricator (person gathering/analyzing the data for the metric) is able to tell whether or not a given information asset is (a) correctly classified and (b) correctly marked.

Part of the concern about the meaning of "information asset" is the determination of what should be assessed and marked: should we classify the filing cabinet, the drawers, the files, the documents or the individual pages?  In some cases, it may be appropriate to classify them all, but there are practical limits in both the micro and macro directions.  The wording of the policies, procedures, examples etc. can make a big difference.

Whereas classification policies are fairly common, the related procedures plus the associated awareness/training and compliance/enforcement activities, are not universal.  This metric could be used to determine the need for additional procedures etc., and with a bit more detail it could help direct resources at the business units, departments, teams or people who evidently need more support.

However, the metric's poor PRAGMATIC score raises concerns:

P	R	A	G	M	A	T	I	C	Score
52	53	63	44	62	13	17	87	44	48%

Low ratings for Accuracy and Genuineness arise from the way the metric would have to be measured.  The third assumption above is the main fly in the ointment, since it is necessary for someone to review a sample of information assets to determine what their classifications should be, and confirm whether they are indeed correctly marked.  This is a tedious process that can result in disagreements regarding the correct classifications and the nature of marking required.

We marked it down on Timeliness since the measurement process would inevitably take days or weeks, during which time incorrectly classified and/or marked information assets would probably remain vulnerable to being mishandled.  Once the final numbers are available, management can take the decisions about additional procedures, awareness and compliance activities, but these will also take time to put into effect.  All in all, there are likely to be significant lags between taking, acting on and adjusting the measurements.

The relatively high Cost of assigning one or more suitable metricators to the job could be offset by reducing the frequency of measurement, perhaps  measuring and reporting this metric just once or twice a year ... but of course that makes the metric less useful as a management tool - it's a trade-off.

The bottom line is that although there are circumstances in which this metric might be worth using, its low score suggests that there are many more PRAGMATIC metrics that should probably take priority.

The ultimate question

In the field of security metrics, much has been written about measuring all manner of detailed parameters that are deemed relevant and important, but the kinds of big-picture strategic questions that matter most to senior management are seldom addressed.  

Take for example the disarming, devilishly simplistic question "Are we more or less secure than we were a year ago?"  

Imagine being asked precisely that question by, say, the CEO or the minister.  How would you actually respond?  What if it turned out the question was not merely an off-the-cuff comment but a deadly serious request for information posed on behalf of a management team struggling to make sense of next year's infosec budget requests in relation to all the other proposals on the top table?  

Go ahead, picture yourself squirming in the hot seat.  What's going through your mind?  Where do you even start to address such a naive question?

For some of us, our knee-jerk reaction is to spew forth a confusing muddle of half-baked assertions and mumbo-jumbo.  We trot out a stream of primarily technical measures, some of which are so narrowly defined as to be of dubious value even to those professionals responsible for managing information security and other risks.  In many areas, we fall back on highly subjective measures that smack of "The answer is 42: what was the question again?"  Faced with a tsunami of dubious numbers, the CEO is left with the overriding impression that he's just been told precisely how fast we are going, and yet we still don't know for sure that we are headed the right way.  That's no way to direct the organization.

Alternatively, we may resort to a purely defensive approach, claiming that the question is unreasonable because infosec is 'too difficult' or 'too complex' to measure, and the situation is 'highly dynamic' to boot.  What may initially appear a perfectly reasonable and honest response from our rational perspective may be counterproductive when viewed in strategic business terms.  With mounting outrage, the CEO may well respond along the lines of "Are you seriously telling me that we don't know whether we are more or less secure than last year because information security is 'special'?  So how come we can measure production, finances, quality and human resources, but we can't measure infosec?"  Oh oh, now we're really in trouble!

Sorry to disappoint you if you are expecting me to answer the CEO's question for you: I won't but perhaps I can help you figure out how to address it yourself.  In fact, that's what I've just been doing.  A useful way to develop worthwhile security metrics is to pose yourself a bunch of rhetorical management questions in order to tease out the underlying concerns.  What are the key security issues facing your organization?  What are the big business drivers this year?  Which aspects matter most to your management: compliance, cost-effectiveness/efficiency, risk, accountability, assurance, adequacy or something else entirely?  

Unless you are a senior manager, or somehow become aware of those earnest discussions in the boardroom, you can only guess at what might be playing on the CEO's mind in respect of information security.  You might therefore get yourself ready to address not one but a whole bunch of potential big-picture questions.  The point is to identify the common themes, and to spot the information (and hence the underlying data) that would be needed to formulate meaningful answers.

In the book, we discuss using the GQM approach in its implied sequence: first determine the Goals, then the associated Questions, and finally the Metrics.  What I'm suggesting here could be termed the Q(GM) approach: first pose those rhetorical Questions, then figure out the Goals, objectives or issues that might have led to them, as well as the Metrics, information and data that would be necessary for the answers.  

Alternatively, do nothing, just sit back and wait for that fateful day when, finally, someone important demands to know "Are we secure enough?" or "How secure are we?" or "Do we really need to spend so much on security? ... or "What has Information Security ever done for us?"

IASAP International Association of Security Awareness Professionals

The CSI Security Awareness Peer Group has reformed itself as a non-profit organization after CSI discontinued the group last year.  IASAP (the International Association of Security Awareness Professionals) offers a supportive forum in which security awareness professionals can meet up and share good ideas.  A former CSI manager remains involved in the role of coordinator and meeting facilitator.

IASAP membership is limited to those who are running security awareness programs within their companies: vendors of security awareness, training and related products are supposedly excluded although, paradoxically, PhishMe Inc. is the group's "exclusive founding sponsor" and "corporate sponsorships" are acknowledged as a source of funding on the IASAP website.  

Perhaps as a consequence of the no-vendor policy, annual membership costs $2,500.  Members presumably anticipate recovering at least as much value through networking and sharing ideas and materials with their professional colleagues, primarily through two-day meetings held twice or three times a year.

The 'international' part of the title currently appears to mean US and Canada but fair enough, that's a start! We wish the group well for the future.

Breaking Sod's Law

A news piece about a failure of the 911 emergency phone services during a storm is valuable security awareness lesson for any organization that relies on a generator-backed UPS to maintain clean, stable power to essential ICT services - meaning practically every large organization and many smaller ones too.

Reading somewhat between the lines, the Fairfax County 911 ICT services depended on a UPS, the batteries in which were supposed to be kept fully charged by two generators in case the mains power failed.  One of the generators worked fine but the other evidently failed to auto-start.  Insufficient capacity in the one running generator meant that the UPS batteries gradually ran down over a few hours until eventually the UPS ran out of juice, the lights went out and the 911 ICT services failed.

Moments after the initial power cut, the technician/s on site were probably relieved that the change-over from mains input to generator input had gone smoothly, and the ICT services carried on running normally.  It is a stressful event.  However, it seems they relaxed too much or perhaps lacked the information to realize that the second generator was not working, and hence the UPS batteries were gradually discharging.

Speaking from personal experience as a former ICT manager, 'running on UPS' is an unusual situation that requires unusual activities.  Whenever the sites I managed experienced power cuts during the normal working day, I relied on the site maintenance people - our wizards - to ensure that the electrical equipment was working correctly, leaving me to worry about the peripheral ICT equipment and services that may not be on UPS, for example, figuring out which bits of kit had failed or might fail soon, and which parts of the business might be affected.  When power cuts happened out-of-hours, there was generally less pressure since the business was relatively quiet but at the same time the wizards were often unavailable, so I've been faced with unfamiliar blinkenlights and contingency situations, leaving me little choice but to muddle through and hope for the best.

Thankfully that haphazard, high-risk approach was good enough at the time but would be quite inappropriate for a critical 24x7 operation ... such as the ICT supporting 911 emergency services to nearly 2.3 million people ...

Verizon had tested (and presumably passed!) the standby power system just three days before the storm - so why did it go wrong on the big day?  According to the Washington Post article, "At the Arlington site, the routine and limited testing had not checked whether a generator could carry a full power load in an emergency".  Oops.  The Post doesn't say why Verizon's testing was limited, but these are the kinds of reasons (justifications or excuses?) I have come across in the course of hundreds of IT installation audits elsewhere:

• "Since the power system was professionally designed and thoroughly tested when it was installed, routine testing is unnecessary" [wrong!  Loading changes, equipment wears out, batteries lose their capacity ...];
• "It was tested 2 or 3 years ago" [see above - and in this particular case, it turned out the previous testing was so limited as to be pathetically inadequate]
• "Full testing is too costly" [unanticipated power incidents can be far costlier];
• "Limited/offline testing is sufficient" [it's useful for some but not all checks, and could even be counterproductive since lightly-loaded generators may accumulate partially-burnt fuel];
• "The power fails every few weeks and whenever it does, the backup power has worked fine - so there's no point in testing it" [testing is an opportunity to check things more thoroughly and if appropriate push things to the limit e.g. simulating extended power outages on full load];
• "We follow the equipment suppliers' recommendations" [strangely enough, when I trotted out the predictable line "Show me", nobody could lay their hands on the mythical guidance documents];
• "Full testing is too risky so we only do very limited testing out of hours" [a scary response: management was clearly afraid their critical backup systems would fail the tests, meaning they lacked the necessary assurance to be confident they would work when actually needed for real, meaning significant business risks were not being properly treated].

Aside from the obvious stuff such as excellent power engineering, automated failovers, over-capacity, proper equipment maintenance, procedures and full on-load testing, there are other ways of reducing unacceptable power risks:

• More than barely adequate funding, in other words treating the complete power system as a vital infrastructure investment; 
• Proper instrumentation, allowing power supplies and loads to be monitored continuously and projected accurately in relation to power system capacity, with suitable alarms and alerts triggering response procedures when the readings head into the amber (don't wait for them to go red - or worse still go out altogether!).  Adequate voltage, current and power metering is hardly rocket-surgery, while temperature monitoring (including the use of thermography) can tell an experienced power engineer a lot about the state of the plant and switchgear;
• Independent power system audits by competent, experienced assessors;
• Productive working relationships between the facilities people, IT people, site and information security people, power people and business people, including the suppliers of specialist UPS, generator and other equipment and, of course, the lines companies and power suppliers.

Backups and contingency arrangements are needed because of Sod's Law or Murphy's Law. Trouble is, backups and contingency arrangements are subject to exactly the same laws (remember how the tsunami flooded the emergency backup generators at Fukushima?).  And so are the tests, by the way.  The trick is to do whatever it takes to make sure the systems will pass their tests with flying colors and have solid contingency plans just in case they don't.

SMotW #20: uptime

Security Metric of the Week #20: ICT service availability ("uptime")

Uptime is a classic ICT metric that is also an information security metric, although it is seldom considered as such.

Uptime is commonly measured and reported in the context of Service Level Agreements or contracts for ICT services, but in our experience this is usually something of a farce.  The IT Department or company generally defines uptime narrowly in ways that suit their purposes rather than being a true reflection of the ICT services actually provided to the business users/customers, while business people don't honestly believe the numbers anyway since (a) they do not reflect their experience as consumers of ICT services, and (b) they are self-assessed and self-reported by the IT people  who clearly have an interest in reporting only good news.  Tying internal ICT cost recovery to uptime  makes things even worse from the security metrics perspective (i.e. providing genuine, fact-based data on which to make business decisions concerning information security), since it places IT and the business in diametrically opposing positions - a recipe for much more heat and smoke than light.

Being rather cynical graybeards, we note that uptime is often defined (by IT) only in relation to ICT service provision during "core service hours" (which IT determines unilaterally).  Exclusions are common, particularly 'scheduled downtime' (as if the fact that IT has decided when to take ICT services down somehow magically allows the business to carry on using them normally), 'change and patch implementation' (because the business wanted the changes or patches, so IT can hardly be blamed for doing what they are asked to do) and backups (again, IT rationalizes this exclusion along the lines of "Backups are required by the business, so they should be bloody grateful!").

Despite the drawbacks that we have just described, uptime turns out to have a pretty good PRAGMATIC score as an information security metric:

P	R	A	G	M	A	T	I	C	Score
84	97	66	78	94	61	79	47	89	77%

The very high 97% rating of uptime for Relevance may come as a surprise to those who are unfamiliar with the modern interpretation of information security as 'ensuring the confidentiality, integrity and availability of information'.  We rated the metric a few percent less than 100 for Relevance to account for the relatively small amount of business information which lies completely outside of IT: we completely accept that paperwork and knowledge need to be available as well as the ICT systems, networks and data, but without ICT support, a lot of the non-IT information is practically worthless to the business as a whole since it cannot be communicated or processed much beyond its immediate location.

The high rating for Meaningfulness reflects the fact that, leaving aside arcane issues relating to the precise definition, uptime is a simple and familiar measure.  

The high Cost-effectiveness score also reflects the metric's simplicity and familiarity: in most organizations, uptime is already being measured, analyzed and reported for purposes other than information security, so the marginal cost to include it in information security reports is negligible.  However, Costs can increase markedly if management decides to measure uptime independently of IT, for example using network and system availability monitoring outwith the department, or independently auditing the figures.

[By the way, in discussing this metric in the book, we refer to the interesting metrics challenges presented by cloud computing when significant parts of the ICT service delivery process depend on third parties and resources well outside the organization's physical and logical boundary.  The PRAGMATIC approach is just as well suited to developing, selecting and/or improving valuable, worthwhile security metrics for cloud computing as for more traditional approaches ... but I'm afraid you'll have to wait for the book to find out exactly what we mean!]

SMotW #19: employee turnover/absenteeism

Security Metric of the Week #19: rate of change in employee turnover and/or absenteeism

In most organizations, employee turnover rumbles along at a 'normal' rate most of the time, due to the routine churn of people joining and leaving the organization.  Likewise, there is a 'normal' rate of absenteeism, due to sickness, holidays/leave and unexplained absences.  Big changes (especially sudden increases) in either set of numbers suggest the possibility that information security risks associated with disaffected or malicious employees might6 have substantially increased, in other words increased turnover and absenteeism may be indicators of a discontented workforce voting with their feet, or indeed of management sacking loads of employees.

Of course there are many reasons why people leave the organization or are temporarily absent, aside from discontent and redundancy, hence the metric is unlikely to be particularly useful in isolation.  We refer to it as an indicator, since an adverse change signals or indicates a situation that merits further investigation to determine the likely reasons.

We calculated the following PRAGMATIC score for this metric:

P	R	A	G	M	A	T	I	C	Score
60	66	20	85	60	80	75	80	91	69%

Being an indicator means it is fairly Predictive and Relevant to information security, but not very Actionable (if only there was some simple and straightforward thing that management could do to improve morale!).  

Assuming that the raw numbers are available from HR (and possibly Procurement if you account for the comings and goings of consultants and contractors, as well as employees), they are likely to be both Genuine and Independent.  

The score for Meaning suffers because of the need to investigate and explain changes in the metric, while the Timeliness suffers because of the inevitable delays in gathering, analyzing, presenting and using the numbers.

In comparison to most other measures of the morale and contentedness of the workforce, this metric has the merit of being low Cost to gather although as we said the analysis does involve a bit of digging to determine the likely reasons for sudden changes, so it is not exactly free.

Organizations that employ seasonal workers and have greater 'normal' variations in employee numbers could still use this kind of metric by normalizing the statistics over successive years, assuming sufficient historical data are available.  You can probably picture a scattergram-type graph showing employee numbers through the year, with a smoothed curve following the mean level and a range of values at any point based on historical data.  Highlighting this year's curve and particularly the current/latest value against the mean and  usual range should show whether or not things are ticking along nicely, or something unusual is going on.

Although the overall PRAGMATIC score of 69% is hardly outstanding, this metric does feature in the top three HR-related information security metrics in our example set.  The HR security maturity metric that we discussed recently on this blog scored over 80% so - given the choice between those two - we would definitely expect that metric to be a better option than this one.

What HR security metrics would you prefer to use?  We welcome your suggestions of totally different metrics and variants of those we have discussed here, particularly those that you feel score substantially better on the PRAGMATIC criteria.  So, over to you ...

Social media naivete

Users of social media sites such as Twitter and Facebook don't always appreciate how much information they are giving away as they naively post updates.  For example, this site figures out where you live from the GPS info you send to Twitter from your Android system, looking up the latitude and longitude on Google Street View, while this site simply filters Facebook updates for potentially incriminating or embarrassing information.

Using social media Application Programming Interfaces to query their public message databases is much the same in principle as using Google to find sensitive stuff published inadvertently on the Web.  Once information is published, it is there for people to use.

Awareness lessons from Wal Mart

This year's social engineering 'capture the flag' competition at the DefCon hackers' conference was won by a contestant who socially engineers his clients' employees for a living.  In the course of a 20 minute phone call, he successfully fooled an unsuspecting Wal Mart employee into revealing potentially sensitive and valuable information, even persuading him to visit a (potentially infectious) website to 'complete a survey'.  Read more about the con here.

The con was cool in the sense that, live on stage, the contestant collected all the flags on his task list, but uncool in the sense that the attack was relatively straightforward and entirely benign, within the strict rules of the competition.  I've read about many similar attacks in books such as The Art of Deception, The Art of Intrusion and Ghost in the Wires by Kevin Mitnick, and Spies Among Us by Ira Winkler.  In No Tech Hacking, Johnny Long writes at length about the ability to research potential targets and identify vulnerabilities, while David Lacey discusses the psychological flaws that open up vulnerabilities in his Human Factor book.  This is not rocket surgery :-)

The troubling part is that actual real-world social engineers are far from benign, and don't follow the rules - in fact, they consciously eschew the rules and take advantage of not always making the anticipated approaches, gaining a significant advantage from being innovative as well as ballsy.  Social engineering is merely one tool in their toolboxes.

As to what Wal Mart might actually do to mitigate the risk of its customer services and other employees being socially engineered for real, reading Rebecca Herold's Managing an Information Security and Privacy Awareness and Training Program would be a great start - but don't get me wrong, a 'training session' for employees is certainly not going to make them immune to such attacks, while even a rolling/continuous security awareness program is not the Ultimate Solution either.

To claim otherwise is as ridiculous as a technical security consultant recently claiming that security awareness is a waste of money since incidents such as this still occur, and hence we must put all our faith - and $$$$ - into technical security controls.  As we say in NZ, "Yeh, right".  Of course you can throw big money down the drain by doing awareness incompetently and badly, in exactly  the same way as you can chuck money at unsuitable technical security controls, or neglect to train people in how to install, use, manage and maintain them properly (which, by the way, is itself a form of security awareness).

Social engineering is one of our most popular and important awareness topics, one that we revise, update and reissue annually.  You can be sure that the Wal Mart incident will be mentioned in the security awareness materials this December, delivering the module in good time for the Thanksgiving/Christmas/New Year holiday season when social engineering attacks are rife.  You can be sure because earlier DefCon Capture The Flag social engineering competition were featured along with various other social engineering incidents in previous modules.  The competition remains a golden opportunity in awareness terms, for those organizations that are far-sighted enough to appreciate its significance to them.

What's more, social engineering is just one of the 40+ awareness topics we cover, and in fact it gets a mention in some form in almost every other module, a practice known professionally as "reinforcement".  The idea is to remind people about various threats throughout the year rather than relying solely on a single awareness/training event.  The same thing applies to other commonplace security issues such as malware: a once-a-year malware focus is woefully inadequate to maintain a sufficient level of awareness.  I completely understand that those organizations who are still stuck in the Dark Ages, believing that an annual lecture to the troops on (usually just IT) security is sufficient, are less than impressed at security awareness.  That may be enough to comply with various badly-written laws and regulations, but it's way  short of good practice in this area.

And, by the way, compliance is another of those 40+ topics we cover!

That said, although we know how to do security awareness well, we also know it's never a perfect control.  We also emphasize the value of other forms of control therefore, ranging from security governance, risk management, policy,  business continuity and other strategic security stuff for management to technical security stuff for the IT professionals.  Managers and teccies also benefit from security awareness, whereas only addressing "end users" (which is itself a demeaning or belittling term for PEOPLE) is definitely missing a trick.  No wonder those old-fashioned "annual security training sessions" give awareness a bad name: they are almost guaranteed to fail.

If, at the end of the day, year-round information security awareness programs are sufficient to make the hackers, crackers, social engineers, industrial spies, identity thieves, organized criminals and security services go elsewhere for easier/softer targets, our customers are happy and our job is done - although naturally we're always hopeful of signing up the likes of Wal Mart and other victims who - finally - appreciate that they could do with some professional help.  

SMotW #18: security spend

Security Metric of the Week #18: information security expenditure

At first glance, this metric looks like it would be ideal for those managers who are obsessed with costs.  "Just how much are we spending on security?" they ask, followed shortly no doubt by "Do we really need to spend that much?"

OK, let's go with the flow and try to get them the figures they crave.

Our first challenge is to define what counts as security spend, or more precisely information security expenditure.  It's pretty obvious that the salaries for full-time dedicated information security professionals go in that particular bucket, but what about the security guards, or the network analysts and systems managers, or the architects and programmers spending some variable proportion of their time developing security functions?  Oh and don't forget managers and staff 'wasting their valuable time' constantly logging back in or changing their passwords or whatever: does that count as security spend?  If so, how much, exactly?  

Then there's the security hardware and software - the antivirus and firewall systems, and backups ... and what about the additional incremental costs of finding and purchasing secure as opposed to insecure systems?

Next, security incidents: these are 'clearly' security costs, aren't they?  Well, no, it could be argued that incidents result from the lack of security, the very opposite.

Issues of this nature fall into the realm of cost accounting, allocating the organization's costs rationally across the appropriate accounting categories.  Given sufficient interest and effort, costs can be allocated although the figures will inevitably be highly subjective depending on exactly what proportion of various costs is labeled 'information security'.  Due to the arbitrary decisions, this is likely to be a significant source of error when trying to compare the figures across successive periods, even if some of the cost allocation decisions are captured in a cost accounting system.  Consequently, the Accuracy rating for the metric is quite low, and the Time and Costs incurred in measuring it are also low-scoring factors on the PRAGMATIC score:

P	R	A	G	M	A	T	I	C	Score
82	94	60	60	89	29	33	49	59	62%

The high scores for Predictiveness, Relevance and Meaning might be worth building upon, in other words would it be possible to alter the metric's definition to improve the lackluster PRAGMATIC score?  Knowing the total expenditure on information security would be fascinating, but unfortunately that's still only half of the value equation.  What about the benefits of information security?  This is where things get really tricky.  The primary benefit of security is a reduction in risks, in other words a secure organization suffers fewer and/or smaller incidents.  Measuring the value of the risk reduction is difficult, involving various assumptions and estimations based on the predicted occurrence and severity of incidents if there was no security in place.  Further benefits are associated with the assurance element of security - the confidence for the organization to be able to do business that would otherwise be too risky.  Again, hard but not impossible to value.