Another day, another survey, another ten failures


An article in an eZine concerning a security survey by PwC, sponsored by Iron Mountaincaught my eye today because they offer to benchmark respondents against others. So, purely in the interest of metrics research, I had a go at the benchmark tool.

First of all, the tool asked me for an email address without explaining why. Fail #1 (see also #6 below).

Thankfully, the email address validation routine is easily fooled. Fail #2 (or possibly Success #1 depending on one's perspective!).

Next the survey asked about 20 questions, mostly lame and some badly worded. There is no explanation about why those 20 questions have been selected. They address only a small part of information security. Fail #3.

All 20 questions have the same set of 4 possible multiple-choice answers, even though the stock answers don't cover all possibilities and don't even make sense for all the questions. The survey design is poor. Fail #4.

At the end of the survey, I was presented with a comparative "index score", in my case 41 (presumably 41%), along with a nasty day-glow bar chart and the following commentary:
"Your risk level is serious. Your score is well below the PwC recommended threshold, and it is only a matter of time before serious problems occur. Your business needs to take action now to improve information security and working practices. Read this report for further insight into your business risk and gain practical advice on how you can increase your index score and reduce your risk."
Our risk level is 'serious', eh?  Spot the scare tactics! FUD! Thanks to those 20 lame questions, they know next to nothing about our true situation, yet they presume to tell me we are "well below the PwC recommended threshold". Bloody cheek! Yes, we are facing various information security threats and have various information security vulnerabilities, and no, we have not implemented all the information security controls that might be appropriate for other organizations, but that's not the same as saying we are at serious risk. As my pal Anton would say, "Context is everything". Fail #5.

Next I was asked for yet more personal information in order to access the 'personalized report'. In reality, of course, this is clearly a marketing initiative so I know what they are up to, though again they don't actually say. Failing to explain why the information is needed conflicts with at least one of the OECD privacy principles concerning personal data collection and I think would be illegal under the privacy laws in most of the world outside the US. Fail #6.  

And again the data entry validation routines are weak. Fail #7.

The 'personalized report', "Your information risk profile", compares my score against "averages" (mean scores?) from the PwC/Iron Mountain survey in a PDF report. Generating the PDF on the fly is cool ... but the actual content is poor. The scope and purpose of the PwC survey are not stated in the 'personalized report', nor is the sample size or other basic information about the survey methods. The entire basis of the benchmarking is dubious, particularly if the PwC survey that generated the comparative data also used the lame 20 question multi-choice method. It's essentially meaningless drivel. Fail #8.

For no obvious reason, the 'personalized report' includes a page stating 4 "worrying facts", the first of which being "88% consider paper to be the biggest threat to information security". Eh? In all my years of information security risk management, I have NEVER heard paper being described as an information security threat. Paper is an asset usually of negligible value, although the information content on paperwork can be extremely valuable and a few, very rare bits of paper are priceless ... oh, hang on, Iron Mountain sponsored this survey, right. Ah yes, I remember, that's the same Iron Mountain whose archive facilities have suffered several serious fires including one recently in Buenos Aires. So much for their security credentials. It could be argued that Iron Mountain is "the biggest threat to [its customers'] information security"! Fail #9.

The 'key findings - your next steps' to the 'personalized report' kind of make sense in so far as they go, but bear no obvious relation to the benchmark, survey or the data.  Although for example I'm personally in favor of step 1 'Take it to the top - Get board level support by taking a strategic approach to information management', I'm also in favor of 'Don't run with scissors' and 'Don't smoke' which make about as much sense in this context. Fail #10.

As to the actual PwC/Iron Mountain survey, I encourage you to take a critical look at the survey report, and make of it what you will. Read past the annoying repetitive references to "the mid market" (which I think must be marketing-speak for medium-sized organizations) and the wrongly-labeled graphs. Set aside the spurious references to additional information and news headlines muddled in with the survey data, and the buzzwords-du-jour. Consider the "comprehensive questionnaire" of just 34 statements and the dubious statistics arising, and see what you have left.

Then read the report's disclaimer very carefully:
"This publication has been prepared for general guidance on matters of interest only, and does not constitute professional advice. You should not act upon the information contained in this publication without obtaining specific professional advice. No representation or warranty (express or implied) is given as to the accuracy or completeness of the information contained in this publication, and, to the extent permitted by law, PricewaterhouseCoopers LLP, its members, employees and agents do not accept or assume any liability, responsibility or duty of care for any consequences of you or anyone else acting, or refraining to act, in reliance on the information contained in this publication or for any decision based on it."
[PwC, I'm very disappointed in you. Your work on the UK DTI/BERR security surveys has been pretty good. Your consultants and auditors are generally competent and well-respected. What on Earth possessed you to get involved in this nonsense? Are you really that hard up these days?]