Smoke-n-mirrors IBM style

I've just been reading the IBM 2015 Cyber Security Intelligence Index, trying to figure out their 'materials and methods' i.e. basic parameters for the survey, such as population size and nature. All I can find are some obtuse references in the first paragraph:

"IBM Managed Security Services continuously monitors billions of events per year, as reported by more than 8,000 client devices in over 100 countries. This report is based on data IBM collected between 1 January 2014 and 31 December 2014 in the course of monitoring client security devices as well as data derived from responding to and performing analysis on cyber attack incidents. Because our client profiles can differ significantly across industries and company size, we have normalized the data for this report to describe an average client organization as having between 1,000 and 5,000 employees, with approximately 500 security devices deployed within its network."

Reading between the lines, it appears that this is a report gleaned primarily from 'more than 8,000 client [network security?] devices' belonging to an unknown number of organizations around the world who are customers of IBM Managed Security Services ... which IBM has described as:

"24/7/365 monitoring and management of security technologies you house in your environment. IBM provides a single management console and view of your entire security infrastructure, allowing you to mix and match by device type, vendor and service level to meet your individual business needs while drastically reducing your security costs, simplifying security management and accelerating your speed to protection."

But, before you delve into the actual report, read that final sentence of the first paragraph again: they have 'normalized the data' (whatever that means) to an 'average client organization' with about 500 security devices ... so given the total of 8,000 devices, and on the assumption that 'average' means 'mean', it appears the survey covers just 16 organizations whose network security devices are managed by IBM. Oh boy oh boy. No wonder they are so reluctant to tell us about the analytical methods!  

The data are from 2014, the report was published in July 2015. Given the miniscule sample, I wonder why it took them 7 months to do the analysis and reporting? Crafting the words to gloss over the glaring flaws, perhaps?

The remainder of the report is pretty humdrum - some superficially interesting graphics and four 'case studies' (three of which - that's 75% or a 'vast majority', IBM - are not actual cases as such but fictional accounts based on the collective experiences of an unknown number of clients). There's nothing particularly unusual or noteworthy in the report, despite the hyperbole (2014 was hardly "The year the Internet fell apart", IBM). The trends and other statistical information is worthless in scientific terms.

Remember this cynical blog piece whenever you see the report quoted. Better still, read the report for yourself and make up your own mind.