What worries CEOs?

According to KPMG, the answer is ...

I've picked out just one of a plethora of gaudily-colored horizontal bar charts from KPMG's 2016 Global CEO Outlook, which I encourage you to download and read. The Now or Never report blends insightful analysis with survey data and anecdotal comments. It's well-written, glossy and eminently readable.

The reason I chose to illustrate this blog piece with that particular chart is that it confirms what I already thought about risks of concern - in other words, I openly admit my 'confirmation bias'. The same could be said for most of the rest of the charts, and indeed the report as a whole, which at least partially explains why I'm blabbering on about it here. It was on-topic for me, interesting enough to read, consider ... and then pick holes in.

Even if you disagree with the report's findings, or have little to no interest whatsoever in the subject matter, it's still worth reading in respect of the manner in which the survey was designed, conducted and reported.

The survey methodology - or rather the surveyed population - is briefly outlined thus in the report:

"The survey data published in this report is based on a survey of 1,268 chief executives from Australia, China, France, Germany, India, Italy, Japan, Spain, the UK and the US. The findings from CEOs in an 18 additional regions and countries can be found in the Appendix section of this report. Eleven key industries are represented, including automotive, banking, infrastructure, insurance, investment management, life sciences, manufacturing, retail/ consumer markets, technology, and energy/utilities and telecom. Two hundred and seventy-five CEOs came from companies with revenues between US$500 million and US$999 million, 595 from companies with revenues from US$1 billion to US$9.9 billion, and 398 from companies with revenues of US$10 billion or more. Eight hundred and ninety-three CEOs came from public companies and 375 from private companies. The survey was conducted during 15 March and 29 April of 2016."  [page 39]

Hmmm, that one paragraph is about all they have to say about the survey, aside from calling out various KPMG people and respondents by name throughout the report (call me a cynic, but I suspect the names were not drawn at random out of a hat!). 1,268 may be a reasonably large size sample for this kind of survey but nevertheless it is an almost infinitesimal proportion of all CEOs in the world. 

Bearing that in mind, take a fresh look at, say, the colorful bar graph above. We can see what it tells us, but what is missing?

For a start, we aren't told how many respondents answered the particular question. Did all 1,268 of the CEO's surveyed answer, I wonder? More contentiously, did any of them answer it? The actual number of responses to that question probably lies between those extremes, quite likely towards the top end, but we simply don't know from the report.

That issue, in turn, has implications on how we interpret the numbers. Are those differences statistically significant or within the margin of error for this survey question? 24-30% is actually quite a narrow range, even for a fairly large scientifically-designed survey. The 1 or 2% differences between the bars, and hence the priority/sequence of their presentation, could easily be an artifact of the survey. We just don't know.

Furthermore, there is no report appendix listing the actual survey questions and (I guess) the multiple-choice answers permitted. Were respondents asked to choose between a list of those 5 particular risks, or were there others listed that didn't even make it into the report? How were those options selected and worded? What, precisely, is "cyber security risk" for instance? Was there a 'something else' option, allowing them to elaborate on whatever burning issues keep them up at night - and if so, how were those free-text responses handled? Were respondents able to select multiple answers, or required to rank them in some way? Were respondents induced, fooled or coerced into answering in a particular way, for example by dint of the context or preamble to the survey or this question, or by the fact that KPMG was asking? Did the bar chart even arise from 'a question', as such, or was it a synthesis of selected data plus some analyst's fertile imagination? Again, we simply don't know from the report.

On a broader note, there is no explanation of how those 1,268 survey respondents were selected (perhaps KPMG clients, or maybe a pool of Forbes Insight contacts?) nor how the survey was conducted (not even the fundamental approach such as on-line or in-person). Were any respondents de-selected i.e. disqualified outright, or their responses to this hypothetical question discounted for some reason? 

Talking of Forbes Insight (whose pretentious name I only spotted when I re-read the report while blabbering on about it here), this is what I've found out about them so far, a byline from the Forbes website:

That's hardly a detailed or convincing statement as to their competence and suitability for such a survey. They might as well have said, lamely. "We do this kind of stuff". Digging just a bit deeper, I find that Forbes is heavily into public relations and overtly self-promotional cobblers like this:

I suspect KPMG paid Forbes Insight either to conduct the entire survey, or maybe to access their mailing list of 'pre-qualified' (pre-duped, softened-up) potential respondents. Note, yet again, that I'm speculating due to the dearth of information in the report itself. There's a lot more omitted than stated ... which perhaps explains the 8 month delay between completing the survey in April and issuing the report in December. The statistical analysis and report-writing would have taken, I guess, about 1-4 months whereas it takes longer to spin a line. 

So much for the forward-looking 2017-2020 focus of the report: the CEO's were surveyed months ago, prior to Brexit (June) and Trump (December). Hey, KPMG, haven't you noticed? Big Things have happened this year.

Oh and another thing. Take a closer look at the survey sample. Why did they only apparently survey respondents from "Australia, China, France, Germany, India, Italy, Japan, Spain, the UK and the US"? What about the other 200 odd countries of the world? And how come the report discusses locations such as Korea, Ireland and East Africa if there were not (according to the report) any respondents from those areas?  Odd, that, very odd. [Cue faint whiff of rodent].

Clearly, I am a cynic. Take my inane comments with a pinch of salt. Argue with or berate me, if you will. But please at least think about what you are reading, both here on the blog and in surveys of the kind I've critiqued. FWIW this is one of the better ones.