How Will the “Age of Big Data” Affect Management?
Summing up: How do we avoid losing useful knowledge in a seemingly endless flood of data? Jim Heskett's readers offer some wise suggestions. What do you think?
Summing up: Will Access to Big Data Further Enable Fact-Based Decision Making or … Analysis Paralysis?
"To T. S. Eliot's prescient words 'Where is the wisdom we have lost in knowledge? Where is the knowledge we have lost in information?' we might now add, 'Where is the information we have lost in data?'"
That's Paul Nicholas' reaction after reading most of the responses to this month's column. It's not a bad "sense of the meeting," in which many contributors offered suggestions to managers wishing to get the most out of so-called big data and avoid paying the price of paralysis in the process.
The challenges Big Data pose for managers include "identifying which data are relevant" (Subrata Chakraborty) and "seeing through the woods to know what to use and what not" (Pieter J de Beer). Scott Waller expressed the fear that "the age of Big Data seems to be arriving at the time of death for the Big Thinker." Gerald Nanninga cautioned us that "the big risk is that it gives executives a false sense of comfort." Clifford Francis Baker added, "My concern is primarily focused on the possibility of complacency … data derived from data analytics must … be handled with care." Mok Tuck Sung commented that "technology and knowledge advancement have again developed faster than the managers' and business owners' capabilities to leverage its usefulness in their decision making process." Philippe Gouamba reminded us that "perfection is but an illusion … A successful outcome is almost never the result of perfect information."
Among the suggestions for managers were these:
- Help "data analyzers consume and translate the data" (Scott Kemme, who also suggested an alternative title for the column, which I instead used above),
- Know "what not to look at." (Phillip Clark)
- Work to reduce turnover among business analysts who "tend to be lower level employees and have a high turnover," creating a "losing battle" through the loss of "institutional data knowledge." (Kim Kraemer)
- Avoid the belief that "whatever is new will solve their problems," concentrating on the appropriate application of Big Data. (Seena Sharp)
- Avoid allowing Big Data to remain the "purview of the select few" only for use for one-off and one-time decisions." (Jonathan Spier)
- Maintain the attitude that "fast is better than perfect." (Mike Flanagan)
- Avoid the temptation of harvesting just the "'low hanging fruit' rather than waiting for the analysts to gain understanding (presumably of the decisions to be made)." (Sean O'Riordain)
- Rather than concentrating on the data, focus on "being able to formulate the right questions to ask at precisely the right moment." (Edward Hare)
That's quite a list.
Tom Dolembo drew a picture of direct process control in which decisions have to be made "in the flow" in situations where there isn't time for conventional analysis (at least by humans). Is this a glimpse into the future of decision-making (without analysis paralysis) based on big data for a widening range of decisions or is it confined to a special set of conditions? Will access to Big Data further enable fact-based decision making or … analysis paralysis? What do you think?
Ideas and trends converge from time to time in a way that suggests the possible shape of the future. Sometimes I think I can comprehend what they may mean. But other times I know I need help. This is one of those times.
Just two decades ago, we didn't have Google and other information sources; storage constraints would not have permitted Google to provide everyday access to the "world's information." If we had had the information, we couldn't have accessed it effectively anyway. Email systems were not widely available, let alone mobile devices with capacity to access the data. Now the capacity to store and access information through cloud computing is so great that we are entering a post-Google era in which new organizations like Factual (founded by a former Google employee) have set as their goal that of providing access to all of the world's facts. Presumably this means data such as the location of every factory in the world, data that has not already been massaged and spun. Some facts have to be acquired and organized. Other facts are generated by so-called digital sensors operating worldwide in industrial equipment, autos, and the like. By linking the sensors, an "industrial Internet" can be created.
These trends appear to have "opportunity" written all over them, particularly for those who are training now for jobs in data analytics. In addition to less wasteful marketing efforts (we should be able to know, for example, "which half" of advertising is effective, thereby making an old marketing saw obsolete), they should produce more effective business strategies and inject added certainty into the appraisal of opportunities for new business startups. Furthermore, analytics (not the data) should be a source of continuing competitive advantage. In his new book, Charles Duhigg describes how the retailer Target uses data on consumption patterns to discern and address promotions to pregnant customers, perhaps even before they've announced their pregnancy to friends (and Target competitors). This is particularly important because pregnancy is one of those life events associated with significant shifts in consumption habits.
A problem is that the shortage of experts in data analytics (some call them "data whisperers") is so acute that it may be years before a sufficient supply can be trained. The McKinsey Global Institute estimates that up to 190,000 are needed now in the US, along with 1.5 million managers capable of using their work. The shortage appears to be growing along with the potential for competitive advantage associated with data analytics.
This all raises many questions. Will the age of big data eliminate most or all uncertainty from business decisions for those most able to make effective use of "all the facts in the world?" Will it fuel the next "gold rush" for talent in a quest for competitive advantage? Will analytics, as well as the supply of analytics-savvy managers, so badly lag "big data" that it will only lead to confusion and misguided decisions? Or is this just the latest management fad? How, if at all, should this affect education for management? What do you think?
To read more:
Charles Duhigg, The Power of Habit: Why We Do What We Do and How to Change It , Random House, 2012.
Quentin Hardy, Just the Facts. Yes, All of Them, The New York Times, March 25, 2012.
Steve Lohr, The Age of Big Data, The New York Times, February 12, 2012.
The challenge—and opportunity—of 'big data', McKinsey Global Institute, May 2011.