KorrawinSUMMING UP
Will Digital Experimentation Take Hold in Industies Outside High Tech?
The several responses to this month’s column generally were favorable to the notion that experimentation is gaining momentum as an everyday habit in many organizations, especially in organizations heavily reliant on the internet and cloud for their existence.
“There is a softening of attitudes towards quants in today’s organization,” Tahir Qazi commented. “ … As more people with an experimentation mindset and grooming move across organizations, it is natural that they will tout their past successes and seed the concept more broadly.” But Ana Frony reminded us to “never forget that even trainees must have some intuition and poetry to develop a hypotheses and design an experiment.”
Nick C raised the question of just how far the practice of routine experimentation might extend. As he put it, “Whilst the relentless quest for adaptation and improvement in the so-called technology sectors seems to fuel the need for experimentation, the nature of experimentation in other sectors, including the high hazard/high reliability industries, may reflect a bias for knowing … the consequences of ‘failure. ’” It would be interesting , he continued, to examine sectors other than technology “for their natural tendencies and what drives such culture and procedural traits.”
Joe noted that some of the principles of experiment design provide “a piece of business wisdom that’s lost on firms who have a rigidly hierarchical bureaucratic structure ill-suited to innovation.”
What are other “natural tendencies” alongside the kind that Nick C mentions? His and Joe’s comments suggest that such natural tendencies may not be enough. Effective adoption of experimentation will require changes in organization “traits” as well.
What kind of “life” will experimentation have outside high tech? What do you think?
Original Post
“Test, then invest” is an increasingly common saying among entrepreneurs and those who finance their startups. The idea is simple: Fast, inexpensive tests in the marketplace can often increase the probability of success for ideas central to a strategy.
Frequent testing has been especially relevant for retailers, whose merchandise displays offer endless opportunities to test product placement ideas. Now organizations whose businesses are primarily online have taken the strategy to a new level. Their work has been both influenced by and reported recently by several academics.
Harvard Business School professors Michael Luca and Max Bazerman track experimentation in decision-making back to the Book of Daniel, BC 167-164. They put forward a context leading up to what they call “the beginning of an experimental revolution,” fueled by data, that has become more available using online platforms that make randomization cheaper and easier. These have fueled a greater appreciation for how experiments can “help to complement intuition with evidence.”
“Learning organizations try a lot of things and keep what works.”
It is not by chance that nearly all of their private sector examples come from high tech, where employees are being trained in large numbers to routinely conduct experiments that appear to provide enterprise agility and competitive advantage. Experimentation is endemic to the so-called learning organization.
Learning organizations try many things and keep what works. This process requires intensive testing of ideas as well as reliance on the data produced by the tests, rather than depending on untested opinions (too often generated by leaders) that dominate decision-making today.
Experimentation in action
Firms competing in today’s high tech environment routinely test such variables as offerings, page design, and service. For example, Booking.com, a Dutch-based travel website, has developed what another HBS Professor, Stefan Thomke, calls an “experimentation organization.” In his book Experimentation Works, Thomke writes that at Booking.com, “experimentation is an integral part of everyday life (where) … anyone can conduct or commission a test (without approval from above).”
This requires that employees receive training in how to design and carry out experiments. Booking.com conducts thousands of tests every month. Evidence from a test always trumps executive opinion. Even the failures that tests often produce are regarded as opportunities for learning rather than costly mistakes. The only failure is a poorly-designed test, one with weak hypotheses, poor data, the lack of a control group, and careless analysis of the results.
Experimentation on a large scale has worked for Booking.com, apparently. The Company has employed this strategy to gain the largest share of the traffic in its industry.
High tech organizations are producing an army of people who will be hired into other organizations based on their skills, successes, and advocacy of experimentation. How will their attitudes play in organizations where intuition is still honored and the poets can still speak louder than the quants? Or are those days in our past? Will the experimentation organization become the competitive gold standard? What do you think?
References:
Michael Luca and Max H. Bazerman, The Power of Experiments: Decision Making In a Data-Driven World (Cambridge, MA: The MIT Press, 2020)
Stefan H. Thomke, Experimentation Works: The Surprising Power of Business Experiments (Boston: Harvard Business Review Press, 2020)