When customers log onto Booking.com—the world’s largest online accommodations platform—they might naturally assume they are seeing the same website as every other customer logged on at the same time. In fact, Booking is running quadrillions (millions of billions) of landing page permutations, with subtle tweaks to everything from varying shades of color to alternative placement of links and menu options for booking properties.
It’s part of an innovative culture of experimentation that pervades every aspect of how the company operates, says Stefan Thomke, William Barclay Harding Professor of Business Administration at Harvard Business School, and author of the forthcoming book, Experimentation Works: The Surprising Power of Business Experiments. Booking isn’t alone. Other companies such as Amazon, Microsoft, and Netflix are also taking advantage of new tools to conduct scientific experiments to optimize customer experience—and their own return on investment.
“Very few people know they are participants in this massive experimentation ecosystem,” says Thomke. And it’s not just online. Retailers such as Walmart and Kohl’s are also running disciplined experiments, investigating customer purchasing decisions based on layouts that change from store to store. “It’s around us now, and it’s growing.”
"If something is very novel, there is little data around it to use."
Thomke first explored the phenomenon in his 2003 book Experimentation Matters, which describes the use of simulation and rapid prototyping to optimize product development and R&D. “After that, I thought I was done,” he says. Since then, however, there has been a rise in new technology platforms that allows experimentation to innovate on an unprecedented scale, in every aspect of business.
One of the biggest problems in innovation is that it can generate uncertainty, says Thomke. Even in an age of Big Data, companies are stymied by the fact that they don’t know what they don’t know. “If something is very novel, there is little data around it to use.” Using experimentation to systematically test new ideas and gauge their effects is a game changer, revealing surprising insights that can lead to incremental changes that can have a cumulatively huge effect on business.
After research that the checkout experience could be improved, for example, Booking’s designers noticed that a drop-down menu asking for children’s ages “at checkout” was potentially vague. They wondered what would happen if they specified the checkout date. Creating two different versions of the same menu, they tested it in real time, and determined that the change had a positive impact on the site’s key metric.
- Insight: User research suggested that the checkout process could be improved.
- Hypothesis: Displaying the checkout date when selecting the number of children improves the user experience (by adding clarity).
- Result: Treatment had a significant positive impact on key metric; hypothesis supported and challenger launched as new champion.
Example of an experiment conducted by Booking.com; from Experimentation Works by Stefan Thomke. Discipline required
In order for such an experiment to work, says Thomke, a company must be disciplined—having a testable hypothesis with a control group (status quo) and one or more variations that are randomly applied to customers. As important, he says, it must also agree to abide by the results. In his book, Thomke includes Booking as a case study, showing how running these kinds of experiments by the tens of thousands helped to transform it from a small Dutch startup to the most successful online hotel platform, with some 1.5 million room-nights booked daily.
“They fundamentally believe that the ability to run large experiments has been critical to their success,” says Thomke. At Booking, experimentation is not the responsibility of a niche division; a full 75 percent of core employees participate, with the ability to pull the trigger independently to virtually test anything at any time.
To illustrate that point, Thomke describes the time an incoming CEO called a meeting to present a new logo design. “The CEO said here’s what we are going to do,” Thomke recounts. “The employees said, that’s great. We’ll check it with an experiment. The experiment determined if the logo could stay.”
As that story illustrates, the role of leadership is very different in an experimentation organization, changing from a person who gathers information and makes top-down decisions, to a person who facilitates the process of decision-making through experimentation. “If you really believe in this, then you have to be a believe that you as a leader don’t have better answers that anyone else,” Thomke says. “You have to leave your ego at the door.”
He quotes Booking’s chief product officer, who says that when it comes to experimentation, “in our industry, you have to embrace it or die.” New CEOs, he continues, needs to ask themselves two big questions: How willing are you to be confronted every day by how wrong you are? And how much autonomy are you willing to give to the people who work for you?
That doesn’t mean leaders are superfluous, however. In order to reap the benefits of experimentation, Thomke says leaders must set out a roadmap to determine a company’s strategic objectives and break those big questions into manageable parts. In addition, they need to serve as a role model, showing employees they live by the same rules and allow themselves to be governed by results as much as anyone else.
5 steps for scaling
While developing that culture may seem daunting for companies that aren’t the size of Booking—let alone Microsoft or Amazon—Thomke lays out a process of five steps by which companies can effectively scale experimentation, using an alphabetical mnemonic: Awareness, Belief, Commitment, Diffusion, Embeddedness.
The five stages of creating an experiment organization, adapted from Experimentation Works by Stefan Thomke.
- Awareness: Management knows that business experiments matter to innovation.
- Belief: Management accepts that a more disciplined approach to experimentation is needed; adopts a rigorous framework and testing tools in small and specialized groups.
- Commitment: Management pledges to make experimentation core to learning and decision-making, requiring input from experiments for some innovation decisions.
- Diffusion: Management realizes that scale is key to having a larger business impact, rolling out formal company-wide programs and standards.
- Embeddedness: Disciplined experimentation becomes deeply rooted and democratized; accessed by a large percentage of employees, who consider it as ordinary as “running the numbers.”
At the final stage, companies become a true experimentation organization, says Thomke. “It’s a little like breathing. People are empowered and capable of running these type of experiments on their own.” The results can be impressive. On average, he says, companies may find that only 10 percent of experiments yield results that improve the bottom line. But multiplied by 1,000 experiments a year means 100 wins that impact the business.
“It’s really the scientific method turbocharged,” says Thomke. “Companies better get on this if they don’t want to be at a major competitive disadvantage.” On the other hand, he says, companies that begin transforming themselves now can reap the benefits of hundreds or even thousands of small improvements. “If you get the small things right then it can scale to have a giant impact.”
About the Author
Michael Blanding is a writer based in Boston.
[Image: ndresr]
Related Reading
- At Booking.com, Innovation Means Constant Failure
- Jumpstarting Innovation: Using Disruption to Your Advantage
- Research Paper: Experimentation and Startup Performance: Evidence from A/B Testing
