Innovation and experimentation are vital to the longevity of any organization, says Stefan H. Thomke in his new book Experimentation Matters: Unlocking the Potential of New Technologies for Innovation. In this interview, Thomke, an associate professor of technology and operation management at Harvard Business School, discusses how businesses can sidestep the often prohibitive costs and time-consuming trials of experimentation by implementing new technologies.
Wendy Guild: We've all heard the old saw, "If it ain't broke, don't fix it." How dangerous is stasis to an organization and what are some of the possible negative consequences of remaining stagnant?
Stefan Thomke: Competitive environments and technologies are constantly changing, which creates both wonderful opportunities to innovate and grave threats if we fail to respond to such changes. For example, my book shows that product and service development is changing; creating the potential for higher R&D performance, innovation, and value creation for customers. The choice is simple: Organizations can either ignore these changes or take action and tap into this new potential.
Q: What are the first steps in assessing which management practices and processes in an organization could benefit from experimentation?
| It is important to understand that experimentation matters to managing change and uncertainty. |
A: It is important to understand that experimentation matters to managing change and uncertainty at four different levels: technical (can it work?), production (can it be produced?), need (does it address customer needs?), and market (is it big enough to justify the investment?). So managers need to assess at which level the uncertainty and opportunity for innovation is the greatest. For example, a set of well-designed business experiments can address new and unknown markets. In contrast, running experiments where early product prototypes are shown to customers can address need uncertainty.
Q: How should organizations respond when experiments fail?
A: We need to appreciate that new knowledge comes as much from failure as it does from success. Innovators learn from failure: Understanding what doesn't work may be at least as important as understanding what does, provided these failures are revealed early in a project and are swiftly reexamined. Learning from failure is a boon at this point: Few resources have been committed and decision making is flexible, meaning that other approaches can themselves be tested. Thus, experiments that result in failure should not be viewed as failed experiments.
Q: New experimentation technologies are helping to reduce the time and cost of innovation. What are some of the new "star" technologies and how can the average business integrate them in order to conduct low-cost, rapid testing?
| We need to appreciate that new knowledge comes as much from failure as it does from success. |
A: Rapid advances in technology are changing the economics of experimentation and are triggering fundamental changes in R&D processes and performance in such fields as integrated circuit design, automotive development, and pharmaceutical drug discovery. Computer modeling and simulation, rapid prototyping, and combinatorial technologies drive down the marginal cost of experimentation and allow companies to create more learning more rapidly, provided that managers can capture that potential. My book provides a roadmap of how the average business can take advantage of these new technologies.
Q: How can businesses shift experimentation to their customers?
A: The final chapter of my book explores how this can be done. Essentially, I show how some companies have abandoned their efforts to understand exactly what products their customers want and have instead equipped them with tools to design and develop their own new products, ranging from minor modifications to major new innovations. The user-friendly tools, often integrated into a "toolkit" package, deploy new technologies (e.g., computer simulation and rapid prototyping) to make innovation faster, less expensive and, most importantly, better, as customers run "what-if" experiments themselves. A variety of industries have started to use this approach. 
[ Order this book ]
Book Excerpt: Six Principles of Experimentation
Experimentation matters because it fuels the discovery and creation of knowledge and thereby leads to the development and improvement of products, processes, systems, and organizations. Put concretely, without experimentation, we might all still be living in caves and using rocks as tools. Anything we use today arrives through a process of organized experimentation, over time; improved tools, new processes, and alternative technologies all have arisen because they have been worked out in various structured ways.
But experimentation has often been expensive in terms of the time involved and the labor expended, even as it has been essential to innovation. What has changed, particularly given new technologies available, is that it is now possible to perform more experiments in an economically viable way while accelerating the drive toward innovation.
This book emphasizes not only why experimentation matters in the largest sense of its connection to innovation, but why and how new technologies are transforming its economics. Not only can more experiments be run today, the kinds of experiments possible are expanding. Never before has it been so economically feasible to ask "what-if" questions and generate preliminary answers. New technologies enable organizations to both challenge presumed answers and pose more questions. They amplify how innovators learn from experiments, creating the potential for higher research and development (R&D) performance and new ways of creating value for firms and their customers. At the same time, this book discusses companies that do not fully unlock that potential because how they design, organize, and manage their approach to innovation gets in the way. That is, even deploying new technology for experimentation, these organizations are not organized to capture its potential value—in experimentation or in innovation. Like any effective experiment, this book will reveal what does and does not work.
Experimentation encompasses success and failure; it is an iterative process of understanding what doesn't work and what does. Both results are equally important for learning, the goal of any experiment and of experimentation overall. Thus, a crash test that results in unacceptable safety for drivers, a software user interface that confuses customers, or a drug that is toxic can all be desirable outcomes of an experiment—provided these results are revealed early in an innovation process and can be subsequently reexamined. Because few resources have been committed in these early stages, decision making is still flexible, and other approaches can be experimented with quickly. In a nutshell, experiments that result in failure are not failed experiments, although they frequently are considered so when anything deviating from what was intended is deemed "failure."
Herein lies the managerial dilemma addressed in this book. A relentless organizational focus on success makes true experimentation all too rare. Because experiments that reveal what doesn't work are frequently deemed failures, tests may be delayed, rarely carried out, or simply labeled verification, implying that only finding out what works is the primary goal of an experiment. If there is a problem in the experiment (and there always is at least one!), it will, under this logic, be revealed very late in the game. But when feedback on what does not work comes so late, costs can spiral out of control; worse, opportunities for innovation are lost at that point—reinforcing the emphasis on getting it right the first time. By contrast, when managers understand that effective experiments are supposed to reveal what does not work early, they realize that the knowledge gained then can benefit the next round of experiments and lead to more innovative ideas and concepts—early "failures" can lead to more powerful successes faster. IDEO, a product development firm discussed in Chapter 6, calls this "failing often to succeed sooner."
But organizing for more frequent, rapid feedback—as powered by these new technologies—is not trivial. Multiple issues can arise, for instance, the "problem" of greater experimental capacity. What do we do with the opportunity to experiment "more"? Consider the attempted integration of computer modeling and simulation—examples of new experimentation technologies discussed throughout the book—in the automotive industry. Car companies have spent hundreds of millions of dollars on computer-aided technologies and employ many engineers and specialists to improve the performance of their complex development processes. By replacing expensive physical testing with virtual models, management hopes not only to save costs and time but also to streamline decision making and coordination among team members.
Principle #1: Anticipate and exploit early information through "front-loaded" innovation processes
Large companies often spend millions of dollars to correct late-stage development problems but underestimate the savings of early experimentation. Moreover, enormous time and energy is expended on these problems, derailing schedules and budgets—taking away "opportunities" for companies to focus their precious resources on other projects. New technologies are most powerful when they are deployed to test what works and what doesn't work as early as possible—the "frontloading" effect. These experiments are not as complete or perfect as late-stage tests, but they are able to direct early attention and integrated problem solving at potential downstream risks. Significantly, these experiments can reveal what does not work before substantial resources are committed and design decisions are locked in. With more experimentation capacity available during early development, teams are also more likely to experiment with many ideas and concepts that will ultimately result in better products and services. The early sections of Chapter 5 show how this is done at companies such as Microsoft, Boeing, and Toyota.
Principle #2: Experiment frequently but do not overload your organization
With the benefits of front-loaded innovation processes, there remains the question of how frequently companies should run experiments. The quest for efficiency, combined with an incomplete understanding or measurement of the benefits of early problem solving, has been driving out experimentation. Money can be saved, so goes the logic, by lumping experiments into one big test or delaying them as long as possible until they are more likely to result in the verification of "success." In contrast, experimenting more frequently reveals what does and does not work with minimal delay and problems can be addressed right away, thus minimizing the cost of redesign. The middle section of Chapter 5 thus suggests that a good experimentation strategy balances the value of early information against the cost of repeated testing. 5 Most companies find, however, that their accounting tracks the latter cost but little formal analysis is done on the former savings, shifting the balance toward too little experimentation. When new technologies dramatically drive down the cost of testing, understanding the need for frequent experimentation becomes more important than ever. As with many other changes, however, companies need also to prepare for the rapid increase in information that will result from more experimentation, information that has to be processed, evaluated, understood, and used in the planning of more experimentation.
Principle #3: Integrate new and traditional technologies to unlock performance
The new experimentation technologies discussed in this book should not be used in isolation; they need to be placed within an innovation system so that they enhance overall performance. While impressive in their potential, new technologies like computer simulation may not achieve more than 70 percent or 80 percent of their traditional counterpart's technical performance—but they can get there more quickly. Thus, by combining new and traditional technologies, organizations can avoid a performance gap while also enjoying the benefits of cheaper and faster experimentation. The later sections of Chapter 5 show how the true potential of new technologies often lies in a company's ability to reconfigure its processes and organization to use them in concert with traditional technologies. In rare instances is the technology so advanced that it instantly displaces its traditional counterpart and all the development experience and engineering knowledge that goes with it. Eventually, a new technology can replace its traditional counterpart, but it then might be challenged by a newer technology that itself must be integrated.
Principle #4: Organize for rapid experimentation
Integral to innovation is the ability to experiment quickly: Rapid feedback shapes new ideas by reinforcing, modifying, or complementing existing knowledge. Indeed, rapid feedback is important to learning, yet far too many developers must wait days, weeks, or months before their ideas can be tested in experiments. Time passes and attention shifts, and when feedback finally arrives, momentum is lost, the link between cause and effect has been severed, and project decisions have been made. To prevent the trail of an idea or inspiration from growing cold, Thomas Alva Edison built and organized his famous West Orange, New Jersey laboratory around the concept of rapid experimentation. Supply rooms and machine shops were close to experimental rooms, libraries and storerooms had diverse supplies, and ample capacity made sure that delays could not slow down people's work and creativity. 6 Chapter 6 shows how the German car company BMW leveraged advances in crash safety modeling and simulation to remove interfaces between functional groups in order to speed up learning from experimentation. The result was faster iterations and more ideas because much of the knowledge required about safety, design, simulation, and testing resided within the small group. In multiple instances, faster experimentation resulted in fundamentally new insights that ultimately made BMW cars much safer.
Principle #5: Fail early and often but avoid "mistakes"
When experiments reveal what does and does not work, the inevitable happens: Novel ideas and concepts fail. Early failures are not only desirable but also needed to eliminate unfavorable options quickly and build on the learning they generate. The faster the experimentation-failure cycle, the more feedback can be gathered and incorporated into new rounds of testing. The problem in many organizations is that knowing what doesn't work can go against a relentless focus on success. Failures expose gaps in knowledge and can lead to embarrassment, and the people "responsible" are unlikely to be promoted within traditional incentive systems. Ask yourself: How often are people rewarded for exposing failure early, thus saving their employer from investing precious resources in opportunities with little promise? This matters a great deal because in innovation processes, experiments that result in failure are not failed experiments. In fact, it was Edison, again, who noted that the "real measure of success is the number of experiments that be crowded into twenty-four hours." 7
Chapter 6 suggests that failures, however, should not be confused with mistakes. Failure can be a desirable outcome of an experiment, whereas mistakes should be avoided as they produce little new or useful information and are therefore without value. Distinguishing between failures and mistakes thus becomes very important, particularly when organizations are integrating new technologies that will increase the number of experiments resulting in failure—in other words, learning what doesn't work. The chapter also presents research on what managers can do to invite experimentation into their companies.
Principle #6: Manage projects as experiments
A final principle for unlocking the potential of new technologies is that projects can be conceived of as experiments themselves. While senior management often considers portfolios of projects in the allocation and management of resources, it rarely applies the same logic to portfolios of experiments. This is somewhat surprising since projects are powerful mechanisms for managing change, knowledge creation, and the introduction of new technologies and processes. After all, how many organizations can point towards a set of fifteen to twenty ongoing and well-designed experiments that are either exploring new markets or are changing the organization itself? Chapter 6 explores how learning from projects can be maximized, using the same factors that drove learning within projects: fidelity, cost, iteration time, capacity, sequential and parallel strategies, signal-to-noise ratio, and type of experiment. Significantly, the chapter also addresses the delicate balance of managing dual objectives: finishing projects on time and budget and using the project as an experiment for learning. Research at Bank of America, BMW, and IDEO shows how this balance is achieved.
[ Order this book ]