Skip to Main Content
HBS Home
  • About
  • Academic Programs
  • Alumni
  • Faculty & Research
  • Baker Library
  • Giving
  • Harvard Business Review
  • Initiatives
  • News
  • Recruit
  • Map / Directions
Working Knowledge
Business Research for Business Leaders
  • Browse All Articles
  • Popular Articles
  • Cold Call Podcast
  • Managing the Future of Work Podcast
  • About Us
  • Book
  • Leadership
  • Marketing
  • Finance
  • Management
  • Entrepreneurship
  • All Topics...
  • Topics
    • COVID-19
    • Entrepreneurship
    • Finance
    • Gender
    • Globalization
    • Leadership
    • Management
    • Negotiation
    • Social Enterprise
    • Strategy
  • Sections
    • Book
    • Podcasts
    • HBS Case
    • In Practice
    • Lessons from the Classroom
    • Op-Ed
    • Research & Ideas
    • Research Event
    • Sharpening Your Skills
    • What Do You Think?
    • Working Paper Summaries
  • Browse All
    Is the 'Experimentation Organization' Becoming the Competitive Gold Standard?
    02 Aug 2020What Do You Think?

    Is the 'Experimentation Organization' Becoming the Competitive Gold Standard?

    by James Heskett
    SUMMING UP: Digital experimentation is gaining momentum as an everyday habit in many organizations, especially those in high tech, say James Heskett's readers.
    LinkedIn
    Email
    Photo of a swan with black feathersKorrawin

    SUMMING UP

    Will Digital Experimentation Take Hold in Industies Outside High Tech?

    The several responses to this month’s column generally were favorable to the notion that experimentation is gaining momentum as an everyday habit in many organizations, especially in organizations heavily reliant on the internet and cloud for their existence.

    “There is a softening of attitudes towards quants in today’s organization,” Tahir Qazi commented. “ … As more people with an experimentation mindset and grooming move across organizations, it is natural that they will tout their past successes and seed the concept more broadly.” But Ana Frony reminded us to “never forget that even trainees must have some intuition and poetry to develop a hypotheses and design an experiment.”

    Nick C raised the question of just how far the practice of routine experimentation might extend. As he put it, “Whilst the relentless quest for adaptation and improvement in the so-called technology sectors seems to fuel the need for experimentation, the nature of experimentation in other sectors, including the high hazard/high reliability industries, may reflect a bias for knowing … the consequences of ‘failure. ’” It would be interesting , he continued, to examine sectors other than technology “for their natural tendencies and what drives such culture and procedural traits.”

    Joe noted that some of the principles of experiment design provide “a piece of business wisdom that’s lost on firms who have a rigidly hierarchical bureaucratic structure ill-suited to innovation.”

    What are other “natural tendencies” alongside the kind that Nick C mentions? His and Joe’s comments suggest that such natural tendencies may not be enough. Effective adoption of experimentation will require changes in organization “traits” as well.

    What kind of “life” will experimentation have outside high tech? What do you think?


    Original Post

    “Test, then invest” is an increasingly common saying among entrepreneurs and those who finance their startups. The idea is simple: Fast, inexpensive tests in the marketplace can often increase the probability of success for ideas central to a strategy.

    Frequent testing has been especially relevant for retailers, whose merchandise displays offer endless opportunities to test product placement ideas. Now organizations whose businesses are primarily online have taken the strategy to a new level. Their work has been both influenced by and reported recently by several academics.

    Harvard Business School professors Michael Luca and Max Bazerman track experimentation in decision-making back to the Book of Daniel, BC 167-164. They put forward a context leading up to what they call “the beginning of an experimental revolution,” fueled by data, that has become more available using online platforms that make randomization cheaper and easier. These have fueled a greater appreciation for how experiments can “help to complement intuition with evidence.”

    “Learning organizations try a lot of things and keep what works.”

    It is not by chance that nearly all of their private sector examples come from high tech, where employees are being trained in large numbers to routinely conduct experiments that appear to provide enterprise agility and competitive advantage. Experimentation is endemic to the so-called learning organization.

    Learning organizations try many things and keep what works. This process requires intensive testing of ideas as well as reliance on the data produced by the tests, rather than depending on untested opinions (too often generated by leaders) that dominate decision-making today.

    Experimentation in action

    Firms competing in today’s high tech environment routinely test such variables as offerings, page design, and service. For example, Booking.com, a Dutch-based travel website, has developed what another HBS Professor, Stefan Thomke, calls an “experimentation organization.” In his book Experimentation Works, Thomke writes that at Booking.com, “experimentation is an integral part of everyday life (where) … anyone can conduct or commission a test (without approval from above).”

    This requires that employees receive training in how to design and carry out experiments. Booking.com conducts thousands of tests every month. Evidence from a test always trumps executive opinion. Even the failures that tests often produce are regarded as opportunities for learning rather than costly mistakes. The only failure is a poorly-designed test, one with weak hypotheses, poor data, the lack of a control group, and careless analysis of the results.

    Experimentation on a large scale has worked for Booking.com, apparently. The Company has employed this strategy to gain the largest share of the traffic in its industry.

    High tech organizations are producing an army of people who will be hired into other organizations based on their skills, successes, and advocacy of experimentation. How will their attitudes play in organizations where intuition is still honored and the poets can still speak louder than the quants? Or are those days in our past? Will the experimentation organization become the competitive gold standard? What do you think?

    References:

    Michael Luca and Max H. Bazerman, The Power of Experiments: Decision Making In a Data-Driven World (Cambridge, MA: The MIT Press, 2020)

    Stefan H. Thomke, Experimentation Works: The Surprising Power of Business Experiments (Boston: Harvard Business Review Press, 2020)

    Post A Comment
    In order to be published, comments must be on-topic and civil in tone, with no name calling or personal attacks. Your comment may be edited for clarity and length.
      Trending
        • 31 Jan 2023
        • Op-Ed

        Can Insurance Technology Solve the Uninsured Driver Problem?

        • 17 Jan 2023
        • In Practice

        8 Trends to Watch in 2023

        • 27 Jan 2023
        • Op-Ed

        Have We Lost Sight of Integrity?

        • 31 Jan 2023
        • Research & Ideas

        It’s Not All About Pay: College Grads Want Jobs That ‘Change the World’

        • 20 May 2019
        • Research & Ideas

        Activist CEOs Are Rising Up—and Their Customers Are Listening

    Max H. Bazerman
    Max H. Bazerman
    Jesse Isidor Straus Professor of Business Administration
    Contact
    Send an email
    → More Articles
    James L. Heskett
    James L. Heskett
    UPS Foundation Professor of Business Logistics, Emeritus
    Contact
    Send an email
    → More Articles
    Michael Luca
    Michael Luca
    Lee J. Styslinger III Associate Professor of Business Administration
    Contact
    Send an email
    → More Articles
    Stefan H. Thomke
    Stefan H. Thomke
    William Barclay Harding Professor of Business Administration
    Chair, General Management Program
    Contact
    Send an email
    → More Articles
    Find Related Articles
    • Product Design
    • Web Sites
    • Market Platforms
    • Marketing Strategy
    • Research and Development
    • Forecasting and Prediction
    • Technology

    Sign up for our weekly newsletter

    Interested in improving your business? Learn about fresh research and ideas from Harvard Business School faculty.
    This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.
    ǁ
    Campus Map
    Harvard Business School Working Knowledge
    Baker Library | Bloomberg Center
    Soldiers Field
    Boston, MA 02163
    Email: Editor-in-Chief
    →Map & Directions
    →More Contact Information
    • Make a Gift
    • Site Map
    • Jobs
    • Harvard University
    • Trademarks
    • Policies
    • Accessibility
    • Digital Accessibility
    Copyright © President & Fellows of Harvard College