Skip to Main Content
HBS Home
  • About
  • Academic Programs
  • Alumni
  • Faculty & Research
  • Baker Library
  • Giving
  • Harvard Business Review
  • Initiatives
  • News
  • Recruit
  • Map / Directions
Working Knowledge
Business Research for Business Leaders
  • Browse All Articles
  • Popular Articles
  • Cold Call Podcast
  • Managing the Future of Work Podcast
  • About Us
  • Book
  • Leadership
  • Marketing
  • Finance
  • Management
  • Entrepreneurship
  • All Topics...
  • Topics
    • COVID-19
    • Entrepreneurship
    • Finance
    • Gender
    • Globalization
    • Leadership
    • Management
    • Negotiation
    • Social Enterprise
    • Strategy
  • Sections
    • Book
    • Podcasts
    • HBS Case
    • In Practice
    • Lessons from the Classroom
    • Op-Ed
    • Research & Ideas
    • Research Event
    • Sharpening Your Skills
    • What Do You Think?
    • Working Paper Summaries
  • Browse All
    Is A/B Testing Effective? Evidence from 35,000 Startups
    Research & Ideas
    Is A/B Testing Effective? Evidence from 35,000 Startups
    11 Jan 2021Research & Ideas

    Is A/B Testing Effective? Evidence from 35,000 Startups

    by Kristen Senz
    11 Jan 2021| by Kristen Senz
    A/B tests help startup founders capitalize on good ideas—and move on from duds—faster, says research from Rembrand Koning. How can established companies benefit?
    LinkedIn
    Email

    Startups that use A/B testing scale more quickly, launch more products, and attract more venture capital, according to recent research.

    But not all startups enjoy these benefits. Early-stage startups using experimentation strategies usually flame out faster, perhaps reflecting management inexperience in how to apply the methodologies. Another possibility? Their ideas just weren’t that good.

    “I think one of the coolest findings in our paper is that experimentation doesn’t necessarily drive startup growth,” says Rembrand Koning, an assistant professor in the Strategy Unit at Harvard Business School. “Instead, we find that firms that adopt A/B testing both scale and fail faster. Sometimes founders just have bad ideas and, in these cases, experimentation helps them move onto their next opportunity faster.”

    Indeed, companies of every stripe have embraced popular experimentation methods, including design thinking and lean startup approaches, that have given rise to some of the world’s biggest technology companies. But does the Silicon Valley mantra “fail fast, fail often,” which encourages early testing of new ideas and products, really help companies succeed?

    To find out, Koning, along with Sharique Hasan and Aaron Chatterji of Duke University’s Fuqua School of Business, examined 35,262 high-tech startups founded globally between 2008 and 2013. They detailed their results in the working paper, Digital Experimentation and Startup Performance: Evidence from A/B Testing.

    The researchers analyzed the characteristics, web metrics, and technology adoption trajectories of the startups over four years starting in April 2015 to determine not only whether A/B testing affected firm performance, but how. What they found was that the ease and low cost of using A/B testing for making incremental changes to their websites, such as improving elements of design and presentation, also facilitated an increased focus on big-picture strategy and innovation, says Koning.

    "A/B testing also allows the firm to free up the time it would have spent tinkering with the small stuff."

    “By making it so much easier to make all these slight improvements, A/B testing also allows the firm to free up the time it would have spent tinkering with the small stuff, which it now delegates to the A/B testing tool,” he says. “The founders can now think about more radical changes, like innovation and new products... You make more changes overall, so [adopting A/B testing] doesn’t come at the cost of big ideas.”

    On average, firms that used A/B testing to select the best features for their websites had higher page views, along with visitors who browsed more pages and stayed longer. But these companies also reached their natural endpoints—either scaling or failing—more quickly.

    “Firms that adopt A/B testing are more likely to hit zero page views, which we take as evidence of shutting down, but they’re also more likely to have extreme page views, ending up with well over 50,000 views a week,” says Koning.

    Isolating the effects of A/B testing

    Using data available publicly through the website BuiltWith, the researchers identified companies that were up to eight years old that had installed A/B testing software. These tools enable firms to randomly assign different versions of their websites to different visitors in order to compare their performance.

    Because BuiltWith data shows when each company installed the software, the researchers could study firm performance before and after testing began using data on weekly website page views from the company SimilarWeb.

    By comparing each company to itself before and after the testing software was implemented, the researchers mitigated the potential for selection bias in their sample—the possibility that companies opting to A/B test are inherently better, for example. To determine whether A/B testing led to more product launches or better VC funding, the researchers examined news articles and CrunchBase data.

    From 2015 to 2018, 4,645 startups adopted A/B testing technology on their websites. On average, those firms had roughly 10 percent more weekly page views, a 5 percent greater likelihood of raising VC funding, and launched 9 to 18 percent more products.

    "If you’re generating more ideas, you’re more likely to generate new products."

    “We think there are more product introductions because when you lower the cost of testing, that makes it more valuable to generate more potential ideas,” Koning says. “If you’re generating more ideas, you’re more likely to generate new products.”

    Further research is needed to determine whether and how established firms might benefit from A/B testing. Koning thinks the beneficial effects might be even greater for such firms because the cost of testing new ideas is historically much higher.

    “These firms often have a ton of employees with great ideas, and if they can get their management strategies right, I think they could benefit the most from this sort of A/B testing technology,” says Koning, who is currently studying Fortune 1000 companies working with a large A/B testing firm.

    Managing experimentation

    For managers, implementing an experimentation strategy is about more than installing software and conducting a single experiment. It requires a long-term commitment. The effects Koning and his colleagues found often did not appear until after a startup had been using A/B testing for six months.

    Embracing experimentation across an organization also requires managers to be strong delegators who are willing to empower employees to select the best ideas.

    “A lot of experimentation is about using the results of the A/B test to make decisions, not about a manager making that decision,” Koning says. “Empowering other people to come up with ideas and test them without interfering in how they’re selected is huge, and it’s incredibly difficult to do, because it changes a lot of the paradigm around management.”

    About the Author

    Kristen Senz is the growth editor of Harvard Business School Working Knowledge.

    [Image: themacx]

    Related Reading

    • How Hackathons Help Decide Platform Winners and Losers
    • Recessions Push Some Entrepreneurs to Launch Too Soon
    • Rethinking Business Strategy in the Age of AI

    How do you approach experimentation and testing?

    Share your insights below.

    Post A Comment
    In order to be published, comments must be on-topic and civil in tone, with no name calling or personal attacks. Your comment may be edited for clarity and length.
      Trending
        • 25 Jan 2022
        • Research & Ideas

        More Proof That Money Can Buy Happiness (or a Life with Less Stress)

        • 16 Mar 2023
        • Research & Ideas

        Why Business Travel Still Matters in a Zoom World

        • 14 Mar 2023
        • In Practice

        What Does the Failure of Silicon Valley Bank Say About the State of Finance?

        • 25 Feb 2019
        • Research & Ideas

        How Gender Stereotypes Kill a Woman’s Self-Confidence

        • 17 May 2017
        • Research & Ideas

        Minorities Who 'Whiten' Job Resumes Get More Interviews

    Rembrand M. Koning
    Rembrand M. Koning
    Assistant Professor of Business Administration
    Contact
    Send an email
    → More Articles
    Find Related Articles
    • Innovation and Invention
    • Technology

    Sign up for our weekly newsletter

    Interested in improving your business? Learn about fresh research and ideas from Harvard Business School faculty.
    This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.
    ǁ
    Campus Map
    Harvard Business School Working Knowledge
    Baker Library | Bloomberg Center
    Soldiers Field
    Boston, MA 02163
    Email: Editor-in-Chief
    →Map & Directions
    →More Contact Information
    • Make a Gift
    • Site Map
    • Jobs
    • Harvard University
    • Trademarks
    • Policies
    • Accessibility
    • Digital Accessibility
    Copyright © President & Fellows of Harvard College