Startups that use A/B testing scale more quickly, launch more products, and attract more venture capital, according to recent research.
But not all startups enjoy these benefits. Early-stage startups using experimentation strategies usually flame out faster, perhaps reflecting management inexperience in how to apply the methodologies. Another possibility? Their ideas just weren’t that good.
“I think one of the coolest findings in our paper is that experimentation doesn’t necessarily drive startup growth,” says Rembrand Koning, an assistant professor in the Strategy Unit at Harvard Business School. “Instead, we find that firms that adopt A/B testing both scale and fail faster. Sometimes founders just have bad ideas and, in these cases, experimentation helps them move onto their next opportunity faster.”
Indeed, companies of every stripe have embraced popular experimentation methods, including design thinking and lean startup approaches, that have given rise to some of the world’s biggest technology companies. But does the Silicon Valley mantra “fail fast, fail often,” which encourages early testing of new ideas and products, really help companies succeed?
To find out, Koning, along with Sharique Hasan and Aaron Chatterji of Duke University’s Fuqua School of Business, examined 35,262 high-tech startups founded globally between 2008 and 2013. They detailed their results in the working paper, Digital Experimentation and Startup Performance: Evidence from A/B Testing.
The researchers analyzed the characteristics, web metrics, and technology adoption trajectories of the startups over four years starting in April 2015 to determine not only whether A/B testing affected firm performance, but how. What they found was that the ease and low cost of using A/B testing for making incremental changes to their websites, such as improving elements of design and presentation, also facilitated an increased focus on big-picture strategy and innovation, says Koning.
"A/B testing also allows the firm to free up the time it would have spent tinkering with the small stuff."
“By making it so much easier to make all these slight improvements, A/B testing also allows the firm to free up the time it would have spent tinkering with the small stuff, which it now delegates to the A/B testing tool,” he says. “The founders can now think about more radical changes, like innovation and new products... You make more changes overall, so [adopting A/B testing] doesn’t come at the cost of big ideas.”
On average, firms that used A/B testing to select the best features for their websites had higher page views, along with visitors who browsed more pages and stayed longer. But these companies also reached their natural endpoints—either scaling or failing—more quickly.
“Firms that adopt A/B testing are more likely to hit zero page views, which we take as evidence of shutting down, but they’re also more likely to have extreme page views, ending up with well over 50,000 views a week,” says Koning.
Isolating the effects of A/B testing
Using data available publicly through the website BuiltWith, the researchers identified companies that were up to eight years old that had installed A/B testing software. These tools enable firms to randomly assign different versions of their websites to different visitors in order to compare their performance.
Because BuiltWith data shows when each company installed the software, the researchers could study firm performance before and after testing began using data on weekly website page views from the company SimilarWeb.
By comparing each company to itself before and after the testing software was implemented, the researchers mitigated the potential for selection bias in their sample—the possibility that companies opting to A/B test are inherently better, for example. To determine whether A/B testing led to more product launches or better VC funding, the researchers examined news articles and CrunchBase data.
From 2015 to 2018, 4,645 startups adopted A/B testing technology on their websites. On average, those firms had roughly 10 percent more weekly page views, a 5 percent greater likelihood of raising VC funding, and launched 9 to 18 percent more products.
"If you’re generating more ideas, you’re more likely to generate new products."
“We think there are more product introductions because when you lower the cost of testing, that makes it more valuable to generate more potential ideas,” Koning says. “If you’re generating more ideas, you’re more likely to generate new products.”
Further research is needed to determine whether and how established firms might benefit from A/B testing. Koning thinks the beneficial effects might be even greater for such firms because the cost of testing new ideas is historically much higher.
“These firms often have a ton of employees with great ideas, and if they can get their management strategies right, I think they could benefit the most from this sort of A/B testing technology,” says Koning, who is currently studying Fortune 1000 companies working with a large A/B testing firm.
Managing experimentation
For managers, implementing an experimentation strategy is about more than installing software and conducting a single experiment. It requires a long-term commitment. The effects Koning and his colleagues found often did not appear until after a startup had been using A/B testing for six months.
Embracing experimentation across an organization also requires managers to be strong delegators who are willing to empower employees to select the best ideas.
“A lot of experimentation is about using the results of the A/B test to make decisions, not about a manager making that decision,” Koning says. “Empowering other people to come up with ideas and test them without interfering in how they’re selected is huge, and it’s incredibly difficult to do, because it changes a lot of the paradigm around management.”
About the Author
Kristen Senz is the growth editor of Harvard Business School Working Knowledge.
[Image: themacx]
Related Reading
- How Hackathons Help Decide Platform Winners and Losers
- Recessions Push Some Entrepreneurs to Launch Too Soon
- Rethinking Business Strategy in the Age of AI
How do you approach experimentation and testing?
Share your insights below.