In 2008, entrepreneur Brian Chesky and his two San Francisco roommates made the rounds of Silicon Valley VC firms with what they thought was a great idea: a website and mobile app that would allow homeowners to open their homes to strangers to sleep on their floor while traveling, in exchange for a nightly fee.
Of course, now we know the idea as Airbnb, a $10 billion business with 1.5 million listings around the world. But back then it must have seemed crazy. The liability issues alone seemed insurmountable—to say nothing of the likelihood that people would be willing to give the keys of their houses to total strangers who may or may not be serial killers.
Five VC firms rejected the nascent company’s pitch outright, and another two didn’t even bother to reply. “Investors must have thought, who would ever do this?” says Assistant Professor Pian Shu, a member of the Technology and Operations Management unit at Harvard Business School. “They didn’t know it would turn out to be a multibillion dollar industry.”
“By definition, when an investor makes an investment, it changes the probability of success”
In a new working paper, Shu asks the fundamental question that cases like Airbnb and other once unlikely, now successful startups (LinkedIn similarly got more than 20 rejections back in 2003) seem to beg: How do you tell a good idea from a bad one?
“With startups, especially high-growth startups, it’s extremely hard to predict the probability of success,” says Shu, who studies innovation and entrepreneurship. When dealing with something truly innovative, it’s difficult to compare it to anything that came before. That uncertainty makes the line between a tremendous success and a phenomenal flop a thin one.
Predicting startup success or failure also turns out to be incredibly difficult to study. When VC companies invest in an idea that later becomes successful, it’s hard to know whether that is because the idea was inherently a good one, or because the investment and mentorship made it good, a self-fulfilling prophecy.
Changing the likelihood of success
In order to come to grips with the question, Shu and co-author Erin Scott, of the National University of Singapore, needed to find a setting in which they could pinpoint the relationship between initial evaluation and future outcomes—that is, one in which experts were evaluating ideas but not funding them or advising them in a way that determined their success.
“By definition, when an investor makes an investment, it changes the probability of success,” says Shu. “You need to find a setting in which you have an evaluation of a startup at an early stage, where the evaluation is not known to the entrepreneurs and doesn’t influence the idea.”
They found that setting in a place close to home for Shu. As a doctoral student at the Massachusetts Institute of Technology, Shu flirted with entrepreneurship herself, even applying to the MIT Venture Mentoring Service (MIT VMS)—a program that connects budding entrepreneurs to successful businesspeople to develop their ideas. Whenever an entrepreneur applies to the program, a staff member writes up a one-paragraph description of the idea in a uniform format, and then circulates it among a pool of more than 100 possible mentors, who may express interest in the idea.
Shu and Scott realized that they had the perfect laboratory for judging the success of ideas. By comparing the number of mentors expressing interest in an idea to the eventual success of that idea, they could see how well the amount of interest predicted that success. At the same time, since the entrepreneurs—who determine how much mentoring they’d like to get—had no idea how many mentors expressed interest, and since MIT offered access to the same amount of resources to each startup, that success wouldn’t become a self-fulfilling prophecy.
Uncanny ability to pick winners
Analyzing the data along with MIT VMS’s Roman Lubynsky, the researchers found that overall, in fact, the mentors had an uncanny ability to predict the success of ideas. Compared to an average venture, which attracted interest from six mentors, a venture that attracted twice as much interest was 27 percent more likely to commercialize (which Shu and her colleagues defined as having multiple repeated sales, an Amazon storefront, or a technology licensing, among other measures of success).
When the researchers drilled down into the data, however, they found some marked differences in the ability to forecast success based on the industry of the proposed idea. Expert interest was highly predictive of success in sectors that were R&D intensive, such as energy, hardware, medical devices, and pharmaceuticals. However, in non-R&D-intensive sectors, such as mobile apps and software, the ability to predict success was no better than random.
Perhaps that is because it is easier to evaluate technology that has a well-defined set of potential market needs, Shu hypothesizes. “With an industry like drug development, it’s not like you can just switch from one disease to another disease,” she says.
While R&D-intensive ventures can shift the application of their underlying technology, Shu continues, the fact they are formed around specific “intellectual assets” makes it unlikely that such shifts will be drastic. On the other hand, “mobile apps don’t require much fixed costs, so you can change the business objective more easily.” (The way Airbnb did from focusing on air mattresses in someone’s living room to renting out furnished rooms and apartments.)
Some investors believe that with early-stage companies, the idea doesn’t matter so much as the quality and passion of the entrepreneur who is pushing it; while others invest in the idea and replace the founder with a professional management team when needed. Shu speculates that with R&D-intensive companies, the initial idea plays a larger role in determining entrepreneurial success than with less R&D-intensive companies. “There’s a long-standing debate in venture financing on whether one should bet on the ‘horse’—the idea—or the ‘jockey’—the team. Our results suggest that the answer varies across industry sectors.”
The other interesting finding from the data had to do not with the industry of those being evaluated, but the industry of those doing the evaluating. The researchers found surprisingly that those within a certain industry were no better at predicting success of an idea in that industry than those outside the industry.
“We’re not saying expertise is bad,” Shu hastens to add. “We just aren’t finding any evidence that industry expertise is required for the collective group to effectively assess the commercial potential of an idea.”
Of course, the mentors assessing ideas in the MIT VMS program weren’t random—they did have entrepreneurial acumen. However, Shu contends that having a large group of people with general business experience evaluate an idea could be more effective than having a small group of people who all come from a particular industry. “That’s counter to the way VC works, where there are usually a small number of partners in a group that all specialize in one industry.”
It may be more important for entrepreneurs to quickly develop a prototype for a minimum viable product and get early feedback from a much wider group of people. That goes both for entrepreneurs honing their idea into a viable company, and for investors eager to spot the next living room air mattress company that turns into a $10 billion empire.