The Novelty Paradox & Bias for Normal Science: Evidence from Randomized Medical Grant Proposal Evaluations

by Kevin J. Boudreau, Eva C. Guinan, Karim R. Lakhani & Christoph Riedl
 
 

Overview — A key task for executives and managers involved with innovation is to evaluate new ideas and proposals. In the sciences, one longstanding hypothesis contends that research ideas outside the mainstream are susceptible to being discounted, rejected, or ignored. These days, expert peer review in academic science is the approach most relied upon for enabling research agendas and providing research funds. Are novel research projects—those deviating from existing research paradigms—treated with a negative bias in expert evaluations? In this paper, the authors investigate how nascent scientific hypotheses are evaluated, specifically looking at the process by which medical research grant proposals are assessed by "gatekeepers": in this case, elite researchers from a leading medical school. Innovation requires novelty—but novelty, as this paper shows, is not appreciated and is in fact penalized. These findings help explain concerns about incrementalism in science and also point at the challenge that most organizations face when dealing with novel topics Key concepts include:

  • Expert peer review of new research proposals in academic science is now a large organized practice in its own right. There are many concerns about the reliability of the peer review process.
  • Novel research proposals are generally discounted—even when accounting for quality and feasibility of ideas. Evaluators also tend to be more critical of proposals that were closer to their area of expertise-and these two effects of novelty and intellectual distance appear to work largely independently of one another.
  • How novel ideas and research hypotheses are treated may be consequential to the organization of innovation in general and peer evaluation in the sciences in particular.

Author Abstract

Central to any innovation process is the evaluation of proposed projects and allocation of resources. We investigate whether novel research projects, those deviating from existing research paradigms, are treated with a negative bias in expert evaluations. We analyze the results of a peer review process for medical research grant proposals at a leading medical research university, in which we recruited 142 expert university faculty members to evaluate 150 submissions, resulting in 2,130 randomly assigned proposal-evaluator pair observations. Our results confirm a systematic penalty for novel proposals; a standard deviation increase in novelty drops the expected rank of a proposal by 4.5 percentile points. This discounting is robust to various controls for unobserved proposal quality and alternative explanations. Additional tests suggest information effects rather than strategic effects account for the novelty penalty. Only a minority of the novelty penalty could be related to perceptions of lesser feasibility of novel proposals.

Paper Information