27 Feb 2012  Research & Ideas

When Researchers Cheat (Just a Little)

Although cases of clear scientific misconduct have received significant media attention recently, less flagrant transgressions of research norms may be more prevalent and, in the long run, more damaging to the academic enterprise, reports Assistant Professor Leslie K. John. Key concepts include:

  • Many of the surveyed research psychologists admitted to bending scientific norms in their work, but most transgressions were innocuous.
  • Over a third of those surveyed said they doubted the integrity of their own research on at least one occasion.
  • Ten percent of the research psychologists had engaged in the most serious practice of using false data.
  • This methodology could help business practitioners learn about undesirable practices inside their companies.

 

Leslie K. John is keenly aware of the pressure researchers feel to get results. When her graduate studies in behavioral decision research didn't produce significant findings that led to publication in a prestigious journal, John felt disheartened.

"The incentive structure is such that you're strongly rewarded for a positive result," says John, now an assistant professor of marketing at Harvard Business School.

This system can drive researchers to bend the rules to get a desirable outcome. Sometimes researchers commit minor infractions just to simplify the process, but even tweaks that don't affect the results can cast a shadow on the credibility of academic research.

In research to be published in a forthcoming issue of Psychological Science, John and coauthors George Loewenstein (Carnegie Mellon) and Drazen Prelec (MIT) write that although cases of clear scientific misconduct have received significant media attention recently, "less flagrant transgressions of research norms may be more prevalent and, in the long run, more damaging to the academic enterprise."

In an attempt to get researchers to honestly report questionable practices they have engaged in, John and her coauthors surveyed more than 2,000 research psychologists at major US universities and told them that the more truthful they were about their transgressions, the more money would be donated to a charity of their choice.

The 10 questionable practices in the study ranged from misdemeanors, as John calls them, such as selectively reporting studies that achieved positive results, to "academic felonies" such as falsifying data.

Measuring truthfulness

The participants' scores were determined by a truth-telling algorithm developed by coauthor Prelec, known as the Bayesian Truth Serum, that compared their own admission rates to their estimates of how likely they thought other researchers were to engage in and admit to the same practices. Higher scores were given to the admissions that were surprisingly common, because, according to the serum's logic, honest answers have the best chance of producing surprisingly common results. The higher the score, the larger the donation to charity.

John says that ideally, participants' answers would have been tested against evidence showing that they had actually performed, or not performed, the acts they were questioned about. But since that wasn't possible, John and her colleagues used the enhanced charitable-giving incentive to encourage honest responses.

The enhanced charity incentive increased the rates at which participating psychologists admitted to engaging in questionable practices themselves, but it didn't change their estimates of prevalence and admission rates among other psychologists. This led the study authors to conclude that the truth-telling incentive was the most effective for the toughest questions—in this case, those that required people to admit to their own wrongdoing.

In all, the truth-telling incentive ended up generating $4,200 for four charities; the receipts were posted online afterward for the participants to see. Almost every respondent admitted to having engaged in at least one of these practices, but it's important to note that some of the practices, such as failing to report all of a study's dependent measures, can be fairly innocuous.

For instance, because it can be expensive to gather a representative sample from the American population, a researcher might elect to conduct one large-scale survey that includes three different dependent measures, each intended to answer a separate research question. The researcher could defensibly write three different papers based on the three different research questions, and fail to report all the dependent measures in each paper—instead just reporting measures that were relevant to the given research question in the given paper.

On the other end of the spectrum, researchers can obtain false positives by running statistical tests over and over until they find the result they are looking for, or by excluding data after looking at how doing so might affect the results.

The truth-telling algorithm had the biggest impact on practices deemed by the respondents to be less defensible. The study estimated that approximately one in 10 research psychologists had engaged in the most serious of the questionable research practices, using false data. "That's very scary to me," John says.

Falsifying data has been a hot topic in the research community lately. Last year, a Boston University cancer researcher was found to have fabricated data in two published papers that were later retracted. Also in 2011, a well-known Harvard University psychology professor was found guilty of scientific misconduct, most likely including fabricating data.

Respondents who admitted to a questionable research practice tended to have a rationalization for doing so, but 35 percent said they doubted the integrity of their own research on at least one occasion—a statistic John finds "disheartening."

This doubt could be because of gray areas that exist in scientific discoveries, John says. These gray areas are necessary for scientific experimentation because if researchers didn't have the freedom to deviate from the norm and try different ways of doing things, they might not make important discoveries.

One way to help keep research methods in check, John says, would be to create a repository where each study is registered, as in clinical trials, and all the measures recorded in order to be compared against the results.

Problematic practices

"Measuring the Prevalence of Questionable Research Practices with Incentives for Truth-Telling" is thought to be the first study to show that using truth-telling algorithms in combination with truth-telling incentives can lead to higher—and probably more valid—estimates of how often researchers engage in the most problematic practices.

Although used to study researchers, John suggests that this methodology could help business practitioners learn about undesirable practices inside their companies. "If you were trying to find out the prevalence of employee theft, or any type of unsavory behavior, and if you preferred to do this by asking people rather than resorting to something like surveillance, this research suggests that you'll get more valid prevalence estimates if you use this method of incentivizing people to tell the truth."

Combined with computer-assisted self-interviewing, which has been found to increase self-reporting, the method could prove even more effective, she adds.

Even when people are given incentives to tell the truth, though, John is surprised at their willingness to admit to dubious acts. "I am constantly surprised at research participants' willingness to pour their hearts out, even for negative, unflattering information," she says.

John holds a PhD in behavioral decision theory from Carnegie Mellon University, where she also earned an MSc in psychology and behavioral decision research.

About the author

Katie Johnston is a writer based in Medford, Massachusetts.

Post A Comment





By hitting “Submit” you agree that your comment, in whole or in edited form, may be posted online. Comments are selected on the basis of relevancy and variety; not all will be posted.

 

Comments

    • Abdulla Al Babul
    • MBA student, Bangaldesh University of Professionals

    Interesting Article. Thruthfullness shows us the way.

     
     
     
    • CJ Cullinane

    Disheartening but good (interesting) research. Creative methodology.

     
     
     
    • David Agogo

    Deep down, there is that deep urge that people have to come clean with immoral behavior. This research brings up the important issue of incentives for academic research... Publish or Perish... and other pressures that structured academia place on the average researcher. This in addition to plain 'hit and miss' research that is difficult to face up to can make otherwise truthful people falsify data out of desperation. I would not worry too much about these findings. Fortunately, the scientific method is built with ample positive feedback loops - a lie cannot lie unnoticed for too long.

     
     
     
    • Anonymous

    This article is totally based on the method for estimating whether or not people are telling the truth. I find that method questionable and see no way of testing its validity. I say this as a professional research psychologist.

     
     
     
    • David Lindsay
    • Tutor/Lecturer, Napier Business School

    I understand only too well the pressure to "produce" whether by an academic authority or a commercial sponsor particularly in field of statistics where inference on certain data can be turned into a political football -but the integrity of not only the researchers but the intellectual chain is put at risk by publishing elequent but not inscrutable data which amy lead to sponsors moving their subsidies and support to somewhere else. Deadlines should be kept and time management one of the governing criteria for a reliable researcher !

     
     
     
    • Paul Nicholas
    • Director, Soul-Chaplain Consultancy

    This is really interesting and thought provoking - thank you.

    All research depends on 'just a little' bit of cheating - and that's why we make progress. Apart from the unconscious biases we are all subject to (including when we read someone else's work), the inherent ambiguity in all communications and the fact that all research is motivated by some kind of ideology and morality - Enrico Fermi liked to point out that all facts are based upon opinions - we have to accept that ultimately all judgements on the honesty and truthfulness of 'objective realities' are themselves subjective experiences.

    Cheating of some kind is inevitable - and it's about time we were honest about it!

     
     
     
    • amitha rani
    • Principal scientist, National Aerospace Laboratories, Bengaluru

    I fully endorse what Leslie having to say about not being afraid to prsent what results we obtain, the pathway to inventions. During my doctoral work, I got results which were totally out of the way.thanks to my supervisor who had a great knack of picking up cooking up data by interpreting /analysing our data in different ways. Finally we used a more sophisticated method to confirm my results, my lab mated blind coded my samples and my supervisor sent me to do the analysis with a message " Don't come back if you don't get the same results, because you won't get your degree" beleive me I got the same results, presented it and won a gold medal for the work. It was just that researchers had not looked into the type of samples I was looking at and hence no reports on that. This experience and guidance from a great supervusor has boosted our confidence to such levels that I am not afraid to tread unwalked paths in research an d truly enjoy my research. Yes, the journey is tough, but at the end of the road you are flying as you have got what you were looking for.Thanks once again to the great master who guided us. So don't be afraid to publish what you get!Leave it open for discussion if you don't understand whats happening.

     
     
     
    • Syanne

    I am wondering whether organisational culture can be the moderating variable in testing the relationship between incentive and honesty. For instance, an organisational culture which is dominated by corruption will undermine the power of incentive in stimulating someone's honesty.

     
     
     
    • Anonymous

    It seems the high-pressure competitive landscape of academic research has just as many incentives to lie and get ahead as any other profession where staying at the top may benefit from extreme, if not ethically perverse measures.

    It sounds like unsound data manipulation or fabrication is just the academic equivalent to steroids for athletes.

     
     
     
    • Michael Biggs
    • Professor of Aesthetics, University of Hertfordshire

    This is an interesting piece, and I agree with the ethical point that honesty is an important part of reporting. However, it also raises the concept of objectivity because it assumes that reporting "failures" as well as "successes" will ensure that the underlying method will deliver what it promises: a reliable [true?] account of external reality. But we could just as reasonably adopt a different method in which the researcher's judgment plays a more active role in achieving "best fit solutions" rather than "objective ones. The ethical point would remain the same, i.e. that one must operate the method in good faith, but what constitutes good faith, and honest dealing, would be different.

     
     
     
    • Kapil Kumar Sopory
    • Company Secretary, SMEC(India) Private Limited

    This research looks at the freshness of research as being tainted by some. Yes there could be instances where total originality may not be there but we must not forget that it is re-search and some element of what has already been looked into may be there. We have, however, to be wary of blatant copying word for word without contributing from own findings, etc.

     
     
     
    • Mrs Dorothy C. Okere
    • Director Director,R?s.&Dev., Tertiary Education Trust Fund, TETFund, Abuja Nigeria

    An interesting article and an eye-opener.Dishonesty and untruthfulness in any syst?m, be it research and/or other Life endeavours have destructive influences,retrogressive growth and hinder further discoveries. Researchers should stick to ethical virtues of honesty,integrity consciousness and truthfulness in everything aspects of their research works to enable new inventions and innovations that really address life challenges,profer appropriate solutions/strat?gies,improve livelihood humanity as a whole.

     
     
     
    • Christine S. Teopiz
    • Student, University of San Carlos-Cebu Philippines

    The article reveals the reality that all of us face. Some people tend to cheat because of the pressure that they experienced. It should be up to us to judge the credibility of the facts presented to us by the researcher. The best method of doing triangulation.