Should scientists handle retractions differently?

It is one of the highest-profile cases of scientific fraud in memory: In 2005, South Korean researcher Woo-Suk Hwang and colleagues made international news by claiming that they had produced embryonic stem cells from a cloned human embryo using nuclear transfer. But within a year, the work had been debunked, soon followed by findings of fraud. South Korea put a moratorium on stem-cell research funding. Some scientists abandoned or reduced their work in the field.

But the case is not so simple: By 2007, other stem-cell researchers had found that the debunked research contained a few solid findings amid the false claims. While prior stem-cell findings remained intact, it took time to rebuild support for the field.

Now a study by MIT scholars quantifies the fallout for scientists whose fields suffer high-profile retractions, with a twist: Even valid older research, when intellectually related to a retracted study, loses credibility — especially if the retracted paper involves malfeasance. The fallout from a retraction does not land solely on the scientists who are at fault, but on people in the field more broadly.

As the new paper contends, “scientific misconduct and mistakes, as signaled to the scientific community through retractions, cause a relative decline in the vitality of neighboring intellectual fields.” This spillover effect, which includes a 6 percent decline in citations relative to similar, unaffected papers, suggests that scientists would benefit by trying to describe the nature of each retraction in more detail.

“A well-functioning, transparent retraction process is actually part and parcel of the scientific system,” says Pierre Azoulay, an economist at the MIT Sloan School of Management, and a co-author of the new study. “We need a system where … journals help the readers spell out the reasons for the retractions, and help the scientific community parse the implications for the forward movement of science.”

Identifying the “stigma story”

The paper, “Retractions,” is published in the Review of Economics and Statistics, a peer-reviewed economics journal. The authors are Azoulay, the Sloan Distinguished Associate Professor of Management; Fiona Murray, the Alvin J. Siteman Professor of Entrepreneurship, associate dean for innovation at MIT Sloan, and co-director of MIT’s Innovation Initiative; Joshua Krieger, a doctoral student at MIT Sloan; and Jeffrey Furman PhD ’02, an economist at Boston University.

Murray and Furman were co-authors of an influential 2012 paper that studied the circumstances in which retractions occur. The new paper, Azoulay notes, is different in that it spotlights “the consequences of retractions, not their antecedents.”

The current study focuses on the life sciences. The researchers examined the effects of more than 1,000 retractions of papers published between 1973 and 2007, and retracted by 2009. They also used the PubMed Related Citations Algorithm to help define the research fields relevant to those retracted papers.

Given those definitions of research fields, the researchers then examined the effect of retractions on intellectually related work. One group of 60,000 papers that they examined experienced the 6 percent decline in citation rates after retractions occurred in the same fields. To establish that fact, the researchers compared the citation trajectories of those papers with a control group of 110,000 papers that were published in the same journals. The data and method draw on work Azoulay has compiled and refined while developing numerous other analyses of citation rates in scientific literature.

But within these numbers lie another story: The citation rates for related papers that are still valid drop more precipitously when papers citing them are retracted for reasons of fraud or other misconduct, as opposed to, say, a laboratory mistake.

“Most of the declines in citations and funding we see are driven by fraud cases,” Krieger notes. “When an honest mistake happens, the related field doesn’t experience this big decline.”

Conversely, this means that, given two equally valid sets of past research, legitimate research cited in a paper later retracted for fraud will be more harshly punished.

Further evidence for the “stigma story,” as Krieger calls it, is that academic papers and nonacademic papers from firms tend to shun retracted papers at similar rates — but papers from commercial firms do not avoid citing older, still-valid related papers to the same extent as academic researchers. This could mean there is an unfounded flight away from those related papers in academic research.

“Our evidence pointing to the stigma story implies that funding agencies and investigators should be more cautious when deciding to abandon a field after a case of research misconduct,” Krieger adds. “We need to be careful in separating what we’ve learned about the field’s scientific status from our strong reactions to the disgrace of misconduct and fraud.”

Limits to knowledge

Other scholars in the field find the study valuable. Ben Jones, a professor at Northwestern University’s Kellogg School of Management, says the study “uses rigorous empirical methods to establish an important fact: Retractions matter for the ensuing progress of a field.” He adds: “There are good reasons to think that stigma effects may help explain these results.”

Jones, who was not involved in this research, was co-author of a 2013 paper indicating that authors of retracted papers avoid relatively larger citation losses when they self-report problems.

As Azoulay and his colleagues acknowledge, there are limits to their study. It’s possible that citations in fields affected by fraud logically should decline more than those marred by honest mistakes, because scientists may rightfully conclude that research areas containing outright fraud are further away from yielding productive results.

“Maybe … fraud retractions and error retractions are different in ways that are correlated for the potential for follow-on research,” Azoulay says.

The researchers tried to account for that in their study, by looking at the number of citations garnered by retracted articles prior to the discovery of problems, then correlating that with the impact on related papers. But the numbers do not show an underlying structural reason why retractions for fraud, as opposed to unintentional mistakes, should generate a larger impact on related prior research.

The researchers have a couple of specific policy suggestions to limit unwarranted damage to the accretion of scientific knowledge. For instance, journals that retract articles should offer detailed explanations of why those papers are no longer valid. It would not be hard, they suggest, to develop a basic system of categories of retractions.

“By lumping together honest mistakes and misconduct, we’re undermining the smooth functioning of the retraction system,” Azoulay says.