Nobel Laureate James Heckman co-authors study examining bias in publishing, impact on tenure
Too often in economics, where you publish can be more important than what you publish.
That’s the theory explored in a new study co-authored by Nobel-winning economist James J. Heckman and Sidharth Moktan, a predoctoral fellow at the Center for the Economics of Human Development. The University of Chicago scholars found that tenure and prize committees often base decisions on how often candidates publish in “top five” journals in the field. That practice not only concentrates career advancement into the hands of a select set of editors—many of whom are long-serving—but does so at the expense of innovative economic research.
“Relying on publication counts in ‘top-ranked’ journals encourages crass careerism among young economists,” Heckman said. “It diverts their attention away from basic research toward blatant strategizing about lines of research and favored topics of journal editors with long tenures.
“Relying on rankings rather than reading to promote and reward young economists subverts the essential process of assessing and rewarding pathbreaking original research.”
The “top five” refers to the leading economic journals most crucial to the academic and professional success of young scholars: The American Economic Review, Econometrica, the Journal of Political Economy, The Quarterly Journal of Economics and The Review of Economic Studies. These journals are chosen by a process that weighs citation counts to all papers in the journal; in other words, it judges a paper by the company it keeps.
The working paper from Heckman and Moktan, released Oct. 1 in the National Bureau of Economic Research, found that scholars who have written one “top five” article are 90 percent more likely to receive tenure in a given year. Those figures balloon to 260 percent for two such articles and 370 percent for three.
“Both junior and senior scholars often bring up the top five when they’re evaluating somebody,” Moktan said. “Even if it’s in a casual conversation, they’ll say, ‘Oh, how many ‘top fives’ do they have?’”
Gauging the role of bias
Writing for a “top five” journal involves more than just producing the best possible piece of research, according to Heckman and Moktan. Their study argues that to optimize chances for placement, scholars are incentivized to tailor their work for individual editors—who, consciously or not, are guided by their own biases.
In addition to tracking tenure rates, Heckman’s and Moktan’s study tracks the author affiliation in “top five” journals from 2000-16. For example, Heckman sits on the editorial board of the Journal of Political Economy, which is published by the University of Chicago Press. Perhaps unsurprisingly, it drew 14.3 percent of its articles in the aforementioned time period from authors connected to the University of Chicago. The Quarterly Journal of Economics, which is edited at Harvard University’s Department of Economics, drew nearly a quarter of its articles (24.7 percent) from its own affiliates, plus another 13.9 percent of its articles from Massachusetts Institute of Technology affiliates.
In contrast, The Review of Economic Studies, which has a higher rate of turnover on its editorial board, has much weaker ties to individual universities. From 2000-16, the publication was most strongly connected to New York University and Northwestern University affiliates, from whom it drew 7.3 percent and 7.0 percent of articles.
The scope of the problem expands when universities use “top five” journals as a proxy for determining tenure. No longer is this simply an issue of who gets published in certain journals, the authors claim; rather, flaws in the editorial process are amplified into career hurdles—ones that can be difficult for outsiders to surmount without connections to “top five” editors and the referees they select.
The “top five” also don’t hold a monopoly on high-quality work. According to Heckman and Moktan, some of the most influential work in economics is published by other journals. Although the “top five” articles produce more citations on average, those numbers are skewed by outliers. Moreover, the senior scholars who rely on the “top five” to judge their colleagues often do not publish in those journals themselves once they are tenured. Relying on the journals, Heckman said, instills caution rather than creativity in young scholars anxious to gain tenure.
The “top five” journals incentivize scholars to focus on follow-up and replication work—research that is easy to assess for immediate publication, but which does not advance the frontiers of economic science. Often, the sorts of innovative projects that would challenge accepted ideas are too long or data-intensive to fit into the format of “top five” journals.
“Research is inherently risky, because you’re trying to find answers to questions that have not been solved,” Moktan said. “Sometimes the answers aren’t exciting. But serious assessments require senior scholars to read papers and understand them, and why that line of research is important.”
Potential solutions
Heckman and Moktan suggest that tenure committees devote more resources to closely reading published and unpublished papers, rather than relying on journal reputation as a substitute for careful reading. That method could prompt each individual institution to pursue more unconventional research instead of leaning on papers funneled through “top five” publications. Expanding the pool of influential publications beyond five journals could ameliorate the problem too.
They also suggest a more radical solution: Shifting away from conventional journals in favor of open-source formats such as arXIV and PLOS ONE, which are used in hard sciences. Such a change would offer scholars an opportunity to share their ideas earlier and get peer review in real time—an approach that might be more welcoming to out-of-the-box ideas.
“The current system of publication and reward does not encourage creativity,” Heckman said. “It delays the publication and dissemination of new ideas. It centralizes power in the hands of a small group of editors, prevents open discussion and stifles dissent and debate. It needs to be changed.”
