The Chrysalis Effect: Publication Bias in Management Research

14523043285_2235b0dbb4_zHow well do published management articles represent the broader management research? To say that questionable research practices impact only a few articles ignores the broader, systemic issue effecting management research. According to authors Ernest Hugh O’Boyle Jr., George Christopher Banks, and Erik Gonzalez-Mulé, the high pressure for academics to publish leads many to engage in questionable research, thereby leading the resulting published articles to be biased and unrepresentative. In their article, “The Chrysalis Effect: How Ugly Initial Results Metamorphosize Into Beautiful Articles,” published in Journal of Management, O’Boyle, Banks, and Gonzalez-Mulé delve into the issue of questionable research practices. The abstract for the paper:

The issue of a published literature not representative of the population of research is Current Issue Covermost often discussed in terms of entire studies being suppressed. However, alternative sources of publication bias are questionable research practices (QRPs) that entail post hoc alterations of hypotheses to support data or post hoc alterations of data to support hypotheses. Using general strain theory as an explanatory framework, we outline the means, motives, and opportunities for researchers to better their chances of publication independent of rigor and relevance. We then assess the frequency of QRPs in management research by tracking differences between dissertations and their resulting journal publications. Our primary finding is that from dissertation to journal article, the ratio of supported to unsupported hypotheses more than doubled (0.82 to 1.00 versus 1.94 to 1.00). The rise in predictive accuracy resulted from the dropping of statistically nonsignificant hypotheses, the addition of statistically significant hypotheses, the reversing of predicted direction of hypotheses, and alterations to data. We conclude with recommendations to help mitigate the problem of an unrepresentative literature that we label the “Chrysalis Effect.”

You can read “The Chrysalis Effect: How Ugly Initial Results Metamorphosize Into Beautiful Articles” from Journal of Management free for the next two weeks by clicking here.

Want to stay current on all of the latest research published by Journal of ManagementClick here to sign up for e-alerts! You can also follow the journal on Twitter–read through the latest tweets from Journal of Management by clicking here!

*Library image attributed to Apple Vershoor (CC)

 

Listen to the Latest Podcast from Journal of Management on “The Chrysalis Effect”

jom coverIn the latest podcast from Journal of Management, Ernest Hugh O’Boyle Jr, lead author of the article “The Chrysalis Effect: How Ugly Initial Results Metamorphosize Into Beautiful Articles” speaks with Journal of Management Associate Editor Fred Oswald about the article’s findings concerning questionable research practices.

The podcast can be downloaded by clicking here and the article can be read for free by clicking here. Follow this link to subscribe on iTunes.

o'boyleeErnest Hugh O’Boyle Jr is an assistant professor of Management and Organizations at the University of Iowa. His research interests include questionable research practices, outcome reporting bias, publication bias, structural equation modeling, meta-analysis, “dark” personality traits, and superstar effects. He has been published in such journals as Journal of Management, Organizational Psychology Review, Family Business Review and International Business Review.

FredOswaldFred Oswald currently serves the Rice University Department of Psychology as Chair, and he is a Professor in the Industrial/Organizational Psychology program. His published research addresses the reliability and validity of tests administered to applicants in organizational, education and military settings. Substantively, his work deals with defining, modeling and predicting societally relevant outcomes (e.g., job performance, academic performance, satisfaction, turnover) from psychological measures that are based on cognitive and motivational constructs (e.g., cognitive abilities, personality traits, situational judgment tests, job knowledge and skill, and biographical data). His statistical work in meta-analysis, structural equation modeling, and adverse impact also informs personnel selection issues and psychological testing in the research, practice and legal arenas.

Want to know about all the latest news and research like this from Journal of Management? Click here to sign up for e-alerts!

Are There Questionable Research Practices in Management Research?

erasure-1123441-mThe publishing industry can be competitive. But how far will a potential researcher go to achieve success? Farther than you would think, according to a forthcoming Journal of Management article entitled “The Chrysalis Effect: How Ugly Initial Results Metamorphosize into Beautiful Articles.”

The abstract:

The issue of a published literature not representative of the population of research is most often discussed in terms of entire studies being suppressed. However, alternative sources of publication bias are questionable research practices (QRPs) that entail post hoc alterations of hypotheses to support data or post hoc alterations of data to support hypotheses. Using general strain theory as an explanatory framework, we outline the means, motives, and opportunities for researchers to better their chances of publication independent of rigor and relevance. We then assess the frequency of QRPs in management research by tracking differences between dissertations and their resultingjom cover journal publications. Our primary finding is that from dissertation to journal article, the ratio of supported to unsupported hypotheses more than doubled (0.82 to 1.00 versus 1.94 to 1.00). The rise in predictive accuracy resulted from the dropping of statistically nonsignificant hypotheses, the addition of statistically significant hypotheses, the reversing of predicted direction of hypotheses, and alterations to data. We conclude with recommendations to help mitigate the problem of an unrepresentative literature that we label the “Chrysalis Effect.”

Leading author Ernest Hugh O’Boyle, Jr. and Journal of Management Associate Editor Fred Oswald discuss the article’s findings and the effects these findings have on management literature in a new podcast. You can listen to the interview for free by clicking here.

Read “The Chrysalis Effect: How Ugly Initial Results Metamorphosize into Beautiful Articles” for free from Journal of Management by clicking here. Want to know about all the latest from Journal of Management? Click here to sign up for e-alerts!