Restricted Variance Interaction Effects

[We’re pleased to welcome authors Jose Cortina of Virginia Commonwealth University,  Tine Koehler of the University of Melbourne, Kathleen R. Keeler of Virginia Commonwealth University, and Bo Bernhard Nielsen of the University of Sydney and Copenhagen Business School. They recently published an article in Journal of Management entitled “Restricted Variance Interaction Effects: What they are and why they are your friends,” which is currently free to read for a limited time. Below, Dr. Cortina reveals the inspiration for conducting this research :]

JOM_44.1_72ppiRGB_powerpointI had read about Mischel’s situation strength notion when I was an undergrad. The idea was that, in strong situations, everyone behaves the same way regardless of individual differences like conscientiousness or extraversion. In weak situations where there aren’t clear norms for behavior, individual differences rule. This phenomenon results in Mischel’s personality by situation interaction such that personality predicts behavior in weak situations but not in strong situations. That made sense to me, and I didn’t giveit much more thought.

Until few years ago. Some of my students were interested in this stuff, so I started reading more about the situation strength hypothesis. Then, as always, I started to question. First, do authors who rely on Mischel’s theory for their hypotheses actually test for variance differences as per the theory? (Spoiler alert-the answer is no, but that paper is under review elsewhere). Second, might it be that this sort of phenomenon goes beyond personality by situation interactions? The more I thought about this second question, the more intrigued I became.

Then I was on sabbatical at the University of Sydney, and I was looking for an excuse to collaborate with Bo Nielsen on something related to international business. It occurred to me that a more general sort of interaction, something that I began calling a restricted variance interaction, was quite common in IB research. So Bo, my longtime partner in crime Tine Kohler, and I published a paper to this effect in JIBS. But the more I thought about it, the more I realized that RV interactions went beyond IB. They were, in fact, everywhere, and at every level of analysis, from within person to between country. If we ever start doing interplanetary research, I bet we find RV interactions there too.
We started fiddling with data and equations, and we discovered that there was a lot of interesting stuff going on with these interactions. First, restriction of variance affects unstandardized weights, but not standardized weights. Second, while restriction on the DV weakens prediction as per Mischel, restriction on the IV actually has the opposite effect! Third, restriction on a mediator has no effect on the indirect effect. Fourth, higher order RV interactions are also entirely possible. Fifth, RV interactions have their own testing requirements. And the more we looked in the literature, the more we found examples of these and other RV interaction phenomena. Put all of this together, add my student Kate Keeler to the team, and you have our JOM paper.

This paper is one of three that Tine, Bo, Kate, and I are working on. The more that people look at the field through an RV lens, the easier they will find it to support their interaction hypotheses. My hope is that, through these various papers, we can generate enough interest in RV interactions that it reaches a tipping point such that everyone gets some exposure to the thinking that underlies these phenomena. Then we will see interaction hypotheses with stronger foundations than is currently the case. Here’s hopin’.

 

Stay up-to-date with the latest research and sign up for email alerts today through the homepage!

The Chrysalis Effect: Publication Bias in Management Research

14523043285_2235b0dbb4_zHow well do published management articles represent the broader management research? To say that questionable research practices impact only a few articles ignores the broader, systemic issue effecting management research. According to authors Ernest Hugh O’Boyle Jr., George Christopher Banks, and Erik Gonzalez-Mulé, the high pressure for academics to publish leads many to engage in questionable research, thereby leading the resulting published articles to be biased and unrepresentative. In their article, “The Chrysalis Effect: How Ugly Initial Results Metamorphosize Into Beautiful Articles,” published in Journal of Management, O’Boyle, Banks, and Gonzalez-Mulé delve into the issue of questionable research practices. The abstract for the paper:

The issue of a published literature not representative of the population of research is Current Issue Covermost often discussed in terms of entire studies being suppressed. However, alternative sources of publication bias are questionable research practices (QRPs) that entail post hoc alterations of hypotheses to support data or post hoc alterations of data to support hypotheses. Using general strain theory as an explanatory framework, we outline the means, motives, and opportunities for researchers to better their chances of publication independent of rigor and relevance. We then assess the frequency of QRPs in management research by tracking differences between dissertations and their resulting journal publications. Our primary finding is that from dissertation to journal article, the ratio of supported to unsupported hypotheses more than doubled (0.82 to 1.00 versus 1.94 to 1.00). The rise in predictive accuracy resulted from the dropping of statistically nonsignificant hypotheses, the addition of statistically significant hypotheses, the reversing of predicted direction of hypotheses, and alterations to data. We conclude with recommendations to help mitigate the problem of an unrepresentative literature that we label the “Chrysalis Effect.”

You can read “The Chrysalis Effect: How Ugly Initial Results Metamorphosize Into Beautiful Articles” from Journal of Management free for the next two weeks by clicking here.

Want to stay current on all of the latest research published by Journal of ManagementClick here to sign up for e-alerts! You can also follow the journal on Twitter–read through the latest tweets from Journal of Management by clicking here!

*Library image attributed to Apple Vershoor (CC)

 

William H. Starbuck on How Journals Can Improve Research Practices in Social Sciences

FullSizeRender

This year marks the 60th Anniversary of Administrative Science Quarterly, presenting an opportunity to not only celebrate the success of the journal and anticipate the promise of what the future holds, but also an opportunity to reflect on areas where the editorial process could be improved. In his essay, “60th Anniversary Essay: How Journals Could Improve Research Practices in Social Science,” published in the Administrative Science Quarterly, William H. Starbuck considers some imperfect properties of current editorial practices and methodology in the social sciences.

ASQ CoverThe abstract from his essay:

This essay proposes ways to improve editorial evaluations of manuscripts and to make published research more reliable and trustworthy. It points to troublesome properties of current editorial practices and suggests that editorial evaluations could become more reliable by making more allowance for reviewers’ human limitations. The essay also identifies some troublesome properties of prevalent methodology, such as statistical significance tests, HARKing, and p-Hacking, and proposes editorial policies to mitigate these detrimental behaviors.

You can read “60th Anniversary Essay: How Journals Could Improve Research Practices in Social Science” from Administrative Science Quarterly free for the next two weeks by clicking here. Want to know all about the latest research from Administrative Science Quarterly? Click here to sign up for e-alerts!

*Gould Reading Room picture credited to eflon (CC)

Listen to the Latest Podcast from Journal of Management on “The Chrysalis Effect”

jom coverIn the latest podcast from Journal of Management, Ernest Hugh O’Boyle Jr, lead author of the article “The Chrysalis Effect: How Ugly Initial Results Metamorphosize Into Beautiful Articles” speaks with Journal of Management Associate Editor Fred Oswald about the article’s findings concerning questionable research practices.

The podcast can be downloaded by clicking here and the article can be read for free by clicking here. Follow this link to subscribe on iTunes.

o'boyleeErnest Hugh O’Boyle Jr is an assistant professor of Management and Organizations at the University of Iowa. His research interests include questionable research practices, outcome reporting bias, publication bias, structural equation modeling, meta-analysis, “dark” personality traits, and superstar effects. He has been published in such journals as Journal of Management, Organizational Psychology Review, Family Business Review and International Business Review.

FredOswaldFred Oswald currently serves the Rice University Department of Psychology as Chair, and he is a Professor in the Industrial/Organizational Psychology program. His published research addresses the reliability and validity of tests administered to applicants in organizational, education and military settings. Substantively, his work deals with defining, modeling and predicting societally relevant outcomes (e.g., job performance, academic performance, satisfaction, turnover) from psychological measures that are based on cognitive and motivational constructs (e.g., cognitive abilities, personality traits, situational judgment tests, job knowledge and skill, and biographical data). His statistical work in meta-analysis, structural equation modeling, and adverse impact also informs personnel selection issues and psychological testing in the research, practice and legal arenas.

Want to know about all the latest news and research like this from Journal of Management? Click here to sign up for e-alerts!

Are There Questionable Research Practices in Management Research?

erasure-1123441-mThe publishing industry can be competitive. But how far will a potential researcher go to achieve success? Farther than you would think, according to a forthcoming Journal of Management article entitled “The Chrysalis Effect: How Ugly Initial Results Metamorphosize into Beautiful Articles.”

The abstract:

The issue of a published literature not representative of the population of research is most often discussed in terms of entire studies being suppressed. However, alternative sources of publication bias are questionable research practices (QRPs) that entail post hoc alterations of hypotheses to support data or post hoc alterations of data to support hypotheses. Using general strain theory as an explanatory framework, we outline the means, motives, and opportunities for researchers to better their chances of publication independent of rigor and relevance. We then assess the frequency of QRPs in management research by tracking differences between dissertations and their resultingjom cover journal publications. Our primary finding is that from dissertation to journal article, the ratio of supported to unsupported hypotheses more than doubled (0.82 to 1.00 versus 1.94 to 1.00). The rise in predictive accuracy resulted from the dropping of statistically nonsignificant hypotheses, the addition of statistically significant hypotheses, the reversing of predicted direction of hypotheses, and alterations to data. We conclude with recommendations to help mitigate the problem of an unrepresentative literature that we label the “Chrysalis Effect.”

Leading author Ernest Hugh O’Boyle, Jr. and Journal of Management Associate Editor Fred Oswald discuss the article’s findings and the effects these findings have on management literature in a new podcast. You can listen to the interview for free by clicking here.

Read “The Chrysalis Effect: How Ugly Initial Results Metamorphosize into Beautiful Articles” for free from Journal of Management by clicking here. Want to know about all the latest from Journal of Management? Click here to sign up for e-alerts!