The Chrysalis Effect: Publication Bias in Management Research

14523043285_2235b0dbb4_zHow well do published management articles represent the broader management research? To say that questionable research practices impact only a few articles ignores the broader, systemic issue effecting management research. According to authors Ernest Hugh O’Boyle Jr., George Christopher Banks, and Erik Gonzalez-Mulé, the high pressure for academics to publish leads many to engage in questionable research, thereby leading the resulting published articles to be biased and unrepresentative. In their article, “The Chrysalis Effect: How Ugly Initial Results Metamorphosize Into Beautiful Articles,” published in Journal of Management, O’Boyle, Banks, and Gonzalez-Mulé delve into the issue of questionable research practices. The abstract for the paper:

The issue of a published literature not representative of the population of research is Current Issue Covermost often discussed in terms of entire studies being suppressed. However, alternative sources of publication bias are questionable research practices (QRPs) that entail post hoc alterations of hypotheses to support data or post hoc alterations of data to support hypotheses. Using general strain theory as an explanatory framework, we outline the means, motives, and opportunities for researchers to better their chances of publication independent of rigor and relevance. We then assess the frequency of QRPs in management research by tracking differences between dissertations and their resulting journal publications. Our primary finding is that from dissertation to journal article, the ratio of supported to unsupported hypotheses more than doubled (0.82 to 1.00 versus 1.94 to 1.00). The rise in predictive accuracy resulted from the dropping of statistically nonsignificant hypotheses, the addition of statistically significant hypotheses, the reversing of predicted direction of hypotheses, and alterations to data. We conclude with recommendations to help mitigate the problem of an unrepresentative literature that we label the “Chrysalis Effect.”

You can read “The Chrysalis Effect: How Ugly Initial Results Metamorphosize Into Beautiful Articles” from Journal of Management free for the next two weeks by clicking here.

Want to stay current on all of the latest research published by Journal of ManagementClick here to sign up for e-alerts! You can also follow the journal on Twitter–read through the latest tweets from Journal of Management by clicking here!

*Library image attributed to Apple Vershoor (CC)

 

How to Grow the Impact of Your Paper: A Step by Step Guide to Using Kudos

[This post comes from the SAGE Connection blog. It was written by Rebecca Wray of Group Marketing Manager, SAGE Publishing. You can find the original blog post here.]

becky wrayAfter an intense period of researching, writing, re-writing, submitting, and proofing an article, authors are then able to experience the delight of seeing it published online. They can now relax, sit back and watch as the downloads and citations stack up. But wait! There are also over 2.5 million other articles publishing this year too. How will people find this paper among this increasing landscape of research output and what can be done to make the author’s article more visible?

This is where Kudos comes inkudos logo

We’ve partnered with Kudos, a service designed specifically for authors, to help them maximize the visibility of their work. SAGE has always put its authors and their content first through supporting both dissemination and accessibility, and we’re really excited by this new partnership that enables us to further achieve our aims.  I’m not a researcher, or an author of any kind (other than this, my first published blog post!), but I’ve been marketing scholarly journals for over 10 years. During this time, I’ve seen that authors can be powerful advocates for their own work, complementing and extending the discoverability, circulation and marketing services that the publisher already provides to ensure their article reaches the widest possible audience. So how can Kudos help authors grow the impact of their papers?

Getting started with Kudos

When an author’s paper is published online on SAGE Journals, they will receive an email from Kudos inviting them to register on the website and ‘claim’ their paper. Authors also have the option to go back and claim all of their past papers that have a CrossRef DOI. Another key thing to point out: Kudos is free for SAGE authors to use!

Once the author has completed the brief registration form, they will have access to their own private author dashboard. Here they will be able to see all of the articles they claim (including those from other publishers) listed out, and track their actions and results.

The 4 stages of Kudos

Kudos offers four author tools, and authors are free to mix and match from the below:

  • Explain: Add a lay summary, impact statement and personal perspective to their Kudos publications page. This will make their article stand-out to researchers within their field, as well as make it more accessible to a broader audience.
  • Enrich: Add supplementary data such as podcasts and videos to enrich their article. This helps to engage readers with their work, and provide them with more context for the research.
  • Share: Kudos generates a trackable URL for the author’s article page. Here they can sync their Twitter, Facebook, and LinkedIn accounts and post directly from Kudos using their trackable link. Kudos also provides a tool to share the link to their paper by email.
  • Measure: Authors can track clicks from their sharing link in step 3, and see the impact of their actions in the dashboard with official citations and Altmetric scores for their article.

Spare a few minutes for many downloads

Authors can spend as little or as long on Kudos as they like. Needless to say, the more time spent on the site, the more they are likely to get out of it. In a pilot program, authors using the Kudos tools saw 19% higher downloads than those in a control group. So, now over to you to get creative with your communications and get Kudos!

Looking for me information on Kudos? Read our interview with Kudos co-founders Charlie Rapple, David Sommer, Melinda Kenneway, along with Ann Lawson, Head of Business Development, on how the key features of the service can help you grow citations for your articles here.

Connecting with the Community: Stephen Pinfield on Institutional Open Access Funds

SGO[We’re pleased to feature an interview, originally posted on the SAGE Connection blog, with Bailey Baumann and Stephen Pinfield. Stephen Pinfield recently published a paper with co-author Christine Middleton about the adoption of institutional central funds for open access publishing in his paper entitled, “Researchers’ Adoption of an Institutional Central Fund for Open-Access Article-Processing Charges: A Case Study Using Innovation Diffusion Theory”] 

In  a new study Stephen Pinfield and his co-author Chris Middleton analyze patterns
to the adoption of the Nottingham central fund by researchers at the university. Curious to learn more, I asked Professor Pinfield to share his thoughts on the adoption of open access funds and open access publishing.

Q. What are some reasons for a having university-owned OA fund?

Universities set up centrally-coordinated open access funds usually to encourage
take-up of OA amongst faculty members by making it easier for them to pay APCs (article-processing charges). A fund is normally made available throughout the institution, often in order to create a “level playing field” for institutional members, enabling a wide range of staff to afford APC payments. Managing the budget at a central level also allows the institution to be clear about how much money is being spent overall so that it can manage budgets at a strategic level. The alternative is to allow APCs to be paid locally by individual researchers from a variety of budgets. This often means that only certain members of faculty can afford to pay APCs and that it is very difficult to understand at an institutional level what is going on.


Q. Why is it important to study the adoption patterns of a central fund?

Universities set up centrally-coordinated open access funds usually to encourage take-up of OA amongst faculty members by making it easier for them to pay APCs (article-processing charges). A fund is normally made available throughout the institution, often in order to create a “level playing field” for institutional members, enabling a wide range of staff to afford APC payments. Managing the budget at a central level also allows the institution to be clear about how much money is being spent overall so that it can manage budgets at a strategic level. The alternative is to allow APCs to be paid locally by individual researchers from a variety of budgets. This often means that only certain members of faculty can afford to pay APCs and that it is very difficult to understand at an institutional level what is going on.

Studying adoption patterns of a central fund can help develop an understanding of the success (or otherwise) of such an approach and also may tell us a lot about acceptance of OA publishing more generally. In our study, we wanted to carry out an analysis of the use of a central fund which had been in operation for a long period (2006 onwards) to see how its use had diffused through the institution over time and what this told us about OA adoption and how it can be influenced.

Q. How might a lack of knowledge about OA publishing keep researchers from taking advantage of a central fund?

Recent studies indicate there is still a great deal of ignorance and misunderstanding amongst researchers about open access. Many faculty members may still be only vaguely aware of OA and may not understand its relevance to them. While this is undoubtedly changing, such attitudes would tend to mean the take-up of any budget to fund OA publishing would inevitably be limited. Researchers have well-established ways of working, often associated with publishing in conventional high-impact-factor journals, with OA featuring very low on their list of priorities. This is beginning to change but it is a slow process. It is influenced by a large and complex set of factors of which the availability and usage of a central fund may have a part to play. In our study, we wanted to understand what kind of role a central fund might perform in an institution and in the organization’s positioning in relation to OA.

Q. What do you think are some common misunderstandings about OA publishing? What would you like researchers who are considering publishing their work in OA journals to know?

Perhaps one of the major concerns that faculty have about OA publication is that of quality. OA is often associated in people’s minds with low quality. Of course, there is no necessary association between OA and low quality (many OA journals are of a very high quality), just as there is no necessary link between traditional publishing and high quality. Furthermore, it is not just about publishing in fully-OA journals. The central fund at Nottingham explicitly allowed payment of APCs for hybrid journals (subscription titles which also allow particular articles to be made OA on payment of an APC). Although hybrid open access is controversial, funding it, at least for the foreseeable future, does help OA to be seen more as mainstream. For many authors, to make their article OA in a familiar high-impact-factor journal makes them feel more comfortable with OA in general, at least at the beginning.

One clear message that emerged from our research was the importance of communication. The benefits of OA in general and of the use of the central fund in particular need to be communicated to individual researchers in a way that is directly relevant to them. Our research indicates in particular the influence of researchers themselves on their own immediate colleagues, in this case their experience of using the central fund, in encouraging wider adoption. Like most of us, researchers listen to those around them and adjust their behavior accordingly, rather than listening to 18682615624_115d0448cb_zpeople coming from ‘outside’ their immediate community. Faculty members listen to other faculty members more than they listen to librarians or research managers.

Q. In your opinion, what makes for the successful operation of a university-owned OA fund? How are librarians best involved in this process?

A central fund needs to be properly resourced and easy-to-use. Why and how to use it needs to be clearly communicated to academic staff – and communication needs to happen on an ongoing basis to ensure the message is heard. In particular, a communication strategy about the fund needs to leverage local support in academic schools and departments, and needs to take on board disciplinary differences. Crucially, it needs to be part of wider institutional strategies and policies on OA implementation which will also include a range of guidelines, processes and systems which together support the institutional response to the OA challenge.

Our research indicates that policies encouraging or requiring OA, especially from funders, are particularly important in influencing adoption. At an institutional level, therefore, there needs to be clear guidance and support to faculty to ensure they can easily comply with relevant policy requirements.

Librarians have been heavily involved over the last decade in promoting OA and supporting its implementation. This has included communication and advocacy, policy development, process design, technology deployment etc. This has undoubtedly worked well and it is testament to the information profession just how successful they have been in raising the profile of OA in the academic community. But there is only so much that one professional group can do alone. There is a clear need to disseminate and embed OA working practices widely in institutions. Librarians alone cannot make OA work – they need to help make OA a sector-wide imperative involving a wide range of stakeholders if it is to work.

Q. It seems like it could take significant resources to build and maintain central funds. How would you suggest smaller universities and universities in the developing world approach central funds?

Resourcing a central fund is obviously a big challenge regardless of the size of the institution. There is money in the system as a whole but it is often not funneled in the right direction. Changing the flow of funding streams is difficult and will only happen over the long term. Pilot funding to get things moving seems to be a good place to start – this is what happened at Nottingham. One of the potential benefits of Gold OA is that publication costs scale with research funding, something which is not necessarily the case with, for example, subscription funding. However, achieving alignment between the research costs and the costs of publishing in institutional budgets is not easy. Developments are required at national, funder and institutional levels in order to work toward achieving this. It will be interesting to see how the systemic change necessary to fund and manage OA will be achieved over the next few years.

You can read “Researchers’ Adoption of an Institutional Central Fund for Open-Access Article-Processing Charges: A Case Study Using Innovation Diffusion Theory” from Stephen Pinfield and Christine Middleton here.

Interested in more interviews like this? You can read more from the Connecting with the Community collection by clicking here.

*Library image credited to ktchang16 (CC)

 

l2Stephen Pinfield is Professor of Information Services Management at the University of Sheffield, UK. He has been involved in research and development in the area of open access for 15 years, including contributions to national and international policy discussions. His recent publications include work on the economics of open access, open data policies, and the global development of open-access repositories. He was previously Chief Information Officer at the University of Nottingham where he also founded the SHERPA OA initiative.

l1Bailey Bauman is the Editorial Assistant for SAGE Open. SAGE Open is a peer-reviewed, “Gold” open access journal from SAGE that publishes original research and review articles in an interactive, open access format.

Journal of Management Education’s Editors on the Invisibility of Reviewers

JMEFor reviewers, anonymity can be both a good thing and a bad thing. While anonymous reviews allow reviewer’s freedom to evaluate submissions solely based on merit, anonymity also means that reviewers are left unrecognized for their thoughtful yet time-consuming work. It would seem that this trade-off has made reviewing less of an attractive opportunity for potential reviewers. In their article, “Harry Potter in the Academy: Reviewing and Our Own Cloak of Invisibility, published in the current issue of Journal of Management Education, Kathy Lund Dean and Jeanie M. Forray offer a thought-provoking discussion of the flaws and merits of the blind review process, including why change is necessary to attract new reviewers. The article begs the question, is it possible that in the future, reviewers will cast off their Invisibility Cloaks, so to speak, and receive more recognition?

From the editorial:

We should not be surprised by the shrinking pool of reviewers for our conferences and publications. Steve Kerr explained for us decades ago how we focus our attention on that which is rewarded at the expense of other activities (Kerr, 1975). Perhaps because of Kerr’s article so many years ago and its continued power to frame reward systems theory and practice, we would expect management academics to, well, understand how not acknowledging the importance of reviewing will lead to precisely the dearth of reviewer pool we and other editors are experiencing, and change the reward system. That reviewing remains largely considered a service activity rather than a bona fide intellectual contribution is a serious issue for the continued health of our field. And the “blind” aspect of reviewing only exacerbates its invisibility—a reviewer devotes many hours, probably closer to a full day, to assessing a manuscript and to crafting a helpful, supportive review, and all that is usually noted on that person’s CV is a single bullet point telling others for which journals she or he serves as a reviewer. This must change, not only due to the inequity between those who author and those who review but also because of its gross distortion of how publication actually comes about.

You can read the rest of the editorial from Journal of Management Education free for the next two weeks by clicking here. Want to know about all the latest research from Journal of Management Education? Click here to sign up for e-alerts!

Behind the Scenes with ASQ Managing Editor Linda Johanson

[We’re pleased to welcome Cassandra Aceves of the University of Michigan Ross School of Business and Linda Johanson, Managing Editor of Administrative Science Quarterly. Cassandra recently sat down with Linda to discuss the peer review process and what it’s like to serve in the role of Managing Editor for Administrative Science Quarterly.]

Republished with permission. The original post was published on the ASQ Blog.

ASQ_v60n1_Mar2015_cover.inddQuestion 1. Many of our readers are beginning Ph.D. students who have yet to submit a paper or participate in the review process. Could you provide a brief introduction to the publication process and your role at ASQ?

I’m usually the first one to see a submitted manuscript. I manage the ASQ editorial and administrative office at the Johnson School at Cornell, which is the owner of the journal. I oversee the budget and finances, contracts, the submission and review system, and copy editing and issue production. For the past year, Joan Friedman, our new Associate Managing Editor, has been working with me on these and other tasks.

When ScholarOne notifies me of a new submission, I open the file, read the author’s cover letter and suggested reviewer and handling editor fields. Then I open the pdf file and read the abstract and first few pages to see if the paper’s appropriate for review and who might best handle it. Paging through the pdf file, I check for anonymity and make sure everything is legible. I assign a handing editor, based on the editor’s expertise, schedule, and current workload. We try to provide authors with the most helpful feedback we can leading to a publishable paper. If I can send the paper to the author’s preferred handling editor, I will. But sometimes the author doesn’t know that a different editor knows the methods or literature better—or is doing heavy teaching or traveling or has the flu.

It helps us all if the cover letter lists the people who’ve already given the authors comments or were on their dissertation committee, especially if any of them are likely to be asked to review for ASQ. Most of them won’t review a paper if they know the author, so it slows down the process if we ask them. It’s a good idea, too, to use the cover letter to explain any “non-preferred reviewers” listed on the submission: Are they people who really do hate the authors’ work or do they know the authors’ work and have already provided useful feedback? “Non-preferred” is an ambiguous term, so it’s good to be clear.

An informative abstract also helps me assign the right editor. Sometimes I have to read through half or even most of the manuscript before I figure out what it’s about and who could handle it. Authors can use published abstracts as guides to how to summarize a paper for readers so they want to read it. A useful abstract includes the theory, context, method, and main findings—but especially the context. A reader shouldn’t have to read 20 pages before learning that the study was conducted in an Indian call center, a Chinese factory, or a Brazilian company. The context can be a real asset—and could determine who should handle or review the paper.

The handling editor who gets the paper either assigns and invites reviewers or writes a “decline to review” letter saying why the paper won’t be going out for formal review. About a third of our submissions are “declines to review,” sometimes because they’re based on literature from another field and aren’t connected to or aimed at contributing to the organization studies literature or speaking to that audience. Sometimes it’s because they’re written for practitioners or policy makers rather than organization theorists. Authors whose papers are sent out for review have made it in the door and will get back a decision letter and set of reviews that should be helpful in continuing to develop the work, either for ASQ for for submission to another journal.

Question 2. In this article you present 8 tips to increase the likelihood that an article will be published: 1) clarify the research question and intended contributions early on in the paper, 2) guide the readers’ understanding of relevant literature, 3) understand readers’ perspectives and anticipate their questions, 4) be aware of and explain how terms and figures are used, 5) obtain collegial feedback on a paper before submitting, 6) carefully proofread the article, 7) use editors’ and reviewers’ feedback to understand how readers make sense of the paper, and 8) space out submissions of papers from the same project to incorporate sensemaking feedback. It has been seven years since you penned these tips. Do you have any additional tips to add?

I’ve always thought that Ph.D. students have a real advantage in that they’re immersed in foundational and current literature and know what’s already been done, what’s hot and what’s not, and have a pretty good mental map of the field. If they’re paying attention to the writing as they read, they can also find templates in published literature for how to structure different kinds of papers, establish a contribution, write up a method section or lay out a data table.

At the same time, I know how hard it is to divide and conquer a topic in the literature review for a paper and not just list everything written on a topic. One strategy is to use a smaller core set of very relevant studies and in reporting findings from those, take a piece of evidence from each one to use in building the argument for the important gap in the literature that really needs to be filled. As one of our editors used to say, “What’s the burning question to which this paper is an answer?” Each relevant piece of literature can become a building block for constructing the motivation and the theory underlying the study. Seeing how others did this in published work can be very helpful. This is where the hints in my 2007 article about sensemaking can be helpful.

Question 3. You note that the target journal determines the conversation that an author will enter and that this could be highly consequential for the success of an article in the review process. Do you have any insights into how students can best identify, follow, and signal that they are a part of that conversation?

Linda M. Johanson

Linda M. Johanson

Authors can identify their potential receptive journal audience by looking at their own reference list. Which articles have you used to ground the arguments? Which journals do you cite most often? Those probably represent the best audiences. Authors who follow and read a journal regularly are more likely to know what constitutes a contribution and to get a paper published there. Prospective authors are joining the conversation of authors who published on their topic. One way to signal that they should be part of that conversation is to make it clear that they know what that conversation is about and have something worthwhile to contribute. It’s a bit like joining an ongoing conversation at a party. To join in, you have to understand what people are talking about.

Question 4. You often host workshops for graduate students who are at the beginning stages of the research process. Do you have any tips for students who are submitting to a journal or going through the review process for the first time?

First, write sincerely. Communicate your ideas to your readers as clearly as you can, rather than impressing them with how many articles you’ve read—especially early in the paper. It’s frustrating to run into dense clumps of 6 to 10 references when we’re trying to figure out what a paper is about. Some authors have told me that they keep a copy of my 2007 article and read it again before they start writing.

You don’t need to write a long cover letter, especially if it repeats the abstract. Do let us know if you have related studies from the same data set published or under review elsewhere that aren’t cited in the paper. If you have a preference for a handling editor, tell us, but don’t insist that your suggestion be honored. We know the editors and their current situations better than most of the authors do. And we’re in the journal business, so we’re all doing everything we can to find and develop the best articles.

And once you’ve submitted, please be patient. Manuscript Central gives you information on what’s happening to your manuscript, but it doesn’t give all the information. For instance, you may see that your manuscript is waiting for reviewers to be assigned. Pretty alarming, right? But in truth that message comes up if even one reviewer is missing—which can happen when a reviewer agreed to review the paper but then had a family emergency. Sometimes we already have other completed reviews, so the problem isn’t as bad as it seems. It’s probably better not to check on your manuscript too obsessively. If it is taking a long time and you get worried about it, though, it’s OK to email me or Joan. We’re happy to check on it and move it along if we can.

When you get the reviews and a decision on the paper—whatever it is—take a deep breath. Then read through it, keeping in mind that the editor and reviewers gave you the best feedback they could with the hope that it will help you move this work forward. Honest and helpful feedback from those who know your topic is a true gift, even if it’s sometimes hard to accept. Use it to improve your scholarship and your writing. If you get an opportunity to revise and resubmit, that’s fabulous. Do your best to take care of the issues and any misunderstandings that came up in the review process and get some colleagues you trust to look at the revision before you send it back. Ask them to tell you if you did what was asked. If your paper was rejected and you need to rewrite it to submit to another journal, be sure to use the feedback you got from the first review process. Your reputation will take a hit if one of the same reviewers gets the paper at a different journal and sees that you didn’t pay attention to his or her review.

Question 5. You have been with ASQ for over 30 years. What is your favorite aspect of working at Administrative Science Quarterly?

That’s a hard question. It’s been such a rich experience. I guess what I’ve enjoyed most is being part of the vibrant ASQ intellectual community. I couldn’t have imagined when I started that I’d have a chance to edit papers on Niagara Falls hotels competing to control tourists’ view of the falls, scandals in Parliament, narcissistic CEOs, wildland firefighters, emotional intelligence, group diversity and performance, contract workers, scads of papers on social networks—an area that was just beginning to attract scholars then—and now sustainability and emerging markets. But it’s also the people in this community that I’ve had a chance to work with and get to know who have so enriched my life. Authors, editors, board members, reviewers, graduate students, faculty members across the globe. And finally, I’ve witnessed enormous growth in the field of organizational behavior and theory as well as dramatic changes in journal publishing and the number and kind of outlets for scholarly work. One of these changes is the growth of the ASQ student blog, and I’m so honored to be asked to contribute. Thank you.

For more about the peer review process from Linda Johanson, click here to read her article “Sitting in Your Reader’s Chair: Attending to Your Academic Sensemakers” from Journal of Management Inquiry.

Writing Your Journal Article in 12 Weeks

Interested in writing an article for a journal but struggling with it?

SAGE published Writing Your Journal Article in Twelve Weeks: A Guide to Academic Publishing Success, by Wendy Laura Belcher, in 2009 for anyone who struggles with academic writing.

“Wendy Laura Belcher’s Writing Your Journal Article in Twelve Weeks: A Guide to Academic Publishing Success is a revolutionary approach to enabling academic authors to overcome their anxieties and produce the publications that are essential to succeeding in their fields. Each week, readers learn a particular feature of strong articles and work on revising theirs accordingly. At the end of twelve weeks, they send their article to a journal. This invaluable resource is the only guide that focuses specifically on publishing humanities and social science journal articles.”

Key Features

  • Has a proven record of helping graduate students and professors get published:This workbook, developed over a decade of teaching scholarly writers in a range of disciplines at UCLA and around the world, has already helped hundreds to publish their articles in peer-reviewed journals.
  • Demystifies the academic publishing process: This workbook is based on actual research about faculty productivity and peer review, students’ writing triumphs and failures, as well as the author’s experiences as a journal editor and award-winning author.
  • Proceeds step by manageable step: Within the context of clear deadlines, the workbook provides the instruction, exercises, and structure needed to revise a classroom essay, conference paper, dissertation chapter, master’s thesis, or unfinished draft into a journal article and send it to a suitable journal.
  • Targets the biggest writing challenges: This workbook focuses squarely on the most difficult tasks facing scholarly writers, such as getting motivated, making an argument, and creating a logical whole.

For more information on SAGE journals, please click here.

Bookmark and Share

Behind the Editorial Curtain

David J. Ketchen, Jr., Jeremy Short and Will Terrell give us a peek behind the editorial curtain in the recent article, “Graphic Truth: Some Hidden Realities of the Review Process” recently published in the Journal of Management Inquiry. Jeremy Short adds:

 

 “This piece was inspired by the warm reception of two graphic novel management texts (Atlas Black: Managing to Succeed and Atlas Black: Management Guru?) I co-authored with Talya Bauer and Dave Ketchen. My co-author and I have been involved in a number of editorial roles over the years and thought we could share some of the stories we’ve encountered behind the editorial curtain. Because touchy matters are sometimes better addressed through using stories rather than finger pointing, we thought the graphic novel format would be ideal for the messages we hope to convey. We were fortunate to be able to engage Will Terrell, an excellent artist, to bring our thoughts to life. We certainly hope other editors, reviewers, and authors can learn from some of the tales we illustrate in our work.”

 Bookmark and Share