No comment

At the risk of beating the issue to death, I offer yet another post on the question, “why don’t scientists comment on scientific articles?” Previous reflections stood within the larger context of scientific impact and article-level metrics, and I’ve also attempted some superficial analysis of commenting behavior at PLoS, BMJ, and BMC. More recently (and this is why the topic is on my mind again), a room full of bright minds at the PLoS Forum (including Cameron Neylon and Jon Eisen) scratched their heads over it and came up with pretty much the same conclusion as everyone else who’s ever thought about the problem — the costs simply outweigh the benefits.

The costs, in principle, are minimal. You might need to register for an account at the journal website and be logged on, but then all that’s needed is little more than what most of us already do multiple times a day with our email — type into a box and click “submit”. (In practice, there may be nonsensical, hidden costs that make you wonder what the folks at those journals were smoking.) So the perception that the cost-benefit equation doesn’t work speaks more to the lack of benefit than anything else.

Photo by jamesclay on flickr



In academia, this translates to “commenting on scientific articles does not advance your career.” In fact, commenting might even have a negative impact if it’s perceived as a frivolous activity (as most online activities are perceived, though this is starting to change as online reference managers and other tools become more popular). Every minute not spent doing research or applying for grants is a minute wasted, despite the fact that all but the most superhuman of us — and we all know one of those — have to sleep, eat, and use the can.

This doesn’t mean that no one comments on scientific articles. They just don’t do it online. Instead, they might spend months slogging through a convoluted review process similar to that for full-fledged papers so that their comments are published and citable in a “traditional” form (granted, some of these “comments” are barely indistinguishable from full-fledged papers). Never mind that by the time the comment is published, the discussion may no longer be relevant. Never mind that only a narrow slice of the possible spectrum of comments is represented.

People prefer this type of commenting to the relative ease and immediacy of online commenting, and it isn’t because of any real utility. Traditionally published comments are widely accepted as contributions to the scientific record, and so might be considered part of someone’s academic record as well. That little distinction is the only reason I can fathom people suffering through even a shadow of what this individual suffered through. The perceived benefit is high enough to offset what is arguably a very high cost.

And that benefit is conspicuously absent from online commenting.

Photo by the-majestic-fool on flickr

Does that have to be the case? No, not necessarily. But that’s how it is right now, and it will remain that way as long as hiring committees, tenure committees and funding agencies rely on traditional measures of academic success and contribution. We could ask courageous, aspiring researchers to embrace online commenting merely on principle, for “openness” and “betterment of science”, but, frankly, they shouldn’t have to.

Naturally, the discussion at the PLoS Forum centered around how to make online commenting more attractive, with the assumption being that providing incentives would shift the cost-benefit equation. But the main incentive suggested — a “karma” type system reminiscent of StackOverflow — still begs the question, “who cares?” Scientists aren’t going to comment simply because they get imaginary points for it — the points would have to translate to real-world recognition.

With all of this in mind, I was struck by the description of PLoS Currents, a new offering from PLoS that is part journal, part blog. The idea is to provide a venue for “fast-breaking” research that simply can’t be delayed by the traditional peer review process. Articles are composed in Google Knol with a blog editor-style interface; after light review by an expert moderator (with typically around 24 hours turnaround time), the articles are posted on the PLoS Currents website. The finished articles are barely distinguishable from articles that might appear in a peer-reviewed journal, with familiar document structure and detailed figures and tables. And, importantly, the articles are indexed on PubMed — making them archived and citable. The costs, clearly, are lower than traditional commenting, but the benefits are arguably similar.

Photo by ubermoogle on flickr

Is this the transitional form that will successfully bring science towards more flexible modes of publishing and professional advancement? I don’t know, but I like the idea. PLoS Currents might just satisfy the ingrained compulsion to make distinct, cohesive contributions to the scientific record but has none of the arbitrariness and convolution of commenting through lengthy peer review.

Only time (and perhaps some proactive efforts from PLoS) will tell. They’ve only launched one Current so far — Influenza, spurred by the H1N1 flu crisis — but it’s not hard to see how the concept could be extended to other topic areas. PLoS has also been fairly reserved in its advertising of Currents given its experimental nature, but I’d love to see what happens when they throw more weight behind it. Maybe, just maybe, it will be contender.

About these ads

9 Responses to No comment

  1. Bee says:

    I think you’ve entirely missed the point. Scientists don’t comment on other’s articles without putting considerable thought and time into it because they’re afraid of damaging their reputation with off-hand (gut-feeling) comments that later turn out to be wrong. That’s why they’ll rather spend several weeks digging through the details and maybe then figuring out their question/comment was irrelevant and not offering it at all. You don’t change that by offering imaginary points. You change that by creating an atmosphere in which it’s okay to be wrong, as long as you’re willing to be corrected. Scientists were educated to be accurate, to make sure everything they say is backed up by references, to never ever make a mistake in their argumentation. Of course they have an opinion on [that recent paper/topic/study], but they’ll rather bite off their tongue than to publicly share it. That’s a generation issue though. It will fade with time, just try again.

    • shwu says:

      Bee, I entirely agree. Every time I write about this, one aspect or another jumps out at me. For this post, I was stuck on the more discrete citation/credit aspect, but the point you bring up about the qualitative measures of reputation and needing a culture that allows room to be wrong is spot on.

      I think can still be rolled under the general umbrella of costs and benefits. The costs associated with a careless comment could be damaging to a career (though I think the risks are generally overblown), and certainly with any commenting system we want to encourage thoughtful ones that contribute to a discussion. But the traditional process is so frustrating it’s a wonder anyone submits comments that way at all. And yet they do. Do people really get that much kudos for publishing comments in journals?

      I think the PLoS Currents model could mitigate both the operational problems of traditional commenting and the cultural reluctance of online commenting. The articles really are more or less complete articles into which much more research, thought and effort goes into than with typical blog comments, and the finished look and PubMed indexing encourages that effort. But the 24 hr turnaround and simple moderation brings it much closer to online commenting. There shouldn’t have to be an artificial barrier to publishing comment-style articles just to provide a sense of security that the comments being published are truly “worth it”.

      I’m hoping that something like PLoS Currents could encourage more people to publish their ideas — even if not quite ready for a full-length paper — and engage in discussion using online media.

  2. Chris Tan says:

    Not sure how feasible is this:

    A solution is create an online currency/record for scientific contribution. Some central authorities (such as NIH) create a point-reward system and online infrastructure. Then, every (ideally) researcher, scientific institution and possibly scientific publisher are allocated some points for given out to others for sharing data, codes, assistance that is not enough for authorship on publication, scientific comment, and other scientific activities that advance science in general. The points given to each activities should be very small to reduce people focusing on accumulating the points rather the activities themselves. Every researcher has an account keeping track of points he accumulated. The central authorities can then issue awards or recognition when researchers accumulate above some number of point. Maybe Google can provide the infrastructure.

    • shwu says:

      Chris,

      Interesting idea. I’m wary of a complicated point system in general, though. It doesn’t seem like we need something that prescribed. Perhaps if the “central authorities” simply started considering more aspects of scientific contribution in their funding decisions and published recommendations for doing so, it would at least start encouraging a culture where scientists can be creative, engaged, and more open about their thoughts and their work. Of course, one of the problems is that there isn’t an easy way to look at all of someone’s blogging behavior, or online commenting behavior, or contributions to databases, etc. So some other tools are needed first.

      • Chris Tan says:

        I agree we do not need a complicated point system. And I think you are right on how do the “central authorities” assess other aspect of scientific contribution objectively. Faced with thousands of grant proposal to review and not an expert in all the proposed projects, I can understand why funding agencies often have to rely partially on measures like impact factor. Hence, innovation tools are needed to solve this problem for them.

        The reward/credit system, though not ideal, is already presented in scientific enterprise in form of publication/citation which has encourage scientists throughout history to share their result/findings. Ideologically, we should cultivate a culture where scientists can be creative, engaged, and more open about their thoughts and their work. However, setting up a infrastructure (which need not be physical) can greatly accelerate that. The reward/credit system in form of publication/citation has served scientific enterprise well, it is just a matter of broadening this scope to accommodate other channel of contribution.

        Just my two cents if it worth that much.

  3. Bee says:

    Hi Shwu,

    Thanks for your reply. It’s probably a combination of all these reasons. I tend to believe that this is where the future will go anyway, it’s just that most people are somewhat slow arriving there. There’s nothing too puzzling or unusual about this, it just takes some patience and persistence, so keep on writing :-) You might be interested in my today’s blogpost that discusses options how to improve peer review. Best,

    B.

  4. BenK says:

    Have you read most of the comments on Nature articles at Nature.com? With the very rare exception, they embarrass the authors of the posts. In general, to properly respond to an article, one either needs give and take (best done via email or privately) or one needs data (best done in a new, reviewed publication).

    • shwu says:

      @BenK,

      I’m not disputing that commenting systems as they are today are woefully inadequate/misutilized for the lofty standards of scientific exchange. It’s also quite possible that some barrier to entry is useful and even desired to avoid regrettable outbursts or ensure a minimum level of investment. This was why I found the PLoS Currents model intriguing as a compromise between the overly bureaucratic traditional mode of publishing formal comments and informal online commenting.

  5. I am a pharmacist and blogger (www.rxbalance.org) who stopped blogging for the same reasons addressed above about commenting. It is extremely time consuming – writing about science and medicine requires a degree of accuracy that goes beyond commenting on everyday topics. From carefully reviewing an article, absorbing the content, formulating your ideas, writing and editing your comments. And than you still have to fact check yourself and style your references. Arrggh. It is really tough if you also have a regular job. Compare this to FaceBook or online news sites, where you can write and post a comment in 3 minutes or less. What we need is a crowdsourcing portal where you can break down the tasks into “fun sized components” (think candy bars!) that are easy to do on the fly.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

Join 25 other followers