A brief analysis of commenting at BMC, PLoS, and BMJ

As announced on FriendFeed and Twitter, a writing collaboration between me and the inimitable Cameron Neylon has just been published at PLoS Biology, “Article-level metrics and the evolution of scientific impact”! (Loosely based on a blog post from several months ago.)

One of the many issues Cameron and I touched on was the problem of commenting. Most people probably aren’t aware of the problem; after all, commenting is alive and well on the internet in most places you look! But click over to PLoS or BioMed Central (BMC) and the comment sections are the digital equivalent of rolling tumbleweed.

As we mention briefly in the article, comments have great potential for improving science. For one thing, they’re a form of peer review, but without the month-long wait and seemingly arbitrary review criteria. Readers, authors, and other evaluators can also get a sense of what people think about the article. The ideal is certainly tantalizing — vigorous, rigorous debates over the finer scientific points as well as the overarching conclusions with participation both from experts in the field as well as informed laypeople, always with intelligence and civility!!!1!11!!one!! But let’s not kid ourselves — the worst-case scenario is all too easy to imagine and would probably look something like the discussions over at YouTube.

And this would be positively urbane. (From PhD comics)



Clearly, few scientists would find such discussions illuminating or useful as a means of evaluating the impact of a particular paper. But the idea of using comments to attain immediate feedback — and, verily, to give it — still beckons. Ironically, the real problem seems not to be with keeping undesirables out, but in enticing potential contributors in.

Despite having commenting platforms up for several years, the vast majority of articles published in PLoS and BioMed Central have no comments, and those that do average in the low single digits. Only a handful of articles at both publishers have comment numbers in the double digits.

What types of articles are these? Looking at BMC’s top commented articles, about 80% are commentaries, editorials, correspondence, etc; in other words, not primary research. If you add in the odd case report or study protocol, that percentage only goes down to 67%. (These numbers were taken from a list of most commented articles as of July 2008, looking at the 12 articles with more than five comments. Obviously, the list may have changed since that time, but I’m guessing not significantly, as very few of the most highly accessed articles on BMC have any comments and comment numbers for the 12 articles I looked at have not gone up for the most part.)

If we instead turn to the top 100 most highly accessed articles at BMC in the last 30 days (data collected around Oct 30), we see a slightly different picture. Here, the balance is approximately 60:40 in favor of research articles. The number of articles in the top 100 that have comments, however, is somewhat depressing — only 12 have any comments at all, and only two of these have more than two. The good news is that 1/2 of the commented articles are research (disproportionately more than for non-research), and research articles averaged 4.3 comments compared to 1.5 for front matter. Taking a quick glance at the all-time most highly accessed articles reinforces the observation that number of views does not seem to correlate with number of comments nor with time since publication.

Turning our attention to a sampling of highly commented papers at PLoS, we see a slightly different pattern. Although the article with the most comments is in PLoS Medicine, the median number of comments per article in PLoS Medicine is lower than that for PLoS ONE. This is to be expected given the fact that PLoS ONE publications have been designed to be highly interactive from the start.

Reassuringly, “Research” articles tend to have more comments than non-research articles, though this may be due to an outlier. The number of highly commented articles also seems to be fairly even for both types overall, though this balance is skewed by the fact that PLoS ONE only publishes primary research. PLoS Comp Bio publishes both types of articles and the highly commented ones tend to be front matter. Articles with comments at PLoS Medicine, on the other hand, seem to be more evenly distributed between research and front matter.

ALM-comments-plos

Interestingly, articles with comments tend to be in the medical or public health fields at both BMC and PLoS. If you go to BMJ, a popular medical journal that also allows comments (“Rapid Responses”), you’ll actually find fairly lively discourse. Yet a bias towards “front matter” like editorials and news items is apparent: out of 159 articles commented on between 9/30/09 and 10/30/09, only 31% were research-type articles (for which I included “Research”, “Methods”, “Analysis”, and “Practice” articles).

The good news is that research articles in BMJ tended to have more comments than front matter articles, though not by much — three per research article commented on in that time period vs. two per front matter article. Research articles also showed a slight trend towards more comments over time which the front matter articles did not exhibit.

BMJ-commentsFrom looking at this data, we can draw several very tentative conclusions*:

  1. Front matter articles attract comments more than research articles.
  2. Articles in the fields of medicine and public health attract comments more than other fields.
  3. Research articles tend to accumulate more comments than non-research articles.

* Note that the sample sizes for all analyses are very small and thus any conclusions are purely speculative.

In the next post, I’ll offer some very informal and more or less unsubstantiated explanations for these observations. One final observation is that none of the three publishers I looked at allows browsing or ranking of articles based on number of comments as far as I could tell, although with PLoS I could certainly go through the ALM data that they’ve kindly made available. Still, I would imagine that having a browsable list of the most commented articles would be very useful just for the casual reader.

About these ads

7 Responses to A brief analysis of commenting at BMC, PLoS, and BMJ

  1. Pingback: Say something « Faculty of 1000

  2. David Crotty says:

    I thought your PLoS article was informative and did a nice, fair job of evaluating some of the metrics offered. Some thoughts on why rating systems are so flawed can be found here:
    http://scholarlykitchen.sspnet.org/2009/11/16/how-meaningful-are-user-ratings-this-article-4-5-stars/

    Two quick thoughts/questions:

    1) There’s a huge disconnect between the commenting system (online in the journal) and where most readers read the article (as a pdf or as a printout of their pdf). Because of this, commenting requires a large activation energy, you have to go back to the journal, find the article, then leave your comment. At that point, it’s just as easy to comment on Friendfeed, a blog, or anywhere else (you’ll note I left a comment here, but not on the actual paper). Would a journal’s comments increase if the journal eliminated pdf versions of articles and forced a reader to read them online at the site itself? Or would people just print out the html version and no difference would happen?

    2) How does the long shelf life of a research paper come into play? I’ve written about it here:
    http://scholarlykitchen.sspnet.org/2009/11/05/participation-value-and-shelf-life-for-journal-articles/

    We are trained that a blog entry is a transient thing. If you want to be part of the conversation, you need to comment fairly quickly after the entry comes out. If you want to gossip about what Angelina did this week, you need to gossip about it immediately, or the conversation will move on to something else. As a researcher though, one will often discover relevant papers that are 5, 10 or 20 years old. You might not get to the new issue of journal X for a few months after it comes out. Because reading of articles is so asynchronous, does this reduce the chances of conversation, as participants are likely to be separated in time?

  3. Pingback: Article-level metrics getting attention « Be openly accessible or be obscure

  4. Pingback: Science Spotlight – November 24th, 2009 | Next Generation Science

  5. Pingback: Mailund on the Internet » Blog Archive » Last three weeks in the blogs

  6. Pingback: No comment « I was lost but now I live here

  7. Pingback: Libre accès, mais ouvert à quel point ? #openaccess | Tout se passe comme si

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.