A brief analysis of commenting at BMC, PLoS, and BMJ

As announced on FriendFeed and Twitter, a writing collaboration between me and the inimitable Cameron Neylon has just been published at PLoS Biology, “Article-level metrics and the evolution of scientific impact”! (Loosely based on a blog post from several months ago.)

One of the many issues Cameron and I touched on was the problem of commenting. Most people probably aren’t aware of the problem; after all, commenting is alive and well on the internet in most places you look! But click over to PLoS or BioMed Central (BMC) and the comment sections are the digital equivalent of rolling tumbleweed.

As we mention briefly in the article, comments have great potential for improving science. For one thing, they’re a form of peer review, but without the month-long wait and seemingly arbitrary review criteria. Readers, authors, and other evaluators can also get a sense of what people think about the article. The ideal is certainly tantalizing — vigorous, rigorous debates over the finer scientific points as well as the overarching conclusions with participation both from experts in the field as well as informed laypeople, always with intelligence and civility!!!1!11!!one!! But let’s not kid ourselves — the worst-case scenario is all too easy to imagine and would probably look something like the discussions over at YouTube.

And this would be positively urbane. (From PhD comics)

Clearly, few scientists would find such discussions illuminating or useful as a means of evaluating the impact of a particular paper. But the idea of using comments to attain immediate feedback — and, verily, to give it — still beckons. Ironically, the real problem seems not to be with keeping undesirables out, but in enticing potential contributors in.

Despite having commenting platforms up for several years, the vast majority of articles published in PLoS and BioMed Central have no comments, and those that do average in the low single digits. Only a handful of articles at both publishers have comment numbers in the double digits.

What types of articles are these? Looking at BMC’s top commented articles, about 80% are commentaries, editorials, correspondence, etc; in other words, not primary research. If you add in the odd case report or study protocol, that percentage only goes down to 67%. (These numbers were taken from a list of most commented articles as of July 2008, looking at the 12 articles with more than five comments. Obviously, the list may have changed since that time, but I’m guessing not significantly, as very few of the most highly accessed articles on BMC have any comments and comment numbers for the 12 articles I looked at have not gone up for the most part.)

If we instead turn to the top 100 most highly accessed articles at BMC in the last 30 days (data collected around Oct 30), we see a slightly different picture. Here, the balance is approximately 60:40 in favor of research articles. The number of articles in the top 100 that have comments, however, is somewhat depressing — only 12 have any comments at all, and only two of these have more than two. The good news is that 1/2 of the commented articles are research (disproportionately more than for non-research), and research articles averaged 4.3 comments compared to 1.5 for front matter. Taking a quick glance at the all-time most highly accessed articles reinforces the observation that number of views does not seem to correlate with number of comments nor with time since publication.

Turning our attention to a sampling of highly commented papers at PLoS, we see a slightly different pattern. Although the article with the most comments is in PLoS Medicine, the median number of comments per article in PLoS Medicine is lower than that for PLoS ONE. This is to be expected given the fact that PLoS ONE publications have been designed to be highly interactive from the start.

Reassuringly, “Research” articles tend to have more comments than non-research articles, though this may be due to an outlier. The number of highly commented articles also seems to be fairly even for both types overall, though this balance is skewed by the fact that PLoS ONE only publishes primary research. PLoS Comp Bio publishes both types of articles and the highly commented ones tend to be front matter. Articles with comments at PLoS Medicine, on the other hand, seem to be more evenly distributed between research and front matter.

Interestingly, articles with comments tend to be in the medical or public health fields at both BMC and PLoS. If you go to BMJ, a popular medical journal that also allows comments (“Rapid Responses”), you’ll actually find fairly lively discourse. Yet a bias towards “front matter” like editorials and news items is apparent: out of 159 articles commented on between 9/30/09 and 10/30/09, only 31% were research-type articles (for which I included “Research”, “Methods”, “Analysis”, and “Practice” articles).

The good news is that research articles in BMJ tended to have more comments than front matter articles, though not by much — three per research article commented on in that time period vs. two per front matter article. Research articles also showed a slight trend towards more comments over time which the front matter articles did not exhibit.

From looking at this data, we can draw several very tentative conclusions*:

1. Front matter articles attract comments more than research articles.
2. Articles in the fields of medicine and public health attract comments more than other fields.
3. Research articles tend to accumulate more comments than non-research articles.
   

* Note that the sample sizes for all analyses are very small and thus any conclusions are purely speculative.

In the next post, I’ll offer some very informal and more or less unsubstantiated explanations for these observations. One final observation is that none of the three publishers I looked at allows browsing or ranking of articles based on number of comments as far as I could tell, although with PLoS I could certainly go through the ALM data that they’ve kindly made available. Still, I would imagine that having a browsable list of the most commented articles would be very useful just for the casual reader.