A brief analysis of commenting at BMC, PLoS, and BMJ

As announced on FriendFeed and Twitter, a writing collaboration between me and the inimitable Cameron Neylon has just been published at PLoS Biology, “Article-level metrics and the evolution of scientific impact”! (Loosely based on a blog post from several months ago.)

One of the many issues Cameron and I touched on was the problem of commenting. Most people probably aren’t aware of the problem; after all, commenting is alive and well on the internet in most places you look! But click over to PLoS or BioMed Central (BMC) and the comment sections are the digital equivalent of rolling tumbleweed.

As we mention briefly in the article, comments have great potential for improving science. For one thing, they’re a form of peer review, but without the month-long wait and seemingly arbitrary review criteria. Readers, authors, and other evaluators can also get a sense of what people think about the article. The ideal is certainly tantalizing — vigorous, rigorous debates over the finer scientific points as well as the overarching conclusions with participation both from experts in the field as well as informed laypeople, always with intelligence and civility!!!1!11!!one!! But let’s not kid ourselves — the worst-case scenario is all too easy to imagine and would probably look something like the discussions over at YouTube.

And this would be positively urbane. (From PhD comics)

Clearly, few scientists would find such discussions illuminating or useful as a means of evaluating the impact of a particular paper. But the idea of using comments to attain immediate feedback — and, verily, to give it — still beckons. Ironically, the real problem seems not to be with keeping undesirables out, but in enticing potential contributors in.

Despite having commenting platforms up for several years, the vast majority of articles published in PLoS and BioMed Central have no comments, and those that do average in the low single digits. Only a handful of articles at both publishers have comment numbers in the double digits.

What types of articles are these? Looking at BMC’s top commented articles, about 80% are commentaries, editorials, correspondence, etc; in other words, not primary research. If you add in the odd case report or study protocol, that percentage only goes down to 67%. (These numbers were taken from a list of most commented articles as of July 2008, looking at the 12 articles with more than five comments. Obviously, the list may have changed since that time, but I’m guessing not significantly, as very few of the most highly accessed articles on BMC have any comments and comment numbers for the 12 articles I looked at have not gone up for the most part.)

If we instead turn to the top 100 most highly accessed articles at BMC in the last 30 days (data collected around Oct 30), we see a slightly different picture. Here, the balance is approximately 60:40 in favor of research articles. The number of articles in the top 100 that have comments, however, is somewhat depressing — only 12 have any comments at all, and only two of these have more than two. The good news is that 1/2 of the commented articles are research (disproportionately more than for non-research), and research articles averaged 4.3 comments compared to 1.5 for front matter. Taking a quick glance at the all-time most highly accessed articles reinforces the observation that number of views does not seem to correlate with number of comments nor with time since publication.

Turning our attention to a sampling of highly commented papers at PLoS, we see a slightly different pattern. Although the article with the most comments is in PLoS Medicine, the median number of comments per article in PLoS Medicine is lower than that for PLoS ONE. This is to be expected given the fact that PLoS ONE publications have been designed to be highly interactive from the start.

Reassuringly, “Research” articles tend to have more comments than non-research articles, though this may be due to an outlier. The number of highly commented articles also seems to be fairly even for both types overall, though this balance is skewed by the fact that PLoS ONE only publishes primary research. PLoS Comp Bio publishes both types of articles and the highly commented ones tend to be front matter. Articles with comments at PLoS Medicine, on the other hand, seem to be more evenly distributed between research and front matter.

Interestingly, articles with comments tend to be in the medical or public health fields at both BMC and PLoS. If you go to BMJ, a popular medical journal that also allows comments (“Rapid Responses”), you’ll actually find fairly lively discourse. Yet a bias towards “front matter” like editorials and news items is apparent: out of 159 articles commented on between 9/30/09 and 10/30/09, only 31% were research-type articles (for which I included “Research”, “Methods”, “Analysis”, and “Practice” articles).

The good news is that research articles in BMJ tended to have more comments than front matter articles, though not by much — three per research article commented on in that time period vs. two per front matter article. Research articles also showed a slight trend towards more comments over time which the front matter articles did not exhibit.

From looking at this data, we can draw several very tentative conclusions*:

1. Front matter articles attract comments more than research articles.
2. Articles in the fields of medicine and public health attract comments more than other fields.
3. Research articles tend to accumulate more comments than non-research articles.
   

* Note that the sample sizes for all analyses are very small and thus any conclusions are purely speculative.

In the next post, I’ll offer some very informal and more or less unsubstantiated explanations for these observations. One final observation is that none of the three publishers I looked at allows browsing or ranking of articles based on number of comments as far as I could tell, although with PLoS I could certainly go through the ALM data that they’ve kindly made available. Still, I would imagine that having a browsable list of the most commented articles would be very useful just for the casual reader.

The big hairy ambitious haircut – finally!

It seems like ages ago that I was fretting over what big hair ambitious haircut to get… because it was, well, ages ago. But I finally went out and got it after determining that you could see my split ends from Google Earth and that brushing my hair required more shoulder flexibility than I possess.

After some back and forth with the stylist over whether I could have my short hair cake and not have to blow dry it too, I ended up with this:

While I think I might have to compromise and use some kind of “product”, it might just work. Well, except for the bits across my face, which are already starting to annoy me. Still, it’s leagues better than the $4 mullet cut I got in college, which was the only other short style I’ve had since I was a toddler.

The jury is still out on whether I can actually play sports and see at the same time.

In memoriam: Warren DeLano

 

PyMOL has starred in many journal covers

On Tuesday, November 3rd, the scientific community suffered a great loss with the passing of Warren DeLano. Most people know him as the creator of PyMOL, a popular and extremely powerful molecular visualization tool, but most – including myself, until recently – may not know all of the other unique qualities that made Warren a mentor, collaborator, inspiration and friend to many. And by making PyMOL open source, Warren demonstrated his generosity and ensured that his work would continue to help future generations of scientists.

 

One visit through the comments at the memorial site set up by his sister is more than enough to illustrate the impact Warren had, not only on the scientific community, but to individuals too many to count. He is also remembered through many blog posts (MacResearch, ByteSizeBio, ChemSpider) and on FriendFeed.

My own memory of Warren is fleeting and I did not fully appreciate the opportunity to engage with him until his untimely passing. I posted the following on his memorial site:

One Tuesday night this past February, a handful of young sciencey types gathered for an open science-themed meetup at the Prolific Oven in Palo Alto. Broadcast at short notice through FriendFeed and blogs, the gathering was small and I didn’t know who was going to show up (if at anyone!). It ended up being two guys from Labmeeting.com (the organizers), a guy from or formerly from Merck, myself – a graduate student at Stanford, …. and none other than Warren DeLano.

At the time, there seemed nothing out of the ordinary. But it was truly a wonderful thing that the creator of PyMOL would take a break from his work to stop by that blustery night (probably just taking a break from his work that evening) and engage in a discussion about open source and open science with a group of young science enthusiasts. I used PyMOL often in my graduate work (indeed, the “pretty pictures” were sometimes the only bright spot in the doldrums of a PhD!) and the fact that he was essentially a one-man show makes his contributions and his commitment to helping others even more incredible.

May all the memories of Warren inspire scientists everywhere to have the same generosity of spirit. His passing is a huge loss to the scientific community but his life was a tremendous gift.

My deepest condolences to his family and friends.

Let us all remember Warren DeLano, who gave so much of himself to science. And if you have your own memories of Warren, please post them at his memorial site.

Posts in the pipeline, and in the meantime

October’s been a busy month and so I haven’t had much time to post. But busy means interesting, and so I have lots of things to write about, it just doesn’t really get done. Some of the posts I have in the pipeline — mostly just as titles with scarce notes to remind myself what they mean:

The commenting conundrum: about where and why scientists do or don’t comment on scientific articles.

Responding to “them”: about the whos, whats, wheres, whens, whys, and hows of criticism and responding (or not) to it; mostly on the web but also off.

A detailed look into PLoS’s article-level metrics data: it’s open, so why not? And the results might just surprise you.

Thoughts from Science Commons Salon: with the amount of brainpower in that room, I’m surprised it didn’t explode. In fact, I’m surprised the whole town of Mountain View hasn’t exploded from sheer intellect yet.

So yeah, plenty to write about, sometime. I saw Pete Binfield of PLoS at the SC Salon and he joked that I was falling behind, reposting things that he’d posted a whole four days ago. Makes me want to start the Slow Blog movement…

Those posts will probably keep simmering for a little while. In the meantime, I haven’t been completely idle – in the last three weeks, I’ve written three blog posts for 23andMe’s Spittoon on genetic association studies on glaucoma, bone mineral density, and blood-related traits. Another one is set to come out early next week. So if you haven’t been tuning in regularly to the Spittoon, now you know where else to find me!

Scripts and hacks for curation

When you curate scientific literature, there are lots of little tasks and procedures and requirements that can all add up into a big inefficient mess if they’re not integrated very well into your workflow. Some of these things are in-house and so you have some level of control over how they are handled (depending, sometimes, on engineering or management), while for others you are at the mercy of the wilderness that is science. It would be great, for example, if all abstracts contained enough information for us to evaluate whether the rest of the article is worth reading – or, in many cases, worth buying in order to read it.

Photo by hbart on Flickr

In the context of genetic association literature, it would also be great if all abstracts used standard database identifiers for SNPs (i.e. rs #s), as this unambiguously defines the variant; used standard means of reporting the association (e.g. odds ratios with 95% confidence intervals and p-values); and mentioned pertinent aspects of the study, such as the population, number of cases and controls, and adjustments for confounders and multiple comparisons. For me, this qualifies as “enough information to evaluate whether the rest of the article is relevant.” I hope that when they do not mention things like the corrected p-values, it is not because their p-values were not significant. When articles cost up to $90 a pop… well, let’s not get me started.

But I digress. The point is that there are lots of things that could make curation a challenge, and consequently there are lots of things that could make curation easier. Standardization of abstracts, while it doesn’t make for juicy reading, makes going through high volumes of abstracts easier (and machine-accessible). Linking articles from journal websites to PubMed would also be useful, as PubMed serves as a portal to many other resources. Currently, almost all journals use digital object identifiers (DOIs), which are unique pointers to objects on the web. But PubMed IDs (PMIDs), like digital object identifiers (DOIs), are a little simpler, and provide a lot of useful functionality through integration with NCBI’s many databases. You can imagine all the little scripts and hacks you could come up with to improve the curation process, using greasemonkey scripts on Firefox, bookmarklets on any browser, and even web apps.

One somewhat mundane task we often have to do is search for a paper on PubMed to get the PMID. This is straightforward given we already know the authors, title, journal, etc, but still kind of a pain. Fortunately, PubMed allows you to search by DOI, which almost all publishers provide. So a slight improvement is to use the DOI as the search term in PubMed, as this will return the exact result if the DOI exists in the database. But you still have to open up a new browser window or navigate to PubMed and copy and paste the DOI into the search bar. To reduce the number of steps even further, we can use a simple bookmarklet containing a bit of javascript (if it looks cut off, you can still double-click copy and paste it):

javascript:var%20t;%20try%20%7B%20t=%20((window.getSelection%20&&%20window.getSelection())%20%7C%7C%20(document.getSelection%20&&%20document.getSelection())%20%7C%7C%20(document.selection%20&&%20document.selection.createRange%20&&%20document.selection.createRange().text));%20%7D%20catch(e)%20%7B%20%20t%20=%20%22%22;%20%7D;%20location.href='http://www.ncbi.nlm.nih.gov/sites/entrez?db=pubmed&term='+t+'%5BAID%5D';

This script extracts whatever text you’ve highlighted on a page and attempts to search PubMed using it as the DOI. So obviously it will only work if you’ve highlighted something and that something is a DOI, and that DOI is in PubMed. But assuming you do and it is, it will send you directly to the PubMed entry for that paper. Save the script as a browser bookmark, put the bookmark in your bookmarks bar, and whenever you’re on an article webpage (or RSS feed) and want to see the PubMed entry for that article, just highlight the DOI and click the bookmark. (Cameron wrote up a pipe on Yahoo!Pipes a while ago that does something similar, which inspired this bookmarklet.)

Clearly even this simple hack can be improved – it would be nice, perhaps, to have it return the PMID in an alert box so you can make a note and then continue doing whatever you were doing, rather than being sent away to PubMed (this might make use of AJAX?). It would be nice if you didn’t have to highlight, but the script would look for and extract the DOI from the page automatically. And I’m sure you could add even more bells and whistles, within reason.

My latest hackneyed…. “hack-need”… is to be able to identify follow-up studies for a particular genetic association. If you read a paper with PMID X saying SNP A is significantly associated with a disease, it would be really useful to know when future studies look into that association and either replicate or contradict the finding. Hopefully when they do so, they cite PMID X and/or mention SNP A. Essentially, I’d like to query PubMed for papers that cite a given PMID or SNP (via rs #). Ideally, I could do this in batch for many PMIDs and many SNPs automatically, and have each query return only results that are newer than the previous query (or query date). Then I set the script running behind the scenes, process the results using another script, and maybe have it send me an email with a list of new PMIDs to look into every week. Can world domination be far behind?*

Seriously though, I am looking for tips on how to do this follow-up identification thing, so any help appreciated. Pierre has given me some useful hints for how to search PubMed for papers citing a given rs #, and it would be great if this could be modified with dates:

http://eutils.ncbi.nlm.nih.gov/entrez/eutils/elink.fcgi?dbfrom=snp&id=1802710&db=pubmed&reldate=7

(insert your favorite rs # as the id)
Update: reldate limits results to those within a number of days immediately preceding today’s date; could also use mindate and maxdate to specify a date range.

* Why stop there, you might ask? I could write a script that downloads the abstracts, “reads” them, filters out the irrelevant ones, summarizes the important information, and populates curation reports. But then I’d be out of a bloody job…

http://www.flickr.com/photos/hbart/ / CC BY-NC-SA 2.0

New job and curation 101

It’s been several weeks now since I started working at 23andMe, a personal genomics company located in Mountain View, CA. Perhaps not coincidentally, it’s also been several weeks since I last blogged. The transition hasn’t been difficult, but it did take some getting used to, mentally and physically. I mean, leaving for work by 8:30am? Regular hours? Commuting??

Ok, so I really have nothing to complain about. 8:30 isn’t that early, and I could shave half an hour off each end of my commute if I didn’t choose to take advantage of bike-friendly roads, good weather, and a company-sponsored free train pass (OMG benefits!?). All in all, things are pretty much fantastic. The work environment is friendly, flexible, and laid-back; we have plenty of food and drink to keep us fueled throughout the day, and regular workouts/yoga if we need to get fired up or mellowed down (and to keep the “Free Food 15″ at bay). Plus, personal genomics is a super interesting and rapidly evolving industry, so there’s really never a dull moment.

So what is personal genomics, anyway? We’ve known for a while that genetics – the sequence of DNA inside our cells – plays an important role in our form and functioning. Many diseases are caused by changes in DNA (often in genes, parts of DNA that code for proteins) that alter the normal functioning of cells, though not all genetic differences lead to negative changes. (Genetics can also tell us about ancestry – who is related to whom and the history of populations – but I won’t be addressing that in this post.) Where it gets personal is when you apply it to individuals, such as when someone gets a genetic test to determine whether they have or are at risk of developing or passing on a particular disease. Where it gets genomics is when we use high-throughput technologies to do what is essentially thousands of genetics tests at once. Put them together, and you get personal genomics.

How do we know what genetic “pieces” correspond to what conditions or diseases? The general strategy is to compare the DNA of a whole bunch of individuals that have that condition (cases) to a whole bunch of individuals that don’t (controls). As long as both groups are similar save for their case-control status, any significant genetic differences between them should have something to do with that condition. We call this a genetic association.

It turns out that there are millions of single locations in the human genome where the exact sequence of the DNA might differ between two people, and these places, called single nucleotide polymorphisms, or SNPs, can contribute to differences we can observe, such as whether you flush when you drink alcohol or how easily you put on weight. 23andMe personal genomics kit determines what your sequence is for a representative subset of SNPs. Many are already known to be associated with certain conditions, and new research is being done every day to uncover more and more of these associations.

So what exactly do I do at 23andMe? My official job title is “Scientist, Content Curation”. Curation, I’ve found, is not very familiar to most people. Most people probably know that there is such a thing as a museum curator, but might not know what they do. Hardly anyone has ever heard of scientific curation. (And I thought explaining what I was studying as a grad student was hard! Biomedical informatics, anyone?)

But it’s really not that complicated. The essence of curation is almost always the same: the selection, acquisition, and management of content. What that content is differs depending on the field – for example, an art curator might look for and organize artwork for exhibition in a gallery, while a curator in the “Ancient Civilizations” department of a museum may be in charge of acquiring, managing, and presenting archaeological artifacts.

In science, curation involves organization of scientific knowledge and data. An area where this has been especially important is the life sciences, as the amount of information being generated by high-throughput experiments, large-scale projects, and scholarly publishing has skyrocketed. In order to manage this information and render it useful to others, the field of biocuration was born. Any database that organizes scientific knowledge – UniProt (the Universal Protein resource), FlyBase (database for that very important model organism, Drosophila), PharmGKB (a database focused on how genes and drugs interact), etc – depends on curators to keep the information up to date and easy to use.

And so it is with 23andMe. The genetic testing kit is one part of the product, but the other part is information – what knowledge is there about associations between the SNPs on our platform and health traits or conditions? What does your particular data mean? The science is far from exhausted on this subject, and in order to stay up to date with the research, 23andMe spends a lot of effort on curating the scientific literature for new genetic associations and presenting the information on our website for our customers.

Day to day, this means that we keep track of papers recently published in scientific journals, skim through to find ones that may have promising findings, and then vet these more thoroughly to see if they pass our stringent scientific standards. If they do, we extract the bits of information we need and put the bits together in reports that will eventually become part of the content on the website. It’s a job that definitely benefits from an organized system and an eye for detail – as well as a sense of curiosity.

After three weeks on the job, I think I’m starting to get the hang of the day to day work. Since my work is even more directly tied to the literature than it was as a graduate student in academia, I’m also developing an enhanced awareness of issues surrounding scientific publishing – those related to standardization and metadata, publication bias towards positive results, and closed vs. open access. The hardest aspect of transitioning from academia to industry hasn’t been the regular schedule, or the work environment, or the work itself, it’s been getting used to being on the other side of the pay-wall of scientific journals.

But that’s a rant for another time.

A Real Guitar Hero – Sungha Jung 12 Year Old Prodigy Fingerstyle Guitarist at LimitlessUnits.com

via limitlessunits.com

This kid is truly talented… the style, skill, originality, I can’t think of anything he’s missing. I’d definitely buy an album of his arrangements!

Sydney International Food Festival: Flags

via ibelieveinadv.com

Whimsical, clever, beautiful and delicious – what more could you want? Food is a common denominator across the world!

How personal genomics is rocking the boat

I’ve been doing some reading on personal genomics, direct-to-consumer genetic tests, and personalized medicine lately, in an effort to steep myself in the science and issues prior to starting work in this field. Today, I read an opinion piece by R. J. Carlson titled “The disruptive nature of personalized medicine technologies: implications for the health care system,” [1] that was especially interesting. Rather than expound on the usual arguments for or against consumer genomics, it laid out several important areas where personalized medicine and genomics technologies would disrupt the current system, often with brutal honesty.

Clearly, one of these areas is private health insurance. Describing private health insurance as “a hybrid of economic ruthlessness and utilitarian social policy … is supposed to perform the social policy role that the public sector can’t or won’t, and
that is to ration,” Carlson points out some sobering scenarios. One is that the Genetic Information Non-discrimination Act (GINA) covers only the underwriting process, and does not guard against denial of coverage or steep increases in premiums once a genetically-suggested condition manifests. Another is the moral and social dilemma posed by the knowledge – on either side – that those with “demonstrably superior health” are subsidizing care for those with “known genetic risk”. And, given the increasing knowledge we’ll have about health risks, it would be ridiculous not to use any of it in designing insurance packages. Carlson doesn’t paint this as a negative thing, necessarily, but instead calls on public policy to “facilitate the constructive uses of these data by shaping financial and access reforms to the genomics medicine that is arriving.”

The debate over health insurance is fairly familiar, however. What Carlson makes very clear in the rest of the piece is that personal genomics takes medicine in a fundamentally different direction than where it has been going for the last half century. Traditional modern medicine has focused on mechanism and reductionism, finding what’s wrong and fixing it, and applying that knowledge to new cases of the same thing. We use the fact that humans are more or less similar to enact standards of care.

But personalized medicine focuses on the differences between people and treats every patient as a unique case. This leads to two natural consequences: it makes medical care more costly, and it renders the standardization of medical practice obsolete, if not impossible. Of course, personalized medicine could conceivably be more cost-effective through better preventative care, but this is only if significant effort goes towards realizing this potential. And although I hadn’t thought about personal genomics in the context of evidence-based medicine, it’s not hard to see the conflict:

There’s the rub: to be effective, a personalized medicine must build on our ever more definitive differences, defying standardization for the very long haul, if ever. Measuring quality in health care under a genomics model is crudely analogous to measuring automobile fuel efficiency when every automobile is assembled from a wide array of materially different but functionally interchangeable parts, performs differently on every trip, and changes in performance with the moods and capacities of every driver.

This article captures the nuances of some very interesting challenges facing health care in response to genomics technologies with a view that is both realistic and optimistic. Carlson recognizes that the era of medical paternalism is giving way to democratization of health information, and we must adapt our policies to reflect this. Indeed, he argues that without active and careful management of this process, we may very well sabotage our ability to reap any rewards from this technology.

Definitely worth a read, and worth thinking about.

[1] Carlson RJ. (2009) The disruptive nature of personalized medicine technologies: implications for the health care system. Public Health Genomics 12(3):180-184.
DOI: 10.1159/000189631 [PubMed] [Journal]

Scientific discourse as an epic FAIL

Download now or preview on posterous

How to Publish a Scientific Comment2.pdf (42 KB)

A post on FriendFeed pointed me to this blog post in Adventures in Ethics and Science discussing a particularly infuriating example of just how broken the current system of scientific publishing can be. The epic tale is presented by Prof. Rick Trebino in a PDF document (above) outlining “How to Publish a Scientific Comment in 123 Easy Steps”. This version includes his second addendum in which he gives many excellent (and some painfully obvious) suggestions for how to improve the system.

Here’s a preview:

1. Read a paper in the most prestigious journal in your field that “proves” that your entire life’s work is wrong.

2. Realize that the paper is completely wrong, its conclusions based entirely on several misconceptions.  It also claims that an approach you showed to be fundamentally impossible is preferable to one that you pioneered in its place and that actually works.  And among other errors, it also includes a serious miscalculation—a number wrong by a factor of about 1000—a fact that’s obvious from a glance at the paper’s main figure.

3. Decide to write a Comment to correct these mistakes—the option conveniently provided by scientific journals precisely for such situations.
…
6. Prepare further by writing to the authors of the incorrect paper, politely asking for important details they neglected to provide in their paper.

7. Receive no response.
…
15. Write a Comment, politely explaining the authors’ misconceptions and correcting their miscalculation, including illustrative figures, important equations, and simple explanations of perhaps how they got it wrong, so others won’t make the same mistake in the future.

16. Submit your Comment.

17. Wait two weeks.

18. Receive a response from the journal, stating that your Comment is 2.39 pages long. Unfortunately, Comments can be no more than 1.00 pages long, so your Comment cannot be considered until it is shortened to less than 1.00 pages long.
…
20. Remove all unnecessary quantities such as figures, equations, and explanations.  Also remove mention of some of the authors’ numerous errors, for which there is now no room in your Comment; the archival literature would simply have to be content with a few uncorrected falsehoods.  Note that your Comment is now 0.90 pages.

21. Resubmit your Comment.

22. Wait two weeks.

23. Receive a response from the journal, stating that your Comment is 1.07 pages long. Unfortunately, Comments can be no more than 1.00 pages long, so your Comment cannot be considered until it is shortened to less than 1.00 pages long.
…

And so the saga begins. Really, the whole thing makes my blood boil.