User Password
Login / Create Account
<< Back to the Blog

The revolution has begun

Tags Post Publication Peer Review PubmedCommons
By: Rafael Najmanovich  
November 6 2013

Personal responsibility is essential in all aspects of science and that includes the communication of science. The logical endpoint of this personal responsibility when it comes to science communication, is openness and non-anonymity. I am a strong supporter of non-anonymous open peer review (NAOPR). NAOPR means that whenever you read a paper you can also judge objectivity (afforded by the lack of anonymity) and the quality of the reviews (as they are open) that lead to its publication. Furthermore, if any petty issues (David vs. Goliath) arise due to anonymity the open aspect guarantees that there is a paper trail documenting the issues. I spoke of this several times, for example here and here. But as much as NAOPR will be beneficial, it represents only half of the story.

The other half of the story is what happens with a paper after it was published. I will not discuss here some interesting initiatives such as that of F1000Research that is implementing a system for follow ups and updates (for more info, see here).

What I am really interested to discuss here is PubmedCommons.

Once a paper has been published, no matter what the quality of its peer review is, it becomes part of the corpus of our scientific knowledge. The problem is that many of these papers are wrong. A paper could be wrong in its conclusions, methodology, claims of originality and many other ways. In some cases this is so problematic that the paper needs to be retracted. However, in the majority of cases, the problems are only partial or not severe enough to require a retraction. But still these papers have problems that need to be highlighted and until now the process of highlighting such problems has been very cumbersome. That is until a couple of weeks ago, because now we have PubmedCommons.

PubmedCommons makes it possible to leave comments that become associated to the article in question and that will eventually (once the system is no longer in beta testing mode) appear to any one who searches that article. For the time being such comments are only visible to people participating. Readers of those comments can assign a comment as useful or not. To participate one needs to have already published an article that is indexed in Pubmed. This allows at once to restrict comments to scientist (necessary in my view for the time being at least) and also makes it more fair as those that comment can be also commented upon - one crucial characteristic of the system is that people need to use their real names. Because it is non-anonymous, it helps prevent abuse.

I think the system lacks one major component that once in place will allow PubmedCommons reach its full potential. This component is the ability for every person commenting on an article to designate the article as Approved (A), Approved with Reservations (AR) or Disapproved (D). In the same way as done in F1000Research. A summary of the number A/AR/D could be displayed at the top of all comments. Immediately one can think that the system could be played by commenters making non relevant comments that count equally to the A/AR/D tally. Here is where the already existing useful/non-useful button associated to every comment plays a part. The vote of that commenter can be normalized by the fraction of readers that found the comment useful and reported comments don’t count until assessed by moderators.

I think that PubmedCommons represents a revolution in science. Here are some reasons why:

1. The number of publications in any one field have exploded and it is only going to grow- A/AR/D values could be used as a new way to sort search results.

2. It is difficult to judge the quality of scientists. Number of citations of articles is a relevant measure but not ideal. Aggregated A/AR/D values or any other time dependent or otherwise measure based on them could be used as a peer judgement of the quality of a scientist. In effect, it is equivalent of having had a pool of experts in the field to assess a large number (eventually all) publications of that individual, associating a quality number to each and given short reasons why. If I put myself in the place of a hiring or grant committee, where it is often the case that not a single member is an expert in the specific sub field of the candidate, I would be happy to have A/AR/D values as an extra tool to evaluate the quality of some of the professional output of the candidate.

3. The problems with journal impact factors are immense and so extensively reported (e.g., here) that I will not discuss them in this post. In the same way as A/AR/D values help judge a paper or an author, they could also be used to aggregate the output of a journal for those that care about it. Now, in a world where you can judge an author based on the A/AR/D values of his own output, the only people that will care about the journal aggregated values will be publishers in judging the quality of their editors and reviewers. Most importantly, it will allow journals to judge the quality of the service they provide to authors and society because myself as an author will not worry to publish in a journal with a high traditional JIF as I will not be judged on that but on my A/AR/D values and therefore, can pick a journal based on what they offer to me and to the society at large. Just as a clarification, I personally already don’t consider JIF in my choice of publication venues.

Before I finish I’d like first to remind people of a few things: PubmedCommons will only work if it is massively used. You don’t need to wait for the beta to end to participate, send me the PMID of one of your papers, your name and email and I will invite you. Most important of all, whenever you comment on any paper, invite the corresponding author (whose email appears in the paper) to participate in PubmedCommons to give her/him the opportunity to reply. And remember, your name will be associated to that comment possibly for a lot longer than you will be around, so be objective, be fair, back up your arguments, be prepared to defend your positions publicly.

There is one thing that PubmedCommons will not do for me. It will not make me read less papers in my field. Ultimately, it is my responsibility to judge the quality of a work that has been published in my field and so even if I come across a paper with a very low A/D ratio, I will still judge for myself the paper and all the comments along with it. So in fact, I will be reading more than before, but I will be wiser for it.

PS: There are some technical minor issues with the system that annoy me like the apparent maximum length of a comment. I had to cut one comment into two. This limitation only adds clutter to the system, it is unnecessary and PubmedCommons should remove it.

Comments (1)
Login to leave a comment.
Pandelis Perakakis - November 7 2013, 11:05:34
Perhaps the true revolution will happen when we manage to dissociate the evaluation from the publication process using author-guided, formal, open peer review on all OA content before, during and after journal submission.