Posted by: Kay at Suicyte | February 28, 2008

Review confidentiality and related issues

It looks like the guys at SART take longer than expected to finish their groundbreaking research. This leaves me some time to blog about another topic that has caused some stir in the blogosphere recently. I just was reminded of the story when reading an editorial by Donald Kennedy published in last weeks Science.

What seems to have happened is that Pfizer is being sued on side effects of Celebrex and Bextra, and some material published in NEJM is to be used as evidence in this case. It now seems that Pfizer tries to force NEJM by means of subpoenas to let the Pfizer lawyers browse confidential NEJM review material. I am neither familiar with this particular case, nor with the relevant US laws. Here is a short paragraph from the editorial:

Pfizer asserts that in some cases plaintiffs are making use of published papers from the New England Journal of Medicine (NEJM). So it wants to dig though the confidential reviews of those papers in search of something to strengthen its defense.

You can find extensive coverage of this incident in a number of blogs, including PharmalotThe Futile Cycle and at Nature network. As can be expected, bloggers and most scientists I spoke to take side with the NEJM. As a (moderately) frequent author of scientific papers, and a (more) frequent reviewer, I also feel inclined to fiercely defend reviewers confidentiality – at least against lawyers and other low life forms.

However, a closer look at the wording made me hesitate. If Kennedy’s editorial is correct, NEJM is not asked for the reviewers identity, but rather for ‘confidential review material‘. And this is – at least in my humble opinion – an entirely different matter. One feature of peer review forms at most journals is a box with the title ‘confidential remarks – not to be sent to the authors’, or similar. I never quite understood what these confidential remarks are good for. As far as I can remember, I never used this feature myself. In general, I think that peer review should be made as transparent as possible. I would strongly prefer that decision about life or death of a submission should be made on the basis of material that can be seen (and understood) by all parties involved. The prospect of some dark collusions going on between editors and reviewers is nothing that any author would fancy.

I have been raised in the scientific working class, i.e. a minor university in some obscure European city. In these circles, you will often find a deeply rooted distrust in the editorial processes at high-profile journals. I am not saying that this distrust is generally justified, but it is rather widespread. Many people won’t even consider sending their good stuff to Nature, Science, Cell. The general feeling is that these journals are exclusively meant for PIs working at prestigious places, who are used to having dinner with chief editors, which whom they are on a first-name-basis anyway. Ah yes, and a flawless scientific pedigree is also required. In addition, everybody knows somebody who can tell a sorry tale of what happens when you dare to swim with the big fish.

To make this posting more interesting, here are my own two contributions to the elegy of the scientific underdog. I must admit that I wasn’t directly involved, but I can guarantee that these are not just urban legends. In one case, I have seen black-on-white evidence, and the other case happened to somebody I consider very trustworthy (and I was a minor co-author of the paper).

Case 1: This incident happened about 15-20 years ago, but trust me – both the scientists and the journals involved are still around. A colleague of mine, I will call him X, submitted a paper to a moderately high-profile journal. It is important to know that at that time manuscript submissions were not done via the internet, but sent by regular mail on real paper. The journal in question offered the option to exclude particular competitors from the reviewing process. I don’t even know if journals are still doing this, but I consider it a nice feature (although i never used it myself). X had a particular competitor Y , who was working at a far more prestigious institute than X himself, and X asked the editor to exclude Y from the reviewing. The big surprise came when X received the returned manuscript (yes, at that time you got the manuscript back, mainly because of the glossy-print figures that had to be supplied by the authors). I don’t remember if the paper had been accepted or rejected, but what I do remember is finding – hidden between the pages of the manuscript – a signed letter from Y addressed to the editor, saying ‘Dear xxxx, thank you for bringing this manuscript to my attention’. I also remember that I thought this was outrageous and never understood why X never did anything about it.

Case 2: This is not quite as bad as the previous case, but it happened recently. Another researcher ‘X’ (different person, obviously) submitted a manuscript to a high-impact journal Y. It did go out to the referees but eventually was rejected on the basis of two mostly negative reviewer reports. This was not entirely unexpected, and in the meantime the manuscript has been published by another decent journal. To X’s surprise, at a recent meeting X was approached by another scientist Z, who asked why the manuscript hadn’t been published in the original journal Y. During the following discussion, Z admitted to being a reviewer of the original submission to journal Y (we all know that Z is not supposed to mention this, but we are all humans, even reviewers). When asked which of the reviewer reports was his, Z answered ‘the enthusiastic one’. Strangely enough, both of the reports were far from being enthusiastic, so chances are that Z didn’t tell the truth. However, X had wondered all the time why the two negative reports used for the rejection were titled ‘reviewer 1’ and ‘reviewer 3’. Did the journal really drop the (possibly positive ) report of reviewer 2? And if so, did they do it on purpose? Probably, they even are entitled to do whatever they want. Most journals have policies saying that rejection or acceptance is solely a matter of the editor’s discretion. In any case, I consider it bad style to make editorial decisions on the basis of material that is hidden from the authors.

I should hasten to add that I also had a good deal of positive experience with the reviewing process, both as an author and as a reviewer. By no means do I consider the two cases typical of what is going on at the major journals of the field. However, both cases serve to underscore my point that the obscurity and secrecy of the editorial decision process should be kept at a minimum. But not below that.

Advertisements

Responses

  1. I don’t know how most journals operate, but for your second story, if two first two reviewers have widely different opinions, perhaps the journal opted to have a third reviewer as a tie-breaker, and then sent the two reviews that were in agreement?

  2. This is well possible, and probably also entirely legal. I wouldn’t consider it good editorial practise, though. But what do i know?


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Categories

%d bloggers like this: