Posted by: Kay at Suicyte | April 13, 2008

How to deal with bad peer review

I have just read Dave Lunt’s text on Anonymous peer review, and I can only second every word he writes. I consider reviewer anonymity as essential for giving me the freedom of saying things that have to be said. Like David, I would never dare telling a person I know very well that their latest manuscript is essentially rubbish. Without any doubt, my case history as a reviewer would have severly limited my chances of later collaboration.  Even very good scientists can sometimes submit very bad manuscripts.

The other side of the coin, giving ‘rogue reviewers’ the chance to hide behind anonymity has to be addressed, although I would rather see this happen without compromising anonymity. I even have an idea how this could be done. I am sure that some clever people have proposed this before. I wouldn’t even exclude that some (other) clever people have given the idea due consideration and found out that it is not going to work. Anyway, this is my suggestion: Journals publishing peer-reviewed research should have a formalized ‘court of appeal’. The authors should have the possibility to do something about reviewer reports that are way off the mark, be it due to hostility, incompetence, or laziness of the reviewer. Obviously, safeguards have to be established that prevent authors from complaining about every negative review they get. However, authors with justified complaints must be guaranteed that some knowledgeable and unbiased person sees to their case. In an ideal situation, the board of appeal could not only overturn the original judgement, but also do something about egregious cases of flawed reviews – something that would actually hurt the reviewer (no, I am not thinking of high voltage here).

Already now, it is possible to complain with the editor about reviewer’s misjudgements, at least at some journals (generally the better ones). However, unless you know the editor in person, or are a big shot working in the right place, your chances of getting the editor to even look at your case are slim. Given that nobody likes getting a bad review and is likely to find the criticism unjustified, the lack of enthusiasm on the editors side to deal with complaints is understandable. However, there are really some bad bad reviewers out there, and something has to be done about this problem. Nowadays, when more and more journals are charging the authors rather than the reader, the paying ‘customer’ should at least have the right to be treated in a fair and transparent manner.

Probably, the two weakest points in my proposal are i) the increased work load put upon the editor(ial board), and ii) the lack of adequate sanctions that can be imposed on bad reviewers. After all, we have to keep in mind that reviewers are not paid for their job, and every action that risks scaring away the good guys is out of question. Maybe, in the more severe and unambiguous cases, at least a stern letter from the editor to the reviewer, mentioning the flaws uncovered during the appeal process, would be indicated. This action would not destroy the career of a bad reviewer, but might nudge him/her back on track.

Advertisements

Responses

  1. Without any doubt, my case history as a reviewer would have severly limited my chances of later collaboration. Even very good scientists can sometimes submit very bad manuscripts.

    Exactly. I notice that many of the people who think anonymity is of no benefit are those who are quite established authorities, generally in a tenured position, and have nothing to fear from the consequences of writing a negative review.

  2. Freedom of speech isn’t always easy to balance. I think it should be incumbent upon the reader who encounters both the report and the review to decide which, if either, is correct. Most intelligent people should be able to discern what is fair treatment of a subject, and what is mere criticism for its own sake. One manuscript should not define a career, and an author who feels misjudged can always write another.

  3. Freedom of speech isn’t always easy to balance. I think it should be incumbent upon the reader who encounters both the report and the review to decide which, if either, is correct.

    All fine and well, but the problem is that the reader does not see both the manuscript and the review, so there is no chance to decide who is right. Moreover, in case of bad review, nobody will see the manuscript at all.

    One manuscript should not define a career, and an author who feels misjudged can always write another

    This might be true in a non-scientific setting. In the scholarly peer-reviewed publication area, what the authors typically do is submit their work to another journal. Unless you have very powerful enemies in the field, you have to be quite out of luck to get the same malicious review from two different journals. The problem here is that not all journals are of equal reputation, and – silly it may sound – there are situations where the choice of journal can have an impact on your further career.

  4. I should add that there is at least one biomed journal, where you do see both the manuscript and the review. It is the Journal of Eugene Koonin Research Biology Direct. Definitively worth having a look!

  5. The repercussions of established scientists on youngsters for criticizing their weak papers sheds light on a problem that we have in the review process.
    While you take a vote on the paper, the review you provide should typically not include any statements about suggestions for acceptance but solely focus on technical issues, interpretation of the results and scientific soundness. In the end, it’s the editor, who has to take the decision and complaining to him might not really be what you want to do. I have seen papers published despite strongly negative opinions of the reviewers and vice versa. The “problem ratio” of poor reviews over anonymity problems is much greater than 1, strongly in favor of open peer review.

    I have no problem criticizing the work of colleague in a closed system and welcome negative comments myself – delays and poor reviews written by dimwits on their commute are more of a concern. Some OA journals provide a opt-out option, which I have occasionally taken, basically to show that I do mean that the paper has value despite my criticism.

    By the way, if you want to shoot down a paper because the main author wore white socks at a conference, make sure you provide very constructive reviews that lead to many experiments and additional analysis. If you ask for the same in a context of a rant, it’s much easier for the editor to disregard your opinion.

  6. There’s no reason for anonymity. Base your comments on fact and don’t be malicious and you shouldn’t have to worry about using your real name. Like you said, even good scientists sometimes submit bad papers, so instead of getting their feelings hurt, they’ll just send it somewhere else.

  7. In these open peer review discussions, everybody appears so noble and courageous. I for my part don’t take negative reviews lightly. If somebody dares to criticize one of my manuscripts, he/she must either be malicious, stupid, or nitpicking. Yes, I know, I have seen many examples of rebuttal letters, where authors thank the reviewer (me!) for helpful suggestions. As much as I might wish this to be true, I know that they are just lying.

    BTW, thanks to all commenters explaining to me how the reviewing process works. Your valuable suggestions are highly appreciated.

    🙂

  8. Your objects to open review are, of course, legitimate. However, they don’t necessarily mean that open review is worse than anonymous review.

    The point is to weigh up the pros and cons. For one, if you fear hurting your collaborators feelings, but maybe you shouldn’t be reviewing their manuscripts to begin with? Open review helps show when manuscripts are being reviewed by collaborators, former students, bosses and acedemic rivals. These are all things that should be avoided.

    I needn’t go into the raft of other advantages in open review.

    But if you’re going to go for anonymity, then why not make it double blind? I often think that might be the best of all. Make it double blind to begin with, and then publish the reviews names if the manuscript is accepted. That way you still won’t know which reviewer said what. Of course, some times the reviewer might guess who the author was, and vice versa, during the review, but I think the doubt might significantly affect cronyism and enmity.

    (For the sake of full disclosure, I had a manuscript rejected for the second time yesterday)


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Categories

%d bloggers like this: