In the world of scientific peer review, I think that there are three kinds of referees: those that help, those than hinder, and those that are, umm, ineffective. Referees that are ineffective do an adequate surface job, looking over papers to make sure that there are no glaring problems and that the manuscript is appropriate for the journal in question, but that's it. Referees that hinder are the annoying ones we all complain about. You know - they're the ones that send in a twelve word review for your groundbreaking submission to Science or Nature after sitting on it for 6 weeks; the review says little except "Meh." and may even indicate that they didn't really read the paper. They're the ones that say work is nice but not really original, with no evidence to back up that statement. They're the ones who sit on papers because they're working on something similar.
Referees that help are the best kind, of course. These are the people who read manuscripts carefully and write reports that end up dramatically improving the paper. They point out better ways to plot the data, or ask for clarification of a point that really does need clarification or improved presentation. They offer constructive criticism. These folks deserve our thanks. They're an important and poorly recognized component of the scientific process.
10 comments:
By strange coincidence, Asymptotia also posts on referees today. With Hitler!
but on the other side it is also frustrating when the authors do not really respond to a diligent and careful review. often i get the impression they just try to avoid additional work and have the paper quickly published instead of really aiming for quality.
Don - Thanks for pointing that out. Love the Downfall parody.
Anon. - I know what you mean. It's particularly frustrating to write a review that says "The authors really need to say something about XYZ" (which might not even be very complicated or require additional effort beyond adding a citation or something) and get back a revised manuscript + response document that just ignores the issue.
I was motivated to write this post because we're just finishing off a big revision (including more experiments, additional figures, etc.). The original version was fine, but the referee comments really brought home that we needed to present things differently (including confirmatory control experiments showing that the apparatus really does what we say it does), and we did need to consider an additional piece of physics. Very helpful, and done constructively.
Publishers should give out awards and special compensation to good referees, at the end of each year or even every 5 or 10 years. This would be based on referee ratings/feedback as well as scientific significance/citations of the papers the referees referee.
In Nature Photonics' website it states that the editors ignore terse comments (affirmative or not) no matter who provides it. What carries weight is lengthy and detailed reports. I also know that experimental papers are not necessarily refereed by experimentalists so if you can manage to penetrate the screening by the editors, refereeing is quite fair in Science/Nature however I increasingly hear that PRL is being a pain in the butt. The papers that get rejected by it are published in journals with higher IF.
I this you have left out the power trippers. They are the ones who utilize their temporary position of power to promote their own agenda, without necessarily hindering (not so much anyway).
They are the ones who are all right with letting your work through, as long as you cite theirs.
They are the ones who take the opportunity to patronize you, to tell you that, you know, you really should not be working on that problem, that it has been "beaten to death" already and that you really should be changing subject, e.g., work on their pet problem on which they have just published a paper which they want to have cited.
They are the ones who remind you that the type of work that you do (numerical, just as a hypothetical example), is not really "real theory" and that you should really seek to collaborate more with someone who does "real theory"...
That last post was spam, delete it.
I don't have much experience with referees, but what I have had has been mostly okay. But recently I had a referee comment that implied such a lack of understanding of the foundations of quantum mechanics that I quite nearly posted it up on my blog for public ridicule.
I think "ineffective" a bit harsh for the group that do a standard job, but nothing exceptional. Catching a few glaring errors is definitely an effect.
Schlupp, you're right. I should've used a less negative word. "Adequate"?
Post a Comment