Wednesday, March 05, 2008

Reviewing- why, how, and how often?

I review lots of papers and proposals for various journals and funding agencies. While time allocation is a continual challenge, and while there is no good framework for rewarding this kind of professional service, I think it's important to do my fair share for several reasons. First, I'd like someone else out there to do me the same courtesy - well written, thorough, timely referee reports almost always improve the quality of scientific papers. Sometimes it's just a matter of the referee having fresh eyes and a different perspective; a referee can point out that something which seems obvious to you may not be clear to others. Second, reviewing is a way to keep abreast of what's going on out there in the community. Third, reading articles and proposals and having to write reviews is intellectually stimulating - it gets me to think about new things and areas that aren't necessarily my primary interest.

When writing a report, I try to produce something that's actually useful to the authors (as well as the editors in the case of journal articles). I briefly summarize the main points of the paper or proposal to indicate that I've actually read it and understand the key ideas. For a paper, I emphasize my overall opinion of the work. Then I point out anything that I found unclear or any parts of the argument that don't seem supported by the data or calculations, with an eye toward what would improve the manuscript. I rarely reject papers out of hand, since I rarely get manuscripts to review that I think are hopeless (though some are submitted to inappropriate journals). On the other hand, it's pretty rare that I think something is absolutely flawless (though if my comments are minor I don't ask to see the paper again). I truly don't understand why some people submit two-sentence referee reports that are dismissive - this doesn't help anyone. I also don't understand why a small number of people can be venomous in reviews. Ok, so you didn't like the work for some reason - why get nasty? Just explain rationally why you don't think the paper is right. The point of refereeing is not to fire off insults under the shelter of anonymity - that's what blogs and internet forums are for.

Proposals can be more work. The big questions are usually (1) Does the PI clearly articulate the science or engineering question that is under investigation? (2) Is the plan well considered and likely to lead to good science? (3) How much of this is new and how much is completely incremental? (4) Did the PI(s) include everything that they were required to (e.g., description of prior work, for NSF discussion of outreach and education)? Grade inflation in proposal refereeing makes this process more painful as well. I am well aware that labeling a proposal as merely "good" is the kiss of death.

I have a tough time saying "no" to refereeing requests, and I need to get better at it. Prompt refereeing is important, and it's better to decline to review than to sit on a manuscript for two months. Still, good refereeing is definitely needed. I'm sure many readers have had the experience of a manuscript sitting with editors for a long time because it's tough to find qualified reviewers with the right expertise who also have enough time to review promptly. It is a shame that there isn't some intelligent mechanism for rewarding refereeing. It shows up as a line or so on your CV, even though it's arguably more valuable than serving on some university committees. Ahh well. Off to write some reviews.

5 comments:

DanM said...

I know that some IEEE journals actually give awards each year to "best referee" or "most dedicated reviewer" or something like that. More journals should do this, and they should tout it more prominently.

Charles Day said...

Physical Review has started a referee recognition program. Outstanding referees will receive a certificate and a lapel pin. I believe the selectees will be announced next week at the APS March meeting.

Doug Natelson said...

Dan, Charles - That's very good to know, and I do always appreciate the email I get from APS at the end of the year telling me the stats on the papers that I refereed. Tt would be nice if there was some way of recognizing what I think is a big chunk of the community who do this kind of service (rather than the very tip top handful), but every bit helps.

Anonymous said...

Make sure to show up at the Physical Review "meet the editors" session next week and drink their free wine...the most tangible benefit you will probably find!

Incoherent Ponderer said...

I am glad to hear you say that you rarely reject papers out of hand.
I rarely do that either: and I usually get pretty reasonable selection of papers for refereeing. I wonder if by not rejecting more papers out of hand, I further dilute standards of the journals.