Observations about NSF panels

I just returned from an NSF review panel. For those of you that don't know, the NSF peer-reviews all grant proposals, and many programs have a panel review system: an NSF program officer will email you or call you and ask if you are available on such-and-such a date for a panel. If you're willing to do it, you say "yes", and then you're given electronic access to about 8 proposals to review. You do your reviews at your leisure over the next few weeks and upload them via the impressively good web-based system, Fastlane. Then you go to Washington (really Ballston, VA) to NSF headquarters at the appointed time, and sit down in a room with about 10 other reviewers plus the program officer. Everyone has a laptop in front of them, and now you can see each other's reviews. You go through all the proposals (usually about 30 for the whole panel), discuss and compare notes, and in the end write up panel summaries of the reviews that eventually get sent to the proposal writers (PIs, or principal investigators). Typically the proposals are grouped into three categories: "highly recommended" (will actually get funded), "recommended" (on the edge, and may get lucky if there's enough money available), and "not recommended" (no chance). These days the yield of "highly recommended" is 5-15% at NSF, depending on the program. The government pays your travel, and you get a nominal stipend that covers hotel and meals.

A few observations:

• The main reason to do this is one of citizenship: you can really see the process work, learn how to improve your own proposals, and reassure yourself that the people reviewing the grants have a clue.
• Why are there never people from top 15 schools at these panels? Are they really only involved in things like site visits for major center proposals? Seriously, I've never seen someone from any Ivy League school, any of the UC schools, MIT, CalTech, Stanford, Illinois, etc. on one of these things. Are they really all that much busier than me?
• It's painful when someone is on a panel that is not technologically literate enough to handle the web-based system.
• It's equally painful when someone bails at the last minute, doesn't review their share, and doesn't show up.
• This is still the best system around. Scary.