Monday, November 29, 2010

Writing exams.

Writing (or perhaps I should say "creating", for the benefit of UK/Canada/Australia/NZ grammarians) good exams is not a trivial task.  You want very much to test certain concepts, and you don't want the exam to measure thing you consider comparatively unimportant.  For example, the first exam I ever took in college was in honors mechanics; out of a possible 30 points, the mean was a 9 (!), and I got a 6 (!!).  Apart from being a real wake-up call about how hard I would have to apply myself to succeed academically, that test was a classic example of an exam that did not do its job.  The reason the scores were so low is that the test was considerably too long for the time allotted.  Rather than measuring knowledge of mechanics or problem solving ability, the test largely measured people's speed of work - not an unimportant indicator (brilliant, well-prepared people do often work relatively quickly), but surely not what the instructor cared most about, since there usually isn't a need for raw speed in real physics or engineering.  

Ideally, the exam will have enough "dynamic range" that you can get a good idea of the spread of knowledge in the students.  If the test is too easy, you end up with a grade distribution that is very top-heavy, and you can't distinguish between the good and the excellent.  If the test is too difficult, the distribution is soul-crushingly bottom-heavy (leading to great angst among the students), and again you can't tell between those who really don't know what's going on and those who just slipped up.  Along these lines, you also need the test to be comparatively straightforward to take (step-by-step multipart problems, where there are still paths forward even if one part is wrong) and to grade.

Finally, in an ideal world, you'd actually like students to learn something from the test, not just have it act purely as a hurdle to be overcome.  This last goal is almost impossible to achieve in classes so large that multiple choice exams are the only real option.  It is where exam writing can be educational for the instructor as well, though - nothing quite like starting out to write a problem, only to realize partway through that the situation is more subtle than you'd first thought!  Ahh well.  Back to working on my test questions.


Anonymous said...

What course are you trying to write exam questions for? Perhaps I can offer some suggestions if you'd like? Not that I don't think you can write fine exam questions, but as you said, it is hard, and I often find that you gain insight from the various creative approaches that different instructors take to making their questions.

Anonymous said...

As a former student, I can say that your exams were definitely among the most reasonable I have ever taken. No part of a problem was so far fetched to seem impossible unless I completely forgot a concept. I can honestly say that on at least one occasion, making the intuitive leap needed to work through one of your multi-step problems left me with genuine feeling of pride and accomplishment. That's a big step from some of the other "soul-crushing" tests that I have taken. Keep up the good work! That small hint of discovery in an exam goes a long way toward showing students that physics doesn't have to be such a grueling experience all the time.

Douglas Natelson said...

Anon1, thanks for the offer, but I'm all set now. I'm teaching honors mechanics for first-year undergraduates.

Anon2, thanks for the kind words!

sylow said...

Hi Doug,
I am teaching the same course this semester. The mean was 12 out of 100 among chemistry majors. I gave 2 hours for 5 questions (2 of them were trivial questions) and I do not feel bad about it. Half of the class was gone after only one hour. Nice, huh?