Talking with a lay audience about whether different kinds of student assessments are credible and useful, Kate Jamentz of the Western Assessment Collaborative at WestEd likes to use this little test-taking fiction:
Crash Gordon has been enrolled in Fly-by-Nite Pilot School for three weeks. The school promises that by successfully completing this course, Crash will be ready to pilot 747 commercial jets. Crash has been told that Fly-by-Nite is a highly regarded school. Its ads report that nearly 95 percent of its graduates score above average on the final exam, and its tuition costs are low. Five years ago, Fly-by-Nite replaced its expensive flight simulators with textbooks and films that explain in detail how to fly a plane.
Today is Crash’s midterm exam. If he passes with a grade higher than 50 points he can skip the rest of the course and get his license right away. If he scores from 25 to 50 he must repeat the course, and if he doesn’t finish or scores below 25, he will be kicked out. The stakes are high.
Crash opens the exam and finds 100 multiple-choice questions that he needs to answer in 30 minutes. One third of them are about the parts of a plane; another third ask about how to read a flight schedule; the final third cover the dress code for pilots. None of these topics has been covered in the course. About half the questions in each section are in French or Spanish-because 747s usually fly international routes, Crash figures. The last questions asks for a brief written answer to the question, “Who has most influenced your life as an aviator? Explain.”
Crash is not daunted. He realizes that he has to answer quickly, so he fills in the answers according to a pattern: every fourth question will be “A,” every third question “B,” and so on. His teacher, Mr. Soar, will be scoring the exam, so Crash uses the essay question to explain how much Soar’s teaching has meant to his career. He finishes the exam with three minutes to spare.
When the exam scores are posted, Crash is elated: his strategy has paid off. He got 29 points, two points above the average score, which was 27. He leaves eager to improve his score next time and become a pilot-so he goes to the airport and sits near the crew lounge, where he knows he can learn more about the pilot dress code.
“How comfortable will you be flying with Crash if he gets his pilot’s license next time?” Jamentz asks the group. “Why? What does this story suggest about assessment information that we trust and can use?” Participants go on to hash out the qualities that make an assessment valid, fair, credible, and useful, then hold up their district’s measures against those definitions. “It’s a way to quickly launch a rich and concrete discussion,” Jamentz notes, “in terms people can easily relate to.”
The Western Assessment Collaborative at WestEd regional educational laboratory is at 730 Harrison Street, San Francisco, CA 94107; (415) 241-2704 (415) 241-2704 .