Friday, April 1, 2011

Tests Test But They Don't Predict

When I read "[a]t my South Bronx elementary school, we had a Teachers College consultant who encouraged us to 'teach tests as a genre of literature,'” I almost lost breakfast and lunch and broke the computer.  After all the Hulk is an acceptable role model.  I'd sooner treat Fox News as real news than entertain the idea that tests can be literature.

Although he doesn't take on NCLB or Race to the Top mandated testing, Jonah Lehrer provides some perspective on testing.  Citing the work of Paul Sackett, psychologist at the University of Minnesota, Lehrer points out that there are two types of performance.  "Maximum" performance occurs when people know they're being tested whereas "typical" performance occurs when people don't know they are being watched.

Lehrer asserts
We live in a society obsessed with maximum performance. Think of exams like the SAT and the GRE. Though these tests take only a few hours, they're supposed to give schools and companies a snapshot of an individual's abiding talents.
Or consider the NFL Scouting Combine, in which players entering the draft perform short physical and mental tasks, such as the 40-yard dash. The Combine is meant to measure physical ability; that's why teams take the results so seriously.
 Although high priced colleges and graduate schools and $9 billion industries like the NFL may like tests because these tests "don't take very long,". . . "can quantify many people," . . . .and "make assessment seem relatively straightforward," Lehrer writes,
Though the SAT does a decent job of predicting the grades of college freshmen—the test accounts for about 12% of the individual variation in grade point average—it is much less effective at predicting levels of achievement after graduation. Professional academic tests suffer from the same flaw. A study by the University of Michigan Law School, for instance, found that LSAT scores bore virtually no relationship to career success as measured by levels of income, life satisfaction or public service.

Even the NFL Combine is a big waste of time. According to a recent study by economists at the University of Louisville, there's no "consistent statistical relationship" between the results of players at the Combine and subsequent NFL performance.
 Citing Angela Duckworth, a psychologist at the University of Pennsylvania, Lehrer claims that "grit" and "self-control" "consistently predict levels of achievement." He concludes,
The problem, of course, is that students don't reveal their levels of grit while taking a brief test. Grit can only be assessed by tracking typical performance for an extended period. Do people persevere, even in the face of difficulty? How do they act when no one else is watching? Such traits often matter more than raw talent. We hear about them in letters of recommendation, but hard numbers take priority.

The larger lesson is that we've built our society around tests of performance that fail to predict what really matters: what happens once the test is over.
Secretary of Education Arne Duncan has asserted, however,  that the Race to the Top tests "will be an absolute game-changer in public education"  Further, and one must cue the original Six Million Dollar Man intro, these tests will "better, smarter assessments -- the kind of tests our teachers want and our students need.” 

Because we have the technology, cue the original Six Million Dollar Man intro again
Much of the testing will take place via computer, allowing teachers to get results more quickly. One coalition would use computer adaptive technology so that students at certain skill levels would skip to appropriate questions to more efficiently pinpoint their strengths and weaknesses.
Of course what has happened is that teachers are going to spend time teaching to a test; the kids will get bored; learning won't take place. Yet, under Race to the Top, "thousands of colleges have agreed use them as one indicator that students are ready for entry-level courses." The NFL makes $9 billion dollars a year.  It tests for easily measurable things like speed and strength.   All one needs to access these qualities is a stop watch and a a lot of weights.  Yet the NFL can't find a test to determine an athlete's opportunity for success.

Every bureaucrat who has forced NCLB's or Race to the Top's tests down schools' throats should be forced to read Lehrer's article every day for the next three years.  There won't be a test.  Instead, we'll see if the habit develops the "grit" necessary to understand that tests cannot predict success.

No comments: