Representing Response-Time Information in Item Banks (CT-97-09)
by Deborah L. Schnipke and David J. Scrams

Executive Summary

As the Law School Admission Council (LSAC) considers computerizing the Law School Admission Test (LSAT), one of the advantages to keep in mind is the availability of item response times, which provide an entirely new type of information about items and test takers. In addition to knowing the accuracy with which test takers answer an item, in computer-administered tests we can investigate the amount of time test takers spend on each item. Researchers have begun to investigate uses of response times for a variety of research topics in the testing field, but to make use of response times in an operational test administration, response-time information about each item needs to be stored in an item bank. Currently there are no guidelines on how to do this. Storing a large number of response-time statistics would be impractical, but simply storing the mean and standard deviation is not sufficient because response times are positively skewed.

The goal of the present study was to develop a method for summarizing response-time information both accurately and concisely. Specifically, we wanted to be able to characterize the entire response-time distribution in terms of a small number of item parameters. Characterizing the entire response-time distribution is necessary so that any desired response-time characteristic can be easily calculated (e.g., median, mean, 95th percentile, dispersion). Doing so with only a few parameters is necessary for easy storage in an item bank.

In this preliminary investigation, we modeled item response times using several statistical distribution functions used by previous researchers to model response times in the testing field. We randomly separated response times to operational items into two samples. We modeled response times in the first sample with the various distribution functions and evaluated the fits. We then fit the models to the second sample using the parameter estimates from the first sample. This allowed us to examine how well the parameter estimates generalize to a new sample. The lognormal distribution provided very good fits for both samples (better than the fits provided by other distribution functions), and the lognormal distribution has only two parameters. Storing these two parameters for every item provides an accurate and concise summary of response-time information.
 

Representing Response-Time Information in Item Banks (CT-97-09)

Research Report Index