Item Response Theory Parameter Estimation with Response Times as Collateral Information (RR-06-04)
Wim J. van der Linden
Rinke H. Klein Entink
Jean-Paul Fox
University of Twente, Enschede, The Netherlands

Executive Summary

The testing industry is always keen to give test takers more precise scores and to reduce the cost of item calibration. This research investigates a method for achieving this goal by making use of response time (RT) information in the item calibration process. Item calibration is the process whereby statistics used to describe the properties of the test questions (items) are calculated. Such calculations can become costly in that large samples of test takers are generally required to assure that the statistics are calculated with adequate precision. Reducing the number of test takers required for calibration amounts to reducing calibration cost. One of the ways of doing so is to make use of test-taker RTs in addition to their item responses (i.e., correct versus incorrect). In computer-based testing, RTs are automatically recorded, and this extra information is thus free.

A statistically proper way of making use of RTs is through a hierarchical model that involves both traditional item response theory (IRT) and RT parameters. (Note that IRT is a mathematical model that is used to analyze test data.) When estimating test takers’ abilities or calibrating the test items, it then becomes possible to “borrow” from the information in the RTs.

The model used in this project was the hierarchical framework for speed and accuracy developed in an earlier project for the Law School Admission Council. It is shown that when estimating the IRT parameters, the information in the RTs allows us to infer an empirical prior distribution for each IRT parameter from the RTs. Unlike the typical common prior for all person or item parameters in traditional IRT estimation, this prior is individual and better reflects the true parameter values as well as our prior uncertainty about them. A simulation study showed that when using RTs as collateral information in this way, under realistic conditions, a reduction in the estimation error for the ability parameters of about 25% is possible.

Item Response Theory Parameter Estimation with Response Times as Collateral Information (RR-06-04)

Research Report Index