A Comparison of Item-Selection Methods for Adaptive Tests With Content Constraints (CT-04-02)
by Wim J. van der Linden, University of Twente, Enschede, The Netherlands

Executive Summary

In adaptive testing, items are selected for an individual test taker with the goal of administering a test that is, as closely as possible, tailored to the ability level of that test taker. The selection is sequential in that one item is selected at a time. At the same time, adaptive tests typically have to meet a large number of content constraints, and this requirement is solved more naturally by simultaneous item selection.

In this project, the three main item-selection methods in adaptive testing for solving this dilemma were investigated: (1) the spiraling method (SM), which moves across content categories of items in the item pool in a manner that is proportional to the numbers of items needed from them during item selection, (2) the weighted-deviations method (WDM), which selects the items using a projection of a weighted sum of the attributes of the entire test, and (3) the shadow test approach (STA), which selects the items based on a projection of the actual items in the entire test.

An empirical comparison among the methods was conducted for an adaptive version of the Law School Admission Test (LSAT). It turned out to be impossible to implement the SM due to the complicated nature of the set of content constraints for the LSAT. The WDM and STA showed equally good item-exposure rates. The STA always met the constraints for the LSAT, but the WDM showed serious violations of some of the constraints as well as a larger degree of bias and greater inaccuracy of the ability estimate.

A Comparison of Item-Selection Methods for Adaptive Tests With Content Constraints (CT-04-02)

Research Report Index