Implementing Content Constraints in Alpha-Stratified Adaptive Testing Using a Shadow Test Approach (CT-01-09)
Wim J. van der Linden, University of Twente, Enschede, The Netherlands
Hua-Hua Chang, National Board of Medical Examiners

Executive Summary

This paper addresses the problems of item compromise and content balancing in computerized adaptive testing (CAT). An effective method of protection against item compromise is to partition the item pool into strata that are based on the discrimination power of the items, and then to expose test items uniformly from each strata during the test. This method has been termed alpha-stratified adaptive testing.

Content balancing in CAT is possible through the imposition of content constraints on the item selection procedure. In the method of content balancing called shadow testing, an entire content-balanced test form is assembled after the administration of each test item. The newly assembled form automatically includes all items that have already been administered to the test taker. The item from the newly assembled test form that best matches the current estimate of the test taker’s ability is then administered to the test taker. This process continues until an entire content-balanced test form has been administered.

This research showed that alpha-stratified adaptive testing can be implemented through a shadow test approach. The result is an item selection algorithm that both balances test content and prevents the overexposure and underexposure of the items in the pool.

An example from the Law School Admission Test (LSAT) was used to demonstrate the advantages of this method. In total, 65 constraints derived from the content specifications of the paper-and-pencil version of the LSAT were combined with the new constraints that introduce alpha-stratification into the item selection procedure. Adaptive tests of 50 items were simulated from the item pool and the effect on the exposure rates of the items as well as the errors in the estimation of ability were studied. The results showed that the item selection algorithm meets the expectations and is practically feasible.

Implementing Content Constraints in Alpha-Stratified Adaptive Testing Using a Shadow Test Approach (CT-01-09)

Research Report Index