A Model for Optimal Constrained Adaptive Testing (CT-97-07)
by Wim J. van der Linden, University of Twente and Lynda M. Reese, Law School Admission Council
The concept of computerized adaptive testing (CAT) was developed to deal with the statistical aspects of ability testing. The test generally starts out with a question of approximately average difficulty. Based on a test taker's response, subsequent items are chosen that are more appropriate for the ability level of the test taker. In applying CAT within a large-scale testing program, the selection of questions for administration to a test taker cannot be based solely on item difficulty, as described above. Constraints must be imposed on the selection of items that assure that every test taker receives a test that appropriately covers the domain of content the test proposes to measure. Issues such as blended reading load and the proper distribution of answer keys must also be addressed. The goal is that a CAT that incorporates these additional constraints should still provide the reduced test length and improved precision that is promised by this technology.
In this paper, an adaptive testing procedure is proposed in which the content distribution, reading load, and answer key distribution of the test are controlled by explicit constraints imposed on the item selection process. The process begins by assembling a full test that meets the constraints and provides the best measurement for the initial ability estimate for the test taker. The item from this full test that best matches the ability level of the test taker is then chosen for administration to the test taker. After the first item is presented to the test taker and scored, the ability level of the test taker is updated. A full test is then reassembled that includes the item already administered, is appropriate for the updated ability estimate, and meets all of the constraints. The question that is most appropriate for the current ability estimate of the test taker is then selected from the newly assembled test. This process continues until a full test has been administered. This procedure assures that the full test will meet all of the necessary test assembly constraints and will be appropriate for the ability level of the test taker.
A simulation study using a pool of 753 Law School Admission Test (LSAT) items was run to assess the practical feasibility of this procedure. Results indicated that the computer processing time needed to reassemble the test and select the next item was always between one and two seconds. Also, for realistic test lengths, the effect of imposing the set of constraints on the item selection process appeared to have no discernible effects on the statistical properties of the final ability estimate.