Violations of Ignorability in Computerized Adaptive Testing (CT-04-04)
by Cees A. W. Glas, University of Twente, Enschede, The Netherlands

Executive Summary

In a computerized adaptive test (CAT), each test taker responds to a limited subset of items from the available item bank, as test items are chosen throughout the testing session to match the most current estimate of the test taker’s ability level. The statistical theory of computerized adaptive testing is based on the assumption that the dependence between item selection and ability estimation does not result in any bias of the ability estimate. This assumption holds because, in general, the missing-data process is ignorable, that is, item selection is dependent only on the test taker’s responses to the items he or she has been administered, and the remaining items in the item pool may be ignored. There are at least two cases for which this assumption does not hold.

1. If item selection is also based on an impressionistic estimate of the test taker’s ability, or on some other covariates that are not explicitly modeled. This case occurs if the first item in the CAT is chosen based on some prior information about the test taker or if the CAT is stratified for the purpose of content balancing.

2. If item review is allowed and test takers change their responses to previous items. If test takers change earlier responses, the item-selection design is no longer completely determined by the observed responses.

The violation of the ignorability assumption for the missing-data process does not automatically lead to bias. In this report, the following two different situations have been examined:

1. Estimation of the ability parameters based on auxiliary information about the test taker’s ability and the allowance of item review. Both analytically and through simulation studies, it was shown that this case of violation of the ignorability assumption did not lead to a gross inflation of bias.

2. Calibration of item and population parameters using a method known as maximum marginal likelihood estimation. Through simulation studies, it was shown that this case of violation of the ignorability assumption did result in bias. An analytical explanation of the result is given.
 

Violations of Ignorability in Computerized Adaptive Testing (CT-04-04)

Research Report Index