Modeling Nonignorable Missing Data Processes in Item Calibration (CT-04-07)
by Rampaign

Executive Summary

In a computerized adaptive test (CAT), each test taker responds to a limited subset of items from the available item bank, as test items are chosen throughout the testing session to match the most current estimate of the test taker’s ability level. The statistical theory of computerized adaptive testing is based on the assumption that the dependence between item selection and ability estimation does not result in any bias of the ability estimate. This assumption holds because, in general, item selection is dependent only on the test taker's responses to the items he or she has been administered, and the remaining items in the item pool may be ignored. We refer to this as the ignorability principle for missing data.

Using auxiliary information about test taker ability and allowing test takers to review and change answers to items they have previously answered during the testing session produces a violation of the ignorability principle for missing data. This violation may bias the statistics derived through the application of item response theory (IRT), a mathematical model that is used to analyze test data. However, violation of ignorability does not automatically lead to bias. In a previous Law School Admission Council (LSAC) report, it was shown that the estimation of test taker ability in computerized adaptive testing using auxiliary information and allowing item review does not lead to a gross inflation of bias. Nevertheless, it was also shown that violation of ignorability in item calibration can lead to severe bias in the estimates of item and population parameters. In principle, the magnitude of the bias can be so large that calibration results are virtually useless.

In this report, it is shown that the problem of nonignorable missing data in the calibration of CAT data can be handled by introducing an IRT model for the missing data indicator. In the first simulation study, it is shown that treating nonignorable missing data as though the ignorability assumption held leads to a significant increase in estimation errors; introducing an IRT model for the missing data process largely solves the problem. In the second simulation study, it is shown that the correction obtained using this IRT model is almost as good as the correction obtained when the factors determining the missingness of the data are actually observed.

Modeling Nonignorable Missing Data Processes in Item Calibration (CT-04-07)

Research Report Index