A Bayesian Approach to Item Calibration and Evaluation of Parameter Drift (CT-00-02)
by Cees A. W. Glas, University of Twente, Enschede, The Netherlands

Executive Summary

In previous reports for the Law School Admission Council (LSAC), methods for evaluating possible differences between item parameter values during pretesting and the operational stage of a Computerized Adaptive Testing (CAT) program were investigated. These differences are often labeled parameter drift and can have various causes. For example, frequent exposure of items may affect their difficulty, as may motivational differences between examinees in the pretest and the operational test.

Earlier tests to detect item parameter drift proposed by the author were based on marginal maximum likelihood (MML) estimation of the values of the item parameters. Recently, however, a Bayesian approach has been proposed as an alternative to the MML framework, which allows for items with a more complicated response format. Applications include items with multiple raters, testlet structures, multidimensional latent abilities, and multilevel structures for the ability parameter.

In the present paper, the evaluation of parameter drift within this Bayesian framework was investigated. A number of Bayesian modification (BM) indices for the three-parameter normal ogive model are presented, which are generalizations of the modification indices proposed by the author for the MML framework. The strong point of the BM procedure is that many model violations for all items can be assessed without complicated, time-consuming computations.

The power of these indices was investigated in a series of computer simulations. As expected, the power was an increasing function both of the size of the examinee samples and the effect size (true amount of item parameter drift). Generally, the power was smaller than the power of the earlier procedure for the MML framework. However, the likelihood of Type 1 errors (false alarms) was virtually equal to zero. It was therefore concluded that the proposed modification indices could serve very well as quick and convenient caution indices, with significant results followed by a more detailed traditional analysis.

A Bayesian Approach to Item Calibration and Evaluation of Parameter Drift (CT-00-02)

Research Report Index