Simple Nonparametric Checks for Model Data Fit in CAT (CT-01-04)
by Rob R. Meijer, University of Twente, Enschede, The Netherlands

Executive Summary

The development and operational use of computerized adaptive testing (CAT) depends heavily on the availability of item response theory (IRT) models. This fact is not surprising, because one of the key features of IRT models is separate parameters for the properties of the items and the ability of the examinee. Due to this feature, the examinee’s ability level can be estimated from different sets of items, and items can be selected to be optimal at ability estimates. Recent developments in nonparametric IRT, however, suggest that techniques derived from this field may also contribute to the improvement of the psychometric quality of CAT. An important feature of these techniques is that they are based on very weak assumptions and therefore nearly always apply.

The aim of this paper is to investigate the possible use of nonparametric IRT in CAT. In particular, we explore the use of some simple diagnostics statistics derived from nonparametric IRT. As shown by numerical examples, these statistics are able to check basic properties of item response functions such as nondecreasing probability of success as the ability of the examinees increases and whether the ordering of the item difficulties in the pool is the same for each examinee. Checks of the latter property are important to finding out, for instance, if the ability measured by the CAT is curriculum-independent.

Simple Nonparametric Checks for Model Data Fit in CAT (CT-01-04)

Research Report Index