Robustness of Person-Fit Decisions in Computerized Adaptive Testing (CT-04-06)
Rob R. Meijer, University of Twente, Enschede, The Netherlands

Executive Summary

Several statistics have been recommended by researchers for identifying test taker responses to test items that are different from what would be expected, given what is known about the characteristics of the items and the estimated ability level of the test taker. Several of these statistics, often called person-fit statistics, are used for evaluating test taker responses to an entire string of items simultaneously. These statistics allow us to conclude that a particular item response theory (IRT) model either does or does not fit a person’s set of responses to items. (Note that IRT is a mathematical model used to analyze test data.) In this sense, these statistics are for use in a global method that only allows us to identify misfitting responses; that is, they do not help us to identify the type of behavior that caused the misfit.

Fortunately, we also have statistics, termed local statistics, that allow us to diagnose the misfit. Such methods may allow us to evaluate if the misfit was caused by violations of one of the assumptions made in applying IRT to the analysis of test data. One such assumption is that of unidimensionality, which requires that a test measure only one ability. This paper focuses on person-fit statistics developed for checking the unidimensionality assumption. Because test data may not be unidimensional, it is worth investigating the effect of unidimensionality violations on the ability of person-fit statistics to identify violations of this assumption.

We applied both global and local person-fit statistics to multidimensional test data from adaptive testing. As may have been anticipated, the results show that some statistics are more robust to unidimensionality violations than others. The context in which certain methods are more useful than others is indicated.


Robustness of Person-Fit Decisions in Computerized Adaptive Testing (CT-04-06)

Research Report Index