Assessing the Dimensionality of Simulated LSAT Item Response Matrices with Small Sample Sizes and Short Test Lengths (CT-96-01)
by André F. DeChamplain
The assumption of unidimensionality must be met in order to legitimately use common IRT models. The validity of score-based inferences rests largely on the extent to which it can be shown that the dimensional structure underlying a test is consistent with the blueprint. Little research has been undertaken to examine the behavior of dimensionality assessment procedures in conditions similar to those encountered in small volume administrations. The purpose of this study was to examine empirical Type I error rates and rejection rates for three dimensionality assessment procedures with data sets simulated to reflect short tests and small samples.
The TESTFACT G2 difference test suffered from an inflated Type I error rate with unidimensional datasets whereas the approximate X2 statistic based on a NOHARM analysis did not. Rejection rates with simulated two-dimensional data sets were high for both procedures. The behavior of the G2 difference test was highly influenced by the independent variables manipulated, which was not the case for the approximate X2 statistic. The implications of these results for small volume administrations are discussed.