Estimation of Item Dimensional Measurement Direction Using Conditional Covariance Patterns (CT-98-02)
by Daniel Bolt, University of Illinois at Urbana-Champaign; Louis Roussos, Law School Admission Council; and William Stout, University of Illinois at Urbana-Champaign

Executive Summary

As proposed in the Computerized Law School Admission Test (LSAT) Research Agenda, the Law School Admission Council (LSAC) is currently investigating the advisability and feasibility of developing a computerized version of the LSAT. An important component of this research effort is the development of new dimensionality estimation procedures. For the purposes of this report, the dimensions of a test may be thought of as the number of statistically detectable skills that the test is measuring. A dimensionality analysis involves determining the number of dimensions (or skills) being measured by the test, the nature of these dimensions, and the degree to which the dimensions are correlated.

The current LSAT has been reliably estimated by several studies to have two dominant dimensions. One of these dimensions is associated with the analytical reasoning (AR) items, while the other is associated with the combination of logical reasoning (LR) and reading comprehension (RC) items. However, previous analyses have also demonstrated other more minor dimensions, for example, the item sets corresponding to the different AR and RC passages. Moreover, in a computerized adaptive test (CAT) setting, new minor dimensions may be introduced because of the new medium of administering the test or because of the adaptive way in which the items are administered. The influence of such minor dimensions are strictly controlled on the current LSAT so that no test takers are unfairly advantaged or disadvantaged. To ensure the continued control over these minor dimensions, new dimensionality estimation techniques should be developed for CAT data so that we can continue to monitor the influence of such dimensions in a CAT setting.

The purpose of the current paper is to develop a new more powerful dimensionality estimation technique. The new technique expands upon previously developed techniques that could only determine whether two items were measuring the same dimension by also measuring the degree of dimensional similarity between two items.
Through analysis of simulated data, the new technique is shown to be able to estimate dimensional similarity between items that previous techniques were unable to measure. Perhaps the most striking results with the new technique were demonstrated in an analysis of LSAT RC data. In this analysis the new technique not only resolved the conflicting results from two previous analyses, it also yielded a clear and compelling explanation as to why the conflict had occurred.

In conclusion, the new technique introduced in this paper shows much promise as a potentially powerful dimensionality estimation tool.
 

Estimation of Item Dimensional Measurement Direction Using Conditional Covariance Patterns (CT-98-02)

Research Report Index