Item Parameter Calibration of LSAT Items Using MCMC Approximation of Bayes Posterior Distributions (CT-00-05)
by Douglas H. Jones, Mikhail Nediak, Rutgers, The State University of New Jersey, Department of Management Science and Information Systems

Executive Summary

Item calibration is historically one of the more difficult and important psychometric activities. Forty years ago, psychometricians turned to modern statistical methods to overcome the deficiencies in classical test theory, developing computationally tractable methods based on maximum likelihood (ML) techniques, a statistically sound body of knowledge. Other sound statistical methodology is based on Bayesian methods, which are much older than maximum likelihood. However, Bayesian methods were computationally intractable, until fifteen years ago. In the mid-eighties, a mathematical algorithm was discovered that could make Bayesian methods computationally feasible for a wide class of statistical estimation problems, including item calibration. This algorithm is called MCMC and stands for Markov Chain Monte Carlo.

There is commercial item-calibration software based on the ML approach, BILOG. The main advantage of MCMC over BILOG is that MCMC promises to yield far more information about the quality of item-parameter estimators than BILOG. BILOG is currently used at Law School Admission Council (LSAC).

This paper reports research on MCMC methods for calibrating Law School Admission Test (LSAT) items. It is the first study to compare established ML methodology with fully Bayes-MCMC methods based on real data. Using actual LSAT data, this study shows that MCMC calibration replicates or surpasses item calibration methodology as practiced at LSAC.

An additional goal of this research was to provide production-level item calibration software, written in the C++ language, for computer system purposes and online item calibration. The main advantage to LSAC having proprietary software is that it could include this software in future online computer systems for computerized adaptive testing (CAT) administration, should a decision be made to implement computerized testing for the LSAT.

Item Parameter Calibration of LSAT Items Using MCMC Approximation of Bayes Posterior Distributions (CT-00-05)

Research Report Index