A Polynomial Logistic Regression Approach to Graphical Differential Item Functioning (CT-99-06)
by Lori D. McLeod, David J. Scrams, and Louis A. Roussos

Executive Summary

Performance on the Law School Admission Test (LSAT) and similar admissions tests may influence high-stakes decisions. The importance placed on these test scores demands high quality, so the Law School Admission Council (LSAC) carefully monitors the LSAT. One of the many statistical item characteristics examined is evidence of differential item functioning (DIF). DIF exists if an item operates differently for two groups in the population, after taking into account any difference in overall proficiency, thus advantaging a particular subgroup.

The most popular DIF procedures are global measures. Global DIF measures provide a single index which represents the overall difference in an item’s performance between two groups. Currently, LSAC’s primary DIF statistic is the Mantel-Haenszel statistic. The Mantel-Haenszel is relatively easy to implement and appropriate for items that exhibit uniform DIF, that is, DIF in which one group is advantaged by a constant amount on a log-odds-ratio DIF scale across the entire proficiency scale. However, some items may exhibit non-uniform DIF. Non-uniform DIF exists when the amount of advantage differs by location along the proficiency scale. A special case of non-uniform DIF is when one group may be advantaged within a low-ability range, and another group may be advantaged within a high-ability range. For these items, the global measure does not provide information about the location(s) of DIF.

Local measures of DIF have been the product of concern over proficiency-specific advantages. These approaches offer information about the location, as well as the extent of DIF present. Graphical display, along with local DIF measures, offers a relatively new way to evaluate DIF. The current work investigates the use of polynomial logistic models to measure local DIF.

Polynomial logistic models are suggested for use with pretest items as an alternative to the more complicated three-parameter logistic (3PL) model used operationally at LSAC. These models are used to estimate the odds-ratio curve for two groups. Simultaneous confidence bands indicate the uncertainty in the estimate. To demonstrate how the new method is applied and to begin to investigate its behavior, the approach is compared to the Mantel-Haenszel classifications for a set of LSAT pretest items. The results from the new approach agree with those of the Mantel-Haenszel for most items. The results differ when evidence of non-uniform DIF is present.

A Polynomial Logistic Regression Approach to Graphical Differential Item Functioning (CT-99-06)

Research Report Index