A Classical Test Theory Perspective on LSAT Local Item Dependence (SR-96-01)
by Lynda M. Reese, Law School Admission Council

Executive Summary

In the analysis of individual Law School Admission Test (LSAT) questions and the assembly and evaluation of LSAT forms, item response theory (IRT) is applied. IRT is a mathematical model that relates the probability that a test taker will answer a test question correctly to the ability level of the test taker. In applying IRT, a formal assumption of local item independence is made. This assumption states that, once the ability level of the test taker is accounted for, the responses of the test taker to individual items on the test should be statistically independent. In other words, test taker ability should be the only factor contributing to the test taker’s performance on an item.

In a test taking situation, many circumstances arise that cause the local item independence assumption to be violated to some degree. For instance, in the Reading Comprehension section of the LSAT, several test questions are based on a common reading passage. Here, regardless of a test taker's reading comprehension ability, individual performance on all of the questions related to a particular passage may be enhanced or hindered by a prior level of knowledge of the subject matter of the passage. To the extent that a factor such as this causes a test taker to perform similarly on certain test questions, the test is said to exhibit some degree of local item dependence (LID).

Because local item independence is formally assumed in applications of IRT, LSAT research related to the local item independence assumption has centered primarily around IRT outcomes and measures. However, a category of statistics commonly known as classical statistics is also applied for certain purposes in the analysis of LSAT questions and test forms. Among the classical statistics applied in the analysis of LSAT data are an item/test correlation statistic (called the r-biserial), an index of reliability, and the percentile rank associated with each LSAT score. The r-biserial is an index of item discrimination used as an indicator of how well a test question distinguishes between more and less able test takers. This statistic is a common index of the quality of a test question. The reliability coefficient, calculated for each form of the LSAT, is an indicator of how consistent an individual's test score would be if they were to take the same form of the test many times. The percentile rank associated with a particular test score is the percent of test takers falling below that test score. Percentile ranks are routinely provided to test takers and law schools.

Although local item independence is not formally required for classical statistics, the impact of LID on these measures should be better understood. That is, to the extent that these statistics are influenced by the unknown effects of LID, the decisions made and procedures carried out using these statistics may be compromised. Therefore, this study extends past LSAT research related to the IRT local item independence assumption into the realm of classical test theory. Initially, results from the LSAT and two other tests were investigated to determine the approximate state of LID found in actual test data. Based on these analyses, four levels of LID (zero, low, medium, and high) were defined and associated data sets generated. Here, the medium LID level was defined to represent the LSAT. The classical statistics described above were studied in order to determine the effect of LID on these measures.

The results indicated that for extreme cases of LID, the discrimination power of individual items and the reliability of the total LSAT are overestimated. Percentile ranks were also clearly affected by the introduction of a high level of LID, indicating that the impact for individual test takers should be of concern. Because the LID became problematic only at the most extreme level simulated, the less than extreme level of LID typically displayed by the LSAT is probably not problematic with respect to these particular outcomes.

A Classical Test Theory Perspective on LSAT Local Item Dependence (SR-96-01)

Research Report Index