LSAT Item-Type Validity Study (TR-98-01)
by Louis A. Roussos and Lynne L. Norton
This study was conducted to evaluate the predictive validity and the interrelationships of the current LSAT item types. The current LSAT consists of three item types: analytical reasoning (AR), logical reasoning (LR), and reading comprehension (RC). Even though the correlation of overall LSAT scaled score with first-year law school grade point average (FYA) is conducted on a regular basis at LSAC, the separate correlations for each of the three item types have only rarely been studied. The interrelationships of the item types have been studied in previous dimensionality analyses of the entire test-taking population for particular LSAT administrations, but no analyses had been conducted to verify that the nature of these relationships is maintained in the applicant pools of the law schools.
The primary purpose of this paper is to provide this information. Using data associated with the 1994, 1995, and 1996 first-year classes in U.S. law schools, correlations of AR, LR, RC, and LSAT total raw scores with FYA were calculated for individual law schools. Also, the correlations of AR, LR, and RC raw scores with each other were calculated for the applicant pool of each law school. All FYA correlations were corrected for restriction of range to provide an estimate of what the correlation would have been if all the applicants had entered a particular law school. Such a correlation is a more accurate measure of the true predictive power of a test score. All correlations involving test sections of different lengths were corrected so that the correlations reflected the value that would have occurred had the individual test sections been the length of the entire LSAT (101 items). This correction allowed for statistically fair comparisons among item-type validity coefficients and between those coefficients and the total test score validity coefficient.
The major results of this paper indicate that each of the operational LSAT item types has a substantial correlation with FYA, and that each is needed to obtain the reported overall correlation because no two item types are perfectly correlated with each other. The item type with the greatest predictive validity was LR with a validity coefficient of 0.483. Even though RC with a validity coefficient of 0.430 had the next greatest value, AR with a validity coefficient of 0.340 makes the greater additional contribution to the validity coefficient of the entire test as it had a much lower correlation with LR than did RC. After adjusting for the amount of predictive validity accounted for by their correlations with LR, the remaining degree of correlation of AR with FYA was 0.124 whereas the corresponding value for RC was 0.107.
The results also verified that the interrelationships among the item types in the law school applicant pools were the same as those previously found for all test takers for a fixed LSAT form. The results verified that LR and RC remain very highly correlated (0.760), while AR is less correlated with LR or RC, but still strongly so, with correlations of 0.510 and 0.459, respectively.
The implications of this study are that all three item types have substantial correlations with FYA and should all remain as part of the LSAT to maintain the current level of overall predictive validity.