The Effect of COMC on the Stability of LSAT Item Parameter Estimates (SR-93-01)
by Lynda Reese and Robert McKinley

Executive Summary

In item calibration using LOGIST (Wingersky, Patrick, & Lord, 1987), when the program determines that it cannot accurately estimate the c-parameter for a particular item due to insufficient information at the lower levels of ability, an estimate of the c-parameter, called COMC, is obtained by combining all such items. The stability criterion for a given item is b - 2/a, where a and b are the estimated parameters for that item. This value represents the ability level for which the proportion of correct responses is only slightly higher than the lower asymptote of the item response function. If there are few test takers whose abilities are lower than this level, then it is not possible to obtain a stable estimate of the c-parameter for the item. A minimum cutoff value called CRITFIXC is set, and if the value of b - 2/a is less than the value of CRITFIXC, the c-parameter is not estimated for that item and the value of COMC determined by combining all such items is assigned. The LOGIST manual suggests that the value of CRITFIXC be adjusted depending upon the sample size, with CRITFIXC being higher for smaller samples (Wingersky, Patrick, & Lord, 1987).

This aspect of the LOGIST program poses an interesting problem for testing programs that employ common item scaling to place different calibrations on a common scale. First, if different values of CRITFIXC are used for two calibrations, some items will have the c-parameter estimated in one calibration but not the other. This may very well produce a different c-parameter estimate for these items for the two calibrations, and since the COMC value tends to fall near the average of the estimated c-values, for high and low c-items, the difference between the two calibrations could be quite large. This appears to violate the IRT assumption of parameter invariance, since in one case the item c-parameter is estimated and in the other case the somewhat poorly defined parameter being estimated is specific to a set of items. Not only has the nature of the parameter being estimated changed, but the parameter being estimated by the COMC estimate is no longer invariant with respect to the other items included on the test, an assumption critical to applications such as adaptive testing. Moreover, this problem may be compounded by the correlated estimation errors among the IRT parameters. Marked changes in the c-parameter estimates such as may be observed when the c-parameter is estimated once but set to COMC in the second calibration may be accompanied by changes in the a- and b-parameter estimates, which perhaps could result in additional scaling errors.

Given these concerns, the purpose of this investigation was to evaluate the effect of using COMC on the stability of IRT parameter estimates. In this study, two LSAT Reading Comprehension pretest sections were administered twice, once in June and again in December of 1991, and new parameter estimates were obtained for each of these administrations. Using the same value of CRITFIXC, different items were set to COMC for the two administrations due to sample fluctuations. Items were categorized as to their pattern of c-estimation, with the three categories being (1) c-parameter estimated at both administrations, (2) c-parameter fixed at one administration, and (3) c- parameter fixed at both administrations. The purpose of this study was to determine the effect of the pattern of c-estimation on the stability of the item parameter estimates obtained using LOGIST.

The Effect of COMC on the Stability of LSAT Item Parameter Estimates (SR-93-01)

Research Report Index