An Integer-programming Approach to Item Pool Design (CT-98-14)
by Wim J. van der Linden, Bernard P. Veldkamp, University of Twente and
Lynda M. Reese
Recently, a body of research on the problem of optimal test assembly has been developing. Researchers have taken various approaches to the problem of automatically assembling a number of test forms from a pool of test items (questions). The result is a prespecified number of test forms that optimally meet a predefined set of test specifications. These test specifications may include a balance of such things as test content, answer key distribution, word count limitations, and statistical specifications.
While many of the approaches to optimal test assembly have been very successful, the results of these methods are limited by the composition of the item pool to which they are applied. While a method of optimal test assembly may result in test forms that are optimal for the item pool, these forms may be unacceptable in important ways if the item pool is lacking. For example, the item pool may contain items of the required content characteristics, but the items may be too difficult or too easy. The end result could be a content-balanced test form that is inappropriate in terms of difficulty level. Also, an item pool may appear to be large, but may contain many items that will never be selected for inclusion on a test form. In such a case, the size of the pool is deceiving from the point of view of test assembly.
The method described in this paper presents an integer-programming approach to item pool design. The result of this approach is a document, referred to here as a "blueprint," specifying what attributes the items in a new item pool or an update to an existing pool should have. The blueprint is specified to allow for the assembly of a prespecified number of test forms. The solution is optimal in that the efforts needed to achieve this item pool are minimized. The number of unused items in the pool is minimized as well.
This study addressed the definition of an item pool to support the assembly of a prespecified number of paper-and-pencil test forms. Paper-and-pencil test assembly represents a simpler problem than the selection of items for inclusion in a computerized adaptive test. Future research on this problem will address adaptive testing. This approach is very promising for the monitoring of a computerized-testing item pool.