Predictive Control of Speededness in Adaptive Testing (RR-07-02)
by Wim J. van der Linden, University of Twente, Enschede, The Netherlands
In computerized adaptive testing (CAT), the correct and incorrect responses of test takers to test items (questions) are used to select subsequent items for administration to the test takers that are matched as closely as possible to their ability level. One of the requirements of CAT is a pool of test items that can support this adaptation of test difficulty to test-taker ability in a way that is equitable for test takers of all ability levels. CAT item pools typically show a large variation in the amount of time their items require to produce an answer. In recent studies of item pools for a few large-scale adaptive tests, the author found that some items required 5–8 times as much time to answer as others. The potential danger in administering a CAT from such pools is that some test takers may run out of time because their selection of items is highly time-intensive, whereas others have ample time to complete the test. This phenomenon is known as differential speededness.
In this research, we explore the behavior of a method of controlling differential speededness in adaptive testing that is based on predictions of test taker response times for the available items in the pool. The items are then selected such that the sum of the actual times on the items already administered with the predictions of response times for the remaining items in the test meets the time limit.
In a study with an adaptive test simulated from an item pool for the Armed Services Vocational Aptitude Battery (ASVAB), the effectiveness of the method was evaluated for two different implementations: (1) updates given the actual response times on the items already administered and (2) updates that use the responses on these items as an additional source of information. Both methods succeeded quite well in removing differential speededness from the test.