A Method to Determine Targets for Multi-Stage Adaptive Tests (CT-02-07)
by Ronald Armstrong,Rutgers University, and Louis Roussos,University of Illinois at Urbana-Champaign

Executive Summary

The last two decades have seen paper-and-pencil (P&P) tests being replaced by computerized adaptive tests (CATs) for many standardized test administrations. CATs have several advantages as compared to conventional P&P tests. CATs determine the items to administer in real-time; thus, each form is tailored to the test taker’s skill level. A test taker’s responses to items are recorded during the test and a regularly updated estimate of the test taker’s ability is maintained. A CAT can acquire more information about a test taker’s ability while administering fewer items. Other advantages of CAT are immediate scoring, more frequent (flexible) administrations, and the ability to utilize constructed response items.

An MFS is an ordered collection of testlets that allows for adaptation based on a test taker’s ability while exposing a pre-set number of items and providing a reasonable number of possible forms. This test structure is a hybrid between the conventional P&P and the CAT where the test is divided into stages and a testlet is administered at each stage. A testlet found in an MFS fits specified target curves. The MFS contains multiple stages and consists of bins in which testlets are placed. The bins at a given stage are arranged in levels corresponding to ability classifications. Each form contains one testlet from each stage.

Unlike most CAT implementations, MFSs utilize Target Testlet Information Functions (TTIFs) and Target Testlet Characteristic Curves (TTCCs). These targets are based on item response theory. Target curves are common in P&P tests because they facilitate accurate test equating by providing parallel tests. As with P&P targets, the MFS targets are chosen to produce reliable tests while utilizing the item pool effectively. However, an MFS is not a linear test and decisions have to be made as to which items to administer. The targets for each bin and the routing rules are interrelated. Routing rules should depend on the targets and vice versa. Specific aspects of the routing methods will be reviewed here and used in target development. The purpose of this paper is to present a method to create targets for the bins of an MFS Design. The ability distribution of the population and an “ideal” item pool determine the targets.

A Method to Determine Targets for Multi-Stage Adaptive Tests (CT-02-07)

Research Report Index