Testlet-Based Adaptive Mastery Testing (CT-99-11)
by Cees A. W. Glas and Hans J. Vos, University of Twente, Enschede, The Netherlands
In mastery testing, the goal is to decide whether an examinee should be classified as a master or a nonmaster. Well-known examples of mastery testing include testing for pass/fail decision-making, licensure, and certification. The focus of this study was on a combination of sequential mastery testing (SMT) and adaptive mastery testing (AMT). In SMT, testing continues by selecting random items until a master or nonmaster decision can be made with prescribed expected loss. The main advantage of SMT is a shorter expected test length. In AMT, items are not selected at random, but to maximize information at the examinee’s current ability estimate. Generally, AMT results in better precision of classifying the examinees as master or nonmaster and a decrease in the cost of testing. A combination of sequential and adaptive mastery testing (ASMT) will be used in this study.
In a previous report by Glas and Vos, a general theoretical framework for ASMT based on Bayesian sequential decision theory and item response theory (IRT) was presented. The performance of ASMT was investigated for the one-parameter logistic (1PL) IRT model both for the selection of individual items and for small intact sets of items (testlets) from the pool. In the present study, the approach was generalized to testlet-based testing under the regular and a hierarchical version of the three-parameter logistic (3-PL) model. Unlike the regular model, which assumes local independence to hold simultaneously within and between testlets, the hierarchical model assumes greater similarity between responses in the same testlets than in different testlets.
The results showed that the expected losses associated with the decisions made by SMT and ASMT conditions were much smaller than for mastery testing based on a linear test. Also, ASMT produced considerably better results than SMT for the two 3-PL models relative to those for the 1-PL model in the previous study. Finally, the 3-PL model with the hierarchical structure that allowed for local dependence in testlets realized a substantial reduction in expected loss relative to the regular 3-PL model. The general conclusion from this study is that mastery testing can be conducted best in the ASMT format under a hierarchical 3-PL model for the item responses.