Effects of Multidimensionality on IRT Item Characteristics and True Score Estimates: Implications for Computerized Test Assembly (CT-97-06)
by Xiang-Bo Wang, Vincent Harris, and Louis Roussos
In an earlier study by Wang, it was reported that the 18,462 test takers of the 1989 Advanced Placement Chemistry Examination, when asked to select three essay questions to write on from among five possible choices, chose in a seemingly diverse fashion. Their average scores on the more popularly chosen essays were lower than those on the less popularly chosen counterparts. Although such findings have been confirmed by other studies, the causes for the disparity between the popularity of choices and performance on those choices is not known. As a continuation from this earlier study, the purpose of this research is to uncover the psychological processes that influence test takers' choices.
This present study, based on 618 students in Hawaii, an experiment that incorporated a mini Advanced Placement Chemistry Test, and a related questionnaire that tapped into students' perception of item difficulty and similarity, has revealed three major findings concerning how and why test takers chose the constructed response (CR) items as they did. First, by asking the 618 Hawaii students to choose three of the same five essay questions of the 1989 AP Chemistry Examination, this study found that this independent group of Hawaii students virtually replicated the entire choice pattern of their 1989 national counterparts. This indicates an inherent psychological process underlying what appears to be haphazard choice patterns.
What is the underlying psychological process? By asking the Hawaii students to rate the difficulty of essay items, this study found, through unidimensional scaling analyses, that students' perceptions of item difficulty can completely predict the choice combinations and choice popularities of the essay items. Essays perceived as easier by the students were chosen more frequently, even though they might not be truly easier. The students tended to associate familiarity with easiness. That is, the more familiar the items, the easier students viewed them to be.
Moreover, by asking the Hawaii students to evaluate the similarities of the essay items, this study revealed that the students' perception of essay likeness could further explain the choice of essay combinations. Test items whose contents reflected similar curricular instruction or exposure tended to be chosen together more often.