Survey Results From Demonstration of the Preliminary Computerized LSAT Prototypes at the 1995-1998 Law School Recruitment Forums (CT-00-01)
by Kimberly A. Swygert and Andrea E. Thornton, Law School Admission Council
This paper is part of the ongoing research endeavor to investigate the feasibility and advisability of computerized testing (CT) for the Law School Admission Test (LSAT). Among the many issues surrounding this research is the development of an acceptable CT interface design. The development of the interface thus far can be divided into two research phases. The first phase of research is described in this paper, and consists of the development and demonstrations of seven LSAT CT prototypes from the years 1995 to 1998. The main purpose of this first development and demonstration phase was to expose potential law school candidates to these prototypes during the Law School Recruitment Forums and elicit their feedback.
A total of 27 prototype demonstrations were conducted during the Forums, which take place in major cities around the continental U.S. and Canada. These day- or weekend-long Forums draw thousands of candidates each year who attend free of charge for the purpose of obtaining information from LSAC and the attending law school recruiters and admission officers; thus, these candidates are the perfect subject pool for the CT demonstrations. The breakdown of ethnicity and sex percentages of the candidates who viewed the seven prototypes over the 27 demonstrations was fairly comparable to the percentages of LSAT test takers, although African Americans were somewhat over-sampled.
The seven prototypes that were used varied across the four years of demonstration, and changes were often made within years and in between Forums in response to candidates’ comments. All seven prototypes included 10 items from one or more of the three LSAT sections, Analytical Reasoning (AR), Logical Reasoning (LR), and Reading Comprehension (RC), and followed the same basic format of introductory screens, tutorial screens, example screens, item screens, and (for some versions) scoring screens. The modifications to the prototypes over the years generally took one of three forms: (1) expanding the tutorial, (2) adding new features or functions, and (3) developing innovative versions of the LSAT AR, LR, and RC item types. The data that were collected regarding these prototypes came in the form of surveys. The surveys were developed to gather a range of information including participants’ prior computer usage and their evaluation of the CT prototypes.
The other types of survey questions asked the candidates to give their opinions on a computerized LSAT in general or rate the specific prototypes they viewed. On the positive side, the prototypes tended to be rated as “Very Easy/Somewhat Easy” to use by a majority of the candidates, and the screen formats and features were also highly regarded. The welcoming pages that described the prototype study and gave some additional LSAC information were not seen as superfluous by the candidates but were instead highly rated.
On the negative side, it was seen that providing additional features as the prototypes were revised sometimes reduced the perceived usability of the prototypes. The highlighting feature was added because paper-and-pencil LSAT test takers tend to mark up passage text, but this feature was not seen as easy to use relative to the other features, perhaps because of the number of steps needed to activate it. Additional candidate concern was expressed over the need for reading lengthy text from the screen, as on the RC section, with 50.5% of the candidates reporting they would find that somewhat or very difficult, and 62.1% reporting they would definitely choose to print out the reading passage if they could do so. Over 70% of the candidates thought the font was too small. Most alarmingly, when test takers were asked to rate their comfort with taking a computerized LSAT both before and after viewing the prototype, they reported being less comfortable after viewing the prototype. This suggests that it isn’t safe to make the assumption that high computer literacy and comfort translates to high comfort with taking a high-stakes standardized test on the computer. It also suggests that the current prototypes should be revised to address this problem.
Despite these concerns, the results as a whole are encouraging for the plan to computerize the LSAT. More development and research is required. The next phase of CT research should take a step back and begin with revision of the current prototypes, followed by formal usability testing, in which the only feedback from the candidates is on each specific feature of the new prototypes. This will give a better picture of which features should be retained, perhaps with modification. Once the usability tests have confirmed that the software interface is usable, then some of the more important underlying concerns can be addressed, the revised prototypes can be formally field-tested, and hopefully, a more stress-free computerized LSAT can be produced.