Using the Internet to Survey College Students About Their Law School Plans (RR-01-04)
Very little is known about the potential pool of law school applicants from which eventual law school attendees come. This is primarily due to the difficulty of collecting systematic information from the college students that make up this group. Large scale surveys of college students are both difficult and expensive and are frequently limited by their cross-sectional design. The increasing technological sophistication of college students and recent improvements in electronic surveys raises the question of whether the Internet—via e-mail and the web—can be used to conduct a large scale survey of the postgraduate plans of college students. In 1998, the Law School Admission Council provided funding for the Career Plans and Undergraduate Attitudes Project to assess the feasibility of such a survey. Part I of this report describes the organization and structure of the Career Plans and Undergraduate Attitudes Project and the results from the first-year survey. Part II examines the results from the second-year survey.
The Career Plans and Undergraduate Attitudes Project was established to conduct a panel study of the postgraduate plans of a single cohort of students from three separate universities. The 1998 cohort of freshmen at three universities would be interviewed via e-mail and the web during the fall semester of each year. A panel study collects information from the same subjects multiple times. Thus at the end of the project it would be possible to describe the characteristics of law school intenders at each stage of the undergraduate experience and to describe how they had changed from the previous year. Panel studies of this kind are difficult to execute yet having individual level data on the nature of how law school plans change among undergraduates over time would be incredibly valuable. Only with such data would it be possible to develop informed intervention programs to increase the size and diversity of the potential law school applicant pool. This project was an attempt to examine whether the Internet could facilitate such a process.
The Career Plans and Undergraduate Attitudes Project was designed as a modified e-mail and web survey. A project website was established and samples of Freshmen students from the three chosen universities (large public research, regional, and urban) were contacted via e-mail with a brief introduction letter explaining the details of the project and encouraging them to use the web to complete a survey. To encourage participation, follow-up e-mails were sent as reminders to students at the regional and urban universities. A hyperlink was included in the e-mail letters which took the student directly to the web survey. The data was collected on the web survey in a form that was immediately ready for analysis. Students provided their university e-mail addresses, which were subsequently used to validate their surveys. Once the students participated, they were entered in a drawing for a gift certificate from their university bookstore. Because the students’ university e-mail addresses would remain the same, it was hoped that attrition of respondents from one year to the next, which is a problem for all panel studies, would be minimized.
Year One Results
The overall response rate for the first year of the project was a meager 20%. However, it did vary across type of institution—research (34%), regional (19%), and urban (7%). Such a low response rate raised a variety of questions about the representativeness of the sample and the ability to generalize to the larger population of freshmen at these institutions. In all three cases, the demographic characteristics of the sample of students completing the web survey were similar to their respective populations. At the regional university, an effort was made to compare the responses of a subset of the web survey questions across different types of surveys, including face-to-face, telephone, and postal. Several statistically significant differences were identified across the different modes of surveys, but the substantive conclusions were left largely unchanged. Interest in law school was greatest at the research university (12%) and lowest at the regional and urban schools (4%).
Year Two Results
The objective of the second year of the Career Plans and Undergraduate Attitudes project was to recontact all of the Year One participants and ask them to take a second survey. We successfully contacted and interviewed 37% of the Year One participants. Once again the response rate varied across type of institution—research (55%), regional (38%), and urban (17%). We lost nearly two-thirds of our Year One participants. What effect did this attrition have on the sample? At the research and urban universities where the sample sizes were largest and smallest respectively, the attrition had little effect on the project demographics. However, at the regional school—which lost nearly two-thirds of its participants—one change was dramatic—at the end of the second year, 84% of the project participants were women. Interest in law school continued to be strong at the research university (12%), increased a bit at the regional university (7%), and dropped by a point at the urban university (3%). It is important to note that at both the regional and urban universities, the sample size was getting extremely small in the Law School category. Perhaps the most important observation is that just 59% of the law school intenders in Year Two expressed the same postgraduate plan in Year One. This finding suggests that there is considerable volatility in the postgraduate plans during the freshman to sophomore transition.
Understanding the composition of and changes in the potential law school applicant pool during the undergraduate experience would provide valuable information that could prove helpful in efforts to increase the number and diversity of law school applicants. The Career Plans and Undergraduate Attitudes Project was an effort to use the Internet to gather such information. E-mail and the web did not turn out to be the panacea we had hoped for. The overall response rates were low and attrition was substantial. These problems, though, are the same ones faced by all survey researchers—especially those interested in difficult to reach populations such as college students. And the benefits were significant. Our test of the potential biases of collecting information over the web suggested that the problems are minimal, though additional effort in this area is clearly warranted. This is particularly important because if one assumes that the sample is small but not biased then information that is readily available about the population of college students can be used to statistically weight the results; a process used by the well known UCLA Higher Education Research Institute. E-mail and the web proved to be economical alternatives to and more feasible than traditional modes of survey research to collect information from college students. At the end of the day, the Internet—like any other method of survey research—presents both opportunities and challenges.