This is the first written public report on a major study of 93 computer-intensive courses taught in 36 of Yahoo’s 100 most wired campuses. The professors who designed and taught these courses were asked why, after observing the results of their change to computer-enhanced instruction, they continued to spend the extra time and effort that computer enhanced teaching requires. From their answers we have been able (inductively) to (1) craft a typology for the assessment of the impact of technology upon learning, and (2) collect a rich array of specific measured results. My brief paper is divided into 3 parts: the typology, stories about five outstanding efforts to measure results, and the measured results from the 93 courses.

The topic, "Assessing the impact of technology upon learning," has many interpretations. If the results of "assessment studies" are to shape decisions on the character of investment in computer infrastructure and the design of individual courses, the studies themselves need to be grouped into meaningful subcategories. This grouping is best achieved by answering three questions:

• What unit? (e.g. whole college vs. chemistry courses vs. Chemistry 101)
• What tested? (i.e., like vs. learn)
• How measured? (i.e., perception vs. behavior vs. outcome)

These typologies are elaborated in Figure 1. This is a report about how learning gains are being tested at the micro (or individual course) level. Here the measures emphasized are printed in bold.

Note: This presentation is extracted from the opening chapter of Interactive Learning: Vignettes from America’s Most Wired Campuses which is to be published by Anker Publishing Company and released in June, 1999.

Before presenting an overview of assessment methods and results from all 93 courses, let me highlight six of them.

• Sally Jackson and her colleagues from the Department of Communication at the University of Arizona tested the effectiveness of their use of "structured online dialogues." 96% of their students agreed that "the lessons added an important component to the course" and 79% disagreed that "having to use the worldwide web was an obstacle to learning." The four instructors authoring this essay conclude that "after three years of development and experimentation with structured online dialogues, we believe them to be the most important element of these critical courses and are continuing to phase them into this key course sequence."

• William Clark from the Chemical Engineering Department at Worcester Tech compared his two sections of the same course, one taught by the traditional lecture-homework format and the other taught by the multimedia-group project format. The number of "agree" and "strongly agree" responses to 14 statements about the instructor’s delivery of the course increased from 80% to 95%. When asked how they "liked" the multimedia format, over 70% of the students chose the top two categories for each of five components of the course (alternative to lecture, alternative to homework, group project in class, group project between classes, and working groups instead of alone). In another survey, the percentage of students answering agree or strongly agree with the statement "I learned a lot in this course" on the standard WPI course evaluations was 77% when traditionally taught and 97% when taught with the new approach. In his analysis of outcome data, specifically the average grades earned by students under the two methods, Clark concludes "the multimedia-group project format did not hinder student learning and may have improved it."

• Ed Carney and his mathematics colleagues at the University of California at Davis evaluated their effectiveness by student/instructor opinion polls, page counts revealing student behavior when using the material, and student performance on online homework problems. Pilot study results show that students are using the optional online homework assignments, those that are using online exercises are getting better grades and feeling that they are learning more.

• Chris Impey, an Astronomy Professor at the University of Arizona, reports that, even with no particular incentive to participate, there is an average of thirty web hits and ten emails per student with no negative effect on class or office hour attendance. Students who used both email and web tolls performed one letter grade higher than students who used neither. And, in entrance and exit polls, student responses to the electronic aspects of the course were "overwhelmingly positive."

• E. Kashy and colleagues from Michigan State’s College of Natural Science report results from teaching a class of 500 students. When compared with previous years, significantly fewer students dropped the course. Significantly more students earned A’s and A’s + B’s. Numerous student surveys confirmed their acceptance of and enthusiasm for the new course format.

• Carl Blyth from the Department of French and Italian at the University of Texas at Austin, working under a extensively evaluated FIPSE Project, reports that there was a statistically significant difference in grades favoring the students enrolled in the computer-enhanced courses. Moreover, student enrolled in the computer-enhanced curriculum reported higher levels of satisfaction with the course and frequently wrote on their course evaluations that the computer lab was the most helpful and interesting part of the course. Blyth credits the new curriculum for the complete turnaround in previously declining enrollments in French.

Six courses have been highlighted. Equally interesting is an overview of the evidence cited by the 93 essayists. Why are so many faculty willing to spend so much time redesigning courses and curricula, redesigning them mostly in ways that are more time intensive?

Educational Philosophy of Early Adopters. The short answer is provided by the title of my forthcoming book, Interactive Learning. The book was retitled from "Computer Enhanced Learning" to "Interactive Learning" only after reading the 93 vignettes. When asked about the educational beliefs that were motivating their computer-based innovations, in one way or another 92 of the 93 essayists (99%) wrote about "learning by doing," "integration of theory and practice," "collaborative learning," "group projects," or "hands on learning." Each in their own discipline and in their own way had concluded those back-and-forth exchanges between professor and student, and among students, are the great facilitators of learning. And, computers can be wonderful new technology that facilitates and enhances interactive learning.

Faculty Perception. Our scholars seeking more interactivity have been richly rewarded by results. Eighty-three of the 93 chapters include comments on assessment. Among the three strategies for assessment, by far the most frequently used is the perception of faculty and students. Seventy-six percent (63 of 83) wrote about ways they observed more learning: the quality of class discussions, the frequency of "ah-ha’s," student eyes lighting up with new insights, performance in more advanced courses, the capacity to customize assignments to student interest and ability level, the popularity of computer-enhanced sections, the quality of term papers, the depth of conversations during office hours, the nature and frequency of questions asked over the internet, etc. These qualitative judgments about the impact of computer-enhancements seem to be the major reinforcers for continued use of the computer. Let me cite a few quotes:

"…the results …were unbelievably wonderful. The simplicity of the assignment let the students build something very complex and multilayered."—Ed Epping, Professor of Art, Williams

"Having the ability to conveniently review their materials is … a great boon to their learning." John Kappelman, Professor of Anthropology, Texas at Austin

"My experience in a large classroom suggests that a combination of chalkboard and computer slides is superior to either format alone… Email not only has provided greater efficiency of teaching, but also has increased the number of students who seek help….The incorporation of technology…allows instructors to review their course content, teaching style, and learning objectives and upgrade them." Andrew Barkley, Professor of Agricultural Economics, Kansas State.

"It is clear that word has spread that technology is a wonderful aid to learning. Students are now able to answer more complex problems with greater success, and more material may be covered in class. Their better understanding of biology is evidence in their abilities to synthesize various thought processes at an earlier point in the semester."

Sarah Lea McGuire, Millsaps College

"The highest quiz average in my course was achieved on a section of materials that depends most heavily on the website. The average scores for that particular quiz have doubled since I created the interactive tutorials. In addition, student response to the exercises has been extremely positive---they’re asking for more." Charles Crisham, professor of Chemistry, Virginia

"The incorporation of technology has made an incredible difference in this class. This difference can be measured not only in the amount and quality of writing produced, but in the enthusiasm [and excitement] students showed toward their assignment." Franziska Lys, German, Northwestern University

"I will continue and will encourage others to use internet-based resources in teaching, in part because of some surprising results. More students completed their research in a timely way, with excellent content. Gail Sherman, English, Reed

Student Surveys. In many instances, these general observations by faculty are reinforced end-of-course evaluations and perception surveys of students. Seventy-three percent of vignettes speaking to assessment mention student surveys. In virtually every instance, students felt that their learning was advanced by technology. Students reported that they prefer courses that include computer-enhancements. Especially the increased communication, collaboration and interactivity were valued by students.

Qualitative Feedback from Students. Elaborating the statistical results from formal student surveys were the qualitative comments by enrolled students and queries from students in other courses. Sixty-three percent of the essays covering assessment cite some of these comments.

Matched Pair Studies. Matched pair analyses, either "before technology vs. after technology" or "with technologies vs. without technology," were quoted by 45%. Although some of these analyses revealed "no significant difference" in "grade earned" or "as a consequence of computer-enhanced lectures," a majority of the studies revealed statistically significant positive differences in favor of technology, and no study suggested a statistically significant advantage for the class taught traditionally.

Other Metadata. The results from their comparisons of outcomes and perceptions were reinforced by a wide variety of observed behaviors. Ten authors reported that their web-based resources were being used at other universities or by other professors within their own institutions. Web citations in research papers were compared with library citations. Successful completions of web-based problem exercises were related to success rates on final exams. One professor studies metadata on the number of times students try and fail practice homework problems to determine where students are experiencing the greatest difficulty. Voluntary website hits on exam-eve were touted as evidence of the useful of web-based resources. Class percentages entering threaded conversations, and item analyses of these conversations, are being used to judge effectiveness. Clearly, the metadata generated by computer usage is becoming an important factor in evaluation, in feedback, and in the improvement of teaching strategies.

Taken together, these assessments--- perceptions, outcomes, behaviors--- are necessarily related to the objectives of the instruction. Although some professors are measuring whether students "like" a course and "have fun learning," most assessment is against objectives relating to understanding the theory, concept, and application of substantive, disciplinary knowledge. However, a significant portion of our early adopters have expanded their course objectives and are also interested in nurturing, and therefore measuring, the capacity to work collaboratively in teams (mentioned by 20%) and in developing computer skills (mentioned by 26%).

Professor Stephen Loomis, who teaches Human Physiology at Connecticut College and reports on his experience, sums it up well: "In overall learning, 68 percent of the students demonstrated an understanding of over 90 percent of the concepts. This compares with a ten-year average of 24 percent using the lecture format. In attitudinal surveys, 96 percent of the students rated this course as very good to excellent and would recommend it to a peer." It is no accident that virtually all professors who have added technological enhancements to their repertoire of teaching tools are persevering. Students win!