Evidence of Success

Mechanics: An article by Richard Hake in the January 1998 issue of American Journal of Physics ("Interactive Engagement Versus Traditional Methods: a Six-Thousand Student Survey of Mechanics Test Data for Introductory Physics Courses, Am. J. of Phys, 66, 1, pp. 64-74) discusses the difference between traditional physics classes and what he calls "interactive engagement" classes in terms of students' pre-instruction and post-instruction performance on a standardized test of newtonian mechanics called the Force Concept Inventory test (see D. Hestenes, M. Wells, and G. Swackhammer, "Force Concept Inventory", Phys.Teach. 30, 3, pp. 141-158 (1992). The FCI test is a multiple-choice test that examines students' abilities to apply the concepts of newtonian physics to everyday situations. The test uses non-technical language to describe the situations, but employs distractors that specifically address common-sense misconceptions about physics.

Hake defines the normalized gain on the FCI test to be the average increase in students' scores on the FCI divided by the average increase that would have resulted if all students had perfect scores on the post-instruction test. After analyzing results from more than 6000 students, Hake argues that the normalized gain is a meaningful measure of how well a course teaches newtonian ideas to students. In particular, Hake finds that introductory physics course that can be characterized as "traditional" in teaching style get normalized gains of 0.230.04, a quite narrow range in spite of the wide range of student initial abilities. By contrast, the normalized gains earned by the courses Hake calls "interactive engagement" courses are in the range 0.48 0.14, a statistically significant difference. Hake examines and discards a number of possible explanations for these results, and concludes that the teaching methods employed in the courses make the difference.

The normalized gains for students taking Pomona College's Six Ideas introductory course during the past few decade are as follows:

Pomona College

  • Fall 1993: 0.46
  • Fall 1996: 0.48
  • Fall 1997: 0.45
  • Fall 1998: 0.55 (estimated)
  • Fall 2000: 0.63
  • Fall 2001: 0.58
  • Fall 2004: 0.59
  • Fall 2006: 0.68

The gain for the fall of 1998 is only estimated because we were unable to offer the test before instruction: the "pre-instruction score" used in calculating the gain was thus taken to be the average pre-instruction score for the three previous classes listed. I have some evidence that the apparent jump in the normalized gain may be real: the post-instruction score for this class was the highest ever recorded (about the same as for the truly extraordinary 1996 class of students), but this class seemed to be quite average (even a bit below average) in general ability. 1998 was the first year that the published textbooks were available to students, and it was also the first year that we were able to fully implement the scheme for homework suggested in the instructor's manual.

As you can see, these results fall in the center to upper end of the interactive engagement class range. This is in spite of the fact that these courses were conducted in a normal classroom with less time spent doing active learning activities and less time giving personal attention to students than in many of the interactive engagement classes Hake describes. Moreover, somewhat less time is spent discussing newtonian mechanics in Six Ideas than in many traditional courses because of the time allotted to special relativity. In short, Six Ideas, as taught at Pomona earns consistently high gains on the FCI test (well above traditional courses) without radical changes in the infrastructure required to teach the course.

Dr. Ulrich Heinz of Ohio State University (Columbus) reported an even better gain for his sections of Six Ideas. He reported to me that his students started the fall of 2001 with considerably lower initial FCI scores (58%) compared to Pomona students (about 75%) and ended with final FCI scores about the same (88% compared to 90%). His reported normalized gain was

Ohio State University (Columbus)

  • Fall 2001: 0.72

This suggests that students that are considerably less prepared than Pomona College students can handle the material in the Six Ideas course and learn it quite well. Note that this gain is well beyond the typical gains for even interactive engagement courses! (Thanks to Dr. Heinz for sending me these results.)

Dr. Vic DeCarlo of DePauw University has also graciously given permission to post the gains achieved at his institution for the past few years:

DePauw University

  • Fall 2000: 0.54
  • Fall 2001: 0.52
  • Fall 2002: 0.58
  • Fall 2003: 0.60

Initial FCI scores at DePauw were in the 40% to 50% range, again indicating that students at all levels of initial preparation can benefit from the Six Ideas course. Also please note that these gains involve results from two different instructors.

Bottom Line: We see one can get excellent FCI gains in Six Ideas courses at a variety of institutions and with a variety of instructors, in spite of spending significantly less time on mechanics than in a traditional course.

Electricity and Magnetism. Last semester, I gave my students the Brief Electricity and Magnetism Assessment test as a post-test. According to published results, pre-test scores differ little from chance in any setting, so rather than focusing on the normalized gains, one can simply focus on the BEMA post-test scores. Students experiencing traditional instruction in E&M at Carnegie Mellon University and North Carolina State University attained average post-test scores of 45% and 39% respectively, and even senior CMU physics majors earned an average score of 80%. Students at Pomona who used the draft 3rd edition of Unit E earned an average score of 58%, significantly above the traditional scores, roughly halfway toward the practical upper limit set by the senior majors. Again, please note that this is with only 5.3 weeks of instruction in electricity and magnetism.

Bottom Line: Students taking Six Ideas learn the classical topics of mechanics and E&M more effectively (at least as measured by FCI and BEMA) than students in traditional courses, in spite of spending significant amounts of time on contemporary physics.

Effect on Pre-Meds. We have also done a preliminary analysis of Pomona pre-meds that displays a strong and statistically significant positive correlation between and the student's MCAT physical science score and whether the student took Pomona's Six Ideas course as opposed to a traditional physics course at another college (usually during the summer). To more strongly justify a genuine causal relationship between the course and the score, we need to control this result for GPA (as it is possible that stronger students might elect to take the course at Pomona), and that analysis has not yet been done. Even so, it is seems clear that taking the Six Ideas course does not hurt students on the MCAT, again in spite of the shorter amount of time spent on MCAT-related topics.

Return to the top.