Seeing Math Research: Promising Gains

By Alvaro Galvis

What teacher gains in math content and pedagogy can be attributed to teacher participation in the Seeing Math professional development courses developed by the Concord Consortium? What student gains in math performance can be attributed to the participation of their teachers in the Seeing Math project? Such questions were the basis of the research in the final year and a half of the Seeing Math project, in response to the U.S. Department of Education’s interest in quantitative results.

Seeing Math Elementary (SME) includes twelve interactive video cases in four NCTM strands for grades 3-6. Three six-week blended (online and onsite) courses in the Number and Operations strand were selected by participating school districts. Math coordinators were trained to facilitate video case-based discussions and to build local communities that reflect on their own practices.

Seeing Math Secondary (SMS) includes nine five-week online courses focused on improving teachers’ understanding of key algebra topics at the secondary level. Interactive tools or applets afford teachers and students multiple approaches for solving math problems and making explicit their algebraic thinking. Classroom video episodes and expert video commentaries invite teachers to discuss content and pedagogical issues, and to listen to student thinking. For research purposes a 13-week family of three units on linear algebra (functions, transformations, and equations) was selected.

A quasi-experimental design created by the Concord Consortium in collaboration with external evaluators Edcentric and Hezel Associates was approved by the Department of Education. It allowed for cross-sectional comparisons (across cohorts within a given year), as well as longitudinal comparisons (within cohorts across consecutive years). The research was implemented in the spring of 2004 and during the 2004-2005 academic year.

The Concord Consortium developed open-ended measures aligned with content and pedagogy being taught in Seeing Math courses. These teacher assessments were field tested and adjusted before they were administered. Teachers took tests before and after participation in a Seeing Math course; the tests were scored with specially designed rubrics.

NWEA, which specializes in the measurement of student math knowledge, created student tests for both SME and SMS courses. Multiple-choice questions measured math knowledge within the NCTM standards that were directly related to the particular Seeing Math courses taken by participating teachers. These tests were based on Item Response Theory (IRT) and yielded three scores: overall, target, and nontarget. (Target items relate to content that treatment teachers were exposed to, while non-target relate to content in the same NCTM strand, though not studied by treatment teachers.)

Cohort 1 teachers participated in Seeing Math courses during spring 2004; they were encouraged to apply course ideas when teaching their students that semester and the following school year. Cohort 2 and Cohort 3 teachers participated in Seeing Math courses during the 2004-05 school year and were asked to apply this knowledge in their classroom. Pre- and posttests for content knowledge were administered to participating teachers and their students.

Seeing Math Elementary findings and discussion

Studies about the effect and impact of the SME course materials on elementary math teachers and their students at three participating school districts indicate that:

Treatment teachers performed significantly better in Pedagogy as well as in the Modeling/Formulating, Transforming/ Manipulating, Inferring/Drawing Conclusions, and Communicating Content areas than did the control teachers.

Student findings were difficult to interpret. In one school district, there were no significant student gains for either year of the study, and no differences between treatment and control groups. In a second district, treatment students made significant gains over the second year of the study, and had signif- icantly higher gains than did the control students. In the third district, treatment students had significant gains in both years in some test scores; however, both mean scores and gains for the control students exceeded those for the treatment students for all scores.

The study tested for statistical differences related to student grade level, ethnic group, gender, teacher gain scores, and teacher experience teaching math. There were no significant differences in gain scores by grade level within any school district, nor between any ethnic group gain scores; males had higher gains than females, though differences were not significant; very low correlation values indicate no relation between student gains and teacher gains or teacher’s math teaching experience.

The above findings confirm that Seeing Math programs can improve teacher pedagogy and content knowledge. Edcentric also found that the courses met or exceeded participant expectations, and that the major elements of the courses—video cases, expert commentary, hands-on math activities with interactives, face-to-face meetings, and discussions—were considered important by the majority of the respondents. Qualitative evaluations also noted anecdotal evidence of teacher change and student change. Some teacher comments illustrate this: “I encourage my students to discuss their strategies more. I am aware of my own questioning and try to echo students, pose questions to clarify misunderstandings, and highlight different strategies.” “I think the students who have more difficulty with math changed most. When I give the students a word problem and tell them to draw it out and use a method that works, they feel they have a choice and they start to solve it in a way that makes sense to them.”

Seeing Math Secondary findings and discussion

Studies on the impact of SMS course materials on secondary education algebra teachers and their students indicate that:

Overall differences on teacher assessment (content and pedagogy) favored the treatment group; significant advantages were found only for the pedagogy subscale.

Less mathematically educated teachers tended to learn more in Linear Functions.

In the year after taking the Seeing Math course, teachers continued to learn in some content and pedagogy areas. (This may be biased by attrition in the sample.)

The treatment on teachers had no effect on student learning in target areas, but had a positive effect in nontarget areas: students of treatment teachers declined less over a year than students of comparison teachers. A pattern in all cohorts shows decline in non-target areas of linear algebra knowledge over a school year; however, this finding is difficult to interpret.

Students of teachers with a mathematics degree gained less in target areas than students of teachers without a degree. In non-target areas, students of treatment teachers declined less when their teachers had a mathematics degree, and declined more when their teachers did not.

Quantitative findings concerning teacher gains are coherent with the qualitative evaluation that found that Seeing Math can be an effective program for experienced middle and high school math teachers in helping them explore algebra instruction for their students. All SMS course completers (41 out of 57 teachers) found online discussions to be valuable in helping with instructional strategies for teaching algebra and with clarifying course content. Ninety-eight percent said they were using or planning to use strategies and activities from the course in their classes.

Student findings are difficult to explain. It is not clear why, independent of treatment, students of teachers with a math degree gained less in target areas than students of teachers without a degree. It is also unclear why, independent of cohort, students, including those of treatment teachers, declined in non-target areas over a school year.

Future research

From these studies we know that teachers made gains in pedagogy, but we also need to know whether they incorporated this learning into their practice, and if so, how? Do the findings imply that more or deeper content knowledge is less relevant than finding ways to translate existing understanding to students in meaningful ways, or understanding how students think and solve problems? A finer-grained look at some of these areas may be more informative than broadly assessing if teachers and their students learned as a result of taking a course, and may allow for more focused development of new materials. While the results are promising, more research is warranted to exploit the best methods for developing and implementing online professional development for math teachers.

Alvaro Galvis (alvaro@concord.org) is Research Director of the Seeing Math project.