More scholarship and less ideology needed on rewarding good teaching

ROY MARTIN and JOHN McCOLLOW examine the research purporting to measure teacher effectiveness.

Recently a study conducted by Andrew Leigh of the Australian National University entitled Estimating Teacher Effectiveness from Two-Year Changes in Students' Test Scores received a good deal of media attention. Leigh's study was widely reported as providing evidence that supported proposals for "performance pay" for teachers. As will be shown, however, the outcomes of Leigh's research are unremarkable and fall well short of evidence that could be used to support the case for performance pay. In fact, if anything, they provide some evidence to the contrary.

What did Leigh's study entail?

Leigh used data from the year 3, 5 and 7 literacy and numeracy tests in Queensland to track individual student performances at each year level. To eliminate the effects of socio-economic or home factors, Leigh uses each student's year 3 result as the base and focuses on the extent to which performance on the subsequent tests improves relative to the "average" performance of the student's age cohort. Leigh is able to match students with the teachers that they had at each year level. He then assigns the change in relative performance (or lack thereof) to the effects of teaching, specifically to the classroom instruction provided by the teacher the student had at the time of the test. He is then able to calculate the correlation between improved test performance and various factors associated with the teacher (for example gender, age, years of experience, possession of a post-graduate qualification).

What did Leigh find?

There was significant variation in the "effectiveness" of teachers as measured by the improvements in test scores of their students. There was also a positive correlation between "effectiveness" as measured by student literacy results and student numeracy results. Of the factors for which correlation was calculated, teacher "experience has the strongest effect", though "most of the differences between teachers are due to factors not captured in the payroll database".

Limitations and problems with the Leigh research

Defining and measuring teacher effectiveness

Leigh defines teacher effectiveness in terms of the performance of their students on literacy and numeracy tests. This is a relatively limited and crude measure. Furthermore, because the tests were administered at two year intervals in August, students will have experienced classroom instruction under at least three teachers in the intervening period between the tests. Leigh assigns the improvement either entirely to the instruction of the teacher at the time of the relevant test or "splits" it 50/50 between that teacher and the teacher in the previous year.

All student test results are attributed to teacher instruction -- no other factors are considered (for example personal, family, school, community circumstances). While Leigh attempts to control for socio-economic and family factors by using each student's own year 3 result as the base, this only works if one assumes these factors have no effect on learning rates -- a highly contestable assumption. Finally, because teacher effectiveness is measured in terms of how their students perform relative to the average performance of their year-level cohort, any improvements in student scores that align with overall improvements for the cohort are not picked up as "value-added".

Overstating the research results

Leigh's research finds that not all teachers are equally effective in terms of the results achieved by their students on literacy and numeracy tests. This can hardly be considered a startlingly new insight. In fact, as Leigh himself admits on his web log: "For what it's worth, I don't think this dispersion is wider than what one would find among plumbers, dentists, architects or bricklayers. But it does indicate that -- at least as measured by test score gains -- all teachers are not created equal."

Similarly, while Leigh's research purports to show a limited connection between factors identified on a payroll database and teacher effectiveness, he identifies and tests no alternative factors that correlate more highly. The factor identified as having the greatest effect (experience) actually argues against an approach to remuneration based on performance pay.

Leigh is prone to some remarkably simplistic statements. For example, although Leigh's study did not examine the time taken by teachers to prepare their students for the tests he states that "a teacher at the 90th percentile can achieve in half a year what a teacher at the 10th percentile can achieve in a full year". This sort of hyperbole is clearly aimed for media consumption rather than as a contribution to scholarly debate.

Implications of measuring teacher effectiveness for performance pay using Leigh's approach

The principle thing this research "proves" is that if one uses a very crude measure, it is possible to identify those teachers who have most "value added". However, in order for this to be linked to a system of performance pay it would be necessary to:

• vastly increase the frequency of tests (really, it would be necessary to test on entry and exit from each year, and presumably every time there was a change of teacher);
• increase the range of tests to anything that was valued in teaching and learning (or at least to recognised subjects).

(Of course, the problems for secondary schools would be even greater).

Besides being incredibly intrusive and time consuming, this would distort the process of schooling to an incredible degree, demanding high stakes testing, high curriculum definition and low teacher trust to become the whole basis of schooling. Evidence cited by curriculum and assessment experts such as Allan Luke, Sharon L Nichols and David C Berliner strongly indicates that this would actually be counter-productive. Experience in the United States and United Kingdom shows the deleterious effects on schools and teaching of an over-reliance on standardised tests. Some of the most successful countries (Finland and Canada) in terms of educational outcomes have low stakes testing, low definition curriculum and high levels of teacher trust.

Conclusion

Roy Martin is AEU Federal Research Officer. John McCollow is a Queensland Teachers Union Research Officer.

©2000-2002 NSWTF Online is a resource for teachers
provided by the NSW Teachers Federation.
[Authorisation of election comment] [Privacy]

http://www.nswtf.org.au/journal_extras/iflaw.html
Last Modified: 21 Jun 2007