Julie Bishop plays politics with performance pay

By Wendy Currie

Not content with her ill-considered threats to tie federal funding to performance pay, Federal Education Minister Julie Bishop announced recently that she would conduct a tender process to engage an expert to develop models of performance-based pay for teachers to be trialled in Australian schools.

Victorian Premier Steve Bracks said of her announcement: "This is just naked politics before a federal election without having regard to what exists already, just simply saying, 'I think this is popular because probably the polling has shown it's popular. Therefore, I'll do it.'"

NSW Education Minister John Della Bosca said that in the absence of federal funding "from the outset, the consultants will be drafting models that require some teachers to be paid less in order to reward others".

It is understood that those under consideration are McKinsey, Deloitte and KMPG.

The Australian Council for Education Research (ACER) does not appear to be in the running, perhaps for two reasons. The first is that they might know too much about what actually goes on in schools for Ms Bishop's comfort. The second is that they have already produced a report for her. In making an earlier announcement, Ms Bishop implied that this report lent weight to her model, but she did not release it. One of the authors, interviewed by the media the following day, warned against a system based on student test results and student and parent surveys.

That same author, Laurence Ingvarson, has since had an article published in The Age in which he seriously criticised the methodology of another research report written by Andrew Leigh.

Leigh has now produced two papers that purport to be about quality teaching but the first advocated performance pay and the second might as well do so.

The quality of Leigh's research is surprisingly poor and is based on erroneous assumptions and reaches conclusions that are not born out by the evidence. An analysis of his first paper, How and why has teacher quality changed in Australia, appeared in Education (September 18, 2006). It was originally commissioned by the federal Department of Education, Science and Training, and was the paper the Federal Government relied on when it first flagged performance pay for teachers.

His second paper, Estimating Teacher Effectiveness From Two-Year Changes in Students' Test Scores, examined results of students in Queensland schools in tests held two years apart and correlated those results against data about the teachers held in the Queensland department's data base. It was this paper that prompted Ingvarson to write his article in The Age.

The article's headline was "Teacher study fails the test" and so it does. Ingvarson, an education academic for decades before moving to ACER, said: "Readers would be forgiven for thinking that the research opens the way for the 'best' teachers to be identified and rewarded on the basis of their students' test performances, as they were at the end of the 19th century. It does nothing of the sort."

The particular flaw he found in Leigh's paper is a basic one. While the study was reported as focussing on gains in achievement over two years, Leigh in fact looked at the changes in the relative positions in classes over two years, a very different thing. Ingvarson went on to say: "Not surprisingly, he found that some classes improved their position within the state results, while others went in the opposite direction. This, of course, was inevitable. For every class that gained in its relative position, another had to go down. This is the nature of relative data."

Ingvarson makes the point that if you're at the top one year, then being at the top in two year's time is not an improvement, with the result that, according to Leigh's criteria, the teacher must be "bad".

Ingvarson concludes: "Dr Leigh's research provides no basis for the identification of effective and ineffective teachers."

There are a number of other flaws in Leigh's research .

Below is an example from Leigh's paper that demonstrates the very tenuous relationship between his data and his conclusions: "Queensland (like other Australian states and territories) administers its statewide standardized test biennially. Thus the question arises of how teachers in the intervening year should be treated. The two most plausible approaches are: (a) ignore the intervening year altogether, or (b) create an assumed test score in the intervening year, which lies at the midpoint of the other two tests. In this section I present both methods."

Leigh recognises the problem of the effect of the teacher in the year between the two tests, but chooses to manipulate the data to introduce two artificial situations, the validity of both of which is debatable and certainly invalid if the result is a determinant of pay. Leigh claims his research shows teachers' additional qualifications to not improve their students' results, without knowing what those additional qualifications are. He fails to consider the variables that might explain some of these results, such as what the qualification is and the context in which that person is teaching. Is the qualification in special education, educational administration, or not linked to education at all? It would be a travesty if these results led to the view that additional qualifications and/or professional development was useless.

Leigh found that there was "a significant effect of experience": "Compared to teachers with 10 years of experience, novice teachers have test score gains that are 1/100ths of a standard deviation lower in literacy, and nearly 2/100ths of a standard deviation lower in numeracy."

Remembering that all the teachers in the sample were qualified, registered teachers, and teacher experience contributed significantly to the results, it's difficult to account for the following paragraph in Leigh's conclusion: "The results from this paper also shed light on the extent to which uniform pay schedules, which reward teaches based solely upon qualifications and experience, capture productivity differences between teachers," and, "Yet while there are some systematic patterns, 99 percent of the variation in teacher performance remains unexplained by differences in teacher demographics."

There could be a range of reasons for this, not the least being that the assumptions on which the whole paper is based are suspect and shallow. But he goes on to say: "This suggests that uniform pay schedules are indeed only picking up a small portion of differences in test score gains across teachers," thus assuming that all the variations in test scores are a result of teacher effect. Clearly the author understands little about education, schools, teaching and learning and the multiple variables that determine student outcomes. And a significant omission from his research is what actually goes on in a classroom.

He finishes with: "These results suggest that it may be worth considering alternative salary structures, as a means of attracting and retaining the best teachers," without any actual link in this research to the issue of teacher pay.

Andrew Leigh is an economist and law graduate with a PhD in public policy. He appears to have little understanding of education. Yet he, like a number of academics particularly in the United States with whom he has professional ties, apply econometrics and market based policy to essentially human activities with an increasing emphasis on education. Their work fails to take into account what motivates teachers, how to determine what is quality teaching and learning, what variables have an impact on student learning, how and under what circumstances teachers work and students learn and what conclusions can be justifiably drawn.

They aim to create a context where their work at least influences, if not determines, public policy. And in this respect they are succeeding.

The flaws in the work of Leigh and others have been identified and cogently described, but that has not stopped the media's interest in it. The Federal Government is adept at using the media to advantage, including whipping up the public's fears about the quality of teachers.

The link to Bishop using management consultants to determine teacher quality and performance pay criteria is clear.

This work is being used by the Federal Government to support its agenda. It appeals well to those who advocate choice as a means of addressing equity, to those who no longer, if they ever did, talk about the common good and the value of collegiality.

We need to preserve at all costs the incremental scale that Leigh's research inadvertently supports by demonstrating the significant effect of teacher experience.

Wendy Currie is a Research Officer.

Business could have role in determining school curriculum

More scholarship and less ideology needed on rewarding good teaching

Business comes out fighting in support of Howard

Local hiring attacks equity and tenure

Shame file

For further information

June 2007 contents