The PGES Student Growth Component

This article was submitted by Cora Wigger, a graduate student in public policy at Vanderbilt University’s Peabody College of Education.

Examining TPGES’s Student Growth Component

The 2014-2015 school year was the first where all schools in the state fully implemented the Kentucky Professional Growth and Effectiveness System (PGES), and beginning in 2015-2016 all schools and districts will be required by the state to use the results of PGES evaluations for decision-making for professional development and retention plans. Now is a particularly critical time for the state to be evaluating both the structure and rollout of PGES in order to make any final changes before stakes are officially attached to the system.

PGES Overview: http://kyedreport.com/?p=148

If you follow teacher evaluation systems in other states or in the national conversation, you’ve probably come across the terms “Value Added Models” (VAM) (a calculation of student test scores that attributes to the teacher the growth of a student beyond what would have been predicted) and “Student Learning Objectives” (SLO’s) (individualized learning goals developed and assessed for each of a teacher’s students). Kentucky uses both of these in the Student Growth portion of their teacher evaluation system (TPGES), but refrains from using the often politicized terms.  Smart, since not all VAMs and SLOs are created equal.

Kentucky’s Student Growth Goals, a take on SLO’s, are a strong pedagogical tool, and Kentucky’s push to use this strategy statewide is ambitious and forward-thinking, because they’re not easy to implement and monitor. Available research generally supports the idea that SLO’s have a positive effect on student learning, and the individualized nature of goal-development promotes teacher buy-in for the evaluation system. However, there is little evidence that SLO’s are a valid or reliable tool for measuring teacher effectiveness for evaluation (see Morgan & Lacireno, 2013). While the process of creating and using these student growth goals may be beneficial for both teacher practice and student learning, their use in TPGES for determining a teacher effectiveness score and subsequent teacher development and retention needs may not end up being a responsible or accurate measurement approach.

The second component of a teacher’s student growth score uses a student’s change in test scores as compared to their academic peers to determine the teacher’s contribution to that student’s academic growth. Kentucky’s approach here maximizes teacher buy-in by limiting the application of test score data to teachers who actually taught the students being tested in a given year (as compared to some systems that hold all teachers in a school accountable for students’ test scores, even those teaching untested subjects). Student Growth Percentiles (SGP) are determined for each tested student in at least their second year of consecutive testing by comparing each students’ current year scores to other students state-wide with the same scores on the previous year’s test. SGP is determined based on what percentile each student falls in according to their current test score compared to other students with the same test score from the year before. A teacher’s Median Student Growth Percentile (MSGP) is determined from the median of all of that teacher’s students’ SGPs. As complicated an explanation as that may be, compared to other Value Added Models, Kentucky’s is extremely simple. Some VAMs, for instance, take student background or teacher experience into account. And by basing the final SGP score on percentiles instead of raw scores, the TPGES model necessitates that there will always be students with a low SGP and a high SGP, even if all students do better (or worse) than would have been predicted.

The SGP approach also limits the years and subjects for which an assessment-based growth score can be calculated, because it requires consecutive years of test data in the same subject, greatly reducing the number of teachers able to receive a score from the pool of teachers who teach tested subjects. It also averages scores over three years, when available, which is statistically great, but for new teachers it makes each year hold more weight than for those with more experience. Overall, the use of SGP’s for student growth measurement is a potentially invalid and unreliable statistical tool that doesn’t utilize much of the available test data for determining teacher contribution to student growth.

However, it may not much matter. Kentucky allows districts to determine the weight that MSGP scores receive, theoretically allowing this score to make up as little as 5% of the overall student growth score for a teacher. So while it may not be as statistically sound or reliable as is ideal, districts have the ability to nearly completely leave it out of teachers’ final effectiveness scores. However, this then places all of the weight (for untested teachers) and nearly all of the weight (for tested teachers in districts that place little importance on MSGP) on student growth goals, which I have already demonstrated may be a flawed source for teacher evaluation.

The theory behind having an effective teacher evaluation model is that you will improve students’ education by improving the teachers – either by changing which teachers are in the work force or by identifying areas of weakness and tailoring professional development around those areas and for those teachers. But I will not be surprised if we come to see that TPGES isn’t so great at the identification of strong and weak teachers and areas of practice, given its not so strong measurement tools. However, if done well, the use of student growth goals accompanying TPGES may directly improve the education our students receive by giving teachers a powerful tool for offering individualized education to every student. And ultimately, that’s the purpose of any teacher evaluation system. I would be wary, however, to overuse TPGES in more high-stakes decisions that impact teachers, like pay scales or dismissals, as the system may not be up to snuff to be able to give us that kind of reliable information.

Morgan, C., Lacireno-Paquet, N. (2013) Overview of Student Learning Objectives (SLOs): Review of the Literature. Regional Education Laboratory at EDC.

For more on Student Learning Objectives and how they may impact teacher performance and student outcomes, see an analysis of Denver’s ProComp System.

For more on some of the challenges of VAM alluded to in Wigger’s analysis, see Some Inconvenient Facts About VAM.

For a look at some of the challenges posed by Tennessee’s relatively sophisticated VAM model, see The Worst Teachers

 

For more on education politics and policy in Kentucky, follow @KYEdReport

Value-Added Caution

Lots of attention in the discussion around teacher quality focuses on value-added data and the ability to determine a teacher’s effectiveness from a single test score.

More recently, a study by researchers at Harvard has received lots of attention because it purports to indicate that replacing a bad teacher with a good one has significant lifetime impact on student earning potential.

Unfortunately, it seems none of the media fawning over this study know how to use a calculator.

So, I break it down here:

This is the study that keeps getting attention around teacher quality and student earning potential. It was even mentioned in President Obama’s State of the Union back in 2012.  It keeps getting cited as further evidence that we need to fire more teachers to improve student achievement.

Here’s the finding that gets all the attention: A top 5 percent teacher (according to value-added modeling or VAM) can help a classroom of students (28) earn $250,000 more collectively over their lifetime.

Now, a quarter of a million sounds like a lot of money.

But, in their sample, a classroom was 28 students. So, that equates to $8928.57 per child over their lifetime. That’s right, NOT $8928.57 MORE per year, MORE over their whole life.

For more math fun, that’s $297.61 more per year over a thirty year career with a VAM-designated “great” teacher vs. with just an average teacher.

Yep, get your kid into a high value-added teacher’s classroom and they could be living in style, making a whole $300 more per year than their friends who had the misfortune of being in an average teacher’s room.

If we go all the way down to what VAM designates as “ineffective” teaching, you’d likely see that number double, or maybe go a little higher. So, let’s say it doubles plus some. Now, your kid has a low VAM teacher and the neighbor’s kid has a high VAM teacher. What’s that do to his or her life?

Well, it looks like this: The neighbor kid gets a starting job offer of $41,000 and your kid gets a starting offer of $40,000.

Wait, what? You mean VAM does not do anything more than that in terms of predicting teacher effect?

Um, no.

And so perhaps we shouldn’t be using value-added modeling for more than informing teachers about their students and their own performance. Using it as one small tool as they seek to continuously improve practice. One might even mention a VAM score on an evaluation — but one certainly wouldn’t base 35-50% of a teacher’s entire evaluation on such data. In light of these numbers from the Harvard researchers, that seems entirely irresponsible.

Perhaps there’s a lot more to teacher quality and teacher effect than a “value-added” score. Perhaps there’s real value added in the teacher who convinces a struggling kid to just stay in school one more year or the teacher who helps a child with the emotional issues surrounding divorce or abuse or drug use or any number of other challenges students (who are humans, not mere data points) face.

Alas, current trends in “education reform” are pushing us toward more widespread use of value-added data — using it to evaluate teachers and even publishing the results.

I can just hear the conversation now: Your kid got a “2” teacher mine got a “4.” My kid’s gonna make 500 bucks more a year than your kid. Unless, of course, the situation is reversed next year.

Stop the madness. Education is a people business. It’s about teachers (people) putting students (people) first.

I’m glad the researchers released this study. Despite their spurious conclusions, it finally tells us that we can and should focus less on a single value-added score and more on all the inputs at all levels that impact a child’s success in school and life.

As Kentucky considers teacher evaluation “reform,” caution should be used when deciding what (if any) role value-added scores will play in new evaluations.

For more on Kentucky education politics and policy, follow us @KyEdReport