Hill, Kapitula, and Umland (2011) compared 24 middle-school mathematics teachers' value-added (VA) scores with survey- and observation-based indicators of teacher knowledge, instructional quality, and student composition. They found that VA scores correlated with teachers' mathematical knowledge and instructional quality but also with the population of students they taught, and that some teachers with high VA rankings displayed poor observed instruction. Framing their study as an argument-based validity argument for using VA in pay-for-performance plans, the authors left four empirical tasks open: (1) identifying the causal sources of divergence between high VA rankings and poor observed instruction; (2) determining whether richer model specifications (multivariate response models, multiyear linked data, student fixed-effect designs) reduce misclassification and separate teacher quality from student composition; (3) replicating the small-sample partial-correlation findings at scale; and (4) developing observational instruments predictive of long-term student learning and testing combined measurement systems for personnel decisions.

Background

Value-added models (VAMs) are statistical models that estimate a teacher's contribution to student learning by relating students' end-of-year test scores to their prior achievement and other controls; a teacher's estimated contribution is their value-added (VA) score. Around 2010, US policies tying teacher pay, tenure, and evaluation to student test results made the validity of VA scores a first-order policy question. A standard framework for judging such uses is the argument-based validity approach (Kane's validation framework; Kane, 2013; applied to VAMs by Hill, 2009): one specifies the chain of inferences required for an intended use — for example, 'VA scores reflect teacher-caused learning, not student composition, and identify teachers whose students learn more' — and then collects evidence for each link.

Hill, Kapitula, and Umland (2011) contributed such evidence for middle-school mathematics. They computed VA scores from a district data set (N = 222) and compared them, for 24 teachers, with survey- and observation-based indicators: teachers' mathematical knowledge for teaching (MKT, a survey-based measure of the specialized mathematical knowledge teaching requires), the mathematical quality of instruction (MQI, a video-based instrument scoring the mathematical work of lessons), and measures of the students assigned to each teacher. They found VA correlated with teacher knowledge and instructional quality, but also with student composition — teachers of more advantaged or higher-prior-achieving students tended to have higher VA scores — and their case studies described teachers with high VA rankings but poor observed instruction. They concluded that VA scores were not yet safe for high-stakes pay-for-performance decisions and enumerated the four uncertainties that define this problem.

Three strands of later work define the current state of knowledge. First, econometric studies of bias: Rothstein (2010) introduced lagged-score falsification tests and student fixed-effect specifications showing that naive VA estimates absorb student sorting; Chetty, Friedman, and Rockoff (2014a) used parent characteristics and teacher-switching quasi-experiments to argue that prior-score-controlled VA is nearly unbiased, while Rothstein's (2017) reanalysis argued the quasi-experiment is confounded and residual bias may be 10–35% as large (in variance terms) as true teacher effects. Kinsler (2012) and Goldhaber and Chaplin (2015) reassessed the falsification tests with mixed conclusions. Second, studies of precision and specification: Schochet and Chiang (2013) showed classification error rates of roughly 25% with three years of data and 35% with one; Kersting, Chen, and Stigler (2013) found sample size matters far more than specification choice; Guarino, Reckase, and Wooldridge (2015) showed in simulations that no estimator is robust across plausible student-grouping scenarios; Mihaly, McCaffrey, Sass, and Lockwood (2013) showed fixed-effect identification is often weak in practice. Third, studies of divergence and combined measurement: the Measures of Effective Teaching (MET) project randomly assigned students to teachers and found that a composite of VA, observations, and student surveys predicted same-year achievement gains (Kane, McCaffrey, Miller, and Staiger, 2013), though Rothstein and Mathis (2013) argued the randomization was compromised and that the measures capture distinct teaching components; Hill, Blazar, and colleagues (2013) found expert observers misjudged teachers' VA quintiles in about half of cases; Blazar, Litke, and Barmore (2016) found VA rankings are not comparable across districts and that instruction differs starkly among similarly ranked teachers; Blazar and Kraft (2017) and Blazar (2018) showed teachers effective at raising test scores are often not effective at raising students' attitudes and behaviors; and Bacher-Hicks, Chin, Kane, and Staiger (2019) found in a four-district experiment that VA and observation scores are unbiased predictors of performance under random assignment. Chetty, Friedman, and Rockoff (2014b) showed VA predicts adult earnings and college attendance, but this long-run evidence concerns VA itself, not observational measures of instruction. A 2023 systematic review of 75 studies (Amrein-Beardsley, Ryan Lavery, Holloway, Pivovarova, and Hahs-Vaughn) concludes that the validity evidence for VAM-based teacher evaluation remains contested.

Problem Statement

Resolve the empirical questions that Hill, Kapitula, and Umland (2011) left open in their validity argument for using teacher value-added scores in high-stakes personnel decisions such as pay-for-performance. The common objective is a defensible validity argument for test-based teacher-effectiveness measures; the four components, all drawn from the source paper's discussion, are:

  1. Causal sources of divergence. Identify, with design-based evidence, which of the following mechanisms drive the cases in which teachers receive high value-added rankings while displaying poor observed instruction — unmeasured compensatory inputs (such as after-school tutoring or parental support), test-focused instruction, stochastic sampling variation, within-school assignment practices, or residual model bias — and, where identifiable, their relative contribution.

  2. Specification and data structure. Determine empirically whether richer model specifications — multivariate response models that use several outcomes jointly, multiyear linked student-teacher data, or student fixed-effect designs (models containing a separate parameter for each student, so that each student serves as their own control across the teachers they experience) — reduce the misclassification of teachers and disentangle teacher quality from student composition, particularly in districts that lack long historical links between students and teachers.

  3. Replication at scale. Establish, in samples substantially larger than the original in-depth study of 24 teachers, whether the partial correlations among value-added scores, teacher knowledge, instructional quality, and student composition replicate — specifically, whether adjusting for measured teacher knowledge and instruction consistently eliminates the association between value-added scores and the composition of students taught.

  4. Instruments and combined systems. Assess the post-2011 evidence bearing on which observable instructional features and observational instruments predict long-term student learning (not only same-year test gains), and on the accuracy of combined measurement systems — value-added scores together with observational and related measures — in personnel decisions; where that evidence is absent, state the gap explicitly.

The four components are independent of one another, and they concern the evidence accumulated since 2011 rather than new primary data or instruments.

References

  1. Heather C. Hill, Laura Kapitula, and Kristin Umland (2011). A Validity Argument Approach to Evaluating Teacher Value-Added Scores. American Educational Research Journal 48(3), 794-831. DOI: 10.3102/0002831210387916, https://doi.org/10.3102/0002831210387916
  2. Heather C. Hill (2009). Evaluating value-added models: A validity argument approach. Journal of Policy Analysis and Management 28(4), 700-709. DOI: 10.1002/pam.20463, https://doi.org/10.1002/pam.20463
  3. Michael T. Kane (2013). Validating the Interpretations and Uses of Test Scores. Journal of Educational Measurement 50(1), 1-73. DOI: 10.1111/jedm.12000, https://doi.org/10.1111/jedm.12000
  4. Jesse Rothstein (2010). Teacher Quality in Educational Production: Tracking, Decay, and Student Achievement. Quarterly Journal of Economics 125(1), 175-214. DOI: 10.1162/qjec.2010.125.1.175, https://doi.org/10.1162/qjec.2010.125.1.175
  5. Josh Kinsler (2012). Assessing Rothstein's critique of teacher value-added models. Quantitative Economics 3(2), 333-362. DOI: 10.3982/qe132, https://doi.org/10.3982/qe132
  6. Peter Z. Schochet and Hanley S. Chiang (2013). What Are Error Rates for Classifying Teacher and School Performance Using Value-Added Models? Journal of Educational and Behavioral Statistics 38(2), 142-171. DOI: 10.3102/1076998611432174, https://doi.org/10.3102/1076998611432174
  7. Nicole B. Kersting, Mei-Kuang Chen, and James W. Stigler (2013). Value-added Teacher Estimates as Part of Teacher Evaluations: Exploring the Effects of Data and Model Specifications on the Stability of Teacher Value-added Scores. Education Policy Analysis Archives 21, Article 7. DOI: 10.14507/epaa.v21n7.2013, https://doi.org/10.14507/epaa.v21n7.2013
  8. Kata Mihaly, Daniel McCaffrey, Tim R. Sass, and J. R. Lockwood (2013). Where You Come From or Where You Go? Distinguishing Between School Quality and the Effectiveness of Teacher Preparation Program Graduates. Education Finance and Policy 8(4), 459-493. DOI: 10.1162/edfp_a_00110, https://doi.org/10.1162/edfp_a_00110
  9. Thomas J. Kane, Daniel F. McCaffrey, Trey Miller, and Douglas O. Staiger (2013). Have We Identified Effective Teachers? Validating Measures of Effective Teaching Using Random Assignment. MET Project Research Paper. ERIC document ED540959. https://eric.ed.gov/?id=ED540959
  10. Jesse Rothstein and William J. Mathis (2013). Review of 'Have We Identified Effective Teachers?' and 'A Composite Estimator of Effective Teaching: Culminating Findings from the Measures of Effective Teaching Project'. National Education Policy Center review. ERIC document ED539299. https://eric.ed.gov/?id=ED539299
  11. Heather C. Hill, David Blazar, Andrea Humez, Erica Litke, Mary Beisiegel, Johanna Barmore, Mark Chin, Douglas Corey, Sara Roesler, Lucas Salzman, David Braslow, and Samantha Rabinowicz (2013). Examining High and Low Value-Added Mathematics Instruction: Can Expert Observers Tell the Difference? CPRE Working Paper WP-07. DOI: 10.12698/CPRE.2013.WP07, https://doi.org/10.12698/CPRE.2013.WP07
  12. Raj Chetty, John N. Friedman, and Jonah E. Rockoff (2014). Measuring the Impacts of Teachers I: Evaluating Bias in Teacher Value-Added Estimates. American Economic Review 104(9), 2593-2632. DOI: 10.1257/aer.104.9.2593, https://doi.org/10.1257/aer.104.9.2593
  13. Raj Chetty, John N. Friedman, and Jonah E. Rockoff (2014). Measuring the Impacts of Teachers II: Teacher Value-Added and Student Outcomes in Adulthood. American Economic Review 104(9), 2633-2679. DOI: 10.1257/aer.104.9.2633, https://doi.org/10.1257/aer.104.9.2633
  14. Dan Goldhaber and Duncan Dunbar Chaplin (2015). Assessing the 'Rothstein Falsification Test': Does It Really Show Teacher Value-Added Models Are Biased? Journal of Research on Educational Effectiveness 8(1), 8-34. DOI: 10.1080/19345747.2014.978059, https://doi.org/10.1080/19345747.2014.978059
  15. Cassandra M. Guarino, Mark D. Reckase, and Jeffrey M. Wooldridge (2015). Can Value-Added Measures of Teacher Performance Be Trusted? Education Finance and Policy 10(1), 117-156. DOI: 10.1162/edfp_a_00153, https://doi.org/10.1162/edfp_a_00153
  16. Heather C. Hill, Kristin Umland, Erica Litke, and Laura R. Kapitula (2012). Teacher Quality and Quality Teaching: Examining the Relationship of a Teacher Assessment to Practice. American Journal of Education 118(4), 489-519. DOI: 10.1086/666380, https://doi.org/10.1086/666380
  17. David Blazar, Erica Litke, and Johanna Barmore (2016). What Does It Mean to Be Ranked a 'High' or 'Low' Value-Added Teacher? Observing Differences in Instructional Quality Across Districts. American Educational Research Journal 53(2), 324-359. DOI: 10.3102/0002831216630407, https://doi.org/10.3102/0002831216630407
  18. David Blazar and Matthew A. Kraft (2017). Teacher and Teaching Effects on Students' Attitudes and Behaviors. Educational Evaluation and Policy Analysis 39(1), 146-170. DOI: 10.3102/0162373716670260, https://doi.org/10.3102/0162373716670260
  19. Jesse Rothstein (2017). Measuring the Impacts of Teachers: Comment. American Economic Review 107(6), 1656-1684. DOI: 10.1257/aer.20141440, https://doi.org/10.1257/aer.20141440
  20. David Blazar (2018). Validating Teacher Effects on Students' Attitudes and Behaviors: Evidence from Random Assignment of Teachers to Students. Education Finance and Policy 13(3), 281-309. DOI: 10.1162/edfp_a_00251, https://doi.org/10.1162/edfp_a_00251
  21. Andrew Bacher-Hicks, Mark J. Chin, Thomas J. Kane, and Douglas O. Staiger (2019). An experimental evaluation of three teacher quality measures: Value-added, classroom observations, and student surveys. Economics of Education Review 73, 101919. DOI: 10.1016/j.econedurev.2019.101919, https://doi.org/10.1016/j.econedurev.2019.101919
  22. Audrey Amrein-Beardsley, Matthew Ryan Lavery, Jessica Holloway, Margarita Pivovarova, and Debbie L. Hahs-Vaughn (2023). Evaluating the validity evidence surrounding use of value-added models to evaluate teachers: A systematic review. Education Policy Analysis Archives 31. DOI: 10.14507/epaa.31.8201, https://doi.org/10.14507/epaa.31.8201
  23. Mark White (2026). The Impact of the Association Between Student Characteristics and Classroom Observation Scores on Different Interpretations of Scores. Journal of Research on Educational Effectiveness 19(2), 570-593. DOI: 10.1080/19345747.2025.2507350, https://doi.org/10.1080/19345747.2025.2507350