A school's value-added (VA) estimate - the school-average residual of student achievement after adjusting for prior attainment and background - is known to be unstable from cohort to cohort, which undermines its use in accountability and school-choice policy. For Italian primary schools, Agasisti and Minaya (2021) showed that this instability does not depend on students' prior achievement or socio-economic background, implicating transitory factors such as changes in teacher effectiveness; however, the INVALSI administrative data they use contain no teacher or principal identifiers, and the observed inter-year variability in school VA estimates has never been decomposed into persistent school effects, transitory cohort- or teacher-specific effects, and sampling or measurement error. The open problem is to identify and quantify these components for Italian primary education - of which attributing the transitory component specifically to teacher turnover and time-varying teacher effectiveness requires teacher-level information such as linked teacher-student administrative data or repeated teacher assignments across cohorts - and to establish whether longer panels or explicit hierarchical time-series models of school effects can recover the persistent component. Studies from the United States, where teacher-student linkage exists, supply the methodological template (variance decompositions at teacher level and school-level sampling-variance analysis) but no answer for the Italian institutional setting, in which the same teachers typically follow a class through the five-year primary cycle.

Background

A value-added (VA) model measures a school's contribution to student learning as the average difference between students' observed achievement and the achievement predicted from their prior test scores and background characteristics; the school effect is the school-mean residual of such a regression, usually estimated with shrinkage (partial pooling toward the overall mean in a multilevel model, so that small noisy schools are not taken at face value). Because VA estimates feed accountability systems, school choice and, in some countries, personnel decisions, their precision and year-to-year stability matter: if a school's VA rank changes substantially from one cohort to the next, the measure cannot support high-stakes use. Kane and Staiger (2002) showed for schools in the United States that year-to-year changes in school-level test scores are dominated by sampling variance - noise arising because a school's score is an average over the small number of students in a grade (the median US elementary school has only about 69 children per grade level) - so small schools are mechanically over-represented among the best and worst performers. Subsequent work found school VA measures only moderately stable in England (Perry 2016; Dumay, Coe and Anumendem 2014) and showed that richer covariate adjustment does not remove the problem (Leckie and Goldstein 2019; Emslander, Levy, Scherer and Fischbach 2022; see Levy, Brunner, Keller and Fischbach 2019 for an international review of VA modelling choices, which found regression and multilevel models dominant and most studies applying no shrinkage). Where student-teacher linkage exists, the analogous teacher-level question has been studied directly: McCaffrey, Sass, Lockwood and Mihaly (2009) decomposed the year-to-year variability of teacher VA estimates in Florida into roughly 30-60 percent sampling error, a persistent teacher component (about half of the remaining variance in elementary school), and residual teacher-level time-varying factors, and found that averaging two years of estimates improves prediction of future performance; Goldhaber and Hansen (2013) rejected the hypothesis that teacher performance is perfectly stable over a ten-year panel while finding a persistent within-teacher component; and Chetty, Friedman and Rockoff (2014) anchored the interpretation of teacher VA in large-scale validation exercises. For Italy, Agasisti and Minaya (2021) study primary schools using administrative data from INVALSI, the national evaluation institute that administers census-based standardized tests whose records can be linked to students and schools across cohorts. They find that the stability of school VA estimates across cohorts does not depend on students' previous achievement or socio-economic background, so composition-based explanations are insufficient and transitory changes - such as changes in teacher effectiveness - are implicated. Their companion study (Minaya and Agasisti 2019) confirms with the same data that school performance estimates are unstable across metrics and that correlations across cohorts decline over time. The Italian primary setting is distinctive because students are typically taught by the same team of teachers for the whole five-year cycle, so cohort-to-cohort changes in a school's VA plausibly reflect teacher turnover and time-varying teacher effectiveness as well as persistent school quality and pure noise. But the INVALSI-based data used in these studies contain no teacher or principal identifiers, and in their absence the relative contributions of these sources have never been identified.

Problem Statement

Identify and quantify the sources of year-to-year variability in school value-added estimates for Italian primary schools, and determine whether the persistent component of school effectiveness can be recovered. Concretely, for the Italian administrative assessment setting, all of the following are required: (1) specify and derive a decomposition of the inter-year variability of school value-added estimates into (a) persistent school effects, (b) transitory cohort- and teacher-specific effects, including those arising from teacher turnover and time-varying teacher effectiveness (whose precise separation requires linked teacher-student administrative data or repeated teacher assignments across cohorts, so its estimand and the data that would identify it are to be specified), and (c) sampling variance and measurement error, including the contributions of small school sizes; and (2) establish separately, for each of pooling over more cohorts than the three fifth-grade cohorts (2013-2015) used in the working-paper precursor (Agasisti and Minaya 2018) of the authors' companion study and explicit hierarchical time-series models of school effects, whether the persistent component is recovered rather than averaged over transitory shocks. The primary target is the derivation and identification of (1) and the recoverability determination of (2); empirical magnitudes for Italian schools are sought where such linked data are available. These deliverables serve one overarching objective - a source attribution of school VA instability in Italian primary education. Evidence from other educational systems and school levels bears on the external validity of the answer.

References

  1. Agasisti, T., & Minaya, V. (2021). Precision and stability of schools' value-added estimates: evidence for Italian primary schools. Applied Economics Letters 28(7), 541-545. DOI: 10.1080/13504851.2020.1763242. URL: https://doi.org/10.1080/13504851.2020.1763242
  2. Minaya, V., & Agasisti, T. (2019). Evaluating the stability of school performance estimates over time. Fiscal Studies 40(3), 401-425. DOI: 10.1111/1475-5890.12201. URL: https://doi.org/10.1111/1475-5890.12201
  3. Kane, T. J., & Staiger, D. O. (2002). The promise and pitfalls of using imprecise school accountability measures. Journal of Economic Perspectives 16(4), 91-114. DOI: 10.1257/089533002320950993. URL: https://doi.org/10.1257/089533002320950993
  4. McCaffrey, D. F., Sass, T. R., Lockwood, J. R., & Mihaly, K. (2009). The intertemporal variability of teacher effect estimates. Education Finance and Policy 4(4), 572-606. DOI: 10.1162/edfp.2009.4.4.572. URL: https://doi.org/10.1162/edfp.2009.4.4.572
  5. Goldhaber, D., & Hansen, M. (2013). Is it just a bad class? Assessing the long-term stability of estimated teacher performance. Economica 80(319), 589-612. DOI: 10.1111/ecca.12002. URL: https://doi.org/10.1111/ecca.12002
  6. Chetty, R., Friedman, J. N., & Rockoff, J. E. (2014). Measuring the impacts of teachers I: Evaluating bias in teacher value-added estimates. American Economic Review 104(9), 2593-2632. DOI: 10.1257/aer.104.9.2593. URL: https://doi.org/10.1257/aer.104.9.2593
  7. Emslander, V., Levy, J., Scherer, R., & Fischbach, A. (2022). Value-added scores show limited stability over time in primary school. PLOS ONE 17(12), e0279255. DOI: 10.1371/journal.pone.0279255. URL: https://doi.org/10.1371/journal.pone.0279255
  8. Gao, C., & Bi, X. (2023). Stability and consistency of school effects: evidence from senior high schools in China. School Effectiveness and School Improvement 34(3), 358-376. DOI: 10.1080/09243453.2023.2206806. URL: https://doi.org/10.1080/09243453.2023.2206806
  9. Gao, C., Zou, J., Zheng, L., & Yuan, A. (2025). An empirical study on the effectiveness of collaborative school implementation in central China: a value-added approach. School Effectiveness and School Improvement 36(1), 71-91. DOI: 10.1080/09243453.2024.2417048. URL: https://doi.org/10.1080/09243453.2024.2417048
  10. Zhao, J., & Su, H. (2026). Covariate selection and school effectiveness measures in value-added models: A systematic review. Applied Measurement in Education 39(3), 232-248. DOI: 10.1080/08957347.2026.2714502. URL: https://doi.org/10.1080/08957347.2026.2714502
  11. Perry, T. (2016). English value-added measures: Examining the limitations of school performance measurement. British Educational Research Journal 42(6), 1056-1080. DOI: 10.1002/berj.3247. URL: https://doi.org/10.1002/berj.3247
  12. Dumay, X., Coe, R., & Anumendem, D. N. (2014). Stability over time of different methods of estimating school performance. School Effectiveness and School Improvement 25(1), 64-82. DOI: 10.1080/09243453.2012.759599. URL: https://doi.org/10.1080/09243453.2012.759599
  13. Leckie, G., & Goldstein, H. (2019). The importance of adjusting for pupil background in school value-added models: A study of Progress 8 and school accountability in England. British Educational Research Journal 45(3), 518-537. DOI: 10.1002/berj.3511. URL: https://doi.org/10.1002/berj.3511
  14. Levy, J., Brunner, M., Keller, U., & Fischbach, A. (2019). Methodological issues in value-added modeling: an international review from 26 countries. Educational Assessment, Evaluation and Accountability 31(3), 257-287. DOI: 10.1007/s11092-019-09303-w. URL: https://doi.org/10.1007/s11092-019-09303-w
  15. Agasisti, T., & Minaya, V. (2018). Evaluating the stability of school performance estimates for school choice: evidence for Italian primary schools. SIEP Working Papers 67, Societa Italiana di Economia Pubblica. URL: http://www.siepweb.it/siep/wp/wp-content/uploads/repec/1517998327Agasisti_Minaya_WP_SIEP_733.pdf