The Education Value-Added Assessment System (EVAAS), the proprietary value-added model sold by SAS Institute Inc., is used for consequential teacher-evaluation decisions in multiple US states and districts and is advertised as producing precise, reliable, and unbiased results. The only external study of actual EVAAS output — categorical ratings for more than 1,700 teachers in the Houston Independent School District, obtained through litigation — found validity and reliability roughly comparable to other value-added models but possible additional bias linked to school-level student composition, and it could not rule out that the observed patterns reflect the vendor's unexamined data processing rather than model behavior. This problem asks for the independent audit the original investigators called for: using continuous EVAAS estimates, the vendor's raw input files and decision rules, and re-estimation of the same cleaned data under alternative value-added specifications, with adjustment for nonrandom assignment of students to teachers, determine — across districts, states, instruments, and the multivariate versus univariate EVAAS model variants — whether EVAAS ratings can be shown to be sufficiently precise, stable, and unbiased for high-stakes use. A 2023 systematic review of 75 studies found no scholarly consensus on value-added model validity, and no audit of EVAAS with vendor-level access has appeared; the question remains open.

Background

Since the 2000s, many US states and school districts have used value-added models (VAMs) — statistical models that estimate a teacher's contribution to student learning by comparing each teacher's students' test-score growth against expectations based on their prior test scores — as inputs to teacher evaluation, sometimes with consequences such as tenure denial, dismissal, or merit pay (so-called high-stakes use). The most widely marketed commercial VAM is the Education Value-Added Assessment System (EVAAS), sold by the business-analytics software company SAS Institute Inc., which advertises "precise, reliable and unbiased results." EVAAS estimates teacher effects with one of two proprietary layered mixed models: the Multivariate Response Model (MRM), which analyzes all tested subjects and school years jointly in one large covariance model, and the Univariate Response Model (URM), which analyzes a single subject at a time; both apply vendor-controlled rules for organizing, imputing, and cleaning the input data, and both shrink small-sample teacher estimates toward the average (an empirical-Bayes-type stabilization). Districts and teachers typically receive only categorical ratings (e.g., whether a teacher's students made above, at, or below expected growth) rather than the underlying continuous estimates, and SAS has not released its full algorithms or data-processing decision rules, which has hindered independent scrutiny since the first published critique (Amrein-Beardsley, 2008) flagged the shortage of external reviews and validity studies of the model. Because the internals stayed closed, most early criticism was methodological or survey-based; the one exception was Houston, where litigation (Houston Federation of Teachers v. Houston Independent School District) eventually gave outside researchers the actual categorical EVAAS ratings of more than 1,700 teachers. Analyzing these ratings, Amrein-Beardsley and Geiger (2020) found EVAAS performed, in terms of validity (whether the ratings measure the teaching effectiveness they claim to measure) and reliability (the stability of ratings across years and across splits of a teacher's students), roughly in line with other VAMs — itself a low bar given the broader literature's concerns about year-to-year instability — but possibly more biased than other VAMs, with teachers' ratings differing by school-level student composition such as poverty, race/ethnicity, and language status. The closest independent econometric work on the model itself, by Vosters, Guarino, and Wooldridge (2018), could evaluate only the URM, using simulations and administrative (non-vendor) data, and found that ordinary least squares regression often performs at least as well as the more complicated EVAAS URM; the proprietary MRM and the vendor's pre-processing pipeline remained unexamined. What the 2020 Houston study could not do — for lack of access to the continuous estimates, the vendor's raw input files and decision rules, the ability to run alternative VAMs on the identical cleaned data, and raw student-level rosters needed to adjust for the nonrandom assignment of students to teachers (teacher sorting) — defines the open question asked here.

Problem Statement

Determine, through an independent audit with the access the original investigators lacked, whether EVAAS teacher value-added ratings are sufficiently precise, stable, and unbiased for consequential high-stakes use, as called for by Amrein-Beardsley and Geiger (2020). Take that access as a stipulated grant of this exercise: assume that the underlying continuous EVAAS estimates, the vendor's raw input files, its decision rules for data organization, imputation, and cleaning, and the raw student-level test data and vendor rosters have been made available to you, so that no external acquisition of vendor material is required. The audit must jointly establish the following scientifically linked results, each of which is independently required. (1) Continuous-estimate replication: using the underlying continuous EVAAS estimates, the vendor's raw input files, and SAS's decision rules for data organization, imputation, and cleaning — rather than only the categorical ratings delivered to districts — determine whether the bias, precision, and reliability patterns observed in Houston replicate, that is, whether they are attributable to the model's mechanics versus the vendor's pre-processing choices. (2) Specification robustness: re-estimate the identical cleaned data under alternative value-added model specifications and test whether the identified bias patterns and levels of reliability persist. (3) Sorting versus performance: using raw student-level test data and the vendor's rosters, adjust for the nonrandom assignment of students to teachers within classrooms, and determine how much of the observed group differences in teacher ratings reflects residual bias versus true differences in teacher performance. (4) Generalization: establish whether these conclusions hold beyond one large urban district — across other districts, states, test instruments, and EVAAS configurations, including the multivariate response model versus univariate response model variants. The overarching question, which all four components jointly serve, is whether EVAAS can be shown — in multiple contexts and under these audit conditions — to meet precision, stability, and bias standards adequate for consequential high-stakes personnel decisions. Because 'sufficiently precise, stable, and unbiased' and 'standards adequate for consequential high-stakes personnel decisions' are judgments rather than fixed numbers, the audit must also state and defend the operational criterion it applies — the levels of imprecision, instability, and bias it treats as tolerable for such decisions — and it must use one consistent criterion set across all four components. Because the stipulated material is not physically attached, the audit is to be carried out on the stipulated material or, where the stipulated data are unavailable, simulated data may be used for the components that depend on it.

References

  1. Amrein-Beardsley, A., & Geiger, T. (2020). Methodological Concerns About the Education Value-Added Assessment System (EVAAS): Validity, Reliability, and Bias. SAGE Open, 10(2). DOI: 10.1177/2158244020922224. https://doi.org/10.1177/2158244020922224
  2. Amrein-Beardsley, A. (2008). Methodological Concerns About the Education Value-Added Assessment System. Educational Researcher, 37(2), 65-75. DOI: 10.3102/0013189X08316420. https://doi.org/10.3102/0013189X08316420
  3. Amrein-Beardsley, A., & Collins, C. (2012). The SAS Education Value-Added Assessment System (SAS EVAAS) in the Houston Independent School District (HISD): Intended and Unintended Consequences. Education Policy Analysis Archives, 20. DOI: 10.14507/epaa.v20n12.2012. https://doi.org/10.14507/epaa.v20n12.2012
  4. Collins, C. (2014). Houston, We Have a Problem: Teachers Find No Value in the SAS Education Value-Added Assessment System (EVAAS). Education Policy Analysis Archives, 22. DOI: 10.14507/epaa.v22.1594. https://doi.org/10.14507/epaa.v22.1594
  5. Vosters, K. N., Guarino, C. M., & Wooldridge, J. M. (2018). Understanding and evaluating the SAS EVAAS Univariate Response Model (URM) for measuring teacher effectiveness. Economics of Education Review, 66, 191-205. DOI: 10.1016/j.econedurev.2018.08.006. https://doi.org/10.1016/j.econedurev.2018.08.006
  6. Amrein-Beardsley, A., & Geiger, T. J. (2019). Potential sources of invalidity when using teacher value-added and principal observational estimates: artificial inflation, deflation, and conflation. Educational Assessment, Evaluation and Accountability, 31(4), 465-493. DOI: 10.1007/s11092-019-09311-w. https://doi.org/10.1007/s11092-019-09311-w
  7. Paige, M. A., & Amrein-Beardsley, A. (2020). "Houston, We Have a Lawsuit": A Cautionary Tale for the Implementation of Value-Added Models for High-Stakes Employment Decisions. Educational Researcher, 49(5), 350-359. DOI: 10.3102/0013189X20923046. https://doi.org/10.3102/0013189X20923046
  8. Amrein-Beardsley, A., Lavery, M. R., Holloway, J., Pivovarova, M., & Hahs-Vaughn, D. L. (2023). Evaluating the Validity Evidence Surrounding the Use of Value-Added Models to Evaluate Teachers: A Systematic Review. Education Policy Analysis Archives, 31(117). DOI: 10.14507/epaa.31.8201. https://doi.org/10.14507/epaa.31.8201
  9. Amrein-Beardsley, A. (2023). A Florida Teacher Wrongfully Terminated? Alleged (Mis)Uses of Value-Added Model (VAM) Estimates for High-Stakes Teacher Evaluation Decisions. Educational Assessment, Evaluation and Accountability, 35(3), 313-352. DOI: 10.1007/s11092-023-09411-8. https://doi.org/10.1007/s11092-023-09411-8
  10. Chen, J., Wan, X., & Luo, Z. (2025). Historical Development of Value-Added Assessment: A Bibliometric-Content Analysis. SAGE Open, 15(4). DOI: 10.1177/21582440251382664. https://doi.org/10.1177/21582440251382664
  11. Amrein-Beardsley, A., & Geiger, T. J. (2026). Using (unreliable) district-level value-added scores to inform school funding (and potentially other) decisions. Educational Assessment, Evaluation and Accountability, 38(2), 233-264. DOI: 10.1007/s11092-026-09481-4. https://doi.org/10.1007/s11092-026-09481-4