A systematic review of psychological research with open data found that measurement invariance (MI) is almost never tested, that when tests are performed invariance usually fails, and that configural invariance fails in the large majority of comparisons — in most of those rejections the data indicated more than one underlying factor, and in nearly half of them the number of factors differed between groups. The source review left two scientifically open questions on this failure side: (1) there are no clear-cut, field-usable guidelines for deciding when statistically detected noninvariance is practically meaningful, i.e., when it biases a substantive group or time comparison enough to change conclusions; and (2) the causes of the frequent configural-invariance failures and group-differences in dimensionality have not been systematically investigated. Later literature extends noninvariance effect-size measures and debates whether MI testing is even appropriate, but supplies neither validated interpretation guidelines nor a causal account, so the problem remains open.

Background

Psychologists routinely compare averages of latent constructs — traits, attitudes, symptoms — across groups, experimental conditions, or time points using multi-item self-report scales. Because the construct itself is not directly observed, such comparisons are only meaningful if the scale measures the same construct with the same measurement properties in every group; this property is called measurement invariance (MI). MI is usually tested in a sequence of confirmatory factor analysis (CFA) models, a statistical framework in which responses to scale items are modeled as indicators of one or more latent (unobserved) factors: configural invariance requires the same factor structure in each group, metric invariance additionally requires equal factor loadings (the regression of items on the latent factor), and scalar invariance additionally requires equal item intercepts, which is the level needed before latent mean differences can be interpreted as true group differences. Failure of MI means observed group differences may partly reflect differences in how the scale measures rather than differences in the construct. In item response theory, the analogous phenomenon of group-dependent item behavior is called differential item functioning (DIF). A systematic review by Maassen, D'Urso, van Assen, Nuijten, De Roover, and Wicherts (Psychological Methods, 2025; first published online December 2023) examined 426 psychology articles with openly available data and 929 group/time comparisons within them: only 4% of comparisons had undergone MI testing, none of the reported MI tests could be reproduced, and of 174 MI tests the reviewers performed themselves, only 26% reached scalar invariance while MI failed completely in 58%. Strikingly, when a one-factor configural model was tested, configural invariance was rejected in 101 of 113 comparisons, and in 84 of those 101 the data indicated more than one underlying factor in the total sample; in 46 of the 101 (nearly half), the number of factors varied between groups, meaning that the scale's dimensionality itself differed between groups. Half of the scales in the sample had been modified ad hoc (items added, deleted, or altered), rarely with justification, and such modifications mostly produced MI violations. The review's discussion identifies the unresolved scientific core on this failure side. First, practical significance: standardized effect-size measures for noninvariance do exist — most prominently dMACS (Nye & Drasgow, 2011; Nye et al., 2018; Gunn et al., 2019), which quantifies the discrepancy in an item's expected observed-score difference across groups — but, as Borsboom (2006) argued early on, there are no clear-cut guidelines for deciding when statistically significant noninvariance is also practically meaningful; the answer is known to depend on the magnitude of the violation relative to the target effect, the number of noninvariant items, and the research context, and researchers still face challenges in determining when noninvariance becomes practically meaningful. Second, causes: the observed epidemic of configural failures and group-differential dimensionality has not been causally explained — candidate causes include ad hoc scale modification, cultural or language differences, treatment-induced shifts in item meaning, and fundamentally misspecified constructs, but no systematic study has adjudicated among them. The discussion also raises a methodologically linked handling concern: the standard remedy of partial invariance — freeing the parameters flagged by modification indices or expected parameter change (statistics that estimate, for each fixed model parameter, how much overall fit would improve and how much the parameter would shift if it were freed) or by residual inspection of misfit patterns (e.g., Muthén & Asparouhov, 2012; Oberski, 2014) — rests on post hoc, data-driven decisions that capitalize on chance and risk overfitting, so models may fail to cross-validate. A lively methodological debate — Welzel et al. (2023) claiming non-invariance is an overstated problem, Meuleman et al. (2023) defending MI testing, and causal and pragmatic reformulations by Rohrer & Paulewicz (2025), Kusano et al. (2025), and Bratt (2025) — presupposes, rather than provides, a quantitative account of when noninvariance actually distorts conclusions.

Problem Statement

The problem, as posed by the source review's discussion of the failure side of measurement invariance, has two jointly linked scientific questions that together make the widespread noninvariance in psychological scales scientifically interpretable. (1) Practical significance of noninvariance: establish determinate, field-usable criteria — expressed in standardized effect-size units for measurement noninvariance (the dMACS family and its generalizations) — for deciding when statistically detected noninvariance in a scale biases a substantive group or time comparison (e.g., a latent mean difference) enough to change the substantive conclusion, together with an account of the factors on which this threshold depends: the magnitude of the noninvariance relative to the target effect, the number of noninvariant items, and the research context (e.g., item content and the goal of the study). The criteria must be validated, not merely proposed: validation requires showing that a criterion correctly identifies whether noninvariance of a given magnitude in a scale with a given number of noninvariant items changes the substantive conclusion, and doing so across the dependency factors named above. (2) Causes of configural failure: systematically investigate why configural invariance fails so often — in particular, why the data so often indicate more than one underlying factor and why the number or structure of factors at times differs between groups — and adjudicate among the candidate causes discussed in the source (at minimum: ad hoc scale modification, response-style differences, cultural or language differences, treatment-induced shifts in item meaning, and fundamentally misspecified constructs), so as to provide an account, grounded in systematic evidence from the source review's open-data corpus or comparable open datasets rather than speculation, of which of these causes are responsible for the observed configural failures and group-differential dimensionality, and to what extent. A methodologically linked concern recognized by the source is handling: current approaches to dealing with partial noninvariance rest on post hoc parameter freeing (modification indices, expected parameter change, residual inspection) that capitalize on chance and risk overfitting, so models may fail to cross-validate; the source raises this as a caution about current practice that contextualizes the two questions, not as a third open question.

References

  1. Maassen, E., D'Urso, E. D., van Assen, M. A. L. M., Nuijten, M. B., De Roover, K., & Wicherts, J. M. (2025). The dire disregard of measurement invariance testing in psychological science. Psychological Methods, 30(5), 966-979. DOI: 10.1037/met0000624. https://doi.org/10.1037/met0000624
  2. Borsboom, D. (2006). When does measurement invariance matter? Medical Care, 44(Suppl 3), S176-S181. DOI: 10.1097/01.mlr.0000245143.08679.cc. https://doi.org/10.1097/01.mlr.0000245143.08679.cc
  3. Nye, C. D., & Drasgow, F. (2011). Effect size indices for analyses of measurement equivalence: Understanding the practical importance of differences between groups. Journal of Applied Psychology, 96(5), 966-980. DOI: 10.1037/a0022955. https://doi.org/10.1037/a0022955
  4. Nye, C. D., Bradburn, J., Olenick, J., Bialko, C., & Drasgow, F. (2018). How big are my effects? Examining the magnitude of effect sizes in studies of measurement equivalence. Organizational Research Methods, 22(3), 678-709. DOI: 10.1177/1094428118761122. https://doi.org/10.1177/1094428118761122
  5. Gunn, H. J., Grimm, K. J., & Edwards, M. C. (2019). Evaluation of six effect size measures of measurement non-invariance for continuous outcomes. Structural Equation Modeling: A Multidisciplinary Journal, 27(4), 503-514. DOI: 10.1080/10705511.2019.1689507. https://doi.org/10.1080/10705511.2019.1689507
  6. Muthén, B., & Asparouhov, T. (2012). Bayesian structural equation modeling: A more flexible representation of substantive theory. Psychological Methods, 17(3), 313-335. DOI: 10.1037/a0026802. https://doi.org/10.1037/a0026802
  7. Oberski, D. L. (2014). Evaluating sensitivity of parameters of interest to measurement invariance in latent variable models. Political Analysis, 22(1), 45-60. DOI: 10.1093/pan/mpt014. https://doi.org/10.1093/pan/mpt014
  8. Jak, S., Jorgensen, T. D., Verdam, M. G. E., Oort, F. J., & Elffers, L. (2021). Analytical power calculations for structural equation modeling: A tutorial and Shiny app. Behavior Research Methods, 53(4), 1385-1406. DOI: 10.3758/s13428-020-01479-0. https://doi.org/10.3758/s13428-020-01479-0
  9. Jobst, L. J., Bader, M., & Moshagen, M. (2023). A tutorial on assessing statistical power and determining sample size for structural equation models. Psychological Methods, 28(1), 207-221. DOI: 10.1037/met0000423. https://doi.org/10.1037/met0000423
  10. McNeish, D., & Wolf, M. G. (2023). Dynamic fit index cutoffs for confirmatory factor analysis models. Psychological Methods, 28(1), 61-88. DOI: 10.1037/met0000425. https://doi.org/10.1037/met0000425
  11. Welzel, C., Brunkert, L., Kruse, S., & Inglehart, R. F. (2023). Non-invariance? An overstated problem with misconceived causes. Sociological Methods & Research, 52(3), 1368-1400. DOI: 10.1177/0049124121995521. https://doi.org/10.1177/0049124121995521
  12. Meuleman, B., Żółtak, T., Pokropek, A., Davidov, E., Muthén, B., Oberski, D. L., Billiet, J., & Schmidt, P. (2023). Why measurement invariance is important in comparative research. A response to Welzel et al. (2021). Sociological Methods & Research, 52(3), 1401-1419. DOI: 10.1177/00491241221091755. https://doi.org/10.1177/00491241221091755
  13. Sterner, P., De Roover, K., & Goretzko, D. (2024). New developments in measurement invariance testing: An overview and comparison of EFA-based approaches. Structural Equation Modeling: A Multidisciplinary Journal, 32(1), 117-135. DOI: 10.1080/10705511.2024.2393647. https://doi.org/10.1080/10705511.2024.2393647
  14. Lasker, J. (2024). Measurement invariance testing works. Applied Psychological Measurement, 48(6), 257-275. DOI: 10.1177/01466216241261708. https://doi.org/10.1177/01466216241261708
  15. Schlechter, P., Hillmann, M., & Neufeld, S. A. S. (2024). Gender, age, and longitudinal measurement invariance of child and adolescent depression scales: A systematic review. Clinical Psychology Review, 113, 102481. DOI: 10.1016/j.cpr.2024.102481. https://doi.org/10.1016/j.cpr.2024.102481
  16. Nilsen, S. A., Stormark, K. M., Bang, L., Brunborg, G. S., Larsen, M., & Breivik, K. (2024). Time trends in adolescent depressive symptoms from 2010 to 2019 in Norway: real increase or artifacts of measurements? Psychological Medicine, 54(14), 3949-3961. DOI: 10.1017/S0033291724002447. https://doi.org/10.1017/S0033291724002447
  17. Byrne, D., Doyle, F., Brannick, S., Carney, R. M., Cuijpers, P., Dima, A. L., Freedland, K. E., Guerin, S., Hevey, D., Kathuria, B., Wallace, E., & Boland, F. (2025). Do complex psychometric analyses really matter? Comparing multiple approaches using individual participant data from antidepressant trials. Psychological Medicine, 55. DOI: 10.1017/s0033291725101785. https://doi.org/10.1017/s0033291725101785
  18. Ozcan, M., & Lai, M. H. C. (2025). Exploring the impact of deleting (or retaining) a biased item: A procedure based on classification accuracy. Assessment, 32(8), 1211-1225. DOI: 10.1177/10731911241298081. https://doi.org/10.1177/10731911241298081
  19. Rohrer, J. M., & Paulewicz, B. (2025). Rethinking measurement invariance causally. Current Research in Ecological and Social Psychology, 9, 100241. DOI: 10.1016/j.cresp.2025.100241. https://doi.org/10.1016/j.cresp.2025.100241
  20. Kusano, K., Napier, J. L., & Jost, J. T. (2025). The mismeasure of culture: Why measurement invariance is rarely appropriate for comparative research in psychology. Personality and Social Psychology Bulletin, 52(8), 2578-2595. DOI: 10.1177/01461672251341402. https://doi.org/10.1177/01461672251341402
  21. Bratt, C. (2025). Benefits of a pragmatic approach: Rethinking measurement invariance and composite scores in cross-cultural research. Sociological Methods & Research. DOI: 10.1177/00491241251405869. https://doi.org/10.1177/00491241251405869
  22. Lai, M. H. C., Zhang, Y., Ozcan, M., Tse, W. W.-Y., & Miles, A. (2025). fMACS: Generalizing dMACS effect size for measurement noninvariance with multiple groups and multiple grouping variables. Structural Equation Modeling: A Multidisciplinary Journal, 32, 638-646. DOI: 10.1080/10705511.2025.2484812. https://doi.org/10.1080/10705511.2025.2484812
  23. Schuhbeck, T. M. B., Sterner, P., & Goretzko, D. (2025). Quantifying measurement non-invariance beyond simple structure: The closed formulas of universal effect size measures for MI (PsyArXiv preprint). DOI: 10.31234/osf.io/cq9vk_v2. https://osf.io/cq9vk
  24. Schuhbeck, T. M. B., Partsch, M., Sterner, P., & Goretzko, D. (2026). Effect-based assessments of intergroup comparability – The practical impact of measurement non-invariance & differential item functioning on psychological tests and scales (PsyArXiv preprint). https://osf.io/sjtp3