Empirical software engineering asks whether measurable properties of a test suite predict the future defect-proneness of the production code it exercises. When Pecorelli, Palomba, and De Lucia (Empirical Software Engineering, 2021) studied eight Apache Commons systems with multivariate models controlling for production-code and process confounders, most classic test-related factors — the presence and executability of tests, line and branch coverage, assertion density, and nearly all test smells — showed no significant relation to post-release defects; test-suite size and the Mystery Guest smell were weak exceptions, while process metrics dominated. The authors concluded that understanding the effect of tests on post-release defects remains an open problem. No published work replicates or generalizes that multivariate analysis beyond Apache Commons or to integration and system tests; adjacent threads (method-level test quality, test-to-code traceability, flakiness prediction, coverage–effectiveness surveys) have advanced the measurement prerequisites but not the core question. The problem remains open: determine which, if any, test-suite properties carry genuine incremental power for post-release defect prediction once confounders are controlled — across ecosystems and testing levels, with accurate test-to-code traceability, with newly designed metrics, and with the hypothesized smell–flakiness–defect pathway actually tested.

Background

A central concern of empirical software engineering is whether the effort spent on regression testing pays off in measurable product quality. The standard observable is the number of post-release defects of a production class — defects reported or fixed after a release, typically identified by mining the version history and linking bug-fixing commits back to the classes they touch (e.g., with SZZ-style algorithms). The question is which properties of the test suite exercising a class, if any, predict that class's future defect-proneness.

The relevant test-suite properties span three levels. Presence-level factors record whether a production class has tests at all and whether those tests are executable. Static factors are computable from the test code without running it: the size of the test suite (lines of test code), assertion density (the number of assertion statements relative to test code size), and test smells — recurring design flaws in test code, such as Mystery Guest, a smell in which a test relies on external resources or state rather than setting up its own fixtures, making its behavior potentially non-deterministic. Dynamic factors require executing the tests: line and branch coverage (the fraction of production code lines or branches executed by the tests) and mutation coverage (the fraction of artificially injected small code changes, called mutants, that the tests detect — a common proxy for the tests' fault-detection power).

Earlier studies generally analyzed these factors in isolation and reported encouraging relations: coverage and assertion counts were linked to fewer defects or higher reliability, and smelly test suites were linked to more fault-prone production code (Spadini et al. 2018; Qusef et al. 2019). Two results complicated this picture: Inozemtseva and Holmes (2014) showed that coverage is not strongly correlated with test suite effectiveness once suite size is accounted for, and Kochhar et al. (2017), in a large-scale study of open source projects, found coverage had an insignificant relation to post-release defects. What was missing was a joint analysis: do test-related factors keep their apparent predictive power when considered together and when the well-known drivers of defect-proneness — production code size, code smells, and process metrics such as the number of pre-release changes and pre-release defects — are controlled for as confounders?

Pecorelli, Palomba, and De Lucia (2021) provided exactly this analysis as a case study on eight Apache Commons systems (1,457 production classes, generalized linear models with progressively richer confounder sets). Their findings were sobering: neither the presence nor the executability of tests was significantly related to post-release defects; line and branch coverage were never significant; assertion density and most test smells were not significant once confounders were included; mutation coverage was significant only when test factors were considered in isolation. The exceptions were the size of the test suite — larger test suites were associated with fewer post-release defects in the full models, suggesting test size as a rough proxy for test robustness — and the Mystery Guest smell, whose significant association the authors tentatively attributed to an indirect relation with test flakiness (the phenomenon of tests non-deterministically passing and failing on the same code), while explicitly leaving this interpretation to further empirical investigation. Process metrics, especially the number of pre-release changes, dominated every model.

The study itself identified the limitations that make its conclusion provisional rather than settling: it covered a single ecosystem (Apache Commons) and only unit-level tests; and it linked production classes to their tests by naming convention (a class Foo is linked to the test class FooTest), which can under- or over-estimate the tests that actually exercise a class — a threat that one interviewed Apache developer concretely documented for a defect-prone class in Commons-Pool. The authors state that the problem of understanding the effect of tests on post-release defects is still open, and set out the following research agenda: replicate and generalize the multivariate analysis; design test-related metrics that actually capture the defect-preventing quality of tests; improve scalable test-to-production-code traceability; investigate the proposed causal pathway through flakiness; and examine the role of test-suite diversity on software quality.

Problem Statement

Determine which measurable properties of software test suites — if any — have genuine incremental explanatory or predictive power for the post-release defect-proneness of the production code they exercise, where either kind of incremental power suffices, once production-code and process confounders are controlled for. The question is left open in exactly this form by Pecorelli, Palomba, and De Lucia (2021), whose eight-system Apache Commons case study found only a limited connection for nearly all classic test-related factors. It comprises four jointly linked facets that share this single objective, and resolving some facets does not discharge the others: (1) Generalization — re-run the multivariate, confounder-controlled analysis of test-related factors (presence and executability of tests; static factors such as test-suite size, assertion density, and test smells including Mystery Guest; dynamic factors such as line, branch, and mutation coverage) against post-release defects on a substantially larger and more diverse set of systems than the eight Apache Commons projects, and beyond unit tests, including integration and system tests, so the conclusion can be attributed to the phenomenon rather than to one ecosystem, testing level, or sample. (2) Measurement validity — establish whether naming-convention test-to-production-code linking distorts the estimated relations between test factors and post-release defects, and in which direction, since misattributed test-code links may distort those relations in either direction. (3) Metric design — establish whether test-related metrics can be designed that retain incremental power against the full confounder set — whether as statistical significance (the explanatory reading) or as an incremental out-of-sample predictive gain over the confounder-only model (the predictive reading) — and thereby capture the defect-preventing quality of tests which coverage, assertion density, and the other standard metrics fail to capture; a single demonstrated metric (or metric family) meeting this standard resolves this facet, with any general claim limited to what the evidence supports. (4) Mechanism — empirically test the hypothesized indirect relation between the Mystery Guest test smell, test flakiness, and defect-related outcomes, whose existence and direction the source proposed as an interpretation but explicitly left unverified. The analysis should determine whether the weak connection observed on Apache Commons reflects an inadequacy of current test metrics, a measurement artifact of traceability and test-level scope, or a genuinely limited role of test suites in post-release quality, whether the hypothesized smell–flakiness–defect pathway holds, and which test properties (if any) defect prediction and automated test generation should rely on.

References

  1. Fabiano Pecorelli, Fabio Palomba, Andrea De Lucia. The Relation of Test-Related Factors to Software Quality: A Case Study on Apache Systems. Empirical Software Engineering 26(2), article 18, 2021. DOI: 10.1007/s10664-020-09891-y. https://doi.org/10.1007/s10664-020-09891-y
  2. Pavneet Singh Kochhar, David Lo, Julia Lawall, Nachiappan Nagappan. Code Coverage and Postrelease Defects: A Large-Scale Study on Open Source Projects. IEEE Transactions on Reliability 66(4):1213-1228, 2017. DOI: 10.1109/TR.2017.2727062. https://doi.org/10.1109/TR.2017.2727062
  3. Laura Inozemtseva, Reid Holmes. Coverage is not strongly correlated with test suite effectiveness. Proceedings of the 36th International Conference on Software Engineering (ICSE 2014), pages 435-445, 2014. DOI: 10.1145/2568225.2568271. https://doi.org/10.1145/2568225.2568271
  4. Davide Spadini, Fabio Palomba, Andy Zaidman, Magiel Bruntink, Alberto Bacchelli. On the Relation of Test Smells to Software Code Quality. IEEE International Conference on Software Maintenance and Evolution (ICSME 2018), pages 1-12, 2018. DOI: 10.1109/ICSME.2018.00010. https://doi.org/10.1109/ICSME.2018.00010
  5. Abdallah Qusef, Mahmoud O. Elish, David Binkley. An Exploratory Study of the Relationship Between Software Test Smells and Fault-Proneness. IEEE Access 7:139526-139536, 2019. DOI: 10.1109/ACCESS.2019.2943488. https://doi.org/10.1109/ACCESS.2019.2943488
  6. Fabiano Pecorelli, Gemma Catolino, Filomena Ferrucci, Andrea De Lucia, Fabio Palomba. Software testing and Android applications: a large-scale empirical study. Empirical Software Engineering 27, article 31, 2022. DOI: 10.1007/s10664-021-10059-5. https://doi.org/10.1007/s10664-021-10059-5
  7. Victor Veloso, André C. Hora. Characterizing high-quality test methods. Proceedings of the 19th International Conference on Mining Software Repositories (MSR 2022), pages 265-269, 2022. DOI: 10.1145/3524842.3529092. https://doi.org/10.1145/3524842.3529092
  8. Robert White, Jens Krinke. TCTracer: Establishing test-to-code traceability links using dynamic and static techniques. Empirical Software Engineering 27, article 67, 2022. DOI: 10.1007/s10664-021-10079-1. https://doi.org/10.1007/s10664-021-10079-1
  9. Valeria Pontillo, Fabio Palomba, Filomena Ferrucci. Static test flakiness prediction: How Far Can We Go? Empirical Software Engineering 27, article 187, 2022. DOI: 10.1007/s10664-022-10227-1. https://doi.org/10.1007/s10664-022-10227-1
  10. Bruno Camara, Marco Silva, Andre Endo, Silvia Vergilio. On the use of test smells for prediction of flaky tests. Brazilian Symposium on Systematic and Automated Software Testing (SAST 2021), pages 46-54, 2021. DOI: 10.1145/3482909.3482916. https://doi.org/10.1145/3482909.3482916
  11. Fabiano Pecorelli, Giovanni Grano, Fabio Palomba, Harald C. Gall, Andrea De Lucia. Toward granular search-based automatic unit test case generation. Empirical Software Engineering 29, article 71, 2024. DOI: 10.1007/s10664-024-10451-x. https://doi.org/10.1007/s10664-024-10451-x
  12. Maryam Barani, Yvan Labiche, Antoine Rollet. On factors that impact the relationship between code coverage and test suite effectiveness: a survey. 2023 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW 2023), pages 381-388, 2023. DOI: 10.1109/ICSTW58534.2023.00071. https://doi.org/10.1109/ICSTW58534.2023.00071
  13. Fabio Palomba, Andy Zaidman. The smell of fear: on the relation between test smells and flaky tests. Empirical Software Engineering 24:2907-2946, 2019 (article retracted; retraction note DOI 10.1007/s10664-020-09821-y). DOI: 10.1007/s10664-019-09683-z. https://doi.org/10.1007/s10664-019-09683-z
  14. Fabiano Pecorelli. Test-related factors and post-release defects: an empirical study. Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE 2019), pages 1235-1237, 2019. DOI: 10.1145/3338906.3342500. https://doi.org/10.1145/3338906.3342500