An empirical case study of the Eclipse IDE (Chioteli, Batas, and Spinellis) correlated 126,026 unique field-crash stack traces with JaCoCo coverage and test-class linkage over 216,392 methods of one mature release and found that unit-tested code does not appear to be associated with fewer field failures. The authors left open whether this null result generalizes across projects, programming languages, releases (including less mature ones), and application domains; which fault-level root causes explain crashes in unit-tested code and which testing techniques would have detected them; how much measurement noise (tests failing to run on historical builds, noisy stack frames) biases the measured association; and whether alternative test-effectiveness measures such as mutation score, assertion density, or richer coverage semantics such as exception-handling coverage predict field crash incidence better than the metrics used. No study is known to replicate the crash-level result beyond Eclipse or to validate any test-effectiveness measure against field crash incidence, so the question of when and how unit testing reduces field failures remains open.

Background

Software engineering research distinguishes pre-release defects (found by developers before shipping) from field failures, which are failures experienced by end users of deployed software. In Java systems such as the Eclipse integrated development environment, the field failures that can be observed at scale are typically uncaught exceptions, automatically reported as stack traces (the sequence of active method calls at the moment of failure) through crash-reporting telemetry; the Eclipse project accumulated millions of such reports in its AERI (Advanced Error Reporting for Eclipse) incident database. Unit testing is the practice of testing individual classes or methods in isolation, usually with a framework such as JUnit, where a production class may be linked to a dedicated test class. The most common quantitative proxy for testing effort is code coverage: the fraction of a program's instructions, lines, or branches executed by the test suite, measured by tools such as JaCoCo through bytecode instrumentation. Because coverage says nothing about whether a test checks the right behavior (the test oracle problem), stronger test-effectiveness measures have been proposed: assertion density, the amount of checking a test suite performs per test, and mutation testing, in which small artificial faults (mutants) are systematically seeded into the program and the mutation score (the fraction of mutants the suite detects) is used as a proxy for real fault-detection capability.

Whether greater testing effort actually yields fewer field failures is empirically contested. Studies of coverage versus post-release defect reports diverge: an industrial multiple-case study by Mockus et al. reported an association between coverage and post-verification defects, while Kochhar et al. found no significant correlation between coverage and defects across 100 open-source Java projects, and industrial studies at Volvo (Gren and Antinyan; Antinyan et al.) concluded that unit-test coverage is not a useful test-effectiveness metric. In controlled settings, Inozemtseva and Holmes found coverage not strongly correlated with test suite effectiveness, Zhang and Mesbah found assertions strongly correlated with it, and the mutant-real fault coupling line (Just et al.; Papadakis et al.; Laurent et al.) supports mutation score as a stronger adequacy criterion, with Google reporting industrial-scale deployment of code-review-time mutation testing (Petrović et al.). A separate thread observes that exception-handling code — the implicit control-flow branches taken when an exception is thrown — is systematically under-tested: criteria and models for exception-flow coverage have long existed (Sinha and Harrold; Najumudheen et al.), a defect classification at ABB identified improper testing of error cases as the most common unit-testing weakness (Robinson et al.), and a measurement of 27 open-source libraries found exception-handling code inadequately covered (Lima et al.). Mainstream coverage tools nevertheless still do not report exception-handling coverage.

Against this background, Chioteli, Batas, and Spinellis asked the question directly at the field level: for a mature release of Eclipse they correlated 126,026 unique failure stack traces (from 2,083,979 crash incident reports) with JaCoCo line, instruction, and branch coverage and with the presence of a linked test class. JaCoCo analyzed 216,392 methods; matching the crash stack traces against the coverage data retained 14,902 crash methods, reduced to 9,523 after excluding methods appearing beyond the tenth stack frame. Under the study's strict definitions, a method counted as crashed when it appeared in the topmost stack frame of an incident report (1,166 of the 9,523), and as unit tested when its class was linked to a test class by the filename/path heuristics and its JaCoCo line coverage exceeded the median of the non-zero covered-line percentages (98.3% in this release; 589 of the 9,523). Only 67 methods were both unit tested and crashed, against 1,099 crashed and not tested; Fisher's exact test gave p = 0.278, and the 67 methods appeared in 10,608 stack traces. They found that unit-tested code does not appear to be associated with fewer field failures. The authors stress the limits of this null result: it concerns one release of one project; the numerous faults eliminated earlier by testing, alpha/beta testing, and production releases do not appear in the data; developers may preferentially unit-test complex, fault-prone code, so the measured association is confounded; and the failures manifested exclusively as unhandled exceptions, a behavior that unit tests rarely check and coverage tools rarely measure. Their paper therefore defines a concrete research agenda: establish whether the result generalizes, explain the faults behind the crashes, harden the measurement, and find the measures of testing that actually predict field failures.

Problem Statement

Determine whether the absence of an association between unit testing and field failures found in the Eclipse case study holds beyond it, and establish what actually predicts and explains field-crash incidence. With respect to the null result of Chioteli, Batas, and Spinellis — no detectable association between unit-tested status (linked test class plus JaCoCo line, instruction, and branch coverage) and field crashes in one mature Eclipse release — the problem asks for:

(1) Generalizability: replicate the association analysis between unit testing, coverage, and field failures (crash stack traces from deployed software) across other projects, programming languages, releases including less mature ones, and application domains, addressing the external-validity dimensions the source paper itself identifies — small projects, other application domains, software written in other programming languages or multi-language projects, and less mature releases — so that it is determined whether the null result is an artifact of one project and one release.

(2) Fault-level explanation: perform root-cause analyses of the actual faults that produced the observed field crashes in unit-tested code, to determine which testing techniques (for example additional unit tests, integration tests, property-based tests, or mutation testing) could have detected those faults. The source study's incident data and replication package are public (AERI stack-trace data at software-data.org; replication package DOI 10.5281/zenodo.3610822).

(3) Measurement robustness: quantify how far the measured association is biased by tests that fail to run or are skipped under historical-release builds and by noisy or non-informative stack frames.

(4) Predictive measures and coverage semantics: evaluate whether alternative test-effectiveness measures — mutation score, assertion density, or richer coverage semantics such as exception-handling coverage and predicate-outcome coverage — provide stronger predictive power with respect to field crash incidence, under an explicitly stated statistical criterion, than the coverage metrics and filename/path-based test-linkage heuristics used in the source study. Here "richer coverage semantics" denotes criteria beyond the instruction, line, and branch counters that JaCoCo reports and that the source study already measured: the source paper contrasts JaCoCo, which "analyses coverage at the level of instructions, lines, and branches", with coverage-criteria families such as decision-to-decision paths (predicate outcomes), predicate-to-predicate outcomes, multiple conditions, loop repetitions, and execution paths, and the official JaCoCo documentation notes that its branch counter does not consider exception handling as branches. Improving coverage-analysis tools such as JaCoCo — which currently report only instruction, branch, line, complexity, method, and class counters — to examine and report exception handling is an enabling step the source paper suggests to tool builders; shipping such tool support is not a required component of an answer, and the measurement needed to evaluate a criterion may be obtained by whatever instrumentation the study validates.

For comparability, the source study's operational definitions serve as the reference baseline for replication: a method counts as crashed when it appears in the topmost stack frame of an incident report (with the top-1/top-6/top-10 frame groupings used as looser variants); a method counts as unit tested when its class is linked to a test class by the filename/path heuristics and its JaCoCo line coverage exceeds the median of the non-zero covered-line percentages (98.3% in the source release); and the analyzed population consists of the crash methods retained within the top-10 stack frames. Studies that adopt different operationalizations of the unit of analysis, crash outcome, tested-status rule, or stack-frame inclusion rule need to state them explicitly and justify the departure.

These four lines of work jointly address one objective: establishing when and how unit testing reduces field failures, and what should be measured to know it in advance. That objective comprises a replication study that reaches beyond a single project and a single release by covering at least two additional projects, at least two programming languages, at least two application domains, and at least one less-mature release; a fault-level explanation of the actual faults that produced the observed field crashes in unit-tested code; the quantification of how far the measured association is biased by tests that fail to run or are skipped and by noisy or non-informative stack frames; and the evaluation of at least one alternative test-effectiveness measure against field-crash incidence under an explicitly stated statistical criterion.

References

  1. Efstathia Chioteli, Ioannis Batas, Diomidis Spinellis. Does Unit-Tested Code Crash? A Case Study of Eclipse. arXiv:1903.04055 (2019); published in Proceedings of the 25th Pan-Hellenic Conference on Informatics (PCI 2021). DOI 10.48550/arXiv.1903.04055, 10.1145/3503823.3503872. https://arxiv.org/abs/1903.04055
  2. Pavneet Singh Kochhar, David Lo, Julia Lawall, Nachiappan Nagappan. Code Coverage and Postrelease Defects: A Large-Scale Study on Open Source Projects. IEEE Transactions on Reliability 66(4), 2017. DOI 10.1109/TR.2017.2727062. https://doi.org/10.1109/TR.2017.2727062
  3. Audris Mockus, Nachiappan Nagappan, Trung T. Dinh-Trong. Test coverage and post-verification defects: A multiple case study. In Proceedings of the 3rd International Symposium on Empirical Software Engineering and Measurement (ESEM 2009). DOI 10.1109/ESEM.2009.5315981. https://doi.org/10.1109/ESEM.2009.5315981
  4. Vard Antinyan, Jesper Derehag, Anna Sandberg, Miroslaw Staron. Mythical Unit Test Coverage. IEEE Software 35(3), 2018. DOI 10.1109/MS.2017.3281318. https://doi.org/10.1109/MS.2017.3281318
  5. Lucas Gren, Vard Antinyan. On the Relation Between Unit Testing and Code Quality. In Proceedings of the 43rd Euromicro Conference on Software Engineering and Advanced Applications (SEAA 2017). DOI 10.1109/SEAA.2017.36. https://doi.org/10.1109/SEAA.2017.36
  6. Laura Inozemtseva, Reid Holmes. Coverage is not strongly correlated with test suite effectiveness. In Proceedings of the 36th International Conference on Software Engineering (ICSE 2014). DOI 10.1145/2568225.2568271. https://doi.org/10.1145/2568225.2568271
  7. Yucheng Zhang, Ali Mesbah. Assertions are strongly correlated with test suite effectiveness. In Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering (ESEC/FSE 2015). DOI 10.1145/2786805.2786858. https://doi.org/10.1145/2786805.2786858
  8. René Just, Darioush Jalali, Laura Inozemtseva, Michael D. Ernst, Reid Holmes, Gordon Fraser. Are mutants a valid substitute for real faults in software testing? In Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering (FSE 2014). DOI 10.1145/2635868.2635929. https://doi.org/10.1145/2635868.2635929
  9. Mike Papadakis, Donghwan Shin, Shin Yoo, Doo-Hwan Bae. Are mutation scores correlated with real fault detection? In Proceedings of the 40th International Conference on Software Engineering (ICSE 2018). DOI 10.1145/3180155.3180183. https://doi.org/10.1145/3180155.3180183
  10. Thomas Laurent, Stephen Gaffney, Anthony Ventresque. Re-visiting the coupling between mutants and real faults with Defects4J 2.0. In 2022 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW 2022). DOI 10.1109/ICSTW55395.2022.00042. https://doi.org/10.1109/ICSTW55395.2022.00042
  11. Goran Petrović, Marko Ivanković, Gordon Fraser, René Just. Practical Mutation Testing at Scale: A view from Google. IEEE Transactions on Software Engineering 48(10), 2022, pages 3900–3912. arXiv:2102.11378. DOI 10.1109/TSE.2021.3107634. https://arxiv.org/abs/2102.11378
  12. S. Sinha, M. J. Harrold. Analysis and testing of programs with exception handling constructs. IEEE Transactions on Software Engineering 26(9), 2000. DOI 10.1109/32.877846. https://doi.org/10.1109/32.877846
  13. E. S. F. Najumudheen, Rajib Mall, Debasis Samanta. Modeling and Coverage Analysis of Programs with Exception Handling. In Proceedings of the 12th Innovations in Software Engineering Conference (ISEC 2019). DOI 10.1145/3299771.3299785. https://doi.org/10.1145/3299771.3299785
  14. Luan P. Lima, Lincoln S. Rocha, Carla I. M. Bezerra, Matheus Paixao. Assessing exception handling testing practices in open-source libraries. Empirical Software Engineering 26, 2021. DOI 10.1007/s10664-021-09983-3. https://doi.org/10.1007/s10664-021-09983-3
  15. Brian Robinson, Patrick Francis, Fredrik Ekdahl. A defect-driven process for software quality improvement. In Proceedings of the 2nd ACM-IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM 2008). DOI 10.1145/1414004.1414072. https://doi.org/10.1145/1414004.1414072
  16. Alireza Aghamohammadi, Seyed-Hassan Mirian-Hosseinabadi, Sajad Jalali. Statement frequency coverage: A code coverage criterion for assessing test suite effectiveness. Information and Software Technology 129, 2021. DOI 10.1016/j.infsof.2020.106426. https://doi.org/10.1016/j.infsof.2020.106426
  17. Samia Al Blwi, Amani Ayad, Besma Khaireddine, Imen Marsit, Ali Mili. Semantic Coverage: Measuring Test Suite Effectiveness. In Proceedings of the 18th International Conference on Software Technologies (ICSOFT 2023). DOI 10.5220/0012063900003538. https://doi.org/10.5220/0012063900003538
  18. Amani Ayad, Samia AlBlwi, Ali Mili. Detecting faults vs. Exposing failures: Orthogonal measures of test suite effectiveness. Journal of Systems and Software 230, 2025. DOI 10.1016/j.jss.2025.112450. https://doi.org/10.1016/j.jss.2025.112450