The 1993 ISSRE paper introducing the ATAC data-flow coverage analysis tool, studied on twelve independently developed versions of a flight-control program, posed a question that still structures software-testing research: is high structural coverage (blocks, decisions, c-uses, p-uses) achieved during testing actually correlated with lower field-fault rates and higher operational reliability, and can a unified quantitative technique combining coverage measurement with reliability modeling decide when a program is acceptable to release? Three decades of follow-up work have produced coverage-based reliability growth models, many test-effectiveness studies on seeded and mutant faults, and industrial post-release-defect studies; yet the best-controlled effectiveness studies find only low-to-moderate correlation once test suite size is controlled, field-defect studies find weak or inconsistent relationships, and no validated quantitative theory links coverage attained in testing to subsequent field reliability. The open problem is to establish, or decisively rule out, a validated quantitative coverage-to-field-reliability relationship sufficient to ground coverage-based software acceptance criteria.

Background

Software testing is the dominant practical mechanism for detecting faults before release, and the standard way to grade a test suite is by coverage: the fraction of designated structural elements of the program that the suite executes. The classical structural criteria are block (statement) coverage, the fraction of basic blocks executed; decision (branch) coverage, the fraction of program branches exercised; and the data-flow criteria introduced by Rapps and Weyuker, which track how values propagate through the program: a c-use is a use of a variable in a computation (for example, on the right-hand side of an assignment), and a p-use is a use of a variable inside a predicate that controls a branch. A suite is adequate with respect to a criterion when it covers all required elements. Software reliability, in turn, is the probability of failure-free operation for a specified time in a given environment; reliability growth models extrapolate the failure process observed during testing to predict residual faults and guide release decisions, but the classical models use execution time or test counts and carry no information about the program's internal structure.

This question was posed by the builders of ATAC (Automatic Test Analysis for C), a Bellcore data-flow coverage measurement tool. Lyu, Horgan, and London applied ATAC to twelve independently developed versions of a critical flight-control program, produced by an N-version programming experiment (a fault-tolerance technique in which several functionally equivalent programs are written independently and their outputs are voted on at runtime) and tested with a common test process. Their conclusions observe that using such tools presupposes an unproven hypothesis: that high data-flow coverage yields software with higher operational reliability. In their own data, coverage measures did not correlate strongly with the faults found across the twelve versions. They therefore ask whether a correlation between good data-flow testing and a low rate of field faults exists at all, and for a unified technique combining testing methodology and reliability theory that would quantitatively answer the program acceptance question. Industrial practice already used coverage-based release criteria heuristically, as in the function-test coverage experience reported by Piwowarski, Ohba, and Caruso, so the scientific gap was sharp: nothing tied achieved coverage to post-release failure behavior.

Subsequent work sharpened but did not close the question. Formal studies of adequacy criteria established which criteria subsume which others but say nothing about field-fault rates. Empirical effectiveness studies, such as those of Hutchins, Foster, Goradia, and Ostrand and of Frankl and Weiss, generally found that fault detection increases with coverage, with gains concentrated at high coverage levels, but used small programs with seeded or experimentally introduced faults rather than field outcomes. Coverage-based reliability growth models (Malaiya, Li, Bieman, and Karcich; Gokhale and Trivedi; Gokhale and Mullen) use coverage as the growth variable in place of time but are fitted to faults found during testing. The best-controlled recent studies, such as Inozemtseva and Holmes's study of 31,000 test suites, find only a low-to-moderate correlation between coverage and the detection of mutants (small syntactic changes injected into a program, used as proxies for real faults) once suite size is controlled, while industrial post-release-defect studies (Gren and Antinyan; Antinyan, Derehag, Sandberg, and Staron; Pecorelli, Palomba, and De Lucia; Andrade, Machado, and Andrade) find weak, moderate, or absent correlations depending on context. What is missing is precisely what the source asked for: a validated quantitative relationship between coverage attained in testing and subsequent field reliability, integrated into an acceptance criterion, or decisive evidence that no usable relationship exists.

Problem Statement

Determine, quantitatively, whether structural coverage attained during testing is related to subsequent field reliability, and whether that relationship can ground principled software acceptance decisions. As posed in the source paper (Lyu, Horgan, and London, ISSRE 1993): (1) establish whether achieving higher levels of block, decision, c-use, and p-use coverage during testing is correlated with lower subsequent field-fault rates and higher operational reliability - the paper's benchmark being that code tested to 85 percent c-use coverage should exhibit a lower field failure rate than comparable code tested to 20 percent - or establish that there is no such correlation; and (2) construct a unified quantitative technique that combines these structure-based coverage measures with software reliability modeling, so that the relationship between test quality and operational failure behavior is expressed quantitatively and supports decisions about when a program is acceptable to release. The two components jointly serve one objective, namely deciding whether coverage can serve as a principled acceptance criterion: a decisive, statistically well-powered negative result on the correlation across realistic contexts would settle the question negatively by undermining coverage-based acceptance, while a positive resolution requires both a validated empirical relationship to field outcomes and its integration into reliability and acceptance modeling. Answers must concern post-release (operational, field) failure behavior rather than faults detected during testing alone.

References

  1. Michael R. Lyu, J. R. Horgan, Saul London, 'A coverage analysis tool for the effectiveness of software testing', Proceedings of 1993 IEEE International Symposium on Software Reliability Engineering (ISSRE 1993), pp. 25-34, 1993. DOI: 10.1109/ISSRE.1993.624271, https://doi.org/10.1109/ISSRE.1993.624271 (full text also at https://www.cse.cuhk.edu.hk/lyu/_media/conference/mlyu_issre1993_a.pdf)
  2. J. R. Horgan, S. London, Michael R. Lyu, 'Achieving software quality with testing coverage measures', IEEE Computer 27(9):60-69, 1994. DOI: 10.1109/2.312032, https://doi.org/10.1109/2.312032
  3. S. Rapps, E. J. Weyuker, 'Selecting software test data using data flow information', IEEE Transactions on Software Engineering SE-11(4):367-375, 1985. DOI: 10.1109/TSE.1985.232226, https://doi.org/10.1109/TSE.1985.232226
  4. P. Piwowarski, M. Ohba, J. Caruso, 'Coverage measurement experience during function test', Proceedings of the 15th International Conference on Software Engineering (ICSE 1993), pp. 287-301. DOI: 10.1109/ICSE.1993.346035, https://doi.org/10.1109/ICSE.1993.346035
  5. M. Hutchins, H. Foster, T. Goradia, T. Ostrand, 'Experiments on the effectiveness of dataflow- and control-flow-based test adequacy criteria', Proceedings of the 16th International Conference on Software Engineering (ICSE 1994), pp. 191-200. DOI: 10.1109/ICSE.1994.296778, https://doi.org/10.1109/ICSE.1994.296778
  6. P. G. Frankl, S. N. Weiss, 'An experimental comparison of the effectiveness of branch testing and data flow testing', IEEE Transactions on Software Engineering 19(8):774-787, 1993. DOI: 10.1109/32.238581, https://doi.org/10.1109/32.238581
  7. Phyllis G. Frankl, Yuetang Deng, 'Comparison of delivered reliability of branch, data flow and operational testing: A case study', Proceedings of the 2000 ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA 2000), pp. 124-134 (also in ACM SIGSOFT Software Engineering Notes 25(5)). DOI: 10.1145/347324.348926, https://doi.org/10.1145/347324.348926
  8. Y. K. Malaiya, Naixin Li, J. Bieman, R. Karcich, B. Skibbe, 'The relationship between test coverage and reliability', Proceedings of 1994 IEEE International Symposium on Software Reliability Engineering (ISSRE 1994), pp. 186-195. DOI: 10.1109/ISSRE.1994.341373, https://doi.org/10.1109/ISSRE.1994.341373
  9. Y. K. Malaiya, M. N. Li, J. M. Bieman, R. Karcich, 'Software reliability growth with test coverage', IEEE Transactions on Reliability 51(4):420-426, 2002. DOI: 10.1109/TR.2002.804489, https://doi.org/10.1109/TR.2002.804489
  10. Swapna S. Gokhale, Kishor S. Trivedi, 'A time/structure based software reliability model', Annals of Software Engineering 8:85-121, 1999. DOI: 10.1023/A:1018923329647, https://doi.org/10.1023/A:1018923329647
  11. Swapna S. Gokhale, Robert E. Mullen, 'The marginal value of increased testing: An empirical analysis using four code coverage measures', Journal of the Brazilian Computer Society 12:13-30, 2006. DOI: 10.1007/BF03194493, https://doi.org/10.1007/BF03194493
  12. L. Briand, Dietmar Pfahl, 'Using simulation for assessing the real impact of test coverage on defect coverage', Proceedings of the 10th IEEE International Symposium on Software Reliability Engineering (ISSRE 1999), pp. 148-157. DOI: 10.1109/ISSRE.1999.809319, https://doi.org/10.1109/ISSRE.1999.809319
  13. Xia Cai, Michael R. Lyu, 'The effect of code coverage on fault detection under different testing profiles', ACM SIGSOFT Software Engineering Notes 30(4):1-7, 2005. DOI: 10.1145/1082983.1083288, https://doi.org/10.1145/1082983.1083288
  14. J. H. Andrews, L. C. Briand, Y. Labiche, A. S. Namin, 'Using mutation analysis for assessing and comparing testing coverage criteria', IEEE Transactions on Software Engineering 32(8):608-624, 2006. DOI: 10.1109/TSE.2006.83, https://doi.org/10.1109/TSE.2006.83
  15. Laura Inozemtseva, Reid Holmes, 'Coverage is not strongly correlated with test suite effectiveness', Proceedings of the 36th International Conference on Software Engineering (ICSE 2014), pp. 435-445. DOI: 10.1145/2568225.2568271, https://doi.org/10.1145/2568225.2568271
  16. Tamas Gergely, Arpad Beszedes, Tibor Gyimothy, Milan Imre Gyalai, 'Effect of test completeness and redundancy measurement on post release failures — An industrial experience report', 2010 IEEE International Conference on Software Maintenance (ICSM 2010), pp. 1-10. DOI: 10.1109/ICSM.2010.5609692, https://doi.org/10.1109/ICSM.2010.5609692
  17. Lucas Gren, Vard Antinyan, 'On the relation between unit testing and code quality', 2017 43rd Euromicro Conference on Software Engineering and Advanced Applications (SEAA 2017), pp. 52-56. DOI: 10.1109/SEAA.2017.36, https://doi.org/10.1109/SEAA.2017.36
  18. Vard Antinyan, Jesper Derehag, Anna Sandberg, Miroslaw Staron, 'Mythical unit test coverage', IEEE Software 35(3):73-79, 2018. DOI: 10.1109/MS.2017.3281318, https://doi.org/10.1109/MS.2017.3281318
  19. Fabiano Pecorelli, Fabio Palomba, Andrea De Lucia, 'The relation of test-related factors to software quality: a case study on Apache systems', Empirical Software Engineering 26(2), 2021. DOI: 10.1007/s10664-020-09891-y, https://doi.org/10.1007/s10664-020-09891-y
  20. Lucas Andrade, Patricia Machado, Wilkerson Andrade, 'Can operational profile coverage explain post-release bug detection?', Software Testing, Verification and Reliability 30(4-5), 2020. DOI: 10.1002/stvr.1735, https://doi.org/10.1002/stvr.1735
  21. Samia AlBlwi, Imen Marsit, Besma Khaireddine, Amani Ayad, JiMeng Loh, Ali Mili, 'Subsumption, correctness and relative correctness: Implications for software testing', Science of Computer Programming 239:103177, 2025. DOI: 10.1016/j.scico.2024.103177, https://doi.org/10.1016/j.scico.2024.103177
  22. Antonia Bertolino, Guglielmo De Angelis, Felicita Di Giandomenico, Francesca Lonetti, 'Cross-coverage testing of functionally equivalent programs', 2023 IEEE/ACM International Conference on Automation of Software Test (AST 2023), pp. 101-111. DOI: 10.1109/AST58925.2023.00014, https://doi.org/10.1109/AST58925.2023.00014