When software-intensive socio-technical systems fail, organizations conduct postmortem analyses and write incident reports containing lessons learned and proposed changes (action items). Yet recording a lesson does not mean the organization has learned it: the same failures recur, lessons stay narrow, and proposed changes compete with costs, shifting priorities, and staff turnover. This problem, stated by Sillito and Pope (CHASE 2024) and left open, asks how incident-derived learnings change and generalize over time into durable, transferable organizational knowledge; it asks for (a) a model of failure-driven system evolution that links incident-triggered changes to larger patterns of system evolution over time, situated in (b) an organizational learning model of the collaborative process that manages learnings and proposed changes, and (c) interventions (process changes, tooling, services) that are developed and tested so that documented lessons are retained, discovered, and implemented without introducing harmful complexity. Follow-up work through 2025 has deepened the empirical base (characterizing 359 proposed changes across 75 public incident reports, and case studies at high-reliability organizations) but has not constructed the models, answered the longitudinal generalization question, or tested interventions, so the problem remains open.

Background

Large software-based systems are operated by organizations through an iterative, collaborative process: when a failure is judged sufficiently severe, a postmortem analysis is conducted and an incident report is written, describing what happened, what was learned, and what changes are proposed to prevent recurrence. Such systems are socio-technical: their reliability depends jointly on technical components and on the people, procedures, and organizations that design, operate, and evolve them. Organizational learning research (e.g., Argote and Miron-Spektor's synthesis of how organizations turn experience into knowledge held by individuals, groups, and routines) shows that learning from failure at the organizational level involves changing attributes of an organization that may resist change, and that a failure's influence on learning depends on goals, slack resources, and the relative emphasis on innovation versus safety. In software engineering, repositories of public incident reports (such as the Verica Open Incident Database, a large public collection of postmortem reports with supporting metadata) make it possible to study this process at scale.

Sillito and Pope (2024) reported preliminary findings from such a study, combining 13 analyzed incident reports with 7 interviews of engineers and managers. Their central observation is that the loop from failure to learning is not closed: recording a lesson in an incident report does not mean the organization has learned it, and analyzing an incident does not guarantee that similar incidents are prevented. They found that lessons are the product of a social process of interpretation and negotiation; that lessons vary in level of abstraction, generality, and maturity, and are often initially narrow, applying only to a small part of a single system; that lessons can be acted on and retained through recorded action items, but these may target core systems, procedures, or processes and sometimes add complexity that itself triggers new incidents; and that dissemination is hard (practitioners cannot find or read the reports relevant to the systems they own). Software evolution — how systems change over time — is well studied, but the social and organizational process by which systems are evolved in response to failures is not.

The authors therefore set out three research ambitions, which together with one explicitly flagged open question define this problem: (1) develop a model of failure-driven system evolution, accounting both for a single failure and its proposed changes and for larger failure patterns and the changes proposed in response to multiple failures of a system over time; (2) build an organizational learning model of the collaborative process that manages learnings and proposed changes over time, situating the first model in an organizational context of costs, competing priorities, employee turnover, and risk; and (3) develop and test interventions (process changes and supporting tooling) that improve the studied processes. Within ambition (2) they flag the literal open question: how learnings change over time as more failures are experienced — how narrow, situated lessons are generalized into durable, transferable organizational knowledge.

Problem Statement

Close the loop between recording lessons from failure and organizational learning that prevents recurrence, as posed by Sillito and Pope (CHASE 2024). The common objective is to explain and improve how incident-derived learnings become durable, transferable organizational knowledge, with three jointly supporting results. (1) Construct a model of failure-driven system evolution that accounts both for a single failure and the changes proposed in response to it and for larger failure patterns and the changes proposed in response to multiple failures of a given system over time. (2) Build an organizational learning model of the collaborative, socio-technical process by which learnings and proposed changes are managed over time, situating the first model in an organizational context with costs, competing priorities, employee turnover, and shifting attention — including an answer to the explicitly flagged open question of how narrow, situated lessons change and generalize over time as more failures are experienced, and of which mechanisms enable documented lessons to be retained, transferred, and operationalized to completion. (3) Develop and test interventions — process changes, tooling, or services — that improve the retention, discovery, and implementation of lessons without introducing harmful complexity, demonstrating that they improve the learning-from-failure process (in the source's own terms: that lessons are acted on rather than merely recorded).

References

  1. Jonathan Sillito, Matt Pope. Learning From Lessons Learned: Preliminary Findings From a Study of Learning From Failure. In Proceedings of the 2024 IEEE/ACM 17th International Conference on Cooperative and Human Aspects of Software Engineering (CHASE '24), pages 97-102, 2024. DOI: 10.1145/3641822.3641867, arXiv:2402.09538, https://doi.org/10.1145/3641822.3641867
  2. Jonathan Sillito, Matt Pope. Failing and Learning: A Study of What is Learned About Reliability From Software Incidents. In 2024 IEEE 35th International Symposium on Software Reliability Engineering Workshops (ISSREW), pages 295-302, 2024. DOI: 10.1109/ISSREW63542.2024.00093, https://doi.org/10.1109/ISSREW63542.2024.00093
  3. Matt Pope, Jonathan Sillito. Post-Incident Action Items: Crossroads of Requirements Engineering and Software Evolution. In Proceedings of the 1st IEEE/ACM Workshop on Multi-disciplinary, Open, and RElevant Requirements Engineering (MO2RE@ICSE 2024), pages 1-7, 2024. DOI: 10.1145/3643666.3648578, https://doi.org/10.1145/3643666.3648578
  4. Matt Pope, Jonathan Sillito. Characterizing the System Evolution That is Proposed After a Software Incident. In 2025 IEEE International Conference on Software Maintenance and Evolution (ICSME), pages 321-332, 2025. DOI: 10.1109/ICSME64153.2025.00037, https://doi.org/10.1109/ICSME64153.2025.00037
  5. Dharun Anandayuvaraj, Tanmay Singla, Zain Hammadeh, Andreas Lund, Alexandra Holloway, James C. Davis. Learning From Software Failures: A Case Study at a National Space Research Center. Accepted at the IEEE/ACM International Conference on Software Engineering (ICSE) 2026; arXiv:2509.06301, 2025. DOI: 10.1145/3744916.3773149, https://arxiv.org/abs/2509.06301
  6. Linda Argote, Ella Miron-Spektor. Organizational Learning: From Experience to Knowledge. Organization Science 22(5):1123-1137, 2011. DOI: 10.1287/orsc.1100.0621, https://doi.org/10.1287/orsc.1100.0621