AI code generation severs the long-standing inference that a developer's authorship footprint in version control is evidence of their understanding of that code, invalidating the truck factor, Degree-of-Authorship, and degree-of-knowledge family as a class. Wheeler's position paper 'The Substrate Collapse' (2026) argues that the replacement must be grounded in evidence of comprehension rather than authorship, and deliberately leaves open the construction of such an instrument. The open problem is to build and validate an instrument that reads the theory retention of a whole team about a whole system — continuously, unobtrusively, and with an explicit treatment of adversarial gaming (comprehension performed without being held). The paper's accompanying falsifiable prediction — that such a measure predicts incident-resolution outcomes where authorship metrics do not — becomes empirically testable exactly when the instrument exists. The 2024–2026 literature names the gap (comprehension debt, cognitive debt, epistemic debt) but does not close it: existing comprehension measures reach only individual or code-fragment scale, and the nearest system-scale artifacts evaluate code or generators rather than the understanding a team retains.

Background

Software engineering organizations have long estimated where knowledge about a system resides by looking at who wrote the code. The truck factor (also called bus factor) is the minimal number of developers whose departure would incapacitate a project; Avelino et al. (2016) gave it a rigorous automated estimator and found that about two-thirds of 133 popular GitHub repositories had a truck factor of two or fewer. Underneath it sits the Degree-of-Authorship metric and the degree-of-knowledge model of Fritz et al. (2014), which infer a developer's familiarity with a region of code from the footprint they leave on it — commits, lines authored, edits, and interaction history — and Rigby et al. (2016) applied the same machinery to turnover-induced knowledge loss. The entire family rests on one inference: authoring a region of code is evidence of understanding it.

That inference had a mechanical cause: for most of software's history, code entered a repository only when a human wrote it, and writing non-trivial code forced at least a transient understanding — a theory, in Peter Naur's (1985) sense of programming as theory building: a living understanding of how the system behaves, why it is shaped as it is, and how it can be changed without coming apart. The version-control footprint was the fossil that theory left in the record.

AI code generation severs this link in the ordinary case. When an AI agent generates a module and a human merges it, the version-control record still attributes authorship, but the same footprint is now compatible with full, partial, or no understanding. Wheeler (2026) argues that this is not degradation of the authorship-based metrics but invalidation of them as a class: no reweighting of an authorship footprint can recover an inference the footprint no longer supports. He calls the quantity that actually matters theory retention — the living understanding of the system held by humans the organization can deploy — and shows that the field's recent measurement strains are consistent with the collapse: the randomized controlled trial by METR, an AI-safety research organization (Becker et al. 2025), whose follow-up could no longer reliably size the productivity effect once developers grew unwilling to work without the tools, and large-scale corpus studies of AI-authored commits (Liu et al. 2026) that track technical debt in the code while measuring nothing about team understanding.

The hazard has been named repeatedly in 2025–2026 as comprehension debt (Gorman 2025; Osmani 2026; Ahmad 2026, a qualitative diary study with 207 students), cognitive debt (Storey 2026, whose Triple Debt Model treats the erosion of shared team understanding as a first-class software-health concern), and epistemic debt (Sankaranarayanan 2026, who demonstrates experimentally that AI-assisted novices can accumulate functional output with collapsed corrective competence). But naming is not measurement. The program-comprehension literature offers validated instruments only at the scale of the individual and the fragment: functional magnetic resonance imaging of comprehension effort (Siegmund et al. 2014), a large-scale field study of professionals' comprehension activity with eye-tracking (Xia et al. 2018), and NRevisit (Gao et al. 2025), a per-reader code-unit understandability score derived from eye-gaze revisit counts. None of these reads the understanding a whole team holds about a whole system, continuously and unobtrusively. The nearest system-scale public artifacts measure the wrong quantity: A.S.E (Lian et al. 2025) is a repository-level benchmark that scores AI code generators on whether regenerated code reintroduces documented vulnerabilities — a property of generated artifacts, not of retained human understanding. An empirical study by Cury and Avelino (2025) corroborates the invalidation direction, showing that knowledge models and truck-factor algorithms are measurably perturbed by ChatGPT-generated code contribution, but proposes no comprehension-grounded replacement.

Wheeler's paper states two honest limits that any solution must confront: a comprehension measure is gameable, because comprehension can be performed without being held (the way a student passes an examination they do not understand), so any such instrument must be treated as an estimate carrying an adversarial discount; and scaling a valid comprehension measure from the fragment to the system and team may not admit a fully satisfactory solution — theory retention may be destined to be estimated coarsely rather than measured precisely. The paper also fixes an external validation target: its central falsifiable prediction is that systems with a healthy authorship-derived truck factor but low comprehension-measured retention will suffer disproportionate incident-resolution failures (longer time-to-resolution, more escalation, more failed first fixes) that the authorship metric does not predict — a prediction that becomes empirically testable exactly when the instrument exists.

Problem Statement

Construct and validate an instrument for measuring theory retention — the understanding of a software system actually held by the humans of a development organization, in Naur's sense of a theory of the system — at the scale of a whole system and a whole team, such that the measurement is:

  1. Grounded in evidence of comprehension rather than authorship: the load-bearing inputs must be produced by the understanding itself (e.g., demonstrated ability to predict system behavior under unseen inputs, to locate where a specified change must be made, to distinguish load-bearing from incidental behavior, or to reconstruct design rationale), and not by any function of version-control authorship footprints, whose link to comprehension AI code generation has severed.
  2. Continuous and unobtrusive: it must operate on an ongoing basis across a production system and its team without disruptive, examination-style testing sessions.
  3. Adversarially honest: it must include an explicit treatment of gaming — the fact that comprehension can be performed without being held — such that the instrument's readings are treated as estimates carrying an adversarial discount rather than clean readings.

Validation must specify and provide evidence for the instrument's reliability and validity as a measure of retained comprehension, and must define the whole-system and whole-team study design that would demonstrate it on real AI-heavy codebases and teams. The source paper fixes an external validation target in its central falsifiable prediction — that a comprehension-grounded retention measure predicts incident-resolution outcomes (time-to-resolution, escalation, failed first fixes) on AI-heavy systems where the authorship-derived truck factor fails — which becomes empirically testable exactly when the instrument exists; the paper marks that prediction as a commitment the construct makes rather than a result it reports.

References

  1. Wheeler, Brett (2026). The Substrate Collapse: AI Code Generation Invalidates Authorship-Based Knowledge Metrics. arXiv:2606.20882, https://arxiv.org/abs/2606.20882
  2. Avelino, Guilherme; Passos, Leonardo; Hora, Andre; Valente, Marco Tulio (2016). A Novel Approach for Estimating Truck Factors. 24th International Conference on Program Comprehension (ICPC), 1–10. arXiv:1604.06766, https://arxiv.org/abs/1604.06766
  3. Fritz, Thomas; Murphy, Gail C.; Murphy-Hill, Emerson; Ou, Jingwen; Hill, Emily (2014). Degree-of-Knowledge: Modeling a Developer's Knowledge of Code. ACM Transactions on Software Engineering and Methodology 23(2), 1–42. DOI: 10.1145/2512207, https://doi.org/10.1145/2512207
  4. Naur, Peter (1985). Programming as Theory Building. Microprocessing and Microprogramming 15(5), 253–261. DOI: 10.1016/0165-6074(85)90032-8, https://doi.org/10.1016/0165-6074(85)90032-8
  5. Rigby, Peter C.; Zhu, Yue Cai; Donadelli, Samuel M.; Mockus, Audris (2016). Quantifying and Mitigating Turnover-Induced Knowledge Loss: Case Studies of Chrome and Apache. 38th International Conference on Software Engineering (ICSE), 1006–1016. DOI: 10.1145/2884781.2884851, https://doi.org/10.1145/2884781.2884851
  6. Siegmund, Janet; Kästner, Christian; Apel, Sven; Parnin, Chris; Bethmann, Anja; Leich, Thomas; Saake, Gunter; Brechmann, André (2014). Understanding Understanding Source Code with Functional Magnetic Resonance Imaging. 36th International Conference on Software Engineering (ICSE), 378–389. DOI: 10.1145/2568225.2568252, https://doi.org/10.1145/2568225.2568252
  7. Xia, Xin; Bao, Lingfeng; Lo, David; Xing, Zhenchang; Hassan, Ahmed E.; Li, Shanping (2018). Measuring Program Comprehension: A Large-Scale Field Study with Professionals. IEEE Transactions on Software Engineering 44(10), 951–976. DOI: 10.1109/TSE.2017.2734091, https://doi.org/10.1109/TSE.2017.2734091
  8. Gao Hao; Hijazi, Haytham; Medeiros, Júlio; Durães, João; Lam, Chan Tong; de Carvalho, Paulo; Madeira, Henrique (2025). NRevisit: A Cognitive Behavioral Metric for Code Understandability Assessment. EASE 2025. arXiv:2504.18345, https://arxiv.org/abs/2504.18345
  9. Sankaranarayanan, Sreecharan (2026). Mitigating "Epistemic Debt" in Generative AI-Scaffolded Novice Programming using Metacognitive Scripts. arXiv:2602.20206, https://arxiv.org/abs/2602.20206
  10. Storey, Margaret-Anne (2026). From Technical Debt to Cognitive and Intent Debt: Rethinking Software Health in the Age of AI. arXiv:2603.22106, https://arxiv.org/abs/2603.22106
  11. Ahmad, Muhammad Ovais (2026). Comprehension Debt in GenAI-Assisted Software Engineering Projects. 30th International Conference on Evaluation and Assessment in Software Engineering (EASE 2026). arXiv:2604.13277, https://arxiv.org/abs/2604.13277
  12. Gorman, Jason (2025). Comprehension Debt: The Ticking Time Bomb of LLM-Generated Code. Codemanship's Blog, 30 September 2025. https://codemanship.wordpress.com/2025-09-30/comprehension-debt-the-ticking-time-bomb-of-llm-generated-code/
  13. Osmani, Addy (2026). Comprehension Debt — The Hidden Cost of AI-Generated Code. 14 March 2026. https://addyosmani.com/blog/comprehension-debt/
  14. Cury, Otávio; Avelino, Guilherme (2025). The Impact of Generative AI on Code Expertise Models: An Exploratory Study. arXiv:2507.08160, https://arxiv.org/abs/2507.08160
  15. Liu, Yue; Widyasari, Ratnadira; Zhao, Yanjie; Irsan, Ivana Clairine; Chen, Junkai; Lo, David (2026). Debt Behind the AI Boom: A Large-Scale Empirical Study of AI-Generated Code in the Wild. arXiv:2603.28592, https://arxiv.org/abs/2603.28592
  16. Becker, Joel; Rush, Nate; Barnes, Elizabeth; Rein, David (2025). Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity. arXiv:2507.09089, https://arxiv.org/abs/2507.09089
  17. Lian, Keke; Wang, Bin; Zhang, Lei; Chen, Libo; Wang, Junjie; Zhao, Ziming; Yang, Yujiu; Lin, Miaoqian; Duan, Haotong; Zhao, Haoran; Liao, Shuang; Guo, Mingda; Quan, Jiazheng; Zhong, Yilu; He, Chenhao; Chen, Zichuan; Wu, Jie; Li, Haoling; Li, Zhaoxuan; Yu, Jiongchi; Li, Hui; Zhang, Dong (2025). A.S.E: A Repository-Level Benchmark for Evaluating Security in AI-Generated Code. arXiv:2508.18106, https://arxiv.org/abs/2508.18106
  18. Zhang, Yujie (2025). Beyond Technical Debt: How AI-Assisted Development Creates Comprehension Debt in Resource-Constrained Indie Teams. arXiv:2512.08942, https://arxiv.org/abs/2512.08942
  19. Lyu, Yunbo; Williams, David; Shi, Jieke; Sun, Zhensu; Peng, Chao; Yang, Zhou; Sarro, Federica; Lo, David (2026). How Do Practitioners Build SE Agents? Insights from a Mixed-Methods Study. arXiv:2607.10856, https://arxiv.org/abs/2607.10856
  20. Borg, Markus; Hewett, Dave; Hagatulah, Nadim; Couderc, Noric; Söderberg, Emma; Graham, Donald; Kini, Uttam; Farley, Dave (2025). Echoes of AI: Investigating the Downstream Effects of AI Assistants on Software Maintainability. arXiv:2507.00788, https://arxiv.org/abs/2507.00788