Open validation and extension agenda for the multimodal bus factor estimator of Jabrayilzade et al. (ICSE-SEIP 2022)

The bus factor (or truck factor) of a software project is the minimal number of engineers whose sudden departure would stall it. Jabrayilzade, Evtikhiev, Tüzün, and Kovalenko (ICSE-SEIP 2022 — the Software Engineering in Practice track of the International Conference on Software Engineering) proposed the first multimodal bus factor estimator, combining version-control commits, code reviews, and meeting records through an extended Degree-of-Authorship formula, and reported a slight improvement (0.34 in mean absolute error) over the version-control-only state-of-the-art of Avelino et al. on 13 JetBrains projects. However, all numerical coefficients of the formula were fixed a priori rather than fitted, the evaluation corpus was small with no held-out projects, no ablation quantifies each modality's contribution, and the estimator works only at whole-repository (per-branch) granularity. The paper itself enumerates the open agenda: extend estimation to modules/subsystems; fit and validate the coefficients on a larger corpus split into training and test sets; incorporate further knowledge modalities (documentation, issue trackers, test cases); and ablate the modalities and the forgetting-curve factors. Sensitivity to the a-priori chosen parameters — including the meeting-to-commit association heuristic and the meeting time cap — is directly implied by its limitations section. No resolution of these items is known in the literature to date.

Background

The bus factor (also called truck factor) of a software project is the minimal number of engineers whose simultaneous departure would leave the project unable to continue — a measure of resilience to sudden personnel loss. Existing estimation algorithms infer it from version control system (VCS) data such as Git commit histories. The dominant line of work builds on the Degree of Authorship (DOA) metric of Fritz et al. (ACM Transactions on Software Engineering and Methodology, 2014), which scores each developer's knowledge of each file as a weighted combination of first authorship, the number of delivered (committed) changes, and the number of file views, with exponential decay of contributions over time. Avelino et al. (International Conference on Program Comprehension, 2016) turned DOA into a truck-factor algorithm: a developer is an author of a file if their DOA exceeds both an absolute threshold and 75% of the highest DOA on that file; authors are removed iteratively, top author first, until the remaining developers author less than half of the project's files; the number of removed developers is the truck factor.

Jabrayilzade, Evtikhiev, Tüzün, and Kovalenko (ICSE-SEIP 2022, i.e. the International Conference on Software Engineering: Software Engineering in Practice track; "Bus Factor In Practice") observed that VCS-only estimation misses knowledge created and distributed through other channels: a survey of 269 engineers identified code reviews and meetings among the highest-impact channels, and a motivating example is a senior engineer who barely commits but reviews code and attends discussions, carrying maximal project knowledge that a VCS-only algorithm cannot see. They extended the DOA formula into a multimodal estimator that adds code-review contributions (weight $1/2$), meeting contributions (minutes capped by a maximum effective time $MTE=240$ minutes per commit, weight $1$), first authorship (weight $3$), and commit contributions (weight $1$), together with logarithmic cross-developer normalization terms with coefficients $2.4$ and $1.2$; every contribution decays exponentially as $\exp(-t/S)$ with inverse decay speed $S=220$ days (knowledge halves in about five months). Meetings are linked to commits by a simple heuristic: a meeting counts toward a commit's knowledge if the committer attended it and it occurred within one week of the commit. On 13 projects developed at JetBrains, evaluated against ground truth collected by surveying the projects' engineers, the multimodal estimator slightly outperformed the Avelino et al. algorithm (an improvement of 0.34 mean absolute error, MAE, in bus factor, with comparable gains in key-developer identification).

The authors themselves flag the limitations that make this evidence preliminary. All numerical coefficients (modality weights, logarithmic coefficients, thresholds, $S$, $MTE$) were fixed a priori rather than fitted — a choice made deliberately to avoid overfitting to the 13 evaluation projects, but one that leaves the estimator's claimed advantage unvalidated: with unfitted coefficients and no held-out projects, it is unknown whether the multimodal advantage is real or an artifact of the particular corpus. Their threats-to-validity section states that "a better strategy would be to break the projects sample into train and test parts and fit the coefficients on the former; however, the sample size did not allow such an approach." The paper's conclusions then enumerate the open agenda: module/subsystem-level bus factor estimation, a larger corpus with train/test coefficient fitting, additional knowledge modalities (documentation, issue trackers, test cases), and an ablation study of the modalities and the forgetting-curve factors. A further, separable strand comes from the paper's survey findings: engineers voiced concerns about the "key engineer" notion, and the authors recommend that any bus factor tool be deployed on an opt-in basis per team — whether and how exposing key-engineer lists affects teams under different governance regimes (opt-in vs. default, team-visible vs. management-only) is untested.

The surrounding field has stayed active but moved along other axes: file-significance weighting of VCS-based estimators (BFSig, European Software Engineering Conference/Foundations of Software Engineering 2023), interactive visualization tools (Bus Factor Explorer, Automated Software Engineering conference 2023), socially aware estimation from GitHub issue interactions (SNet, Automated Software Engineering conference 2025), large-scale empirical application (truck factors of 36,000 OSS projects, 2024), and a unified combinatorial framework with NP-hardness results and a network-robustness measure (Piccolo et al., Information Sciences 2026) — the last of which explicitly criticizes the heterogeneity of prior measures as making evaluation and comparison difficult, reinforcing rather than closing the validation gap for the multimodal estimator.

Problem Statement

The source paper asks, as its own open agenda, whether its multimodal bus factor estimator — combining VCS commits, code reviews, and meeting minutes through an extended Degree-of-Authorship formula — can be turned from a promising 13-project prototype into a validated and adoptable assessment instrument. Concretely, the requested open results, all stated in or directly implied by the paper's future-work and limitations sections, are:

1. Held-out validation with fitted coefficients. Enlarge the evaluation corpus beyond 13 projects, split it into training and test subsets, fit the estimator's numerical coefficients on the training part — modality weights (first authorship 3, commits 1, reviews $1/2$, meetings normalized by $MTE=240$ minutes), the logarithmic coefficients $2.4$ and $1.2$, the authorship thresholds ($DOA \ge 1$ and normalized $DOA \ge 0.75$), and the inverse decay speed $S=220$ — and evaluate on held-out projects, against both the a-priori coefficient setting and the VCS-only baseline of Avelino et al., using bus-factor MAE and key-developer identification accuracy against survey-collected ground truth.
2. Ablation. Quantify the relative contribution of each knowledge modality (commits, code reviews, meetings) and of the forgetting-curve parameter $S$ to predictive accuracy.
3. Additional modalities. Incorporate further knowledge channels — documentation, issue-tracker activity, test-case authorship — and evaluate their marginal utility.
4. Module/subsystem-level estimation. Extend the estimator to compute bus factors of separate modules or subsystems rather than only whole repositories per branch.
5. Robustness of association choices. Assess sensitivity to the meeting-to-commit association heuristic (meetings attended by the committer within one week of the commit) and to the meeting time cap $MTE$.

Items 1–5 jointly answer one scientific question — does multimodal knowledge measurement genuinely improve turnover-risk estimation, and how much of the improvement each signal contributes — and together define the scope of this agenda.

The governance and key-engineer-exposure strand raised in the Background is a separate organisational question and is out of scope for this agenda. The matched meetings–reviews–commits records and per-project survey ground truth used by the source study are not openly available, so a resolution of items 1–5 must not depend on that closed source data and must preserve the multimodal setting — version-control commits, code-review records, and meeting records.

References

1. Elgun Jabrayilzade, Mikhail Evtikhiev, Eray Tüzün, Vladimir Kovalenko. Bus Factor in Practice. Proceedings of the 44th International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), 2022. DOI: 10.1145/3510457.3513082. https://doi.org/10.1145/3510457.3513082 (preprint: arXiv:2202.01523, https://arxiv.org/abs/2202.01523)
2. Guilherme Avelino, Leonardo Passos, Andre Hora, Marco Tulio Valente. A novel approach for estimating Truck Factors. 2016 IEEE 24th International Conference on Program Comprehension (ICPC), 2016. DOI: 10.1109/ICPC.2016.7503718. https://doi.org/10.1109/ICPC.2016.7503718
3. Thomas Fritz, Gail C. Murphy, Emerson Murphy-Hill, Jingwen Ou, Emily Hill. Degree-of-knowledge: Modeling a developer's knowledge of code. ACM Transactions on Software Engineering and Methodology (TOSEM), 23(2), 2014. DOI: 10.1145/2512207. https://doi.org/10.1145/2512207
4. Egor Klimov, Muhammad Umair Ahmed, Nikolai Sviridov, Pouria Derakhshanfar, Eray Tüzün, Vladimir Kovalenko. Bus Factor Explorer. 2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE), 2023. DOI: 10.1109/ASE56229.2023.00015. https://doi.org/10.1109/ASE56229.2023.00015
5. Vahid Haratian, Mikhail Evtikhiev, Pouria Derakhshanfar, Eray Tüzün, Vladimir Kovalenko. BFSig: Leveraging File Significance in Bus Factor Estimation. Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE), 2023. DOI: 10.1145/3611643.3613877. https://doi.org/10.1145/3611643.3613877
6. Alexis Butler, Dan O'Keeffe, Santanu Kumar Dash. Measuring Software Resilience Using Socially Aware Truck Factor Estimation. 2025 40th IEEE/ACM International Conference on Automated Software Engineering (ASE), 2025. DOI: 10.1109/ASE63991.2025.00318. https://doi.org/10.1109/ASE63991.2025.00318
7. Sebastiano A. Piccolo, Pasquale De Meo, Giorgio Terracina, Gianluigi Greco. The theory and practice of computing the bus factor. Information Sciences, 2026. DOI: 10.1016/j.ins.2026.123353. https://doi.org/10.1016/j.ins.2026.123353 (preprint: arXiv:2603.07845, https://arxiv.org/abs/2603.07845)
8. Sebastiano A. Piccolo, Andrea Tagarelli. The CriticalSet problem: Identifying Critical Contributors in Bipartite Dependency Networks. arXiv:2604.21537, 2026. https://arxiv.org/abs/2604.21537
9. Sebastiano A. Piccolo, Giorgio Terracina. Project resilience as network robustness. arXiv:2604.17417, 2026. https://arxiv.org/abs/2604.17417
10. Olivier Nourry, Masanari Kondo, Shinobu Saito, Yukako Iimura, Naoyasu Ubayashi, Yasutaka Kamei. Myth: The loss of core developers is a critical issue for OSS communities. arXiv:2412.00313, 2024. https://arxiv.org/abs/2412.00313