GitHub deployed three platform-wide security interventions — dependency security alerts (2017), automated security updates via Dependabot (2019), and CodeQL-based code scanning (2020) — and a Bayesian structural time-series analysis of more than 50,000 repositories found positive but wildly unequal effects: roughly a 149% increase in manual security updates from alerts, an additional 31.7% from automated updates, but only about a 2% gain in patched vulnerabilities from code scanning. The study's authors themselves identified what remains unresolved: the causal attribution rests on observational counterfactual prediction that cannot exclude unobserved confounders; short-term improvements (including an early novelty effect for CodeQL followed by stagnation and seasonal fluctuation) may not persist; most repositories do not respond at all while a few respond strongly, for reasons only partially understood; and several subgroup analyses were underpowered. No published study strengthens the causal identification of the platform rollout, establishes long-run persistence, or delivers adequately powered subgroup conclusions; heterogeneity has only qualitative and correlational progress. The problem of establishing, with internally valid causal evidence, whether these interventions durably reduce vulnerability exposure at ecosystem scale — and what would convert short-term gains into durable ones — remains open.

Background

Modern software is assembled from open source dependencies, so a large share of vulnerabilities enters a project through third-party libraries rather than newly written code. GitHub, the dominant host for open source development, addressed this with three staged platform-wide interventions. First, security alerts (November 2017): automatic notifications when a repository's dependencies contain versions with known vulnerabilities, as recorded in vulnerability databases such as the GitHub Advisory Database. Second, security updates (May 2019, after GitHub acquired the Dependabot tool): for each vulnerable dependency, an automated pull request proposing the minimal patched version, which a maintainer accepts with one click. Third, code scanning (September 2020), powered by CodeQL — a static analysis engine that queries source code for dangerous constructs — which flags weaknesses classified under the Common Weakness Enumeration (CWE), a community catalog of software security flaw types, in languages such as JavaScript and Python.

Because all three interventions were switched on for the platform at large, no randomized controlled comparison (a treated group versus an untreated control group) was possible for outside researchers; such experiments could only be run internally by GitHub. The largest published evaluation of the interventions as a whole is the CCS 2023 study (ACM's Conference on Computer and Communications Security, a leading computer-security venue) by Fischer, Höbenreich, and Grossklags. It reconstructed the full commit history of a sample of over 50,000 GitHub repositories, derived time series of security-relevant events (security updates applied, vulnerable dependencies newly committed, CWE weaknesses detected by CodeQL), and applied counterfactual prediction with a Bayesian structural time-series model: the model forecasts what the post-intervention series would have looked like without the intervention, fitted on a pre-intervention period and correlated control series, and the estimated causal effect is the gap between forecast and observation. This approach found that all three interventions had statistically significant positive effects overall, but of very different sizes: security alerts raised manual security updates by about 149%, automated updates added another 31.7%, while code scanning produced only about a 2% gain in patched JavaScript CWE-079 (cross-site scripting) weaknesses, with an insignificant effect on Python CWE-020 (input validation) weaknesses.

The study also surfaced warning signs about durability and reach. For code scanning, vulnerabilities dropped steeply right after activation — an apparent novelty effect, plausibly from initial motivation that then wore off — followed by stagnation and periodic (seasonal) fluctuation consistent with weekly scheduled scans accumulating problems between runs. Response was highly unequal: study-wide, only 15.9% of repositories with detected vulnerabilities improved under code scanning, 67.6% did not improve, and 16.4% got worse; within the DOM-based cross-site-scripting subtype of CWE-079, 71.7% of repositories fixed none of the detected problems, while about 44% of those that responded at all removed 95% of issues. Subgroup analyses were underpowered: only 398 repositories used customized scan schedules, against 8,543 on the default weekly schedule (too few for the model to identify an effect); popular repositories with more than 10 stars showed no significant effect (p = 0.59); and rare weakness types fluctuated without interpretable results (Stored XSS was present in only 11 repositories, too small a sample). A complementary empirical literature studies Dependabot in isolation: Mohayeji and colleagues showed that most Dependabot security pull requests in maintained JavaScript projects are merged within days and that projects with tests and continuous-integration (CI) tooling — automated pipelines that build and test every proposed change — merge more of them, while rejected ones are often fixed manually over months; Rebatchi and colleagues found, over 1.7 million projects, that project, developer, and dependency features correlate with whether security pull requests are accepted and how fast; He and colleagues found Dependabot adoption reduces technical lag (the distance between used and newest dependency versions) but leaves developers with notification fatigue; and an operations-management survival analysis by Jeong and Yoo found JavaScript packages adopting Dependabot resolve vulnerable dependencies about 60% faster, yet with a median resolution time still at 82 days. All of these are observational and conditioned on self-selected adoption, so they describe how the tools are used rather than what the platform-wide rollouts causally caused. Interviews and surveys of maintainers by Ayala and colleagues add qualitative evidence that lack of awareness and the perception that platform security features are unnecessary are major adoption barriers.

Problem Statement

The overarching open question is whether GitHub's security interventions — dependency security alerts, Dependabot security updates, and CodeQL code scanning — cause durable reductions in vulnerability exposure at ecosystem scale, and what would convert their observed short-term effects into larger long-term ones. The CCS 2023 evaluation answered this only within the limits of observational counterfactual prediction, and four linked questions remain open. (1) Causal attribution: establish the interventions' effects with an identification strategy stronger than observational time-series counterfactuals that credibly addresses unobserved confounders such as the simultaneous rise of third-party tools (e.g., Snyk), coordinated security campaigns, or developer security training, which observational counterfactual prediction can only argue to be implausible rather than exclude. (2) Long-term persistence: determine whether the early improvements persist, stagnate, or decay — in particular whether the code-scanning intervention's initial novelty effect can be turned into sustained vulnerability reduction through engagement or design changes. (3) Heterogeneity: provide a causal account of why most repositories show no response (study-wide, only 15.9% of repositories with detected vulnerabilities improved under code scanning; within the DOM-based cross-site-scripting subtype, 71.7% fixed none of the detected problems) while a minority respond thoroughly, going beyond the speculative explanations available in the source (post-activation development activity adding new vulnerabilities, a false sense of security, false positives contributing to a habituation effect, and low salience or accessibility of the alerts). (4) Underpowered subgroups: deliver statistically adequate conclusions for custom scan schedules, rare CWE weakness types, and highly popular repositories, which the original study could not resolve at its sample sizes and post-period lengths. A resolution requires (1) together with (2) — a causally identified estimate of the interventions' effects that is also shown to persist — whereas (3) and (4) extend this core rather than substituting for it: (3) is resolved by a falsifiable causal-mechanism account rather than a correlation, and (4) by an adequately powered estimate for at least one of the named subgroups. The shared objective across these facets is an internally valid, long-horizon account of what these interventions causally achieve and how their design can secure durable reductions, rather than only re-describing usage patterns in self-selected adopters. Safety conditions for automated or model-generated fixes, regression prevention, and reviewer complacency are not part of this open target: in the source, automated patching appears only as a recommended design improvement for the code-scanning intervention, and LLM-based patch generation from the study's vulnerable/patched code pairs only as a future opportunity.

References

  1. Felix Fischer, Jonas Höbenreich, Jens Grossklags. The Effectiveness of Security Interventions on GitHub. In Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security (CCS '23), pp. 2426-2440, 2023. DOI: 10.1145/3576915.3623174 (also arXiv:2309.04833). https://doi.org/10.1145/3576915.3623174
  2. Hamid Mohayeji, Andrei Agaronian, Eleni Constantinou, Nicola Zannone, Alexander Serebrenik. Securing dependencies: A comprehensive study of Dependabot's impact on vulnerability mitigation. Empirical Software Engineering, 30(3), 2025. DOI: 10.1007/s10664-025-10638-w. https://doi.org/10.1007/s10664-025-10638-w
  3. Hocine Rebatchi, Tégawendé F. Bissyandé, Naouel Moha. Dependabot and security pull requests: large empirical study. Empirical Software Engineering, 29(5), 2024. DOI: 10.1007/s10664-024-10523-y. https://doi.org/10.1007/s10664-024-10523-y
  4. Runzhi He, Hao He, Yuxia Zhang, Minghui Zhou. Automating Dependency Updates in Practice: An Exploratory Study on GitHub Dependabot. IEEE Transactions on Software Engineering, 49(8):4004-4022, 2023. DOI: 10.1109/TSE.2023.3278129. https://doi.org/10.1109/TSE.2023.3278129
  5. Seongkyoon Jeong, Eunae Yoo. Managing Software Component Quality with Automation: Evidence from Dependabot. Manufacturing & Service Operations Management, 2026. DOI: 10.1287/msom.2024.0827. https://doi.org/10.1287/msom.2024.0827
  6. Jessy Ayala, Yu-Jye Tung, Joshua Garcia. A Mixed-Methods Study of Open-Source Software Maintainers On Vulnerability Management and Platform Security Features. arXiv:2409.07669, 2024. https://arxiv.org/abs/2409.07669
  7. Dmytro Kashchuk, Tyler Moore. Measuring Enterprise Software Supply Chain Security using Public Repositories. In Proceedings of the 2025 Workshop on Software Supply Chain Offensive Research and Ecosystem Defenses (SCORED '25), pp. 57-63, 2025. DOI: 10.1145/3733827.3765529. https://doi.org/10.1145/3733827.3765529
  8. Raula Gaikovina Kula. Reducing Alert Fatigue via AI-Assisted Negotiation: A Case for Dependabot. In 2025 IEEE/ACM International Workshop on Bots in Software Engineering (BotSE), 2025. DOI: 10.1109/BOTSE67031.2025.00010 (also arXiv:2502.06175). https://doi.org/10.1109/BOTSE67031.2025.00010