A single-case study of an introductory undergraduate engineering course found that students' formative coding-exercise performance rose after generative AI (GenAI) tools were integrated into instruction, but the design could not separate the tool's contribution from novelty effects and the heightened attention of an instructor and teaching assistants experimenting with the technology, and it said nothing about other courses, institutions, or cohorts. The open problem is to establish, through controlled and multi-site longitudinal designs, whether GenAI integration causally improves engineering students' learning, whether such effects replicate across engineering course types, levels, institutions, and multi-year cohorts, and under what conditions integration moves beyond replacing or amplifying existing instruction to genuinely transforming it. Controlled randomized evidence now exists in adjacent settings (secondary-school mathematics, general undergraduate coursework) and the meta-analytic evidence base is contested by a retraction and a publication-bias re-analysis, but no controlled causal evaluation of GenAI integration in engineering education had appeared as of August 2026.

Background

Generative AI (GenAI) refers to machine-learning systems, typically large language models (LLMs) — neural networks trained on large text corpora that produce natural-language answers, explanations, and code — such as ChatGPT and GitHub Copilot. Since 2023, engineering instructors have begun integrating these tools into courses, and a large discussion literature has weighed their promise and pitfalls for engineering education (Qadir, 2023). Empirical evidence from real engineering classrooms, however, has come almost entirely from uncontrolled, single-site studies.

The source study (Kozan, Keles, Jian, Ayvaz, Sierszecki and Joo, 2026) is an embedded descriptive single case study — an in-depth qualitative and descriptive-quantitative investigation of one bounded case — of one instructor and seven teaching assistants (TAs) integrating GenAI for the first time into a single-semester introductory undergraduate computer systems course with formative practice coding exercises (low-stakes exercises whose purpose is to give students feedback during learning rather than to grade them). Students' formative exercise performance increased after GenAI integration, students' use of the technology weakened over time, and the integration remained at the level of replacing other instructional methods or increasing the efficiency of coding work, in the sense of a widely used three-level technology-integration framework that distinguishes replacement (the tool substitutes for an existing method without changing the task), amplification (the tool does the same work more efficiently), and transformation (the tool enables new tasks, roles, or instructional practices). The authors explicitly caution that the observed improvement may have been produced by factors that were not controlled: the novelty effect — a well-documented pattern in which performance and engagement rise transiently when users first encounter a new technology and then decay as the novelty wears off, shown for consumer activity trackers (Shin, Feng, Jarrahi and Gafinowitz, 2019) and for a mobile learning device used by radiology residents (Björkman, Spångeus and Woisetschläger, 2019) — and the instructor's and TAs' own increased effort while learning to integrate GenAI. Their stated suggestions for further research call for multi-year, multi-site evaluations across engineering course contexts and institutions, research on instructors' GenAI-enhanced content design, domain-specific GenAI models trained on engineering datasets, students' perspectives, and broader student use cases.

The surrounding quantitative evidence is only now maturing. Meta-analyses — statistical syntheses of prior studies, reporting standardized mean-difference effect sizes conventionally denoted g — of randomized chatbot and ChatGPT experiments report overall positive effects on learning outcomes with education level and intervention duration as moderators (Wu and Yu, 2024; Deng, Jiang, Yu, Lu and Liu, 2025). But this literature has been destabilized: the meta-analysis by Wang and Fan (2025) reporting a large learning-performance effect (g = 0.867) was retracted in April 2026, and Bartoš, Martinková and Wagenmakers (2025) showed that once publication bias (the tendency for studies with null results to go unpublished, which inflates meta-analytic averages) is modeled, the evidence for ChatGPT's benefits in that literature effectively disappears, prompting an explicit call for high-quality, pre-registered experiments (studies whose analysis plan is time-stamped and lodged publicly before data collection). Field experiments in adjacent domains show both sides of the phenomenon: a randomized study of roughly a thousand high-school mathematics students found large performance gains while AI-tutor access was available but worse performance than never-treated students for the unguarded version once access was removed (Bastani, Bastani, Sungu, Ge, Kabakcı and Mariman, 2025), and a randomized experiment with undergraduates found test-score gains of 0.27 standard deviations persisting one week after AI-assisted study, with gains concentrated among students who used AI to explain concepts rather than to generate text (Contractor and Reyes, 2026). None of these studies concerns engineering courses. In engineering education itself, the most recent empirical work remains uncontrolled — for example, a multi-phase study of 340 first-year engineering students completing a time-constrained debugging task with GenAI as the only permitted assistance found that mere access did not ensure task success, which instead tracked prompting efficiency and verification practices (López-Vargas et al., 2026) — and registered protocols for randomized college-student experiments on GenAI and performance exist (Chen et al., 2025) without engineering results yet. A randomized controlled trial (RCT) is a study design in which participants are assigned by chance to receive or not receive the intervention, so that differences in outcomes can be attributed causally to the intervention rather than to pre-existing differences; a quasi-experiment approximates this without random assignment.

Problem Statement

For GenAI integration in engineering courses, where single-case studies report post-integration improvements in students' formative exercise performance: (1) Causal attribution — determine, with randomized or rigorously controlled designs, whether GenAI integration itself improves engineering students' learning outcomes, as opposed to the improvement being an artifact of novelty effects, increased instructor and teaching-assistant attention, or other concurrent factors. (2) Generalization — establish whether the integration effects replicate across different engineering course types, levels, institutions, and multi-year cohorts, which the source study explicitly identifies as requiring multi-year, multi-site evaluation. (3) Conditions for transformation — characterize under what circumstances (longer-term instructor redesign of GenAI-enhanced content, domain-specific GenAI models trained on engineering datasets, and broader student use cases such as design and development work rather than only coding) integration moves beyond replacing or amplifying existing instruction to transforming instruction and learning, complemented by students' own perspectives on integration. These components jointly serve one overarching objective: a causally identified, generalizable account of whether and under what conditions GenAI integration improves student learning in engineering education. These three components are conjunctive rather than a menu of alternatives: the research question is jointly about causal attribution (1), generalization (2), and transformation conditions (3), with component (2) concerning replication of whatever effect component (1) establishes, and with each of the enumerated circumstances of component (3) — longer-term instructor redesign of GenAI-enhanced content, domain-specific GenAI models trained on engineering datasets, broader student use cases such as design and development work rather than only coding, and students' own perspectives on integration — in scope for that component. Because no controlled causal evaluation of GenAI integration in engineering education yet exists, the objective is addressed by a defended research answer rather than by a new empirical finding.

References

  1. Kozan, K., Keles, O., Jian, S., Ayvaz, S., Sierszecki, K., & Joo, S. (2026). GenAI Integration into Engineering Education: A Case Study of an Introductory Undergraduate Engineering Course. arXiv:2603.13269. DOI 10.48550/arxiv.2603.13269, https://arxiv.org/abs/2603.13269
  2. Qadir, J. (2023). Engineering Education in the Era of ChatGPT: Promise and Pitfalls of Generative AI for Education. 2023 IEEE Global Engineering Education Conference (EDUCON), 1–9. DOI 10.1109/educon54358.2023.10125121, https://doi.org/10.1109/EDUCON54358.2023.10125121
  3. Wu, R., & Yu, Z. (2024). Do AI chatbots improve students learning outcomes? Evidence from a meta-analysis. British Journal of Educational Technology, 55(1), 10–33. DOI 10.1111/bjet.13334, https://doi.org/10.1111/bjet.13334
  4. Deng, R., Jiang, M., Yu, X., Lu, Y., & Liu, S. (2025). Does ChatGPT enhance student learning? A systematic review and meta-analysis of experimental studies. Computers & Education, 227, 105224. DOI 10.1016/j.compedu.2024.105224, https://doi.org/10.1016/j.compedu.2024.105224
  5. Wang, J., & Fan, W. (2025). RETRACTED ARTICLE: The effect of ChatGPT on students' learning performance, learning perception, and higher-order thinking: insights from a meta-analysis. Humanities and Social Sciences Communications, 12(1), 621 (retracted 2026; retraction note DOI 10.1057/s41599-026-07310-z). DOI 10.1057/s41599-025-04787-y, https://doi.org/10.1057/s41599-025-04787-y
  6. Bartoš, F., Martinková, P., & Wagenmakers, E.-J. (2025). Adjusting for Publication Bias Reveals No Evidence for the Effect of ChatGPT on Students' Learning Performance, Learning Perception, and Higher-Order Thinking. PsyArXiv preprint. DOI 10.31234/osf.io/8vs32_v1, https://doi.org/10.31234/osf.io/8vs32_v1
  7. Bastani, H., Bastani, O., Sungu, A., Ge, H., Kabakcı, Ö., & Mariman, R. (2025). Generative AI without guardrails can harm learning: Evidence from high school mathematics. Proceedings of the National Academy of Sciences, 122(26), e2422633122. DOI 10.1073/pnas.2422633122, https://doi.org/10.1073/pnas.2422633122
  8. Contractor, Z., & Reyes, G. (2026). Experimental Evidence on the Learning Impact of Generative AI. arXiv:2607.08849. DOI 10.48550/arxiv.2607.08849, https://arxiv.org/abs/2607.08849
  9. Chen, Y., Wang, Y., Wüstenberg, T., Kizilcec, R. F., Fan, Y., Li, Y., Lu, B., Yuan, M., Zhang, J., Zhang, Z., Geldsetzer, P., Chen, S., & Bärnighausen, T. (2025). Effects of generative artificial intelligence on cognitive effort and task performance: study protocol for a randomized controlled experiment among college students. Trials, 26(1), 244. DOI 10.1186/s13063-025-08950-3, https://doi.org/10.1186/s13063-025-08950-3
  10. López-Vargas, A., Rodríguez-Vidal, J., García-Beltrán, Á., Vigara, P., de León-Torres, L., & López-Vargas, A. (2026). Performance, interaction, and ethical evaluation in the use of generative artificial intelligence in engineering education. Education and Information Technologies, online first 2026-08-14. DOI 10.1007/s10639-026-14112-y, https://doi.org/10.1007/s10639-026-14112-y
  11. Shin, G., Feng, Y., Jarrahi, M. H., & Gafinowitz, N. (2019). Beyond novelty effect: a mixed-methods exploration into the motivation for long-term activity tracker use. JAMIA Open, 2(1), 62–72. DOI 10.1093/jamiaopen/ooy048, https://doi.org/10.1093/jamiaopen/ooy048
  12. Björkman, A.-S., Spångeus, A., & Woisetschläger, M. (2019). Mobile learning device increased study efficiency for radiology residents but with risk of temporary novelty effect. Acta Radiologica Open, 8(11). DOI 10.1177/2058460119889871, https://doi.org/10.1177/2058460119889871