Whether cross-entropy computed by large language models (LLMs) — a 'naturalness' measure of code — is a valid signal of expert-assessed code maintainability remains an open empirical question. A 2024 study (Dillmann, Siebert, Trendowicz) let ten pretrained LLMs generate code for each of 304 expert-rated Java classes and found that the average cross-entropy of the generated code predicts maintainability only when class size (logical lines of code) is controlled for, with the association reversing otherwise, and one model behaving anomalously. The source left a number of open questions: the appropriate aggregation statistic for the signal, the effect of fine-tuning on maintainability-labelled code, empirical verification of the assumption that LLM-generated code is a maintainability baseline, the signal's complementarity or redundancy with structural code metrics (coupling, cohesion, cyclomatic complexity), generalisation across model types, settings, and languages, and the treatment of further model types such as masked language models, which the source set aside on runtime grounds. The later literature has not resolved these questions, though work on snippet-level perplexity aggregation for understandability sharpens the aggregation concern.

Background

Software maintainability — the ease with which a code base can be understood, modified, and extended (a quality characteristic in the ISO/IEC 25010 standard) — is classically assessed either through structural code metrics or through expert judgment. Cyclomatic complexity counts the number of independent control-flow paths through a program; coupling measures how many dependencies a module has on other modules; cohesion measures how strongly the elements of a single module belong together. Such metrics have been linked to maintenance outcomes for decades, but their usefulness has repeatedly been criticized, while expert rating studies remain expensive. Learned predictors have also been tried: for example, the DeepM framework of Hu et al. (2023) predicts maintainability from lexical (text), structural (abstract syntax tree), and code-metric features, training on automatically collected code and evaluating on a manually curated dataset of Java classes. A second, data-driven tradition exploits the 'naturalness' of code: because programmers repeat themselves, source code is highly predictable, and statistical language models trained on large code corpora can assign a probability to a piece of code. The cross-entropy of a code fragment under a language model is the average negative log-likelihood the model assigns to its tokens — lower cross-entropy means the model finds the code more probable, i.e. more 'natural'; its exponential, the perplexity, is a standard measure of how 'surprised' the model is by the text. Building on this, Sengamedu and Zhao (2022) hypothesised that large language models (LLMs), having trained on public repositories, act as oracles of good programming practice: comparing the probability of LLM-generated code (a counterfactual) with the probability of the human-written code can indicate quality problems. Dillmann, Siebert, and Trendowicz (2024) tested this hypothesis for maintainability. They let ten pretrained autoregressive LLMs (models that generate text one token at a time, based on GPT-2, BLOOM, CodeGPT, CodeGen, and Code Llama) generate code for each of the 304 Java classes of the expert-maintainability benchmark of Schnappinger et al. (2020), in which experts rated readability, understandability, complexity, modularisation, and overall maintainability on a four-point Likert scale (ordered agreement categories). For each class they computed the average next-token cross-entropy of the generated code — classes were segmented into chunks respecting each model's maximum input length, and chunk cross-entropies were averaged weighted by token count — and related it to the expert ratings. The ratings were used as binary outcomes: P(strongly agree) >0.5> 0.5 for overall maintainability, readability, and understandability, and P(strongly disagree) >0.5> 0.5 for complexity and modularisation, whose statements are negatively worded, so that strong disagreement indicates maintainable code; log-transformed, standardised cross-entropy and LLOC were then used as classifier features and compared with an LLOC-only baseline. Controlling for the number of logical lines of code (LLOC, the count of non-empty, non-comment lines), higher cross-entropy predicted lower maintainability (e.g., the log-cross-entropy coefficient in a logistic regression of expert agreement on overall maintainability was −0.80-0.80 with p=0.009p=0.009 for their reference model M2, Bloomz-1b1); without the LLOC control the association reversed — a Simpson's-paradox-type confounding, in which an aggregated association flips sign once a subgrouping variable is controlled for, because both the maintainability ratings and the cross-entropy depend on class size. Prediction performance barely differed across the ten models, but one model (CodeGPT-java-10.2) showed an inverted association the authors could not explain, and cross-entropy alone did not outperform the trivial LLOC baseline. The paper closes by stating that further research is needed to make best use of this information in practice, leaving the validity question open. Among the future directions it sketches, the source also suggests examining the assumption that LLM-generated code constitutes a maintainability baseline through assessment of that generated code by subject-matter experts.

Problem Statement

Establish whether — and through which aggregation, training, and modeling choices — cross-entropy derived from large language models is a valid, practically useful signal of expert-assessed code maintainability. The single scientific objective — validating the LLM cross-entropy signal of maintainability — is elaborated by six questions the source left open: (1) Aggregation: is the class-level average (token-weighted) cross-entropy used in the source the most appropriate statistic, or do alternative aggregations — method-level, weighted by semantic units, or token-normalised measures — improve validity? (2) Fine-tuning: does fine-tuning pretrained LLMs on maintainability-labelled code or on curated high-quality code change the predictive power of cross-entropy and mitigate confounding effects? (3) Baseline assumption: does LLM-generated code actually constitute a reasonable 'baseline' of maintainability? (4) Metric relation: does cross-entropy carry information complementary to, or redundant with, established structural code metrics such as coupling, cohesion, and cyclomatic complexity? (5) Generalisability: does the association hold across other model types, settings, training regimes, application domains, programming languages, and larger datasets, and are model-specific anomalies, such as the inverted behaviour of CodeGPT-java-10.2, systematic or idiosyncratic? (6) Segmentation and model coverage: how can the segmentation of classes for cross-entropy computation be refined, and can the scope be extended to further model types — including masked (fill-in) language models, which predict a hidden token from its surrounding context and which the source disqualified after a fill-mask run took roughly 58 minutes where autoregressive generation took under one minute?

All six components concern the same signal and the same validity question; in the source they appear as open questions, as examples of aspects to be investigated in the future, and as threats to be addressed by future research. In this statement a 'valid' signal means one whose association with the expert ratings of readability, understandability, complexity, modularisation, and overall maintainability is not an artefact of class size (LLOC), on the source's outcome convention (agreement binarised at probability 0.5, with reversed polarity for the negatively worded complexity and modularisation statements); 'practically useful' means informative for maintainability assessment beyond the class-size baseline. The central validity question concerns a property of the benchmark data: whether the LLOC-controlled association between cross-entropy and the expert ratings holds on the benchmark's ratings rather than being quoted from the 2024 study; the six components above are the open directions the source left for future research.

References

  1. Dillmann, Marc; Siebert, Julien; Trendowicz, Adam. 'Evaluation of large language models for assessing code maintainability.' arXiv:2401.12714, 2024. DOI: 10.48550/arXiv.2401.12714. https://arxiv.org/abs/2401.12714
  2. Sengamedu, Srinivasan H.; Zhao, Hangqi. 'Neural language models for code quality identification.' Proceedings of the 6th International Workshop on Machine Learning Techniques for Software Quality Evaluation (MALTESQUE '22), ACM, pp. 5-10, 2022. DOI: 10.1145/3549034.3561175. https://doi.org/10.1145/3549034.3561175
  3. Schnappinger, Markus; Fietzke, Arnaud; Pretschner, Alexander. 'Defining a Software Maintainability Dataset: Collecting, Aggregating and Analysing Expert Evaluations of Software Maintainability.' Proceedings of the 2020 IEEE International Conference on Software Maintenance and Evolution (ICSME), IEEE, pp. 278-289, 2020. DOI: 10.1109/ICSME46990.2020.00035. https://doi.org/10.1109/ICSME46990.2020.00035
  4. Hu, Yamin; Jiang, Hao; Hu, Zongyao. 'Measuring code maintainability with deep neural networks.' Frontiers of Computer Science 17, article 176214, 2023. DOI: 10.1007/s11704-022-2313-0. https://doi.org/10.1007/s11704-022-2313-0
  5. Khanfir, Ahmed; Jimenez, Matthieu; Papadakis, Mike; Le Traon, Yves. 'CodeBERT-nt: code naturalness via CodeBERT.' arXiv:2208.06042, 2022. DOI: 10.48550/arXiv.2208.06042. https://arxiv.org/abs/2208.06042
  6. Yang, Chen; Chen, Junjie; Jiang, Jiajun; Huang, Yuliang. 'Dependency-Aware Code Naturalness.' arXiv:2409.00747, 2024. DOI: 10.48550/arXiv.2409.00747. https://arxiv.org/abs/2409.00747
  7. Abdelsalam, Youssef; Peitek, Norman; Maurer, Anna-Maria; Toneva, Mariya; Apel, Sven. 'How do Humans and LLMs Process Confusing Code?' arXiv:2508.18547, 2025. DOI: 10.48550/arXiv.2508.18547. https://arxiv.org/abs/2508.18547
  8. Rong, Xiaokai; Sefidi Esfahani, Mohammadali; Yadavally, Aashish; Rigby Peter; Nguyen, Tien N. 'Can Perplexity Serve as a Cognitive Signal for Code Understandability?' arXiv:2608.00624, 2026. DOI: 10.48550/arXiv.2608.00624. https://arxiv.org/abs/2608.00624
  9. Yetiştiren, Burak; Özsoy, Işık; Ayerdem, Miray; Tüzün, Eray. 'Evaluating the Code Quality of AI-Assisted Code Generation Tools: An Empirical Study on GitHub Copilot, Amazon CodeWhisperer, and ChatGPT.' arXiv:2304.10778, 2023. DOI: 10.48550/arXiv.2304.10778. https://arxiv.org/abs/2304.10778
  10. Santa Molison, Alfred; Moraes, Marcia; Melo, Glaucia; Santos, Fabio; Assuncao, Wesley K. G. 'Is LLM-Generated Code More Maintainable & Reliable than Human-Written Code?' arXiv:2508.00700, 2025. DOI: 10.48550/arXiv.2508.00700. https://arxiv.org/abs/2508.00700
  11. Becha, Rahma; Sellami, Asma; Bouassida, Nadia; Idri, Ali. 'An Automated Evaluation of LLM Generated Code Maintainability Based on Multi-Level Measures and Comparative Analysis.' Proceedings of the 21st International Conference on Evaluation of Novel Approaches to Software Engineering (ENASE 2026), SciTePress, pp. 951-958, 2026. DOI: 10.5220/0014973100004015. https://doi.org/10.5220/0014973100004015
  12. Zheng, Jiasheng; Cao, Boxi; Ma, Zhengzhao; Pan, Ruotong; Lin, Hongyu; Lu, Yaojie; Han, Xianpei; Sun, Le. 'Beyond Correctness: Benchmarking Multi-dimensional Code Generation for Large Language Models.' arXiv:2407.11470, 2024. DOI: 10.48550/arXiv.2407.11470. https://arxiv.org/abs/2407.11470
  13. Shivashankar, Karthik; Martini, Antonio. 'Better Python Programming for all: With the focus on Maintainability.' arXiv:2408.09134, 2024. DOI: 10.48550/arXiv.2408.09134. https://arxiv.org/abs/2408.09134
  14. Matias, Bruno Claudino; Freire, Savio; Freitas, Juliana; Fronchetti, Felipe; Damevski, Kostadin; Spinola, Rodrigo. 'A Survey on Large Language Model Impact on Software Evolvability and Maintainability: the Good, the Bad, the Ugly, and the Remedy.' arXiv:2601.20879, 2026. DOI: 10.48550/arXiv.2601.20879. https://arxiv.org/abs/2601.20879