Deep classifiers are evaluated both on classification accuracy and on confidence calibration — whether predicted probabilities match empirical correctness frequencies. Whether these two properties are fundamentally coupled, or whether calibration error is decoupled from accuracy, remains unsettled: a 117,702-network architecture benchmark finds a positive accuracy–calibration association only among high-accuracy (>90% on CIFAR-10) architectures and no consistent association across the full search space; comparisons of Bayesian inference methods show an apparent trade-off across methods; and hyperparameter-weighted training objectives exhibit a controlled calibration–accuracy trade-off within a single method. Later work supplies partial theory — the calibration–refinement decomposition of proper losses, accuracy–preserving post-hoc calibration, and accuracy-confounding of calibration metrics — but no unified characterization exists. The open problem is to characterize whether any method-independent relationship between accuracy and calibration error exists in deep classifiers, and what mechanism governs the regime-dependence of the association that is observed.

Background

A classifier is “well calibrated” when its predicted probabilities match empirical correctness frequencies: among test points assigned confidence near 0.90.9, roughly 90% should indeed be correct. The standard summary statistic is the expected calibration error (ECE), which bins predictions by confidence and averages the gap between bin confidence and bin accuracy. Guo et al. (2017) reported that modern deep networks, unlike older ones, are strongly overconfident despite high accuracy, and popularized temperature scaling, a post-hoc (applied after training, without retraining) calibration method that rescales all logits by a single temperature; because the softmax ordering is unchanged, accuracy is exactly preserved while ECE typically drops. This hints that at inference time the two properties are separable, yet the empirical literature repeatedly suggests tension: training with focal loss (a loss that down-weights easy, correctly classified examples) yields calibrated networks with essentially preserved accuracy (Mukhoti et al., 2020); soft calibration losses buy an 82% ECE reduction for about a 0.7% relative accuracy decrease (Karandikar et al., 2021); and joint accuracy-versus-uncertainty losses were proposed precisely to avoid degrading either (Krishnan and Tickoo, 2020). Against this, Minderer et al. (2021) systematically related calibration and accuracy across recent architectures and found the most accurate recent models (notably non-convolutional ones) are among the best calibrated, making architecture, not accuracy, a major determinant of calibration. Three findings sharpen the puzzle. First, a large-scale benchmark over the NATS-Bench neural-architecture-search space (a standardized database of automatically searched network architectures; 117,702 trained networks; Tao et al., ICLR 2024) shows a positive accuracy–calibration correlation only among architectures exceeding 90% accuracy on CIFAR-10, with no consistent association across the full topology search space — a regime-dependent relationship, and explicitly a purely empirical one with no theoretical analysis, as the authors themselves note; the same authors caution that their findings predominantly relate to image classification and that extrapolation to other domains should be approached with caution. Second, in Bayesian neural networks — networks with probability distributions over weights, approximated by methods such as variational inference (VI, which optimizes a tractable surrogate objective) and Hamiltonian Monte Carlo (HMC, a sampling method for the posterior) — Mohan and Scaife (2024) find “a certain degree of trade-off” across inference methods for radio galaxy classification: VI is most accurate, while HMC without data augmentation is best calibrated at only 2.5% lower accuracy. Third, in graph learning, Wu et al. (KDD 2023) train rare-category models with a weighted objective (1−λ)LCE+λLEICE(1-\lambda)\mathcal{L}_{CE}+\lambda\mathcal{L}_{EICE} and observe that increasing the calibration weight λ\lambda improves calibration while degrading accuracy. Recent theory clarifies pieces of this landscape without unifying it. Proper losses — losses for which the true class probabilities are the unique minimizer — such as cross-entropy decompose into calibration error and refinement error (the ability to distinguish classes; related to, but distinct from, 0-1 classification accuracy); Berta et al. (2025) give a variational formulation of this decomposition and prove, with empirical support, that the two components are not minimized simultaneously during training, so validation-loss early stopping sits at a compromise point — and they propose to refine during training and calibrate post hoc. Post-hoc calibration with proven accuracy preservation has been developed further: Esaki et al. (2024) place a probabilistic model on the probability simplex whose predictive distribution is independent of the calibration temperature parameter, so the predicted class — and hence the classification accuracy of the pretrained network — is preserved (their Theorem 1); Zhao et al. (2026) apply an invertible transformation to the logits whose monotonicity, and hence argmax preservation, is softly encouraged via a paired inverse network rather than enforced, with accuracy preservation reported empirically in every setting they evaluate. Together with the argmax invariance of temperature scaling, these results establish that inference-time decoupling is achievable, at least for calibrators whose prediction invariance is proven. Separately, Yang et al. (2026) show theoretically and empirically that global calibration metrics such as ECE and the Brier score (the mean squared error between predicted probabilities and correctness indicators) are confounded by model accuracy — accuracy-controlled evaluation frequently reverses calibration rankings — so part of any observed accuracy–calibration association may be a metric artifact; that result is established for cross-model comparison of large language models, so transferring it to an architecture search space is a hypothesis to be tested rather than an accepted explanation. Meanwhile, applied work continues to manage trade-offs rather than characterize them: exemplar selection for in-context learning (choosing the demonstration examples that guide a large language model's predictions) is posed as an accuracy–calibration Pareto-frontier problem, where a Pareto frontier is the set of operating points at which neither objective can improve without worsening the other (Luo and Deshwal, 2025), and unified calibration losses are designed to mitigate a claimed stability–performance trade-off between two-phase and single-loss calibration training (Gómez-Gálvez et al., 2026). What is missing is an account of whether the observed association between accuracy and calibration error is a genuine property of learning systems or an artifact of how calibration is measured, and of what governs its regime-dependence across training methods, architectures, and accuracy levels.

Problem Statement

Characterize the relationship between classification accuracy and calibration error in deep neural network classifiers. Concretely: (1) does any method-independent relationship between accuracy and calibration error exist at all — i.e., does the empirical association (such as the positive accuracy–calibration correlation observed only among architectures above 90% accuracy on CIFAR-10, and its absence across the full architecture space) reflect a genuine property of learning systems, an artifact of accuracy-confounded calibration metrics and bin-dependent estimation of the expected calibration error (ECE), or a mixture of both; and (2) what mechanism governs this regime-dependence, and can it be predicted rather than only observed. The trade-offs observed during training — under objectives of the form (1−λ)LCE+λLcal(1-\lambda)\mathcal{L}_{CE}+\lambda\mathcal{L}_{cal} that weight calibration against cross-entropy, and across Bayesian inference methods with differing posterior-approximation quality — belong to the empirical landscape surrounding (1)–(2); the sources reporting them present them as observations inside their own applications rather than as open problems.

Two conventions fix what is being asked. “Accuracy” means 0-1 classification accuracy; refinement error, the class-distinguishability component of the calibration–refinement decomposition, is a related but distinct quantity. “Calibration error” is not a single functional: the binned ECE defined above, the Brier score, and accuracy-controlled variants can order the same models differently, and in the benchmark evidence the strength — and in some comparisons the sign — of the accuracy–calibration association depends on the functional, on the ECE bin size, and on whether predictions are taken before or after post-hoc calibration. A relationship between accuracy and calibration error is therefore a statement relative to a specified calibration functional and estimator, a specified calibration state of the predictions, and a specified class of training methods, architectures, and datasets; which class a method-independent claim ranges over is part of what the claim must fix, not a choice to be made after the result is known. That class must be nondegenerate: a method-independent claim must range over at least two substantively different training methods or architecture families, and a characterization that holds only within a single training method or a single architecture family does not establish method-independence.

References

  1. Linwei Tao, Younan Zhu, Haolan Guo, Minjing Dong, Chang Xu, 'A Benchmark Study on Calibration', ICLR 2024, arXiv:2308.11838, https://arxiv.org/abs/2308.11838
  2. Devina Mohan, Anna M. M. Scaife, 'Evaluating Bayesian deep learning for radio galaxy classification', arXiv:2405.18351, https://arxiv.org/abs/2405.18351
  3. Longfeng Wu, Bowen Lei, Dongkuan Xu, Dawei Zhou, 'Towards Reliable Rare Category Analysis on Graphs via Individual Calibration', KDD 2023, DOI:10.1145/3580305.3599525, https://arxiv.org/abs/2307.09858
  4. Chuan Guo, Geoff Pleiss, Yu Sun, Kilian Q. Weinberger, 'On Calibration of Modern Neural Networks', ICML 2017, arXiv:1706.04599, https://arxiv.org/abs/1706.04599
  5. Matthias Minderer, Josip Djolonga, Rob Romijnders, Frances Hubis, Xiaohua Zhai, Neil Houlsby, Dustin Tran, Mario Lucic, 'Revisiting the Calibration of Modern Neural Networks', arXiv:2106.07998, https://arxiv.org/abs/2106.07998
  6. Jishnu Mukhoti, Viveka Kulharia, Amartya Sanyal, Stuart Golodetz, Philip H. S. Torr, Puneet K. Dokania, 'Calibrating Deep Neural Networks using Focal Loss', NeurIPS 2020, arXiv:2002.09437, https://arxiv.org/abs/2002.09437
  7. Archit Karandikar, Nicholas Cain, Dustin Tran, Balaji Lakshminarayanan, Jonathon Shlens, Michael C. Mozer, Becca Roelofs, 'Soft Calibration Objectives for Neural Networks', NeurIPS 2021, arXiv:2108.00106, https://arxiv.org/abs/2108.00106
  8. Ranganath Krishnan, Omesh Tickoo, 'Improving model calibration with accuracy versus uncertainty optimization', NeurIPS 2020, https://proceedings.neurips.cc/paper_files/paper/2020/hash/d3d9446802a44259755d38e6d163e820-Abstract.html
  9. Linwei Tao, Minjing Dong, Chang Xu, 'Dual Focal Loss for Calibration', ICML 2023, arXiv:2305.13665, https://arxiv.org/abs/2305.13665
  10. Eugène Berta, David Holzmüller, Michael I. Jordan, Francis Bach, 'Rethinking Early Stopping: Refine, Then Calibrate', arXiv:2501.19195, https://arxiv.org/abs/2501.19195
  11. Yasushi Esaki, Akihiro Nakamura, Keisuke Kawano, Ryoko Tokuhisa, Takuro Kutsuna, 'Accuracy-Preserving Calibration via Statistical Modeling on Probability Simplex', arXiv:2402.13765, https://arxiv.org/abs/2402.13765
  12. Lening Zhao, Qipeng Zhan, Li Shen, 'Invertible Logits Transformation for Accuracy-Preserving Post-Hoc Uncertainty Calibration', arXiv:2608.10372, https://arxiv.org/abs/2608.10372
  13. Zhichao Yang, Caiqi Zhang, Ruihan Yang, Chengzu Li, Nigel Collier, Deqing Yang, 'When Calibration Rankings Reverse: Accuracy-Controlled Evaluation for Fair Comparison of LLMs', arXiv:2606.30814, https://arxiv.org/abs/2606.30814
  14. Gaoxiang Luo, Aryan Deshwal, 'COM-BOM: Bayesian Exemplar Search for Efficiently Exploring the Accuracy-Calibration Pareto Frontier', arXiv:2510.01178, https://arxiv.org/abs/2510.01178
  15. Sandra Gómez-Gálvez, Tobias Olenyi, Gillian Dobbie, Katerina Taškova, 'Socrates Loss: Unifying Confidence Calibration and Classification by Leveraging the Unknown', arXiv:2604.12245, https://arxiv.org/abs/2604.12245
  16. Jinxu Lin, Linwei Tao, Minjing Dong, Chang Xu, 'Uncertainty Weighted Gradients for Model Calibration', arXiv:2503.22725, https://arxiv.org/abs/2503.22725
  17. Eugène Berta, David Holzmüller, Francis Bach, Michael I. Jordan, 'CalArena: A Large-Scale Post-Hoc Calibration Benchmark', arXiv:2605.30188, https://arxiv.org/abs/2605.30188