End-to-End Planner Integration and Error-Propagation Modeling for EXPLAIN-only Residual Cardinality-Estimation Corrections in TiDB (the TiCard Open Questions)

TiCard (arXiv:2512.14358, 2025) learns multiplicative residual corrections to a database's native cardinality estimates using only EXPLAIN-derived features, with offline EXPLAIN ANALYZE labels, and shows large operator-level tail Q-error reductions on TiDB with TPC-H and the Join Order Benchmark under a low-trace regime. The paper itself leaves the deployment questions open: it never injects corrected cardinalities into TiDB's plan enumeration and costing loop, predicts each operator's correction independently, and treats robustness (zero cardinalities, extreme tails, out-of-distribution plan motifs, drift under optimizer evolution) as unresolved. The problem is whether, and under which integration policies, such per-operator residual corrections translate into attributable end-to-end plan-quality gains when used inside the optimizer, together with modeling the propagation of and dependencies among per-operator estimation errors through the plan tree, and with characterizing the stability of the corrected estimates in deployment (zero-cardinality and extreme-tail behavior, out-of-distribution operators and rare plan motifs, and drift under optimizer evolution). No resolution is known in the literature to date; these questions remain open.

Background

Cost-based query optimizers, following the classical Selinger design, enumerate candidate execution plans for a query and rank them with a cost model whose most consequential inputs are cardinality estimates: predicted row counts of intermediate results such as filter outputs and join outputs. Production estimators rely on per-column statistics (histograms, sketches) and independence assumptions, which miss multi-column and cross-table correlations; the Join Order Benchmark (a suite of analytical SQL queries over a real database) study of Leis et al. (2015) showed empirically that cardinality-estimation errors are widespread and the dominant factor behind poor plans, while cost models and enumeration strategies matter comparatively less, and Ioannidis and Christodoulakis (1991) formally analyzed how such errors propagate into the sizes of join results.

A large research line therefore applies machine learning to cardinality estimation (learned CE): query-driven methods learn a supervised mapping from query features to cardinalities, while data-driven methods such as DeepDB and NeuroCard learn a model of the joint data distribution from which subplan cardinalities can be derived. Despite consistent improvements in estimation accuracy (measured by Q-error, $\max(\hat{c}/c, c/\hat{c})$, the factor by which an estimate $\hat{c}$ deviates from the true cardinality $c$), a decade-later retrospective by Leis et al. (2025) concludes that learned approaches have not been widely adopted in industry, citing training and inference cost, difficulty adapting to changing workloads, black-box unpredictability, and the risk of performance regressions; the named open challenges are robustness, adaptive execution, and realistic workloads.

TiCard (Wang, 2025) targets exactly this deployability gap with a correction-based design: rather than replacing the optimizer's estimator, it learns multiplicative residual corrections to the native estimate $E(q)$ of each operator node. Its features come from EXPLAIN output (the planned query operator tree with estimated rows, obtained without executing the query), and its training labels come from EXPLAIN ANALYZE runs (which execute the query and report actual row counts) collected offline. Two instantiations are studied: a gradient boosting regressor (an ensemble of shallow regression trees trained stagewise on residuals, giving sub-millisecond inference) and a tabular foundation model adapted in-context by refreshing a small reference set instead of gradient retraining. On TiDB (an open-source, distributed SQL database) with the TPC-H decision-support benchmark and the Join Order Benchmark, in a deliberately low-trace regime (263 query executions in total, 157 used for learning), TiCard reduces operator-level tail Q-error substantially, e.g. P90 Q-error from 312.85 (native) to 13.69, and proposes conservative integration policies such as correcting only join operators, clamping correction factors, and semantic consistency guards.

The evaluation is, however, deliberately offline: corrected cardinalities are never fed back into TiDB's plan enumeration and costing loop, so no end-to-end evidence (chosen join orders, query latency, plan stability) exists; each operator's correction is predicted independently even though cardinality errors are known to compound along the plan tree; and the paper reports open stability problems, including zero-cardinality handling (a two-stage zero-classifier did not help), large errors on out-of-distribution operators and rare plan motifs, and drift of the learned residuals whenever the optimizer's own estimator or cost model changes. TiCard's roadmap for these stability gaps consists of uncertainty estimates with a principled fallback to native estimates for out-of-distribution operators and rare plan motifs, plus operational procedures (versioning, refresh, drift detection) for shifts in the residual distribution; its offline analysis also reports propagation-related proxies — the root-node (plan-output) Q-error and the query-level worst-case proxies of maximum operator Q-error and maximum join-node Q-error — as partial evidence that estimation errors compound along the plan tree, without itself modeling that dependence. For staged deployment the paper proposes a shadow-mode study in TiDB built on SQL Plan Management (SPM, a mechanism to pin, or bind, a specific execution plan for a query and unbind it to roll back), holding the plan enumerator and cost model fixed so that only the cardinality interface varies; this study is specified but not executed.

Problem Statement

The problem, as posed by TiCard's own limitations and integration roadmap, is to make EXPLAIN-only per-operator residual cardinality corrections a deployable, in-planner component of a cost-based optimizer and to establish, with clean attribution, their end-to-end value. Concretely, for a system in which a learned model corrects a database's native per-operator cardinality estimates (as TiCard does for TiDB): (1) End-to-end integration and attribution: determine whether, and under which integration policies, per-operator residual corrections, once injected into the optimizer's plan enumeration and costing loop, yield attributable end-to-end plan-quality gains — in join order, latency (median and tail), compile-time overhead, and regression rates relative to the native estimator. (2) Error propagation and dependence: determine whether and how per-operator estimation errors propagate through and depend across the plan tree, and whether joint modeling of those dependencies improves propagation-related and end-to-end outcomes over independent per-operator corrections. (3) Deployment stability: characterize and mitigate zero-cardinality and extreme-tail behavior, the behavior of corrected estimates on out-of-distribution operators and rare plan motifs, and the stability of the corrected estimates under shifts in the residual distribution when the optimizer's estimator or cost model changes. These three items correspond to distinct open limitations and future-work directions in TiCard; each is an independent open question that stands on its own. The end-to-end claim in (1) is refutable and may fail to hold under the integration policies actually tested.

References

1. Qizhi Wang. TiCard: Deployable EXPLAIN-only Residual Learning for Cardinality Estimation. arXiv:2512.14358 (2025). https://arxiv.org/abs/2512.14358
2. Viktor Leis, Andrey Gubichev, Atanas Mirchev, Peter Boncz, Alfons Kemper, Thomas Neumann. How good are query optimizers, really? Proceedings of the VLDB Endowment 9(3):204-215 (2015). DOI: 10.14778/2850583.2850594. https://doi.org/10.14778/2850583.2850594
3. Viktor Leis, Andrey Gubichev, Atanas Mirchev, Peter Boncz, Alfons Kemper, Thomas Neumann. Still Asking: How Good Are Query Optimizers, Really? Proceedings of the VLDB Endowment 18(12):5531-5536 (2025). DOI: 10.14778/3750601.3760521. https://doi.org/10.14778/3750601.3760521
4. Yannis E. Ioannidis, Stavros Christodoulakis. On the propagation of errors in the size of join results. In Proceedings of the 1991 ACM SIGMOD International Conference on Management of Data, SIGMOD '91, pages 268-277 (1991). DOI: 10.1145/115790.115835. https://doi.org/10.1145/115790.115835
5. Benjamin Hilprecht, Andreas Schmidt, Moritz Kulessa, Alejandro Molina, Kristian Kersting, Carsten Binnig. DeepDB: Learn from Data, Not from Queries! Proceedings of the VLDB Endowment 13(7):992-1005 (2020). DOI: 10.14778/3384345.3384349. https://doi.org/10.14778/3384345.3384349
6. Zongheng Yang, Amog Kamsetty, Sifei Luan, Eric Liang, Yan Duan, Xi Chen, Ion Stoica. NeuroCard: One Cardinality Estimator for All Tables. Proceedings of the VLDB Endowment 14(1):61-73 (2020). DOI: 10.14778/3421424.3421432. https://doi.org/10.14778/3421424.3421432
7. Parimarjan Negi, Ryan Marcus, Andreas Kipf, Hongzi Mao, Nesime Tatbul, Tim Kraska, Mohammad Alizadeh. Flow-Loss: Learning Cardinality Estimates That Matter. arXiv:2101.04964 (2021). https://arxiv.org/abs/2101.04964
8. Ryan Marcus, Parimarjan Negi, Hongzi Mao, Nesime Tatbul, Mohammad Alizadeh, Tim Kraska. Bao: Making Learned Query Optimization Practical. In Proceedings of the 2021 International Conference on Management of Data, SIGMOD '21, pages 1275-1288 (2021). DOI: 10.1145/3448016.3452838. https://doi.org/10.1145/3448016.3452838
9. Yuxing Han, Haoyu Wang, Lixiang Chen, Yifeng Dong, Xing Chen, Benquan Yu, Chengcheng Yang, Weining Qian. ByteCard: Enhancing ByteDance's Data Warehouse with Learned Cardinality Estimation. arXiv:2403.16110 (2024). https://arxiv.org/abs/2403.16110
10. Guido Moerkotte, Thomas Neumann, Gabriele Steidl. Preventing bad plans by bounding the impact of cardinality estimation errors. Proceedings of the VLDB Endowment 2(1):982-993 (2009). DOI: 10.14778/1687627.1687738. https://doi.org/10.14778/1687627.1687738
11. Kangfei Zhao, Jeffrey Xu Yu, Zongyan He, Hao Zhang. Uncertainty-aware Cardinality Estimation by Neural Network Gaussian Process. arXiv:2107.08706 (2021). https://arxiv.org/abs/2107.08706
12. Lyu Yi, Weiqi Feng, Yuanbiao Wang, Yuhong Kan. MonoM: Enhancing Monotonicity in Learned Cardinality Estimators. arXiv:2512.22122 (2025). https://arxiv.org/abs/2512.22122
13. Junyi Zhao, Kai Su, Yifei Yang, Xiangyao Yu, Paraschos Koutris, Huanchen Zhang. Debunking the Myth of Join Ordering: Toward Robust SQL Analytics. arXiv:2502.15181 (2025). https://arxiv.org/abs/2502.15181
14. Asoke Datta, Brian Tsan, Yesdaulet Izenov, Florin Rusu. Analyzing Query Optimizer Performance in the Presence and Absence of Cardinality Estimates. arXiv:2311.17293 (2023). https://arxiv.org/abs/2311.17293
15. Pengyuan Zou. DBSel-CV: Residual-Learned Cardinality Estimation with Control-Variate Correction over SQLite. Proceedings of the 2025 3rd International Conference on Mathematics and Machine Learning, pages 85-90 (2025). DOI: 10.1145/3783779.3783794. https://doi.org/10.1145/3783779.3783794
16. Ilsun Chang. Risk-Aware GPU-Assisted Cardinality Estimation for Cost-Based Query Optimizers. arXiv:2512.19750 (2025). https://arxiv.org/abs/2512.19750
17. Madhulatha Mandarapu, Sandeep Kunkunuru. When Does q-error Predict Plan Regret? Three Regimes of Cardinality-Estimation Error. arXiv:2606.15600 (2026). https://arxiv.org/abs/2606.15600
18. Mihail Stoian, Tiemo Bang, Hangdong Zhao, Jesús Camacho-Rodríguez, Yuanyuan Tian, Andreas Kipf. The Case for Cardinality Lower Bounds. arXiv:2601.13117 (2026). https://arxiv.org/abs/2601.13117