Patel et al. (2026) showed that programmatic tool calling — invoking tools through generated Python scripts rather than JSON call objects — matches or exceeds native JSON tool calling on 11 of 14 models on BFCL v4, with advantages on chaining, parallel fan-out, and robustness to context flooding. But their evaluation uses echo-return stubs that never execute tools and score argument serialization only, and models can produce correct aggregate answers from parametric knowledge without executing the calls; the paper's stated limitations leave open how to design paradigms that ensure genuine tool execution, and the paper leaves unexplained a generation-linked failure in which three older OpenAI models emit literal \n escape sequences instead of real newlines in generated code. The surrounding literature supplies building blocks — evaluation-validity audits and anti-memorization benchmarks — but no work yet establishes whether programmatic tool calling's advantages survive an execution-faithful, shortcut-immune end-to-end evaluation. That question remains open.

Background

Large language models (LLMs) act as agents when they invoke external tools — calculators, APIs, database lookups — rather than answering only from what they learned during pretraining. Agent frameworks offer two main interfaces for this. In JSON tool calling, the model emits structured JSON objects naming a function and its arguments, which the host application parses and executes. In programmatic tool calling (PTC), tools are exposed as typed Python function stubs and the model writes a Python script that calls them, so that chaining (using one call's result as the next call's argument) and parallel fan-out (issuing many independent calls) are expressed in ordinary code within a single agent turn; this line descends from CodeAct, which showed that executable code actions outperform JSON-formatted actions on multi-tool tasks.

The Berkeley Function Calling Leaderboard (BFCL) is the standard benchmark family for this capability; its version 4 (BFCL v4) scores predicted function calls against ground-truth calls with a deterministic argument-matching scorer. Patel et al. (2026) compared PTC against JSON tool calling on 309 BFCL v4 entries across 14 models from two families and found: PTC matches or exceeds JSON accuracy on 11 of 14 models; its chaining advantage grows with chain length (an 18.8% absolute gap at chain lengths of 12 or more, attributed to the extra inference turn JSON needs per link); JSON tool calling silently drops calls above a model-specific fan-out threshold (N = 70–72 for Claude Sonnet 5) while PTC holds 100% enumeration accuracy at N = 100; and PTC is stable under context flooding (many decoy tool schemas injected into the prompt) where JSON degrades by 2.3% on average.

Two properties of this evaluation limit what the results certify. First, BFCL v4 uses echo-return stubs: each function returns its own arguments verbatim instead of executing real logic, so the score measures argument serialization accuracy, not end-to-end tool use in which return values feed later calls. Second, in their parallelism ablation the authors scored enumeration accuracy (did the model issue all N required calls?) separately from aggregation accuracy (did it produce the correct aggregate answer?), and found that models frequently produce the correct aggregate — for example the top three countries by population — from parametric knowledge, meaning facts stored in the model's weights, without executing the calls. A third anomaly is unexplained: three older OpenAI models (GPT-4o, GPT-4.1, GPT-5.4-mini) emit literal two-character \n sequences in place of real newlines in generated scripts, causing syntax errors on any multiline code — a failure that tracks model release generation rather than model size, and whose cause the authors explicitly leave out of scope.

Adjacent work has approached pieces of this evaluation-validity problem. ToolQA (Zhuang et al., 2023) built a question-answering dataset specifically designed so that questions cannot be answered from the model's internal knowledge and require tools. A validity audit (Bhat et al., 2026) found an 18.5% evaluator–human misalignment rate across four tool-calling benchmark families including BFCL v4's LLM-judge mode, and argued for decomposed metrics that separately measure tool invocation, task completion, and outcome verification. ToolFailBench (Soni, 2026) uses parametric traps — mock tool returns that contradict plausible memorized priors, plus no-tool control tasks — to label Tool-Skip, Result-Ignore, and related failure modes across 19 models. ToolMATH (Choi et al., 2026) executes real Python tools with intermediate-output reuse for long-horizon diagnostic evaluation. None of these addresses the paradigm comparison itself: whether the advantages measured for code-based tool calling under echo-return scoring survive in an environment where tools execute for real and correct answers cannot be produced from memory.

Problem Statement

The open problem left by Patel et al. (2026) arises from what the echo-return BFCL v4 evaluation cannot certify.

Execution-faithful paradigm evaluation. Design a tool-calling evaluation paradigm — benchmark, harness, and scoring protocol — in which a correct final answer certifies genuine tool execution: tools execute real logic, their return values are required for downstream calls, and ground-truth answers are constructed so that they cannot be produced from parametric knowledge alone. Using such a paradigm, determine whether programmatic tool calling's measured advantages over JSON tool calling — on sequential chaining, parallel fan-out, and robustness to context flooding — transfer from echo-return scoring to end-to-end tool use, or whether they are artifacts of the serialization-based evaluation.

The objective is an execution-faithful account of programmatic versus JSON tool calling, spanning the paradigm and the transfer verdict across the three ablation structures. A solution is not required to resolve the cause of the generation-linked newline-encoding failure described in the Background.

References

  1. The Bitter Lesson of Tool Calling (Ishan Patel, Sahil Sen, Elias Lumer, Vamse Kumar Subbiah, 2026), arXiv:2608.06370, doi:10.48550/arXiv.2608.06370, https://arxiv.org/abs/2608.06370
  2. Benchmarking the Benchmarks: A Validity Audit of Tool-Calling Evaluation (Vishvesh Bhat, Jay Vaghasiya, Muhammad Ahmed Mohsin, Asad Aali, 2026), arXiv:2607.02577, doi:10.48550/arXiv.2607.02577, https://arxiv.org/abs/2607.02577
  3. ToolFailBench: Diagnosing Tool-Use Failures in LLM Agents (Harsh Soni, 2026), arXiv:2607.04686, doi:10.48550/arXiv.2607.04686, https://arxiv.org/abs/2607.04686
  4. ToolQA: A Dataset for LLM Question Answering with External Tools (Yuchen Zhuang, Yue Yu, Kuan Wang, Haotian Sun, Chao Zhang, 2023), arXiv:2306.13304, doi:10.48550/arXiv.2306.13304, https://arxiv.org/abs/2306.13304
  5. Executable Code Actions Elicit Better LLM Agents (Xingyao Wang, Yangyi Chen, Lifan Yuan, Yizhe Zhang, Yunzhu Li, Hao Peng, Heng Ji, 2024), arXiv:2402.01030, doi:10.48550/arXiv.2402.01030, https://arxiv.org/abs/2402.01030
  6. ToolMATH: A Diagnostic Benchmark for Long-Horizon Tool Use under Systematic Tool-Catalog Constraints (Hyeonje Choi, Jeongsoo Lee, Hyojun Lee, Jay-Yoon Lee, 2026), arXiv:2602.21265, doi:10.48550/arXiv.2602.21265, https://arxiv.org/abs/2602.21265
  7. Diagnosing Knowledge Gaps in LLM Tool Use: An Agentic Benchmark for Novel API Acquisition (Jinnuo Liu, Yue Peng, Jinhan Niu, Hongyi Wen, 2026), arXiv:2606.03657, doi:10.48550/arXiv.2606.03657, https://arxiv.org/abs/2606.03657
  8. ToolSense: A Diagnostic Framework for Auditing Parametric Tool Knowledge in LLMs (Ashutosh Hathidara, Sai Shruthi Sistla, Sebastian Schreiber, Sahil Bansal, 2026), arXiv:2606.12451, doi:10.48550/arXiv.2606.12451, https://arxiv.org/abs/2606.12451