← logs/

Function-calling evaluation at the mechanism layer

Most function-calling evals report one number: accuracy. That number is a compound of at least four different decisions the model makes, and they fail for different reasons, get fixed in different places, and deserve different oracles. A harness that reports a single score tells you something is wrong. It does not tell you where.

Four decisions, one number

  • Tool selection: which of the available functions matches the intent. Fails when schemas overlap or descriptions are vague.
  • Parameter fill: the right values in the right slots. Fails on unit mismatches, aliases, and missing information.
  • Invocation judgment: call now, ask a clarifying question, or refuse. Fails on over-eager calls (charging before confirmation) and on under-calling (never acting).
  • Multi-step chaining: state carried across calls, where the second call depends on the first result. Fails on lost state, redundant calls, wrong order.

A model can be strong at selection and fill and useless at judgment and chaining in production. The benchmark has to separate them, or the fix loop is blind.

Deterministic judging versus LLM judges

Two families of oracles.

Deterministic checking compares the emitted call against a reference: AST matching for structure, executable evaluation for runtime behavior, state comparison for postconditions. Cheap, reproducible, no judge variance. The catch: the oracle encodes the semantics, so you only measure what you encoded.

LLM judging scores free-form behavior when the correct action is itself a judgment call, for example "did the agent ask the right clarifying question?" Flexible, but it drifts and it costs. Never the first choice; always the residual category.

The best harnesses layer them: deterministic where the semantics are checkable, LLM judgment only for what escapes it.

Close read: BFCL

The Berkeley Function Calling Leaderboard is the de facto standard, and as of mid-2026, V4 is an agentic evaluation. The BFCL paper (ICML 2025) describes the core: AST-based evaluation that scales to thousands of functions, expert-curated plus user-contributed functions, and an explicit test of abstention: the model should decline to call when no function fits, in a stateful multi-step setting.

The version history is the field's history. V1 introduced AST scoring for simple, parallel, and multiple calls. V2 added enterprise and community-contributed functions. V3 added multi-turn, multi-step calls where the model goes back and forth with the user, including asking clarifying questions. V4 is a holistic agentic evaluation: web search, memory read and write, and format sensitivity, all verified with AST or state-transition checks for determinism.

One caution before comparing numbers across that history: V4's agentic turn invalidates V1 through V3 score comparisons, because the earlier versions graded single calls against fixed schemas while V4 grades a whole loop with web search and memory, so a lower V4 score can mean a weaker agent, not a weaker caller.

The category list already encodes the mechanism layer: which function among many (selection), argument correctness (fill), abstention categories such as miss-function and miss-parameter (judgment: do not hallucinate a call when nothing matches, or when a required parameter is missing), and multi-turn base cases (chains). The benchmark family is a failure taxonomy in disguise, and the miss categories exist because those are the failures that matter in production.

Close read: tau-bench

tau-bench is a benchmark for tool-agent-user interaction in real-world domains (Yao, Shinn, Razavi, Narasimhan, 2024). Retail and airline customer service. A simulated user holds a dynamic conversation with an agent that has domain-specific API tools and policy guidelines.

The scoring move is the important one. It does not grade individual calls; it compares the database state at the end of the conversation against the annotated goal state. Did the booking happen, with the right flight, under the right policy? Call-level accuracy is a proxy. State equality is the ground truth of whether the agent finished the job.

The pass^k metric measures reliability over repeated trials, and the finding was blunt: gpt-4o succeeded on under 50% of tasks, with pass^8 under 25% in retail. One-shot accuracy overstates production readiness; agents fail differently across runs.

tau-bench is the benchmark that says: evaluation at the mechanism layer means evaluating the chain, not the links.

Close read: ToolSandbox

ToolSandbox (Lu et al.; Findings of NAACL 2025) evaluates tool use in a stateful, conversational, interactive setting: stateful tool execution with implicit dependencies between tools, a built-in user simulator for on-policy evaluation, and dynamic evaluation of intermediate and final milestones.

Its task classes map cleanly onto the four decisions. State dependency: the output of one tool changes what the next call must do (chaining). Canonicalization: the user says "the blue one" and the agent must resolve the alias into the exact parameter (fill). Insufficient information: the agent must recognize it cannot act and ask (judgment).

The design point I take is milestones, not just final states. A final-state check tells you the agent failed; intermediate milestones tell you where in the chain it failed. That is the mechanism layer.

Scoring the four decisions

A harness with four sub-scores, one per decision:

selection    which function, among N
fill         argument correctness per call
judgment     call / ask / refuse, plus timing
chain        state consistency across calls

Deterministic oracles first: schema validation for fill, overlap tests for selection, state diffs for chains, tau-bench style. LLM judgment reserved for the residual, with a fixed rubric and a held-out human sample to audit the judge itself.

Design notes: a retail-domain harness

I designed an internal function-calling eval harness at PiPlan.ai, in the retail domain, with a 12-way failure taxonomy — design considerations only; no public metrics.

The taxonomy came before any dataset, and three considerations drove it.

Each class must imply a fix location: the prompt, the schema, the tool router, the policy file, or the tool implementation. A class that does not point at a fix is decoration.

Classes must be mutually exclusive so annotation is cheap and the counts mean something. If a failure can plausibly land in two classes, annotators will disagree, and the taxonomy's counts lose meaning.

Retail is policy-heavy: pricing, inventory, shipping constraints. The failures that matter are rarely malformed JSON; they are correct JSON with the wrong action: charging before confirmation, applying a discount the policy forbids, committing inventory before the user agrees. Invocation judgment dominates, which is exactly why the taxonomy needs more than selection and fill categories.

The taxonomy and the judging strategy are one design. Structured fields get deterministic checks; judgment failures get rubrics. Reporting per-class counts beats one accuracy number, because a number cannot tell you which layer to fix. For where this sits in the wider harness picture, see the agent taxonomy overview.

Sources linked in this post were fetched and verified.