LLM Function Calling and Tool Use: A Production Architecture (2026)

Most teams ship their first LLM function calling feature in an afternoon and spend the next three months fixing it in production. The demo works: the model emits a neat JSON blob, you call an API, you paste the result back, the user gets a grounded answer. Then real traffic arrives. The model invents an argument that was never in your schema. It picks the wrong tool out of forty. Two tool calls race and corrupt shared state. A prompt buried in a retrieved document convinces the model to call delete_account. None of these are prompt-engineering problems. They are orchestration and validation problems wearing a prompt costume.

That is the thesis of this piece: function calling is not a clever prompt that makes a model “use tools.” It is a distributed system in which a non-deterministic component proposes actions and your deterministic infrastructure decides whether, when, and how those actions execute. Treat it as the former and you get a fragile demo. Treat it as the latter and you get something you can put on a pager.

What this covers: the tool schema contract, the request/response loop, parallel calls, routing at scale, argument validation, error handling and retries, streaming, security, observability, how all of this underpins agents, and where constrained decoding and the Model Context Protocol fit.

Context and Background

Function calling arrived as a first-class API feature in mid-2023 and has since converged across vendors into a recognisable shape. You declare a set of tools, each with a name, a natural-language description, and a machine-readable parameter schema. You send those declarations alongside the user’s message. The model, instead of answering in prose, may respond with a structured request to invoke one or more of those tools with specific arguments. Your code runs the tool and returns the result. The model then incorporates that result into its next turn.

The three major APIs express this with slightly different vocabularies. OpenAI’s Chat Completions and Responses APIs use a tools array of function definitions and return tool_calls on the assistant message, with a matching tool role for results. Anthropic’s Messages API uses tool_use content blocks that arrive with stop_reason: "tool_use", and you reply with tool_result blocks matched by tool_use_id. Google’s Gemini API uses functionDeclarations and returns a functionCall part. The names differ; the control flow is identical. Learn one and you have learned all three, which is exactly why treating the pattern architecturally — rather than memorising one vendor’s field names — pays off.

What has changed by 2026 is the surrounding rigour. Strict schema enforcement, parallel tool calls, and standardised tool servers via the Model Context Protocol have moved the hard problems from “can the model emit valid JSON” to “how do I route, validate, secure, and observe hundreds of tool invocations per second.” The model is now the easy part. Everything around it is the engineering. For the authoritative field definitions, the OpenAI function calling guide and Anthropic’s tool-use docs remain the primary references, and both are updated more often than most blog posts.

Reference Architecture: The Loop Is the Product

A production tool-use system is best understood as a loop with four participants: the caller (your application), the model, an executor, and the tools themselves. The model never touches your tools directly. It emits intent — a structured description of a call it wants made — and your executor decides how to honour that intent. That separation is the single most important design decision in the whole system.

Direct answer: LLM function calling works by sending tool schemas with the prompt; the model returns a structured tool call instead of prose, your executor validates the arguments and runs the tool, and the result is appended to the conversation so the model can produce a grounded final answer — a loop that may repeat several times per user turn.

Figure 1: The end-to-end tool-use loop. The model emits a tool call, the executor validates and runs it, and the result is fed back for the model to reason over.

Figure 1 traces one full cycle. The user’s prompt goes to the model along with the tool schemas. The model responds not with an answer but with a tool call — say get_weather with {"city": "Bengaluru"}. The executor validates those arguments against the schema, calls the real weather API, and gets back 31C, sunny. It appends that result to the conversation history and calls the model again. Now the model has the fact it lacked and produces the final answer. The user never sees the intermediate turns.

The model proposes, the executor disposes

The model’s output is a proposal, not a command. This framing dissolves most safety and reliability confusion. When the model says “call transfer_funds with amount 5000,” it has not moved any money. It has expressed a wish. Your executor is the authority that checks the allow-list, validates the amount against a policy, confirms the user is authorised, and only then executes. Every guardrail lives in the executor, because the executor is deterministic and the model is not. If you find yourself trying to prompt the model into never doing something dangerous, you are guarding the wrong layer.

The conversation is append-only state

The loop is stateful, and the state is the message list. Each turn appends: the user message, the assistant’s tool-call message, the tool-result message, the next assistant message. This append-only transcript is what lets the model chain calls — look up a user, then use that user’s ID to fetch their orders, then summarise. It is also your primary debugging artifact. When something goes wrong, the message list is the flight recorder. Persist it. In production you will replay these transcripts more often than you expect, so store them with enough fidelity to reconstruct exactly what the model saw.

Termination is your responsibility, not the model’s

Nothing in the protocol guarantees the loop ends. The model can request tool call after tool call indefinitely. Your executor must impose a ceiling — a maximum number of iterations per user turn, a wall-clock budget, and a token budget. A common production default is 8 to 12 iterations before you force a summarising response or return a graceful failure. Without this ceiling, a confused model will happily spin until it exhausts your rate limit or your patience. Treat the iteration cap the way you treat a database connection timeout: non-negotiable infrastructure.

Deeper Walk-through: Schemas, Parallelism, Routing, Streaming

Tool schemas are a contract in JSON Schema

A tool definition has three parts, and every one of them is load-bearing. The name is an identifier. The description is natural-language documentation the model reads to decide whether and when to call the tool. The parameters object is a JSON Schema document describing the arguments — types, enums, required fields, formats, and constraints. Both OpenAI and Anthropic accept a JSON Schema subset for this; OpenAI names the field parameters, Anthropic names it input_schema.

The description is where most reliability is won or lost, and teams consistently underinvest here. The model’s tool-selection accuracy depends far more on clear descriptions than on any prompt trick. “Get weather” is a bad description. “Return current temperature and conditions for a city. Use only when the user asks about present or forecast weather. Do not use for historical climate questions.” is a good one — it tells the model both the capability and the boundary. Write descriptions as if onboarding a new engineer who will be fired for guessing.

Use the schema to constrain aggressively. An enum for a status field means the model physically cannot invent a fourth status. A format: "date" on a string signals the expected shape. Marking fields required and setting additionalProperties: false closes the door on hallucinated extras. OpenAI’s strict: true mode, introduced in late 2024, guarantees the emitted arguments conform exactly to the schema by constraining decoding at generation time — the same mechanism family covered in our piece on constrained decoding for structured output. Strict mode eliminates the entire class of malformed-argument bugs, at the cost of some flexibility: notably, strict mode and parallel tool calls have historically been mutually exclusive on OpenAI, so you choose one guarantee per call.

Parallel tool calls change your executor’s shape

When a user asks “compare the weather in Delhi, Mumbai, and Chennai,” a sequential executor makes three round-trips and pays three latencies back to back. Parallel tool calling lets the model emit all three get_weather calls in a single assistant turn, and your executor fans them out concurrently. OpenAI enables parallel calls by default; Anthropic returns multiple tool_use blocks in one turn and explicitly leaves execution order to you. The latency win is real — independent read-only calls collapse from a sum to a max — but it forces two design decisions.

First, concurrency safety. Parallel is safe for independent, read-only operations. It is dangerous for calls with side effects or shared state. If the model emits book_flight and charge_card in one turn, running them concurrently invites a partial-failure mess where the flight books but the charge fails. The general rule: run reads in parallel, serialise writes, and never assume the model understood your ordering constraints. Encode ordering in the executor, not in a hopeful prompt.

Second, result correlation. When you return results, each must be matched back to the specific call that produced it — by tool_call_id on OpenAI, tool_use_id on Anthropic. Get the correlation wrong and you feed the Delhi temperature into the Mumbai slot, producing a confidently wrong answer with no error anywhere in your logs. This is a silent-corruption failure mode, which makes it far more dangerous than a loud crash.

Figure 2: Routing at scale. When the tool count is large, a semantic router shortlists candidate tools so the model sees only a handful of relevant schemas.

Routing at scale: when forty tools become a problem

With five tools, you inline every schema into every request and move on. With forty, or four hundred, you have two compounding problems. Schemas consume context tokens on every call, and selection accuracy degrades as the model chooses among more near-synonymous options. Figure 2 shows the standard answer: a routing layer that narrows the candidate set before the model ever sees it.

The most common approach is semantic retrieval over tool descriptions. You embed each tool’s description once, embed the incoming query at request time, and retrieve the top-k most relevant tools — often k of 5 to 10 — to inject as schemas. This is retrieval-augmented tool selection, structurally identical to the retrieval step in agentic RAG architectures. A second pattern is hierarchical routing: a cheap first-stage classifier picks a category (billing, weather, calendar), and only that category’s tools are exposed. Anthropic’s platform has moved toward a native tool-search capability that does this narrowing server-side, so the model can discover tools without you paying the full schema-token tax on every turn. Whichever you choose, the router is now a component you must test and monitor in its own right — a bad shortlist caps the model’s ceiling before it even reasons.

Streaming tool calls

When you stream responses for latency, tool calls stream too — and they arrive fragmented. The function name comes first, then the arguments accumulate token by token as a partial JSON string. You cannot parse or execute until the arguments are complete, so your client must buffer the fragments per call index and only dispatch when the tool-call delta closes. Teams that naively json.loads a half-streamed argument string get intermittent parse errors that are maddening to reproduce because they depend on chunk boundaries. Buffer first, parse once, then execute. Streaming improves perceived latency for the surrounding prose; it does not let you start a tool early.

Argument validation and error handling as a pipeline

Everything the model emits passes through a validation pipeline before it becomes an action, and that pipeline has a fixed order. Figure 3 lays it out. First, does the argument string parse as JSON at all? Then, does it satisfy the JSON Schema — types, enums, required fields? Then coerce and normalise: trim a stray space, cast a numeric string, canonicalise a date. Then the business-rule gate the schema cannot express — is this amount within policy, is this record owned by the requesting user? Only then does the handler run, and even the handler’s own failure — a timeout, a 500 from the downstream API — routes back into a bounded retry.

Figure 3: The validation and error-handling pipeline. Each failed check returns a structured error the model can read and retry, and handler failures fall through a bounded retry rather than crashing the loop.

The design principle is that failures are fed back, not thrown away. When validation fails, return a structured error message describing what was wrong (“amount exceeds the 10000 limit”) as the tool result. The model reads it and can correct itself on the next turn — this is where a well-designed loop self-heals. Retries must be bounded, though: a handful of attempts with backoff, then a terminal error, never an open-ended retry that becomes an infinite loop.

Trade-offs, Gotchas, and What Goes Wrong

Function calling fails in a small number of recognisable ways, and naming them is half the defence.

Hallucinated arguments. The model emits an argument that violates the schema — a made-up enum value, a missing required field, a string where a number belongs. Strict/constrained decoding prevents most of this at generation; where it is unavailable, schema validation in the executor catches the rest. Never trust arguments. Validate every one, every time, as covered in depth in our output validation and guardrails guide.

Wrong tool selection. The model calls search_web when it should have called search_internal_docs, usually because the two descriptions overlap. The fix is not a better prompt; it is disambiguating descriptions, tighter routing, and sometimes merging two confusable tools into one with a mode parameter. Measure tool-selection accuracy explicitly — it is a metric, not a vibe.

Infinite and oscillating loops. The model calls a tool, dislikes the result, calls it again with a trivially different argument, and repeats. Or two tools ping-pong forever. The iteration cap catches the simple case. For the oscillating case, detect repeated identical or near-identical calls within a turn and break the loop with an instruction to proceed or fail. A loop that has made the same call three times will not make it work on the fourth.

Schema drift. You change a tool’s parameters, but cached prompts, evaluation fixtures, or a downstream service still expect the old shape. The model, following the new schema, emits arguments the old handler cannot parse. Version your tool schemas and treat a schema change like a database migration — with a compatibility window, not a hard cutover.

Silent result mis-correlation. As noted under parallel calls, mismatched IDs feed the right data into the wrong slot. This produces no error and is caught only by end-to-end evaluation on multi-tool prompts. It is the most expensive bug in the category precisely because nothing crashes.

Security: Excessive Agency Is the Real Risk

Figure 4: Defence in depth for tool calls — allow-list, schema validation, argument policy, human approval for sensitive actions, scoped credentials, sandboxed execution, and audit logging.

The OWASP Top 10 for LLM applications names “excessive agency” as a distinct risk, and function calling is exactly where it materialises. The moment a model can trigger real actions, prompt injection stops being a content problem and becomes a privilege-escalation problem. A malicious instruction hidden in a retrieved web page or a user-supplied document can attempt to steer the model into calling a destructive tool. Because the model treats all text in its context as potentially instructive, you cannot rely on it to refuse.

Figure 4 shows the layered defence. Allow-listing means the executor only dispatches calls to explicitly registered tools — an injected run_shell call for an unregistered tool dies at the gate. Argument policy enforces limits the schema cannot express: a transfer under a threshold, a query scoped to the current user’s own data, a file path inside an allowed directory. Human-in-the-loop approval gates any sensitive or irreversible write; the executor pauses and surfaces the proposed call for confirmation rather than executing it. Scoped, least-privilege credentials ensure a compromised tool call can only reach what that specific tool needs — the calendar tool holds no database credentials. Sandboxed execution contains code-running tools. Audit logging records every proposed and executed call as a trace span, so an incident is reconstructable. The through-line: never let the model’s proposal become an action without a deterministic policy check in between.

Practical Recommendations

Start with the smallest surface that works and grow deliberately. If you have one or two tools, inline them and skip the router entirely — routing infrastructure for three tools is over-engineering. Reach for a semantic router only when tool count or schema-token cost actually hurts, and reach for a standardised tool server such as an MCP server when you need to share the same tools across multiple applications or teams rather than re-implementing them per app.

The decision, in one table:

Situation	Pattern	Why
1–8 stable tools, one app	Inline schemas, direct executor	Simplest; no routing overhead
Many tools or high schema-token cost	Semantic / hierarchical router	Keeps context small, selection accurate
Tools shared across apps or teams	MCP server	Standard interface, write once, reuse
Irreversible or sensitive actions	Human-in-the-loop + scoped creds	Contains excessive agency
Latency-sensitive multi-fetch	Parallel read calls, serial writes	Collapses latency without race risk

Before you ship, walk this checklist:

• [ ] Every tool has an unambiguous description with an explicit “use when” and “do not use when.”
• [ ] Every argument is validated against JSON Schema in the executor; enable strict/constrained decoding where available.
• [ ] An iteration cap, wall-clock budget, and token budget bound every user turn.
• [ ] Writes are serialised and gated; reads may run in parallel.
• [ ] Sensitive tools require approval and use scoped, least-privilege credentials.
• [ ] Every proposed and executed call is logged as a trace span with inputs, outputs, and latency.
• [ ] Multi-tool and injection prompts are in your evaluation suite, not just happy-path cases.

Treat these as the floor, not the ceiling. Illustrative figures to calibrate expectations, clearly labelled as illustrative: a semantic router might trim per-turn tool tokens from several thousand to a few hundred, and parallel execution might cut a three-fetch turn from roughly 2.4 seconds to 0.9 seconds. Measure your own numbers before quoting any of these — every workload is different.

How This Underpins Agents

An agent is nothing more exotic than the loop in Figure 1 running until a goal is met rather than for a single turn. Give the model tools, a termination condition, and permission to iterate, and the tool-use loop becomes an agent. Everything in this article — routing, validation, iteration caps, security layers, observability — is therefore agent infrastructure. A team that has built a disciplined function-calling layer has already built most of an agent platform; a team that has built a fragile one has built a liability that gets worse the more autonomy it is given. That is the real reason to treat function calling as an orchestration and validation problem from day one: the blast radius only grows.

Constrained decoding and MCP sit on either side of this picture. Constrained decoding is a generation-time guarantee about the shape of the arguments — it makes the model’s proposal well-formed. MCP is a transport and packaging standard for the tools themselves — it makes a tool reusable across applications. Neither replaces the executor, the router, the validation, or the security layers. They make specific parts of the loop more robust; the loop, and the responsibility for it, remain yours.

Frequently Asked Questions

What is LLM function calling, in one sentence?

LLM function calling is an API capability where you describe tools to a model using structured schemas, and the model responds with a structured request to invoke one of those tools with specific arguments, which your code executes and feeds back. It converts a text-only model into one that can trigger real actions and fetch live data. Crucially, the model only proposes the call; your application decides whether and how to run it, which is why validation and orchestration matter more than prompting.

Is function calling the same as structured output?

They are related but distinct. Structured output constrains the model to emit JSON matching a schema — useful when you want a typed answer. Function calling uses that same schema mechanism but adds the semantics of invocation: the model chooses which tool and when, then consumes the result in a follow-up turn. Structured output is a shape guarantee; function calling is a control-flow pattern. Modern strict/constrained decoding underlies both, which is why the two topics are often confused despite serving different purposes.

How do parallel tool calls work and when should I avoid them?

Parallel tool calls let the model request several tools in a single turn, which your executor runs concurrently to save latency. They are ideal for independent, read-only operations such as fetching three cities’ weather at once. Avoid them for operations with side effects, shared state, or ordering requirements — booking and charging, for instance — where concurrency risks partial failure. Run reads in parallel, serialise writes, and always match each result to its call by ID to avoid silent mis-correlation.

How many tools can a model handle before accuracy drops?

There is no hard limit, but selection accuracy degrades as the tool set grows and descriptions start to overlap, and each schema costs context tokens on every call. Many teams see the pain around a few dozen tools. The fix is a routing layer — semantic retrieval or hierarchical classification — that shortlists a handful of relevant tools per request. This keeps the prompt small and the choice clear. The threshold depends on how distinct your tool descriptions are, so measure selection accuracy directly rather than guessing.

How do I stop prompt injection from triggering dangerous tool calls?

You cannot rely on the model to refuse, because it treats injected text as potentially instructive. Instead, defend in the executor: allow-list registered tools, enforce argument policies the schema cannot express, require human approval for sensitive or irreversible writes, and give each tool scoped, least-privilege credentials. Sandbox any code-running tool and log every proposed and executed call. The principle is that a model proposal must always pass a deterministic policy check before it becomes a real action.

What is the difference between function calling and MCP?

Function calling is the per-request protocol between your application and the model — declaring tools and handling the tool-call loop. The Model Context Protocol is a standard for packaging tools behind a server so multiple applications can reuse them over a common interface. MCP does not replace the function-calling loop; your application still routes, validates, secures, and observes the calls. Use plain function calling for app-specific tools, and an MCP server when the same tools must be shared across apps or teams.

Further Reading

• Agentic RAG architecture: retrieval agents in 2026 — how retrieval and tool use combine into iterating retrieval agents.
• Anthropic Model Context Protocol (MCP) architecture 2026 — the standard for reusable, shareable tool servers.
• Constrained decoding and structured LLM output architecture 2026 — the generation-time mechanism behind strict tool arguments.
• LLM output validation, structured outputs and guardrails 2026 — validating and coercing tool arguments in the executor.
• External: OpenAI function calling guide, Anthropic tool use overview, and the JSON Schema specification.

By Riju — about