15 Jul 2023 AI Engineering 4 min read

Function Calling: From Free Text to Structured Tool Use

OpenAI added function calling to GPT-4 and GPT-3.5 on 13 June 2023. We explain what the new functions parameter enables — tool use, structured extraction, first agent loops — how the call cycle works, and where it fails: hallucinated arguments, unenforced schemas, prompt injection. With validation patterns that hold up in production.

The gap between text and structure

Language models emit text. Applications need structure. Until last month, every production integration of GPT-4 or GPT-3.5 bridged that gap with prompt engineering: instruct the model to answer only with JSON, then parse, repair, and retry. The failure modes were constant. Markdown fences around the payload. Trailing commas. Explanatory prose before the object. A field renamed on every twentieth call. Robust parsers and retry loops helped, but the interface itself remained an informal convention.

Tool use had the same gap. ReAct (Yao et al., October 2022) showed that a model can interleave reasoning steps with actions. Toolformer (Schick et al., February 2023) showed that models can learn to insert API calls into text. In practice, developers extracted lines like `Action: search[...]` from free-form completions with regular expressions. That approach is fragile by construction.

A task arrives — the agent plans its next step. 1/4

What OpenAI shipped on 13 June 2023

On 13 June 2023, OpenAI added function calling to the Chat Completions API. There are two new request parameters: `functions`, a list of function definitions expressed as JSON Schema, and `function_call`, which can force the model to call a specific function. Two new model snapshots understand them: `gpt-4-0613` and `gpt-3.5-turbo-0613`. Both are fine-tuned for two tasks: detecting when a function should be called, and emitting a JSON object with the arguments for it.

The same announcement shipped `gpt-3.5-turbo-16k` with a 16,384-token context window, cut input-token prices for `gpt-3.5-turbo` by 25 percent, and cut the price of the `text-embedding-ada-002` embeddings model by 75 percent. The stable aliases `gpt-4` and `gpt-3.5-turbo` have pointed at the 0613 snapshots since 27 June.

Model	Context window	Input per 1K tokens	Output per 1K tokens
gpt-3.5-turbo-0613	4,096 tokens	$0.0015	$0.002
gpt-3.5-turbo-16k	16,384 tokens	$0.003	$0.004
gpt-4-0613	8,192 tokens	$0.03	$0.06

The function call loop

The mechanics form a loop. Step one: send the conversation plus the function definitions. If the model decides a call is needed, the response arrives with `finish_reason: "function_call"`; the message contains the function name and the arguments as a JSON string, and `content` is null. Step two: your code executes the function. Step three: append the result as a message with role `function` and call the API again. The model then either answers the user or requests the next call.

The division of labor is the point. The model never executes anything. It proposes a call; the application decides whether to perform it. Authorization, rate limits, timeouts, and error handling remain in ordinary code that you can test, review, and log.

Structured extraction without an agent

A function definition does not need a function behind it. Define a single function such as `extract_contacts` whose parameters are exactly the schema you want, and force it with `function_call: {"name": "extract_contacts"}`. The model responds with arguments that match the schema. This turns GPT-3.5 into a capable extraction engine for invoices, support tickets, and e-mails — with no regular expressions in the post-processing.

The output remains a prediction, not a parse. Fields can be empty, mistyped, or invented, and nothing guarantees that the JSON is syntactically valid. Treat every result as untrusted input and validate it against the schema before it enters your pipeline, exactly as you would validate a web form.

Hallucinated arguments and other limits

OpenAI's API reference states the core limit itself: the model "does not always generate valid JSON, and may hallucinate parameters not defined by your function schema". This is observable daily. If the user has not supplied a required argument, `gpt-3.5-turbo-0613` frequently invents one — the example value from the parameter description, or a plausible placeholder name. The `required` list in the schema is a hint to the model, not a constraint enforced by the API.

Two further limits deserve naming. Function definitions are injected into the system message and count against the context window, so large schemas compete with your actual prompt. And tool results are untrusted content: when a function returns text from the open web, that text can steer the model — prompt injection. OpenAI recommends user confirmation before actions with real-world impact. For anything that writes, we treat confirmation as mandatory.

Patterns that survive production

Validate every call against the JSON Schema before executing it. On failure, do not abort. Append a role-`function` message that describes the error and let the model repair its own call; a single corrective round trip resolves most malformed calls. For extraction tasks, set the temperature to 0.

Keep the tool list short and the descriptions precise; selection quality degrades as functions multiply. Separate read functions from write functions and gate every write behind explicit confirmation. Make write operations idempotent, because the model will occasionally request the same call twice. Log the complete argument payload of every call. None of this is glamorous. All of it is the difference between a demo and a system.

Outlook from July 2023

Function calling is a native, fine-tuned implementation of the ReAct idea, and that makes it the first stable substrate for agent-like systems. Auto-GPT and LangChain agents demonstrated the appetite this spring; they also demonstrated the failure rate of long autonomous chains. We expect short supervised loops — three to five tool calls with a human checkpoint — to reach production well before open-ended agents do.

We also expect the interface itself to spread. Schema-described tools are too useful to remain a single-vendor feature; other model providers will follow, and constrained decoding should eventually guarantee schema-valid output instead of merely encouraging it. That would make the validation layer cheaper. The confirmation layer stays. Models propose; systems decide.