Tool calling

What it is #

You describe some functions. The model, instead of answering, may reply that it would like one of them run and with what arguments. You run it, hand back the result, and ask again. It answers — or asks for another.

Nothing about this is Maxlayer-specific. tools, tool_choice and parallel_tool_calls pass to the model unchanged, and responses come back in OpenAI’s shape, so an existing implementation works by changing the base URL.

The loop #

Three steps, shown as raw JSON because the wire shape is the thing worth learning — an SDK's types hide exactly the fields that have to line up.

1. Send the question and the tools #

parameters is a JSON Schema. The description fields are not decoration: they are the only thing the model has to decide whether a tool applies and what to put in it, and vague descriptions are the most common reason a model picks the wrong tool.

{
  "model": "$MAXLAYER_MODEL_ID",
  "messages": [
    { "role": "user", "content": "Why did build 4182 fail?" }
  ],
  "tools": [
    {
      "type": "function",
      "function": {
        "name": "get_build",
        "description": "Fetch a build by id, including its status and error output.",
        "parameters": {
          "type": "object",
          "properties": {
            "id": { "type": "string", "description": "The build id, e.g. 4182" }
          },
          "required": ["id"]
        }
      }
    }
  ]
}

2. Read what it asked for #

finish_reason is tool_calls and content is usually null. Note that arguments is a JSON string, not an object — it needs parsing, and it is generated text, so it can be malformed.

{
  "choices": [
    {
      "finish_reason": "tool_calls",
      "message": {
        "role": "assistant",
        "content": null,
        "tool_calls": [
          {
            "id": "call_a1b2c3",
            "type": "function",
            "function": {
              "name": "get_build",
              "arguments": "{\"id\":\"4182\"}"
            }
          }
        ]
      }
    }
  ]
}

3. Run it and send the result back #

The next request carries the whole conversation: the original question, the assistant turn including its tool_calls, and one tool message per call carrying the tool_call_id it answers. The endpoint is stateless, so anything you leave out is gone.

{
  "model": "$MAXLAYER_MODEL_ID",
  "messages": [
    { "role": "user", "content": "Why did build 4182 fail?" },

    // The assistant turn goes back verbatim, tool_calls and all.
    {
      "role": "assistant",
      "content": null,
      "tool_calls": [
        {
          "id": "call_a1b2c3",
          "type": "function",
          "function": { "name": "get_build", "arguments": "{\"id\":\"4182\"}" }
        }
      ]
    },

    // One tool message per call, carrying the id it answers.
    {
      "role": "tool",
      "tool_call_id": "call_a1b2c3",
      "content": "{\"status\":\"failed\",\"error\":\"npm ERR! missing script: build\"}"
    }
  ],
  "tools": [ /* the same definitions, every time */ ]
}

A working loop #

The same thing as code. It is a loop rather than two requests because the model may ask again after seeing a result — that is the useful part, and the part a two-request implementation quietly truncates.

import OpenAI from "openai";

const client = new OpenAI({
  apiKey: process.env.MAXLAYER_API_KEY,
  baseURL: "https://inference.maxlayer.cloud/v1",
});

const tools = [
  {
    type: "function" as const,
    function: {
      name: "get_build",
      description: "Fetch a build by id, including its status and error output.",
      parameters: {
        type: "object",
        properties: { id: { type: "string" } },
        required: ["id"],
      },
    },
  },
];

// Your side of the contract: what each tool name actually does.
const handlers: Record<string, (args: any) => Promise<unknown>> = {
  get_build: async ({ id }) => fetchBuild(id),
};

const messages: any[] = [
  { role: "user", content: "Why did build 4182 fail?" },
];

// Bounded, not "while (true)". A model that keeps asking for tools is a real
// outcome, and an unbounded loop bills for every turn of it.
for (let turn = 0; turn < 8; turn++) {
  const completion = await client.chat.completions.create({
    model: process.env.MAXLAYER_MODEL_ID!,
    messages,
    tools,
  });

  const message = completion.choices[0]!.message;
  messages.push(message);

  const calls = message.tool_calls ?? [];
  if (calls.length === 0) {
    console.log(message.content);
    break;
  }

  // Every call gets a reply, including the ones that fail — a missing
  // tool_call_id is a malformed conversation on the next request.
  for (const call of calls) {
    let result: unknown;
    try {
      const args = JSON.parse(call.function.arguments || "{}");
      const handler = handlers[call.function.name];
      result = handler
        ? await handler(args)
        : { error: `unknown tool: ${call.function.name}` };
    } catch (cause) {
      result = { error: `could not run tool: ${String(cause)}` };
    }

    messages.push({
      role: "tool",
      tool_call_id: call.id,
      content: JSON.stringify(result),
    });
  }
}

Controlling the choice #

tool_choice decides how much say the model has:

• "auto" — the default, and right nearly always. The model decides whether any tool applies.
• "none" — answer in prose regardless. Useful for a final turn where you want a summary rather than another call.
• "required" — some tool must be called. Useful when a bare answer is never acceptable.
• A named function — { "type": "function", "function": { "name": "get_build" } } forces that one, which turns the model into a parameter extractor for a call you have already decided to make.

parallel_tool_calls: false asks for at most one call per turn, which is worth setting when your tools have side effects and you would rather sequence them yourself.

Parallel calls #

tool_calls is an array, and a model may put several in one turn — three independent lookups it can see it will need. Run them however you like, including concurrently, but the reply rules do not bend:

• Every call gets exactly one tool message.
• Each carries the tool_call_id it answers. Order does not identify them; the id does.
• Failures are results too. A tool that threw still needs a message — send the error as its content. Silence leaves an unanswered call in the history, and the next request is malformed.

Streaming with tools #

Streaming and tools work together, with one wrinkle worth knowing before you debug it: arguments arrive in fragments. Each chunk carries a slice of the JSON string, so nothing can be parsed until the call is whole. Accumulate by index, and act when finish_reason is tool_calls.

const stream = await client.chat.completions.create({
  model: process.env.MAXLAYER_MODEL_ID,
  messages,
  tools,
  stream: true,
});

// Arguments arrive as fragments of a JSON string across many chunks, so
// nothing can be parsed until the call is complete.
const assembled: Record<number, { id: string; name: string; args: string }> = {};

for await (const chunk of stream) {
  for (const delta of chunk.choices[0]?.delta.tool_calls ?? []) {
    const slot = (assembled[delta.index] ??= { id: "", name: "", args: "" });
    slot.id += delta.id ?? "";
    slot.name += delta.function?.name ?? "";
    slot.args += delta.function?.arguments ?? "";
  }

  if (chunk.choices[0]?.finish_reason === "tool_calls") {
    // Now every slot holds a whole call, and JSON.parse is safe.
    for (const call of Object.values(assembled)) run(call);
  }
}

If you are streaming to a user interface, this is why a spinner reading “calling a tool” should key off the first delta rather than off parsed arguments — the name usually completes long before the arguments do.

Which models support it #

The catalogue records a function-calling capability per model, and the marketplace filters by it. Filter first: this is not a feature every model has.

Support is also a matter of degree among models that have it — some are markedly better at choosing between many similar tools, or at filling a deep schema. If a model picks badly, try fewer tools and sharper descriptions before trying a bigger model.

What it costs #

Tool calling multiplies requests, and each is billed in full. Three things follow, and none of them is obvious from a single call working:

• Definitions are input tokens, on every request. They go up with the question and again with each follow-up. A dozen tools with long descriptions is a fixed cost paid per turn, so prune the set you send to what the task could plausibly need.
• The history grows every turn. Each turn resends everything before it, so a five-turn loop costs far more than five times the first request. This is the single biggest surprise on a tool-calling bill.
• Results are input tokens too. Returning a whole API response where three fields would do is paid for on that turn and every turn after it.

Dashboard → AI → Activity shows requests as well as spend, which is the view that makes a runaway loop obvious — a jump in request count with no corresponding jump in what you asked for. See what a call costs for the rest.

Failure modes #

Worth handling before you meet them, since each looks like something else:

• Malformed arguments. It is generated text, so JSON.parse can throw. Catch it and return the error as the tool result — models usually correct themselves on the next turn given a clear message.
• Arguments that parse but are wrong — a missing required field, a string where a number belongs, an id that does not exist. Validate against your own schema rather than trusting the model to have honoured the one you sent.
• A tool name you never defined. Rare, and worth handling as a result rather than a crash.
• A loop that will not end. Bound the turns, and log when the bound is hit — that is a prompt or a tool-description problem, and it is invisible without the log line.
• An unanswered call. Every id in tool_calls needs a tool message before the next request. This one usually appears as a confusing upstream validation error rather than as anything naming the real cause.