The Scaling Trap: Why Agentic Systems Need Small Language Models

Why one frontier model running every step of an agent loop breaks on cost and reliability, and how to build a heterogeneous system where a large model plans and small language models execute.

TL;DR

  • Stop paying frontier-model prices for repetitive, low-reasoning execution steps.
  • Let a large model plan and small language models (SLMs) execute.
  • Use grammar-constrained decoding so SLMs cannot emit invalid structured output.
  • Distill the large model's reasoning into narrow specialists, then prove them on real traffic.

The Economic Cliff of Agentic Loops

If you assume token costs scale linearly with user growth, you will underbudget by a wide margin. Most cost models were built for chat: one question, one answer. An agent does not work that way.

Volume is the first problem. According to Gartner's March 2026 analysis, agentic models require between 5 and 30 times more tokens per task than a standard chatbot. A single agentic task, with its tool calls, retries and context reloads, can consume 10,000 to 50,000 tokens, or $0.10 to $0.50 per request.

Waste is the second. Research from the Stanford Digital Economy Lab found that re-sent context accounts for 62% of total agent inference bills. Each loop iteration or tool call tends to resend the conversation history and the system prompt, so you pay again for tokens the model has already read. Loops, retries and context reloads multiply costs 3 to 7 times before any optimization.

Now put one frontier model behind every step of that loop. You pay frontier prices for the hard planning step and for the hundred trivial formatting steps after it. That is an architecture problem that looks like a scaling problem, and more hardware will not fix it. Gartner forecasts that 40% of AI agent projects will be cancelled by 2027 due to cost overruns alone.

The Mathematical Reality of Error Propagation

A 95% accurate step sounds good enough. Chain two routing hops at 95% each and you get roughly 90% end to end (0.95 × 0.95 = 0.9025). Ten such steps land near 60%. Every step you add multiplies in its own failure rate, and an error made early in a multi-step plan compounds into a failure later on. If the planner misreads a user constraint at step one, every agent after it works hard on the wrong problem.

Adding more agents without structure makes this worse. Kim et al. report up to 17.2× error amplification in poorly coordinated networks, while centralised coordination contains this to about 4.4×. On strictly sequential planning tasks they found every multi-agent variant degraded performance, because coordination overhead consumed budget without providing a real parallel advantage.

The fix is a topology rather than a head count. A manager with a team limits chatter and contains failures. A flat swarm lets noise grow faster than capability.

The Heterogeneous Architecture: Planning vs. Execution

The fix for both cost and reliability is to stop treating one large model as the brain for every micro-task.

The Hierarchical Split

Most of what an agent does is not hard. The majority of agentic subtasks in deployed agentic systems are repetitive, scoped and non-conversational: extract these fields, call this tool with these arguments, classify this request. That work needs a model that is cheap and predictable.

So give it one. A large model sits at the top and does what only it does well: decompose the goal, choose the order of work, and decide when a plan has failed. Below it, SLMs do the execution. An SLM fine-tuned to a single output format is preferable to a general-purpose LLM for this kind of step. The cost gap is large: serving a 7B SLM is 10 to 30 times cheaper in latency, energy and FLOPs than a 70 to 175B LLM.

The split also buys you control. An executor with one job can get narrow tool permissions and strict output validation, which you cannot give a model that does everything.

The Planner-Executor-Evaluator Loop

In practice I run the hierarchy as three roles. The planner turns the objective into a sequence of concrete steps. SLM executors carry out each step. An evaluator checks each result against what the plan expected and, on a mismatch, sends the work back to the planner instead of passing it on. That check stops one bad step multiplying through the rest of the chain.

When a Large Model Still Earns Its Place

SLM-first is a default, and there are four cases where it loses.

  • Open-ended reasoning and dialogue. Where general-purpose conversational ability is essential, keep an LLM in the system: SLMs by default, with the LLM called only where it is needed.
  • Planning a task the system has never seen. Decomposing a novel goal is the general reasoning SLMs are weakest at, so that is the planner's seat.
  • Low volume. Fine-tuning, evaluating and hosting a specialist is fixed work. If a step runs a few hundred times a day, the frontier API is probably cheaper than your engineering time.
  • Serving economics. It is harder to keep an expert SLM endpoint fully utilized and load balanced than a generalist LLM endpoint. Ten specialists at low utilization can cost more than one busy shared model.

A useful rule: if you can write a schema for a step's output and collect a few hundred real examples of it, it is an SLM candidate. If you cannot describe the output in advance, leave it with the large model.

Engineering High-Performance Specialists

You do not train a specialist from scratch. Knowledge distillation trains a small student model on knowledge from a large teacher. The usual workflow has two steps: generate synthetic data from the teacher with an elaborate, task-specific prompt, then train the student with a short one.

Distilling Step-by-Step

Do not train only on final answers. In distilling step-by-step, you ask the teacher for its answer and the rationale behind it, using Chain-of-Thought (CoT) prompting, and train the student to produce both. The result: a 770M parameter T5 model outperformed the few-shot prompted 540B PaLM model using only 80% of the examples in a benchmark dataset. The caveat: distillation works best when the student has a specific job. Do not expect the student to generalize past it.

Rapid Iteration

Parameter-efficient fine-tuning such as LoRA takes a few GPU-hours for an SLM, so you can add, fix or specialize a behavior overnight rather than over weeks.

Solving the Reliability Gap with Constrained Decoding

Prompting an SLM to "always return JSON" is a losing game. Larger models follow format instructions more reliably, but even they fail occasionally. In a compound system, agents pass structured data between pipeline stages, and invalid output from one agent breaks the entire workflow.

Specialization helps. An LLM might occasionally drift into malformed output where an SLM trained on a single format will not, because it has never seen any other output format. To close the remaining gap, use grammar-constrained decoding. It enforces validity at the token level during sampling, before there is any output to clean up: invalid tokens have their logits set to negative infinity, so the sampler cannot select them. The model still decides what data to generate, but the structure is guaranteed, which is what tool calling needs: the arguments must be valid JSON matching the tool's parameter schema.

The mechanism in miniature: a toy model that prefers chatty prose, and a schema whose retry_count must be 0 to 3.

import json
import math
 
# A toy vocabulary and a toy "model" whose logits favour chatty prose.
VOCAB = ['Sure', '! Here it is: ', '{"command": "', 'cleanup', 'restart',
         '", "retry_count": ', '0', '1', '2', '3', '7', '}', '<eos>']
PREFERENCES = {'Sure': 5.0, '! Here it is: ': 4.0, '7': 3.0, 'restart': 1.0}
 
def model_logits(prefix):
    # A real model conditions on the prefix; this one just has fixed tastes.
    return [PREFERENCES.get(tok, 0.5) for tok in VOCAB]
 
# The schema, compiled to the set of outputs it allows: command is one of two
# tool names, retry_count is an integer from 0 to 3 (so "7" is out).
VALID = {f'{{"command": "{cmd}", "retry_count": {n}}}'
         for cmd in ('cleanup', 'restart') for n in '0123'}
 
def allowed(prefix, tok):
    if tok == '<eos>':
        return prefix in VALID
    return any(v.startswith(prefix + tok) for v in VALID)
 
def decode(constrained, max_steps=8):
    out = ''
    for _ in range(max_steps):
        logits = model_logits(out)
        if constrained:
            # The whole trick: invalid tokens get -inf before sampling.
            logits = [l if allowed(out, t) else -math.inf
                      for l, t in zip(logits, VOCAB)]
        tok = VOCAB[max(range(len(VOCAB)), key=logits.__getitem__)]
        if tok == '<eos>':
            break
        out += tok
    return out
 
print('free:       ', decode(constrained=False)[:40])
text = decode(constrained=True)
print('constrained:', text)
print('parsed:     ', json.loads(text))
free:        SureSureSureSureSureSureSureSure
constrained: {"command": "restart", "retry_count": 0}
parsed:      {'command': 'restart', 'retry_count': 0}

The model still picks restart because it prefers it, and it would have picked 7 retries if the mask allowed it. Production engines compile the schema into a grammar instead of enumerating outputs, but the masking step is the same. With the grammar in place a small model's structured output is always valid, so a step that would otherwise need a large model for reliability can go to the SLM.

The Engineering Roadmap to Production

Do not start by fine-tuning an SLM and hoping. Earn each switch with data.

Phase 1: Prototype and Generate Data

Start with a frontier LLM and define the task through iteration. That gives you a baseline before you have mapped the full input distribution. Once the workflow is stable, use the LLM to generate synthetic training data, with CoT so the student learns the reasoning as well as the answers.

Phase 2: Add a Routing Layer

With specialists trained, you need to send traffic to them. An LLM router is flexible but adds cost and latency. Semantic routing, which embeds each prompt and matches it by similarity, is cheaper. For high volume, a common 2026 pattern is a fine-tuned small model, such as Llama 3.2 3B or Phi-4, as a classifier that routes each request to one of N task categories.

Phase 3: Evaluate, Then Harden

Before a specialist takes traffic, build a test set of 200 to 500 examples from real production traffic and run both the frontier model and the SLM on it with the same evaluators. Switch only the steps where the SLM holds up.

Then harden the routing. Start with autonomous routing, log every supervisor decision and outcome, and promote the dominant pathways into hardcoded workflow edges. Each promoted path is one fewer probabilistic hop in the chain, so the same multiplication that eroded accuracy across ten steps now works in your favour.

Sources

aiarchitecturellmslmengineering

All writing