A Small Model and a Stack of Rules: Our Run at STATE-Bench

We entered Microsoft's STATE-Bench Agent Learning Track with a single small model, gpt-5.4-nano, and a frozen rulebook of plain if/then rules, mined in code from the training conversations the benchmark allows. Two submissions, one whose design was informed by looking at the test results and one that never saw the test set, reach 0.64 and 0.62 pass@1 on v0.8.0, above the verified entries on that version. Both are our own runs on the locked protocol, and both are submitted and currently under verification by Microsoft. What interests us is less the ranking than the shape of the system: a benchmark built for large agent architectures can be met by something small that you can read with your own eyes.

Before we describe what we built, it is worth saying plainly what STATE-Bench tests. A simulated customer — a language model playing a person who is sometimes patient and sometimes difficult — wants to cancel an order, match a price, rebook a flight, or return a pair of shoes. The agent under test works with a set of tools for looking up the order, checking the policy, and issuing the refund. It has to listen, gather the facts, propose the right action, and only then act. When the conversation ends, a judge model reads the whole transcript and asks two questions: did the right things actually change in the database (the state requirements), and did the agent do what the user needed (the task requirements). It also rates the interaction experience. The benchmark covers three domains — customer support, travel, and shopping — with fifty held-out tasks each, run five times over and scored by GPT-5.4 acting as both the user and the judge.

That sounds easier than it is. The tasks are built to punish the agent that acts too soon, discloses the wrong number, skips the policy gate, or confuses one customer's order with another's.

STATE-Bench was designed with large, complex agent architectures in mind: systems with neural memory stores, planning loops, and the machinery that lets an agent carry experience forward across tasks. The Agent Learning Track rewards exactly that kind of learning and remembering.

We went the other way and built something small on purpose, which we called gaia-nano.

The whole system runs on a single compact model, gpt-5.4-nano, with nothing larger anywhere in the pipeline. There is no second model doing the planning behind the curtain. More importantly, the learning phase uses no language model at all: we read the allowed training conversations and mine them deterministically, in code rather than by prompting, into plain human-readable rules — explicit if/then preconditions, the preview-then-confirm discipline, the order in which to consult the tools, and the policy branches the environment itself requires. At test time, the small agent carries that frozen rulebook and nothing more, and a thin retrieval hook hands it the three most relevant pages for whatever the customer just said.

No ground-truth labels ever touch the agent or the learner, neither from the training set nor from the test set. That boundary is enforced in code and machine-checked: the training examples our rules lean on have zero overlap with the test tasks. We wrote the proof down and shipped it alongside the results, so anyone can verify it.

We submitted two versions, and the difference between them is a matter of honesty rather than cleverness:

• gaia-nano is our stronger run. It is label-free, but we will not pretend it never saw the test set. An earlier iteration was run against the test tasks, our engineers read those transcripts, and from reading them we refined the prompt and conduct-rail wording. No labels and no test fields ever entered the machine. Still, the design was informed by looking, and we say so plainly.
• gaia-nano-v3 is the clean one. Its rules were authored from a held-out slice of the training data only, frozen before any evaluation, any judge, and any simulator call, and never touched again. It never saw the test set in any way. It is the honest baseline: what a small model achieves purely from reading allowed demonstrations.

Five runs per task, fifty tasks per domain, judged by the locked GPT-5.4 evaluator. pass@1 is how often a single attempt passes, averaged across the five runs. pass^5 is the strict measure: all five attempts on a task have to pass without a single slip, which measures reliability. UX is the judge's rating of the interaction on a one-to-five scale.

Per domain, for the curious:

Read those numbers soberly. The agent solves roughly six in ten tasks on a single try, and it holds steady across all five tries on roughly four in ten. The most revealing line is the gap that is barely a gap: the clean, strictly blind version trails the design-informed version by less than two points on average, and on travel it actually edges ahead. Looking bought us very little; honesty cost us almost nothing.

STATE-Bench keeps a public leaderboard for the Agent Learning Track, and only entries evaluated on the same release, v0.8.0, make a fair comparison. Here are the verified Agent Learning Track entries on v0.8.0 alongside ours:

On the same benchmark version, both of our small-model runs land well above a full-size GPT-5.4 paired with Microsoft's dedicated Foundry Memory system, and above the no-memory GPT-5.4 baseline, at a fraction of the model size and inference cost. For wider context, the strongest full-size model on the separate Main Track, GPT-5.5 at high reasoning, sits at 58.9% pass@1 on v0.8.0.

Why does a small model carrying a plain-text rulebook beat a frontier model paired with neural memory? Because vector memory blurs hard conditional orderings — such as checking return eligibility and getting explicit customer confirmation before triggering a refund — whereas deterministic rules preserve the exact conditions under which an action is valid.

Even so, the consistency column keeps us honest. A pass^5 near 39% means that on roughly six tasks in ten, the model still slips on at least one of its five tries when rules are only suggested inside a prompt. Closing that remaining gap means enforcing hard policy and tool-order boundaries outside the model rather than hoping the model remembers to obey them every time — a direction we explore further in Note #07.

• Full submissions, scored trajectories, and compliance documents (GitHub)
• Our official submission thread (microsoft/STATE-Bench · issue #51)
• The STATE-Bench repository (microsoft/STATE-Bench)
• STATE-Bench Agent Learning Track Leaderboard