Learned guidelines for agents: ALTK-Evolve's numbers, read closely
On this page
Rules that an agent learns from its own past runs can make an older model more reliable. IBM Research reports that a basic GPT-4.1 agent ranks first on the retail part of a customer-service benchmark. It gets there with this method and no weight updates. I read the post, the mining and retrieval code, and the status of the two leaderboard pull requests. The method is worth testing on my own agents. The ranking, though, rests on small test sets and on submissions that were still open when I read them.
What the post claims
The benchmark is τ²-bench, a customer-service test from Sierra. The agent must follow a written policy and change a database correctly. A task passes only when the final database state is right.
The agent is a basic ReAct agent. ReAct means the model alternates between a short written thought and a tool call. The one change is a list of guidelines added to the system prompt, which is the fixed instruction text sent before each conversation. The guidelines come from the agent’s own runs on training tasks, and the same model writes them.
The post reports these results on held-out tasks. Held-out tasks are tasks the rules were not learned from. Each task ran four times.
| Setup | Tasks | Accuracy | Pass^4 |
|---|---|---|---|
| Retail, no guidelines | 40 | 78.1% | 57.5% |
| Retail, 31 high-support guidelines | 40 | 78.8% | 60.0% |
| Retail, all 292 guidelines | 40 | 82.5% | 67.5% |
| Airline, no guidelines | 20 | 57.5% | 35.0% |
| Airline, 14 high-support guidelines | 20 | 60.0% | 45.0% |
| Airline, all 101 guidelines | 20 | 65.0% | 40.0% |
Pass^4 is the share of tasks that pass in all four runs. I think it is the right number to watch. A refund that works three times in four still fails one customer in four.
With gpt-5.5 as the agent, the airline gain is larger. Accuracy goes from 72.5% to 82.5%, and Pass^4 from 55.0% to 70.0%, with 146 guidelines.
Cost is the other side. In retail, the full list raises the cost per conversation from $0.084 to $0.139, about 65% more. The post says prompt caching, which reuses the same prompt text across calls at a lower price, can recover much of that extra input cost.
How it works
The code is in the AgentToolkit/altk-evolve repository. It is Python 3.12 or later, under the Apache-2.0 license, a permissive open-source license that allows commercial use.
A trajectory is the full record of one run: the messages, the tool calls and the results. The mining step is in altk_evolve/llm/guidelines/guidelines.py. A model reads one trajectory and writes short rules from it. The prompt changes with the outcome. A successful run and a failed run get different instructions. The code also caps how many rules each run can produce. Splitting a long run into sub-tasks is off by default.
Each rule keeps a support count. That is the number of distinct tasks it came from. A rule with high support came up again and again, so it is more likely to apply to new tasks. The default threshold for a core rule is 3, set in altk_evolve/config/evolve.py.
The repo has two ways to use the stored rules:
- A flat playbook. Every rule above a minimum support goes into the prompt. The τ² runs in the post used this.
- A retrieval index, in
altk_evolve/llm/guidelines/retrieval.py. The core rules always go in. Then up totop_kmore rules go in, wheretop_kis a fixed limit. These extra rules come from the past tasks most similar to the new one.
I think the retrieval path is the better design for a long rule list. The post does not test it.
What the numbers show
The test sets are small. Retail has 40 tasks and airline has 20. One task moves Pass^4 by 2.5 points in retail and by 5 points in airline. So the airline gap between the 14-rule core (45%) and the full list (40%) is one task. I would not rank those two on that gap.
The retail gain, from 57.5% to 67.5%, is four tasks. That is a real signal, but it comes from one run over 40 tasks. The post gives no confidence intervals. It also has no control. It never tests a block of generic rules of the same length. Part of the gain may come from any longer, more careful prompt.
The leaderboard numbers need a closer look. The post says its held-out results have no test-set leakage. The leaderboard, though, scores the full base split, which is training plus test tasks. The post says that is how τ²-bench requires submissions to be scored. It does not say which tasks the leaderboard runs learned their rules from. If they learned from the full split, the score counts tasks the agent learned from. That is a different test from the held-out one.
Both leaderboard entries are pull requests: PR #542 for retail and PR #551 for airline. When I read them, both were still open. So “ranks first” means “is in open submissions” on the day I checked.
What the repo shows
The repo has code for mining, consolidating and exporting rules. It has a tau-retail adapter that reads τ-bench run files. An adapter turns one dataset format into the records the miner reads. By default the adapter skips tasks marked as test, and it accepts a split file. That keeps held-out tasks out of mining, which is the right default.
The repo has no harness that computes Pass^4, and no τ² run files. So I can read how mining works, but I cannot reproduce the table from it.
Two more details matter in a real setup. The default storage backend uses simple text matching, and semantic search needs an extra install. The top-k path depends on similarity, so I would check retrieval quality before I trusted it. The README also says the local REST API has no authentication. Any local process can read and change the stored rules. For an agent that writes its own rules, that is a real risk, because a bad rule goes straight into the prompt.
My take
I would try this pattern before I would fine-tune a model. Prompt rules are cheap to add, easy to measure and easy to remove. For an agent with a clear pass or fail check, it is a sensible first step.
I would not adopt this on the leaderboard result alone.
In practice I would start with the smaller core and add rules only when the extra gain pays for the extra tokens. I would also review each rule before it goes into a production prompt.
What I’d test
I would run this on one of my own agent workflows, not on τ²-bench. The repo ships adapters for τ-bench retail runs, and for AppWorld and CUGA runs. AppWorld is a benchmark of agent tasks in simulated apps, and CUGA is an agent whose AppWorld runs the adapter reads. For my own project I would write one more. The plan:
- Pick one agent task with 40 cases and an automatic pass or fail check.
- Split them into 20 training cases and 20 held-out cases before any mining.
- Run the baseline four times per case with one model. Record Pass^4, mean accuracy, cost per case and turns, meaning model calls per conversation.
- Mine rules from the training runs with the repo’s
minecommand. Export a flat playbook and a retrieval index. - Run four arms on the held-out cases: baseline, flat playbook, retrieval index, and a same-length block of generic rules I write.
- Compare the arms in cases, not percentages.
Thanks for reading, Quang