---
title: 'ALTK-Evolve: How learned memory tops the τ²-bench leaderboard on customer service tasks without a frontier model'
link: https://huggingface.co/blog/ibm-research/altk-evolve-bc
source: huggingface-co
published: 2026-10-20T15:19:48Z
updated: 2026-10-20T15:19:48Z
first_seen: 2026-10-20T17:55:58.869157456Z
content: extracted
html: 2026-10-20-altk-evolve-how-learned-memory-tops-the-t2-bench.html
remote_preview:
  url: https://cdn-uploads.huggingface.co/production/uploads/6435a1131860001f144239ea/rzXYJh7e7UdoYgCLwOZjH.jpeg
---

*ALTK-Evolve learns and applies a playbook using GPT-4.1, outperforming the retail entries on τ²-bench, a customer service agent benchmark—without updating model weights.*

## Learned memory takes a basic ReAct agent with GPT-4.1 to the top

Using GPT-4.1, an older, non-reasoning model, a basic customer-service agent reaches **67.5% Pass^4 on τ²-bench retail** — meaning it solves the task on *all four* independent attempts for 67.5% of tasks. It scores above every retail entry in our October 12, 2026 leaderboard snapshot. It runs on the benchmark's stock ReAct scaffold, with learned guidelines appended to its system prompt.

**[ALTK-Evolve](https://github.com/AgentToolkit/altk-evolve) builds that playbook using GPT-4.1.** It analyzes the agent's training-task attempts and uses the model to distill successes and failures into short behavioral rules: how to authenticate a customer, confirm a transaction, or handle a return. Those guidelines are then added to the agent's prompt for subsequent conversations. The same model powers both guideline extraction and the agent; no weight updates or human annotation are needed.

Our [previous work](https://huggingface.co/blog/ibm-research/altk-evolve-sldd) demonstrated this approach on coding agents. Here, we test it on customer service, where success requires following domain policies, understanding a customer's request, and executing the right tool actions.

## τ²-bench tests policy-bound customer service

[τ²-bench](https://github.com/sierra-research/tau2-bench) (from Sierra) measures how well an agent handles **customer-service conversations** under real constraints. Each task pairs the agent with an LLM-simulated **customer** who has a goal in mind, a set of **domain tools** (look up an order, issue a refund, rebook a flight), and a written **policy document** the agent must follow. The agent has to draw out the right details, stay within policy, and take the correct — often irreversible — actions; a task counts as solved only when the final database state matches the expected outcome, a strict binary reward. We use its two customer-service domains, **retail** and **airline**.

We run τ²-bench's own implementation of a basic ReAct agent with a two-phase **THINK → ACT** loop, using `gpt-4.1` as the agent model and `gpt-5.5` as the customer-simulator model; **4 trials** per task. **The only change we make** is a block of learned guidelines appended to the system prompt — everything else (policy, tools, scaffold) is stock τ²-bench. Concretely, the learned guidelines are tacked on after the domain policy as a plain markdown list:

```text
You are a customer service agent that helps users according to the policy below.
Always follow the policy strictly. Use the provided tools when needed.

<policy>
# Retail agent policy

As a retail agent, you can help users:
- cancel or modify pending orders
- return or exchange delivered orders
- modify their default user address
…
</policy>

## Guidelines learned from past task attempts (follow these carefully):
- Authenticate the user before taking any order-related action. If email is unavailable, verify identity via full name and zip code using the find_user_id_by_name_zip API.
- Always confirm all intended changes (modify, cancel, return, exchange) with a clear, structured summary, and require explicit user approval before executing any database-updating action.
- When processing a return or exchange, communicate the refund with an itemized breakdown and clarify refund timing — don't wait for the user to ask.
- Take one action per step: never combine a tool call with a user-facing message in the same turn, and wait for the tool's result before replying.
- Before looking a user up, parse and clean their name — e.g. split a combined handle like `daiki_johnson_9523` into first/last name — and after a couple of failed lookups, state the exact format you need.
…
```

We extract guidelines from the agent's trajectories on τ²-bench **train** tasks using **`gpt-4.1`**, the same model that powers the agent, then apply them unchanged on the held-out **test** set.

## Guidelines improve capability and reliability

In a [prior post](https://huggingface.co/blog/ibm-research/altk-evolve-hmm), we talked about how different models / tasks require different dosage of guidance. Inline with the idea, we also evaluate different amounts of guidelines provided to the model.

Each guideline carries a **support** count — the number of distinct training tasks it was extracted from, so higher support means a rule that generalized across more tasks rather than a one-off. We compare three configurations: **baseline** (no guidelines), **sup3** (keep only guidelines with support ≥ 3 — a small, high-confidence core), and **lossless** (keep every guideline retained after consolidation, i.e. support ≥ 1).

**Table 1. Guideline configurations and retained guideline counts by domain.**

| Variant      | Guidelines included in context      | Retail | Airline |
| ------------ | ----------------------------------- | ------ | ------- |
| **baseline** | no guidelines                       | 0      | 0       |
| **sup3**     | support `n ≥ 3` (high-support core) | 31     | 14      |
| **lossless** | all guidelines                      | 292    | 101     |

**The metrics we track.** Every task is attempted **4 times**, and we read those trials two ways:

- **Mean Accuracy** — the average success rate across the 4 trials. This is the **capability** axis: *how often does the agent get this right?*
- **Pass^4** — the percentage of tasks that passed across all 4 trials. This measures along the **reliability** axis and the metric the τ²-bench leaderboard reports. It matters here because a support agent that resolves a refund 3 times out of 4 still fails a quarter of real customers — Pass^4 asks whether you can trust the agent *every* time, not just on average.

[![Mean Accuracy and Pass^4 on the held-out test tasks, by variant, for Retail and Airline. Lossless lifts retail accuracy +4.4 pp and Pass^4 +10.0 pp; airline accuracy rises +7.5 pp with lossless while sup3 gives the best Pass^4 at +10.0 pp.](https://cdn-uploads.huggingface.co/production/uploads/6435a1131860001f144239ea/HtL2uTn98-xHkcTxKqkFY.png)](https://cdn-uploads.huggingface.co/production/uploads/6435a1131860001f144239ea/HtL2uTn98-xHkcTxKqkFY.png)

*Figure 1. Mean Accuracy and Pass^4 on held-out retail and airline tasks for agents using `gpt-4.1` across four trials per task.*

*Agent model: `gpt-4.1` (user simulator: `gpt-5.5`).*

**Table 2. Held-out agent performance using `gpt-4.1` across four trials per task. Parentheses show percentage-point changes from baseline.**

| Domain (test tasks) | Config       | Mean Accuracy    | Pass^4            |
| ------------------- | ------------ | ---------------- | ----------------- |
| **Retail** (40)     | baseline     | 78.1%            | 57.5%             |
|                     | sup3         | 78.8% (+0.7)     | 60.0% (+2.5)      |
|                     | **lossless** | **82.5% (+4.4)** | **67.5% (+10.0)** |
| **Airline** (20)    | baseline     | 57.5%            | 35.0%             |
|                     | sup3         | 60.0% (+2.5)     | **45.0% (+10.0)** |
|                     | **lossless** | **65.0% (+7.5)** | 40.0% (+5.0)      |

Mean Accuracy climbs in both domains — retail **+4.4 pts** (78.1 → 82.5) and airline **+7.5 pts** (57.5 → 65.0). Reliability moves further: the strict **Pass^4** rises **+10 pts** in retail (57.5 → 67.5) and **+10 pts** in airline (35.0 → 45.0). And the gains come from better-informed calls, not longer conversations — we look at conversation length directly in [Better performance costs more tokens, not more turns](https://huggingface.co/blog/ibm-research/altk-evolve-bc#better-performance-costs-more-tokens-not-more-turns) below.

### The best amount of guidance depends on the domain

The two domains disagree about *how much* guidance pays off. In **retail**, more is better on every metric — the small sup3 core barely beats baseline on accuracy, and you need the **full lossless set** for the real jump. In **airline** the picture flips at the reliability end: lossless leads on accuracy (65% vs 60%), but the compact **sup3 core (14 guidelines) is the more *reliable*** one, taking Pass^4 to **45%** against lossless's 40%. Echoing the earlier finding that [delivery is a dial, not a constant](https://huggingface.co/blog/ibm-research/altk-evolve-hmm), **support count is a knob worth tuning per domain**, not a value to hard-code.

## Better performance costs more tokens, not more turns

Guidelines aren't free, but the cost lands where you'd expect — in **prompt input tokens**, not in a longer, more meandering conversation:

[![Cost per conversation grouped by Retail and Airline. Lossless increases cost by 65% and 32%, respectively, over the no-guideline baseline.](https://cdn-uploads.huggingface.co/production/uploads/6435a1131860001f144239ea/MsqhICXsHtpJozVFZYEm7.png)](https://cdn-uploads.huggingface.co/production/uploads/6435a1131860001f144239ea/MsqhICXsHtpJozVFZYEm7.png)

*Figure 2. Total LLM cost per conversation by domain and guideline configuration on the full base split.*

**Table 3. Total LLM cost per conversation (agent and user simulator) on the full base split across four trials. Lower is better.**

| Domain      | Config       | Cost / conversation | vs. baseline |
| ----------- | ------------ | ------------------- | ------------ |
| **Retail**  | baseline     | $0.084              | —            |
|             | sup3         | $0.095              | +12%         |
|             | **lossless** | $0.139              | +65%         |
| **Airline** | baseline     | $0.081              | —            |
|             | sup3         | $0.097              | +19%         |
|             | **lossless** | $0.107              | +32%         |

Does the agent get there by *talking longer*? To answer that cleanly we compare **like for like**: the task runs solved under **both** baseline and the guideline variant — identical scenario and seed, a true "same task, done right" set — and count assistant turns on each side.

**Table 4. Assistant turns per conversation for matched runs solved under both settings on the test split, pooling four trials.**

| Domain      | Shared set              | Solved under both (n) | Baseline turns | +Guidelines | Δ         |
| ----------- | ----------------------- | --------------------- | -------------- | ----------- | --------- |
| **Retail**  | baseline ∩ sup3         | 108                   | 10.40          | 10.35       | −0.05     |
|             | baseline ∩ **lossless** | 111                   | 10.59          | 10.16       | **−0.43** |
| **Airline** | baseline ∩ sup3         | 38                    | 8.97           | 8.53        | **−0.45** |
|             | baseline ∩ **lossless** | 40                    | 9.15           | 8.82        | −0.33     |

The agent isn't talking longer — on the shared set, guidelines make conversations **slightly shorter**: lossless trims ~0.3–0.4 turns in both domains (retail 10.59 → 10.16, airline 9.15 → 8.82), and no variant lengthens them. That matters beyond cost: each turn is a sequential LLM round-trip, so fewer turns means **lower end-to-end latency — a faster time-to-completion** for the customer, not just a cheaper one. The only real price is on the input side — the larger system prompt riding on every LLM call: retail's 292-guideline prompt pushes cost from ~$0.084 to ~$0.139 per conversation, while airline's smaller playbook is cheaper. And in production even that is largely reclaimable with **prompt caching**, since the guideline block is identical across every call.

So the trade is a favorable one: for a **slightly higher input cost** (much of it cached away in practice) the agent lands **higher accuracy, higher reliability, *and* lower time-to-completion** at once — not the usual quality-for-latency tradeoff, but all three moving the right way together.

## Learned guidelines also improve agents using frontier models

So far, we have used an older, non-reasoning model — `gpt-4.1`. A fair objection is that learned guidelines might have less to add to an agent using a *frontier* model: maybe they only patch the gaps of a weaker model. So we tested it directly — we reran **airline** using **`gpt-5.5` for both the agent and guideline extraction**, on the same τ²-bench ReAct scaffold, with guidelines re-mined from the agent's *own* runs. One detail changes with the model: none of the guidelines extracted in this setup reached support ≥ 3 (they top out at support 2), so the high-support core here is **sup2** rather than sup3.

**Table 5. Held-out airline performance for the `gpt-5.5` agent across four trials per task.**

| Airline · `gpt-5.5` (20 held-out test tasks) | Accuracy          | Pass^4            |
| -------------------------------------------- | ----------------- | ----------------- |
| baseline (0 guidelines)                      | 72.5%             | 55.0%             |
| sup2 (103 guidelines)                        | 80.0% (+7.5)      | 70.0% (+15.0)     |
| **lossless (146 guidelines)**                | **82.5% (+10.0)** | **70.0% (+15.0)** |

The learned guidelines improve this setup too. On the same held-out test tasks, appending them to the prompt lifts **accuracy +10 pts** (72.5 → 82.5) and the strict **Pass^4 by +15 pts** (55.0 → 70.0) — a *larger* reliability gain than we observed with `gpt-4.1` on these very tasks (+10 pts), and from a much higher floor. Learning guidelines *around* the model can improve an agent even when it starts from a stronger base. And where the baseline's reliability swung between reruns, the lossless configuration reproduced exactly — the gain it buys is dependable, not a lucky draw.

## ALTK-Evolve tops the leaderboard comparison

Submissions are scored on the **full base split** (train ∪ test), as τ²-bench requires, and reported at **Pass^4**. Because that covers the full task set rather than the held-out test split, these figures differ from the test-task numbers reported above. This is where the practical upshot shows up — prompt-level memory carries agents using each base model *into*, or *past*, current frontier company:

- **[Retail: #1](https://github.com/sierra-research/tau2-bench/pull/542).** On the base split the agent with learned guidelines reaches **Pass^4 = 67.5%**, ahead of every frontier model *and* ahead of **RAFT-30B-A3B** (61.4%), a model **fine-tuned on synthetic τ²-bench retail data**. Using stock `gpt-4.1` and a prompt-appended playbook, it outperforms that fine-tuned system by **+6.1 points**.
- **[Airline: #1](https://github.com/sierra-research/tau2-bench/pull/551) using `gpt-5.5`, #8 using `gpt-4.1`** Against the published board, the compact **sup3 core** gives the agent using `gpt-4.1` **Pass^4 = 44%** — **8th**, beating GPT-5.2 (no reasoning) and nearly matching Claude Sonnet 4.5. Swap in the stronger base and the same approach tops the field outright: **the agent using `gpt-5.5` reaches 82.0%**, clearing every frontier entry (66–72%).

[![Saved Pass^4 leaderboard comparison for Retail and Airline, highlighting ALTK-Evolve runs at 67.5% in Retail and 82.0% and 44.0% in Airline.](https://cdn-uploads.huggingface.co/production/uploads/6435a1131860001f144239ea/iOmv6o938CgNfFlULP-VV.png)](https://cdn-uploads.huggingface.co/production/uploads/6435a1131860001f144239ea/iOmv6o938CgNfFlULP-VV.png)

*Figure 3. Pass^4 comparison between ALTK-Evolve runs and the τ²-bench leaderboard snapshot from October 12, 2026.*

Same message in both domains: **learning guidelines *around* the model works across different base models** — bringing an agent using an older, non-reasoning model into frontier company on retail, and taking an agent using a frontier model to the top on airline.

## Agents can improve without weight updates

Fine-tuning a model on a benchmark is expensive, brittle, and doesn't transfer to the next domain. Here, ALTK-Evolve improves a **stock agent on a stock model** in both domains — and matches or beats a fine-tuned system on retail — using only **prompt-level guidelines extracted from the agent's own runs**, automatically, without human annotation, and without any test-set leakage.

* * *

### Links:

- **Prior posts**
  1. [Thinking of ACE? We Can Do It with Fewer Tokens](https://huggingface.co/blog/ibm-research/altk-evolve-sldd)
  2. [How much memory does your agent actually need?](https://huggingface.co/blog/ibm-research/altk-evolve-hmm)
- **ALTK-Evolve library** — [https://github.com/AgentToolkit/altk-evolve](https://github.com/AgentToolkit/altk-evolve)
- **τ²-bench** — [https://github.com/sierra-research/tau2-bench](https://github.com/sierra-research/tau2-bench)
- **Retail leaderboard submission** — [τ²-bench PR #542](https://github.com/sierra-research/tau2-bench/pull/542)
- **Airline leaderboard submission** - [τ²-bench PR #551](https://github.com/sierra-research/tau2-bench/pull/551)