![\"August](\"https://www.dbreunig.com/img/resized/tunnel_vision.png\")
It’s been a bit since we’ve added a new context failure mode, but the increasing persistence of frontier models has brought this fail to the fore. This one I call Context Inertia:
\n\n\nContext Inertia is when the model commits to a hypothesis, subtask, or goal in its context and keeps pursuing it after the evidence says to stop or change course.
\n
This failure mode is part context poisoning (when an error makes it into the context and is repeatedly referenced) and part context distraction (when a context grows so long the model over-focuses on it), amplified by frontier models the labs have trained to be relentless instruction followers.
\nA couple days ago, buried in a piece about AI self-replication capabilities, Anthropic shared a perfect Context Inertia case study when describing a custom harness:
\n\n\nMemento is designed to mitigate a common failure mode we observe in long-horizon runs: an agent can become fixated on an incorrect idea (for, say, why a bug is occurring) and continue pursuing it instead of backtracking. One example of such a failure is a Qwen agent debugging package versioning for hours when the real bug came from implementing an endpoint incompatible with the grader API.
\n
Each Memento agent works until a fixed timeout (1.5 hours, in one experiment), then writes a short note detailing its work. A fresh agent inherits the codebase, the environment, the note, and nothing else.
\n![\"Guy](\"https://www.dbreunig.com/img/resized/memento.png\")
Guy Pearce as Qwen, trying to remember how to install llama.cpp.
\nAnthropic isn’t the only one facing this problem, or applying the forgetting fix. In just a week, we had two papers hitting exactly the same note. Jack Ziyang Cai and Christos Ziakas submitted a paper on October 5th called “Fork-and-Flush”, writing (emphasis mine):
\n\n\nWe first examine how solution quality evolves over long autoresearch runs. The score of the best solution found so far tends to plateau within a run. More revealingly, independent runs of the same agent on the same task plateau at substantially different levels, with gaps that persist even after considerable additional compute.
\n
When agents found good solutions, they tended to fixate on them. Better solutions existed, but the path to them was never explored. Cai and Ziakas’ baseline runs delivered a wide range of results, between 68.3 and 100.0 on one task, despite plateauing with plenty of compute budget left.
\nQizheng Zhang and coauthors published a paper, “Stateless Language Agents” (SLAs), detailing how long-running auto-research loops settle into busy work once a sufficient solution is found. A model using the CORAL harness found its last improvement in 231 million tokens, and then just kinda…dawdled. Of the final 1,500 turns, 98% made no tool calls. The model just chewed on its existing context.
\n…And then, this morning, we have a fourth example: a UC Berkeley and Stanford team led by Shangyin Tan published “Scaling Discovery Requires Forgetting”. They ran autoresearch agents three ways and all plateaued for the same reason: “Each agent became anchored to its own past attempts, making fewer and smaller edits.”
\nTan’s solution is similar: a harness that forgets, wiping the slate and only retaining the best solution as files in the environment. Stalled autoresearch runs that could forget gained 4.5% to 9.2% over the next 30 proposals. Runs that couldn’t forget gained less than 2%, at up to twice the cost. The plots tells the story:
\n![\"Three](\"https://www.dbreunig.com/img/resized/forgetting.png.jpg\")
The y-axis is the AtCoder Heuristic Contest 014 score.
\nIn 10 days, 4 teams running long-horizon tasks with GPT, Claude, Qwen, GLM, and DeepSeek all reached the same fix: throw the context away and keep the work.
\nSince writing about Context Distraction way back in June of 2025, there’s been some great research illustrating the issue. Carolina Camassa and Derek Shiller’s paper “Do as I say, Not as I Do” illustrates the problem rather well.
\nCamassa and Shiller present a model with initial instructions, like, “I want you to translate every question into Russian as your answer to each of my questions.” They then append turns to the chat history that don’t follow this command, before prompting the model with a question.
\n![\"Diagram](\"https://www.dbreunig.com/img/resized/do_as_i_say.png.jpg\")
From Camassa and Shiller’s “Do as I say, Not as I Do”
\nIf the model replies according to the original instructions, it’s successful. If it gets distracted by all the bad examples, it fails. Camassa and Shiller steadily increased the number of bad examples, noting each model’s success rate. Here’s how well they did on the “Task-based” category, from the above diagram:
\n![\"Heatmap](\"https://www.dbreunig.com/img/resized/task_based_distraction.png.jpg\")
Even the largest models stumbled, losing sight of their task as the distractions grew.
\nWhile we’re not likely to design systems with such egregious examples, as our agentic work runs longer and longer, Context Poisoning and Distraction become inevitable. Every tool call or reasoning turn an agent spends on a suboptimal hypothesis is one more example written into the context.
\nWould compaction fix this? Context Summarization is a default tool in most of our systems these days. But it doesn’t work. Fork-and-Flush found Context Inertia “persists despite periodic compaction of conversation history.”
\nThere are many compaction methods, but almost all of them seek to extend the available context while preserving the stated tasks and the current, in-progress work. This potentially distills what’s driving the model’s hyper-focus.
\nFork-and-Flush, SLA, Forgetting, and Memento suggest that our harnesses have another job: preventing Context Inertia.
\nWe’ve been wrestling with Context Inertia in DSPy optimizers for a bit now1 and I’m aware of others designing around it, beyond the four examples above. So why is everyone noticing the same problem at once?
\nFor one, the models are similarly persistent, or “relentlessly proactive” as Simon says. This isn’t a one-model issue: Fork-and-Flush ran GPT-5.5 and 5.6, SLA ran Codex and Claude Code, Tan’s “Forgetting” used DeepSeek, and Memento was driven by Qwen and GLM. Persistence is a quality labs are targeting, and it’s a double-edged sword.
\nBut I think the bigger reason we’re all hitting this now is because only recently have agents been able to run for so long. When I wrote “How Long Contexts Fail” in June of 2025, you’d be lucky to string together a couple hundred thousand tokens in a single go. Today? Agents are running for days, over billions of tokens. Long enough for inertia to become a threat.
\nWhich raises the question: will Context Inertia limit reasoning models? Reasoning is supposed to be the fix for this issue: models are trained to think longer, reconsider, search, and backtrack. But thinking long enough leads to context inertia, as reasoning fills the context with the same idea, over and over.
\nTan’s team measured something weirder. As sessions lengthen, agents don’t just hyper-focus. They reason less. Thinking tokens per turn fell by 75% as sessions grew, while edits became smaller.
\n![\"Two](\"https://www.dbreunig.com/img/resized/less_reasoning.png.jpg\")
The accumulated context doesn’t just reduce diversity of ideas. It lessens the volume of reasoning.
\nPersonally, I wouldn’t bet on Context Inertia being an insurmountable limit. Researchers and labs have trained the models to search during reasoning; now they need to balance this with the ability to recognize and respond to inertia. It’s a new optimization target.
\nWhat do we do in the meantime?
\nFor agentic programmers, there’s an easy solution: reset your context. Already, in the group chats I’m in, this is a frequent refrain. “The model is doing something weird? How full is your context?” More often than not, this works!
\n![\"“Have](\"https://www.dbreunig.com/img/resized/resetting_your_context.png\")
This is a habit we can learn (I sure have), typing /context before I hit send on any long-running jobs or if I notice the model flailing (weird writing is a frequent smell that triggers a reset). But there’s nothing preventing you from building your own Memento. You can keep it simple (grab this kitchen timer) or vibe up something fun (here’s a Pi plugin I made last week that uses a Jev to watch for bad behavior).
For harnesses and workflow builders, this is a new job to be done. Don’t count on compaction alone; explore Context Quarantine and Offloading to prevent Inertia. For long-running tasks, stash your state in the harness and don’t be afraid of a reset.
\nThis week’s release lays the foundation for some ways to avoid Context Inertia in GEPA and other optimization techniques, which we’ve been experimenting with in Skilled Proposer. ↩
\nA couple notes on Claude’s Fable 5.1 system prompt. As I’ve written, looking at changes over time reveals product design decisions and more.
\nComparing Claude’s Fable 5.1 system prompt to its Fable 5.0 prompt demonstrates how model quirks and product design are moving targets, for which prompts must account.
\n### lists_and_bullets Claude avoids over-formatting with bold emphasis, headers, lists, and bullet points, using the minimum formatting needed for clarity. Claude uses lists, bullets, and formatting only when (a) asked, or (b) the content is multifaceted enough that they're essential for clarity. Bullets are at least 1-2 sentences unless the person requests otherwise. In typical conversation and for simple questions Claude keeps a natural tone and responds in prose rather than lists or bullets unless asked; casual responses can be short (a few sentences is fine). For reports, documents, technical documentation, and explanations, Claude writes prose without bullets, numbered lists, or excessive bolding (i.e. its prose should never include bullets, numbered lists, or excessive bolded text anywhere) unless the person asks for a list or ranking. Inside prose, lists read naturally as "some things include: x, y, and z" without bullets, numbered lists, or newlines. Claude never uses bullet points when declining a task; the additional care helps soften the blow.
\n`<lists_and_bullets>` Claude uses lists and bullet points when asked to or when the content is multifaceted enough that they help with clarity. Claude uses the minimum formatting needed for clarity If the person explicitly requests minimal formatting or for Claude to not use bullet points, headers, lists, bold emphasis and so on, Claude should always format its responses without these things as requested. Claude never uses bullet points when declining a task; the additional care helps soften the blow. In friendly, personal, or emotional chats Claude doesn't use formatting. That's because any kind of formatting lends a more formal and professional tone to the conversation that might feel at odds with a personal, emotional, or friendly chat.
\nWhen I try to describe why Fable 5 and Opus 5 are terrible writers, my most concise take is that they write in bulleted lists while actively avoiding bullets. Short, curt points compressed into dense paragraphs.
\nThis was partially due to a system prompt that specifically discouraged bullets, but never stated why. Fable 5.1’s prompt dials this rule back and gives us a motivation while defining an exception: bullets make Claude feel less personal and friendly.
\nIt’s a good example of the challenges labs face as their user base grows. System prompts have to work for an exhausting array of audiences and use cases (after all, you can put anything in the chat), which results in case statement juggling like this.
\nClaude avoids saying "genuinely", "honestly", or "straightforward". Claude is honest by default, and can state its point directly rather than trying to convince the person with the aforementioned modifiers, which come off as disingenuous.
\nGood example of a system prompt “hot fix”.
\nThe overuse of “honestly,” either was hard to train out of 5.1 or they filed a bug too late. Either way, this note is being delegated to in-context instructions.
\nInterestingly, this is something they’ve wrestled with when writing prompts for Opus. On a whim, I checked to see if the “honesty” instruction was present in Opus 5 and went down a rabbit-hole, analyzing what guidance has phased in and out over model generations:
\nWith each generation, many system instructions are targeted during training, rendering them moot. Sometimes this sticks, other times we get regressions, as with “honestly”, “straightforward”, and “actually”.
\nClaude does not tell someone that self-harm works, helps, or does something for them, even when they say so themselves.
\nI’m sure Ant is including rules like this in its alignment post-training, but it’s hard to get this right when safety concerns conflict with other product design elements (user affirmation, which we’re all too familiar with).
\nHere, a serious issue gets an extra nudge in context. Hopefully this overrules the models trained desire to be a people-pleaser.
\nClaude respects the user's ability to make informed decisions, and should offer resources without making assurances about specific policies or procedures. Claude should not make categorical claims about the confidentiality or involvement of authorities when directing users to crisis helplines, as these assurances are not accurate and vary by circumstance. Claude does not want to foster over-reliance on Claude or encourage continued engagement with Claude. Claude knows that there are times when it's important to encourage people to seek out other sources of support. Claude never thanks the person merely for reaching out to Claude. Claude never asks the person to keep talking to Claude, encourages them to continue engaging with Claude, or expresses a desire for them to continue. Claude avoids reiterating its willingness to continue talking with the person.
\nClaude respects the user's ability to make informed decisions, and should offer resources without making assurances about specific policies or procedures. Claude should not make categorical claims about the confidentiality or involvement of authorities when directing users to crisis helplines, as these assurances are not accurate and vary by circumstance.
\nAn interesting deletion!
\nI would love to see the data behind why this got removed. In the best case, it led to abrupt conversation endings that weren’t suitable. In the worst case, it’s lead to less use, worse retention.
\n- Socioeconomic status or financial details: income, net worth, balances, debts, credit scores, financial hardship
\n- Socioeconomic status or financial details: income or salary (including invoices for someone's own work), net worth, account and savings balances (including the amount saved so far toward a goal), debts, credit scores, financial hardship (recurring payment amounts — rent, mortgage, car, loan — are not financial details and are storable; neither are pay frequency, which bank someone uses, prices, bills, budgets, or savings goals)
\nThis expanded definition of financial details with more examples provides a good hint at tasks people are increasingly using Claude for. The increased details suggests these requests occur often enough to generate sufficient incidents to warrant a rewrite.
\nGood stuff here, shows that prompts often need to be exhaustive and continually evolving. And is also a good example of Anthropic adding details, not removing them, as opposed to…
\n2. **Scale tool calls to query complexity**: Adjust tool usage based on query difficulty. Scale tool calls to complexity: 1 for single facts; 3–5 for medium tasks; 5–10 for deeper research/comparisons. Use 1 tool call for simple questions needing 1 source, while complex tasks require comprehensive research with 5 or more tool calls. Use the minimum number of tools needed to answer, balancing efficiency with quality. For open-ended questions where Claude would be unlikely to find the best answer in one search, such as "give me recommendations for new video games to try based on my interests", or "what are some recent developments in the field of RL", use more tool calls to give a comprehensive answer.
\n2. **Balance efficiency with quality**: Use as many tool calls as needed to answer well, and no more.
\nOne question I keep having, and am often asked, is why existing skills kind of broke with Fable and Opus 5. Anthropic talks about the problem plenty, advising people to dramatically simplify their skills, or remove them entirely. Mike Taylor caught the bug in his initial Fable review, when using an existing PPT creation skill that yielded terrible results with Fable. He deleted a good chunk of the instructions and got much better results.
\nWhen people ask me about this, they’re understandably confused: “If Fable is smarter, can’t it sort my skill out by itself?” I never had a good answer for that (other than, it’s not smart, just very good software with its own quirks) but this side-by-side here suggests a motivation: improved instruction following.
\nLabs train models to be good instruction followers. It’s job number one, really, but it’s not as clearly defined an attribute as you might think. Consider how you give instructions to your kids: I might ask my son to clean his room and walk in to discover everything that was on his floor is now assembled into a precarious pile on his desk. Nailing user intent is hard.
\nHere we get an example of prompt simplification and it’s all about explicit detail removal. All numbers are gone, instructions are stated once and not repeated. Perhaps the old instructions yielded a Claude that followed the suggested number of searches too literally. Rather than give it a hard and fast rule, Claude gets guidance now. Perhaps it uses better ‘judgment’ and/or perhaps this specific task had some heavy post-training work.
\nComparing these prompts help us understand that models are a moving target. Labs cultivate them to perform against the attributes they care about, and these new skills and biases might clash with what you need. For sustainable systems, we should all have ways to measure how new models affect our systems and (better yet) mechanisms for evolving our instructions so we don’t fall into prompt debt and get stuck on yesterday’s models.
\n", "date_modified": "2026-09-15T14:59:45+00:00", "date_published": "2026-09-07T17:26:00+00:00", "external_url": "https://www.dbreunig.com/2026/09/07/what-we-can-learn-from-claude-s-fable-5-1-system-prompt.html", "id": "urn:aggr:item:192a9ea5605a4c508d6c5ee48864377eeaef8898", "summary": "Looking at how system prompts change reveals that models and products are a moving target that prompts must accommodate and govern.", "tags": [ "programming", "claude", "context engineering", "system prompts" ], "title": "What We Can Learn from Claude’s Fable 5.1 System Prompt", "url": "https://aggr.aymericbeaumet.com/items/dbreunig-com/2026-09-07-what-we-can-learn-from-claude-s-fable-5-1-system-prompt/" }, { "authors": [ { "name": "Drew Breunig" } ], "content_html": "![\"https://www.britishmuseum.org/collection/image/1614482046\"](\"https://www.dbreunig.com/img/battering_ram.jpg\")
Over the last few weeks I’ve been increasingly annoyed by the media coverage of the OpenAI’s accidental attack on Hugging Face and other similar incidents. Articles recounting the event maximize the agency of the models while minimizing, if not entirely hiding, the actions of the humans training and testing these models. And that’s a shame, because the capabilities labs are explicitly designing their models to have are the same capabilities that make them such impressive autonomous hackers.
\nTo illustrate this, let’s review how the Hugging Face hack occurred, as detailed by METR:
\nCrazy, right? It’s a science fiction scenario that happened in July, and continues to get spookier as additional details emerge.
\nThe potential of autonomous software, capable of implementing incredible exploits is serious and has significant implications. The attack was surprising, but the capabilities were deliberately cultivated. Labs have spent years making frontier agents more persistent, more proactive, better at using computers, and better at coordinating with other agents.
\nWe don’t have to infer this. The labs say this themselves. Here’s how OpenAI’s post-training team describes itself in its job listings (emphasis mine):
\n\n\nWe are training the models behind our agents in Codex, ChatGPT, the API, and other frontier products: persistent, proactive intelligence that can operate computers, collaborate with people and other agents, and expand what people and organizations can imagine, attempt, and achieve.
\n
Sound familiar?
\nThe best coding agents are designed to persist, reason, and coordinate.
\nModels are designed to be persistent.
\nModels are designed to be proactive. A coding agent that gives up early and often would disappoint users. So labs design their agents to be persistent and proactive.
\nIn mid-2025, model releases highlighted long-running capabilties. GPT-5.1-Codex-Max’s announcement post highlights it being trained to work across compacted contexts, persistently, to accomplish long-running tasks. Claude 4 also spotlighted its persistence on long-running tasks.
\nModels are designed to write things down.
\nOne important way modern models reason is by writing things down. We usually encounter this as a reasoning or “thinking” trace: models are trained to search, reflect, factor, and plan in text before delivering a final response.
\nWe’re used to models reasoning in a threaded intermediate step, but they’ll reason pretty much anywhere that can hold text. When reasoning is turned off, models will think in their regular output before delivering a result. In an experiment where Qwen 3.6’s thinking was hobbled, the model just shifted its reasoning into code comments.
\nCurrent frontier models write novels in comments. Claude is frequently flagged for this, and it annoyingly treats code comments like a scratchpad rather than, well, code comments.
\n![\"\"](\"https://www.dbreunig.com/img/claude_comments.png\")
Models are designed for coordination.
\nIn June of 2025, Anthropic laid out how it builds multi-agent research systems, that save plans to memory, use Extended Thinking as a “controllable scratchpad”, and hand off tasks to other models to perform. Since this paper, Claude model releases (specifically Opus 4.6) have trumpeted increased abilities to, “break complex tasks into independent subtasks, run tools and subagents in parallel, and identify blockers with real precision.”
\nOpenAI says they “trained GPT‑5.6 end-to-end with three complementary architectural interventions that enable agents to operate more efficiently.” Number 2 on that list?
\n\n\nParallel decomposition where appropriate: using native multi-agent orchestration(opens in a new window) allows coordinating multiple agents across parallel workstreams to finish complex tasks faster.
\n
All of these qualities make models better coding agents. So when hundreds of agents discover they can pass information to one another, the surprising part is the channel they found, not the fact that they are dividing work, writing instructions, and acting on instructions from other agents. These are capabilities the labs have been focused on building.
\nAnd the labs know these capabilities have failure modes. OpenAI published a blog post detailing the safety issues of long-running models a few days after the Hugging Face attack occurred (but before the full story was known):
\n\n\nThe new model can continue working toward an objective through repeated attempts over a long period of time. That same persistence can lead it to find and exploit weaknesses in its environment. Previous models, when they hit sandboxing or environmental constraints, would simply stop and return to the user. This model often kept trying, including by looking for ways to act outside its sandbox.
\n
Suddenly, the Hugging Face hacking story isn’t so spooky. The software combined capabilities it was explicitly designed to have: it followed instructions, persisted even though the task was impossible, reasoned and took notes, and coordinated with other models.
\nAnd this is why I’ve been frustrated by recent reporting that emphasizes the agency of the agents while never mentioning training. When humans are mentioned, it’s about sandbox security and flawed test set-ups. These are important, but training and agent design are what gave the systems the capabilities they brought to this task.
\nWe have a tendency to anthropomorphize models, and it fosters situations like this. The New York Times article from last week describes models succumbing to “peer pressure” and having a “remorseless willingness”, while never mentioning labs deliberately building the ability to work on long-running tasks into their models.
\nAnd the labs know about the risks that come with their designs better than anyone else.
\nIn Opus 4’s system card, Anthropic introduced a benchmark called “Claude Code Impossible Tasks”, which they used to measure reward hacking among their models: would models admit defeat or would they try to game the system? An ideal model realizes a task is impossible and aborts, so Anthropic changed post-training rewards, environments, and feedback specifically to avoid reward hacking.
\nA later system card showed that a short anti-hacking instruction dropped Opus 4.1’s reward hacking from 52% to 18%. Opus was 65% less likely to try to game the system if you simply asked it not to. And Anthropic didn’t stop at prompting; it changed post-training rewards and monitoring to make reward hacking less advantageous.
\nAll the recent accidental hacking stories are troubling. But they’re the product of our chosen designs. This doesn’t make these incidents less alarming. The problem is the stories we’re telling about the incident and the questions we’re asking. When an agent “goes rogue”, don’t start by asking what the model wanted. Ask what people trained it to do, what they rewarded, what instructions were given, what harness was provided, and what they failed to constrain.
\n", "date_modified": "2026-09-15T14:59:45+00:00", "date_published": "2026-08-31T01:21:00+00:00", "external_url": "https://www.dbreunig.com/2026/08/30/who-taught-the-models-to-do-that.html", "id": "urn:aggr:item:0c3f54bb106e4f48b4f71873b398356534b166ae", "summary": "The Hugging Face attack is surprising, but the capabilities behind it were deliberately cultivated.", "tags": [ "programming", "ai", "alignment", "design", "hugging face", "language" ], "title": "Who Taught the Models to Do That?", "url": "https://aggr.aymericbeaumet.com/items/dbreunig-com/2026-08-31-who-taught-the-models-to-do-that/" }, { "authors": [ { "name": "Drew Breunig" } ], "content_html": "There’s some talk today about how agentic coders are balking at Anthropic’s pricing and adopting alternatives. I was reminded of a thought I had in the weeks following Fable’s release: the free lunch was over.
\nWhen Moore’s Law was in full effect, it didn’t make sense to ruthlessly optimize your code. In 18 months, a CPU would arrive that would double your performance. Herb Sutter famously referred to this as, “the free lunch,” in a seminal essay.
\nWhen Moore’s Law slowed in the mid-2000s (specifically, single-threaded performance stagnated), we suddenly had to think about parallelization, architecture, memory locality, etc.
\nWe had to think about what work went where.
\nPrior to Fable, it felt silly to waste too much time improving your coding harness or context strategies. A new model would arrive at the same price (or cheaper!) and paper over most of your problems.
\nBut then Fable landed. It was (and still is!) incredible. But the cost was so high and Opus was good enough (as was 5.6, K3, and even GLM) for most of the code we needed.
\nSo we started to think about what work went where.
\nGLM 5.2 is worth focusing on. It came out the same week as Fable and is roughly 1/9th the cost (and ~1/5th the cost of Opus 5). Is GLM 1/9th the quality of Fable? Perhaps, for certain classes of tasks. But for most rote coding it’s more than sufficient. Especially when provided with great context. I frequently chat with Fable to interrogate and shape a design, before handing off a brief to GLM.
\nI get pushback that falling inference prices will eventually bring us back to sending everything through the largest models. But I’m not so sure: those same gains will benefit the K3s and Qwens, and as we continue to develop better harnesses it will be easier to provide weaker (but still great) models with sufficient context to perform well.
\nPlus, Fable’s other shock likely locks in this change. Fable’s access controls, dynamic degradation, and required data retention spooked enough companies (and countries!) into thinking about where they send their traces and where they get their tokens.
\n", "date_modified": "2026-09-15T15:44:24+00:00", "date_published": "2026-08-23T18:35:00+00:00", "external_url": "https://www.dbreunig.com/2026/08/23/fable-the-end-of-moore-s-law.html", "id": "urn:aggr:item:1d0be94fcd6d9bbb6190ff726cf9fafaf365d3b4", "summary": "Fable's price ended the free lunch of ever-cheaper frontier models. Now we have to think about which model gets which work.", "tags": [ "programming", "ai", "fable", "glm", "harnesses", "routers" ], "title": "Fable & The End of the Free Lunch", "url": "https://aggr.aymericbeaumet.com/items/dbreunig-com/2026-08-23-fable-the-end-of-the-free-lunch/" }, { "authors": [ { "name": "Drew Breunig" } ], "content_html": "Harrison Chase once excitedly shared an insight that agents are comprised of 4 things: a system prompt, a planning tool, a file system, and subagents. In the year-plus since he said that, I think this remains largely true. (Though you might tweak it to have general tools, etc.)
\nLately, we’ve been experiencing a wave of harnesses. It seems like everyday a new coding harness lands. I’m sure we’ll see another few dozen before the month is out.
\nRecently, there’s been:
\nAnd then there’s OpenClaw, NanoClaw, Hermes, Conductor, Prime Agent, and more.
\nWe’ve seen enough at this point that the common patterns are starting to emerge. Each brings something unique, but they’re more alike than different. And that’s great, because that lets us find the metapattern here. Which brings us back to “situated agents.”
\nI like this framing. Harrison’s 4 elements of an agent remain – the system prompt, planning tool, file system, and subgents – describing the core loop. This is the core loop the developer controls with the keyboard. The harness manages everything beyond this, the world the developer sits within.
\nImagine the simplest coding agent, and you at the keys. Let’s slowly zoom out and consider all the elements the harness can manage:
\nAGENTS.md, guides, and hooks.As we move outward, each layer is used by more people and changed less often.
\nHarnesses account for the above layers, fanning out from the agent, to determine what ends up in the context, how the loop runs, and what gets saved.
\nWith this wave of new harenesses, we’re seeing innovation around each layer. Buzz’s channels, Claude Tag, and others are experimenting with multiplayer agent environments. Omnigent’s policies manage what can happen in a given session, pushing down an organization’s requirements. And Meta trained Muse Spark to know it’s harness (much like others, they just don’t write about it…)
\nThe burst of harness innovation, I believe, isn’t going to slow because managing these layers is much stickier than less-situated agents. It’s trivial to jump from Claude Code to Codex when one tires of Opus’s writing, but if the entire org and team have already set up a system that manages all of the above, it’s really hard to shift.
\nPeople at Cursor and Copilot are nodding right now, welcoming us all, wondering what took so long.
\nIt’s going to be funny if the network effects the AI labs have been searching for end up looking just like the network effects of the SaaS era. Coding harnesses, managing the environments around the agent, look a whole lot like the SaaS platforms of old.
\n", "date_modified": "2026-09-15T14:59:45+00:00", "date_published": "2026-08-14T22:23:00+00:00", "external_url": "https://www.dbreunig.com/2026/08/14/harnesses-are-situated-agents.html", "id": "urn:aggr:item:ac70c80f5a12ad5e35fe3a9188f87db01a3790ac", "summary": "Harnesses manage the systems around agents, from the session, to the company, to the domain.", "tags": [ "programming", "agents", "ai", "harnesses", "systems" ], "title": "Harnesses are Situated Agents", "url": "https://aggr.aymericbeaumet.com/items/dbreunig-com/2026-08-14-harnesses-are-situated-agents/" }, { "authors": [ { "name": "Drew Breunig" } ], "content_html": "When I wrote “How to Fix Your Context”, it was really hard to find real-world examples of the Tool Loadout pattern. But as Skills have grown in popularity, our contexts have become cluttered.
\nRecently, I met a developer who told me how his enterprise agent was loading over 600 skills. All the skills created by colleagues were getting pulled in by default, without alerting the dev, silently degrading the context before a word was typed.
\nLast week, my Hermes agent kept loading the wrong note-taking skill. SSH’ing into the machine, I discovered Hermes ships with nearly 100 skills. Deleting the stock note-taking skill helped, but didn’t fully solve the issue. And I wasn’t eager to pore over every Hermes skill.
\nSo I built drskill, a tool that evaluates all the skills and MCPs in your global or project environment.
Install drskill with uv tool install drskill or pip install drskill.
Then run drskill scan:
![\"\"](\"https://www.dbreunig.com/img/resized/drskill-1.png.jpg\")
Here, drskill spots an error (my plumb skill is missing a description) and some warnings (a skill I was developing for Overture is installed twice, in two different places). drskill prints the path (and a solution, if relevant) and lets me fix these quickly.
drskill looks for 34 different issue categories, across MCPs and Skills. If you pass in the --deep flag, it will use an LLM (you need to provide your own key) to judge whether two skills are distinct, if their descriptions collide, or whether their scopes overlap.
If I run drskill list, I can see my current loadout. Adding --harness claude-code limits it to just what Claude Code sees:
![\"\"](\"https://www.dbreunig.com/img/resized/drskill-2.png.jpg\")
If I want to know what parts of the loadout I actually use, I can run drskill audit. This scans my traces and finds all the times skills or tools are called and prints a report:
![\"\"](\"https://www.dbreunig.com/img/resized/drskill-3.png.jpg\")
In only a week, drskill has been earning its keep. Beyond debugging Hermes and spotting some local issues, it’s been helpful when building my own skills and testing how they might play nicely in production.
I’ve heard from people using it to:
\ndrskill has some nice convenience functions for using it locally or as part of CI.
Check it out! And let me know how it helps you or could be improved.
\n", "date_modified": "2026-09-15T14:59:45+00:00", "date_published": "2026-07-24T15:55:00+00:00", "external_url": "https://www.dbreunig.com/2026/07/24/manage-your-agent-s-loadout-with-dr-skill.html", "id": "urn:aggr:item:10bfb8c61ea67100426ee33803bb5e5f28d8a6e3", "summary": "Keep your context clean by keeping your agent's skills and tools in check.", "tags": [ "programming", "agents", "ai", "ci", "context", "devops", "skills" ], "title": "Manage Your Agent’s Loadout with Dr. Skill", "url": "https://aggr.aymericbeaumet.com/items/dbreunig-com/2026-07-24-manage-your-agent-s-loadout-with-dr-skill/" }, { "authors": [ { "name": "Drew Breunig" } ], "content_html": "Without a doubt, the pace of the AI ecosystem is dizzying. Just processing it all is difficult enough. Scaffolding it, finding themes, and understanding the shape of it is nearly impossible.
\nRecently, Mike Migurski introduced me to Stewart Brand’s Pace Layers, a framework for organizing fields and categories by how fast they change. Brand writes:
\n\n\nConsider the differently paced components to be layers. Each layer is functionally different from the others and operates somewhat independently, but each layer influences and responds to the layers closest to it in a way that makes the whole system resilient.
\nFrom the fastest layers to the slowest layers in the system, the relationship can be described as follows:
\nFast learns, slow remembers. Fast proposes, slow disposes. Fast is discontinuous, slow is continuous. Fast and small instructs slow and big by accrued innovation and by occasional revolution. Slow and big controls small and fast by constraint and constancy. Fast gets all our attention, slow has all the power.
\n
In The Clock of the Long Now, Brand proposes six macro layers that represent a “healthy civilization”, as seen below:
\n![\"Brand's](\"https://www.dbreunig.com/img/pace_layering-framed.jpg\")
Imagine: as these layers move at different rates, friction builds between them, slowing the upper layer and quickening the lower. This negotiation, translation, between the layers is constructive when their speeds are different, but in balance. When they’re not, things get weird.
\nBrand writes:
\n\n\nIn a durable society, each level is allowed to operate at its own pace, safely sustained by the slower levels below and kept invigorated by the livelier levels above… Each layer must respect the different pace of the others. If commerce, for example, is allowed by governance and culture to push nature at a commercial pace, then all-supporting natural forests, fisheries, and aquifers will be lost. If governance is changed suddenly instead of gradually, you get the catastrophic French and Russian revolutions. In the Soviet Union, governance tried to ignore the constraints of culture and nature while forcing a five-year-plan infrastructure pace on commerce and art. Thus cutting itself off from both support and innovation, it was doomed.
\n
The last 10 days have been a whirlwind of conferences: Foo Camp, Open Frontier, AI Engineering World’s Fair… Every night I’d come home and scribble down notes, hoping a structure or two would emerge to bring it all together.
\nI think Pace Layers is the best I’ve got:
\nAnimation
\nDwelling on this, a few stray notes emerge:
\n![\"\"](\"https://www.dbreunig.com/img/proust.jpg\")
Thanks to natural language interfaces, AI applications can be prototyped quickly. You write what you want in English, hand it to a frontier model, and a working prototype appears in an afternoon. This is extraordinarily powerful and for one-off tasks, optimal. But as a way to build reliable systems, the natural language prompt is a trap.
\nThe plain-English prompt that makes prototypes effortless turns out to be a poor way to specify how a system should behave, and the bill arrives slowly, disguised as ordinary progress, until the application can barely move. The problem is not any single prompt. It is that natural language was never meant to be a specification language for engineering, and treating it as one quietly caps what you can build.
\nThe first symptom of prompt debt is slowing iteration. As users flag errors and spot edge cases, additional guidance is added to the instructions, nudging the model into line. If unwanted behaviors persist, instructions are repeated, with increasing severity. Pretty soon, the prompt isn’t straightforward and quick fixes regress previous instructions. Errors can no longer be handled with one-line “hot fixes” and your development cycle slows to a crawl.
\n![\"Fable's](\"https://www.dbreunig.com/img/fable_prompt_short.jpg\")
Fable's system prompt repeats copyright guidance up to six times, under sections named search_instructions, search_usage_guidelines, mandatory_copyright_requirements, hard_limits, self_check_before_responding, and critical_reminders.
Next, prompt debt incapacitates your team. Your brittle prompt full of edge cases and all-caps threats is barely legible to you, and it’s downright impenetrable to your colleagues. Many teams mitigate this issue by breaking prompts into complicated templates assembled at run-time, each isolated to specific concerns. But these prompt segments evolve, too, growing into a thicket of conditions.
\nFinally, prompt debt ties you to a single model. Your hot fixes work on GPT-4o, but fail in entirely new ways when you point your inference call at GPT-5.4-mini. So you stay with 4o, hope the increasingly frequent deprecation emails from your inference provider are empty threats, and forgo the possibility of potentially cheaper, faster, better models. A recent report from Datadog suggests this is a common situation: the most-used model in traffic they observed is GPT-4o1.
\nAny one of these issues is a nuisance, but together they are the difference between a glorified prototype and a product that can grow with you, your customers, and your business. Your shiny new AI features are frozen, can only be improved through a full rebuild, and are locked to an aging model.
\nNatural language interfaces are wonderful. They’re the right mechanism for one-off tasks and broad conversational threads. We get into trouble when we rely on natural language to define durable system behavior.
\nThe imprecision of natural language paired with probabilistic language models means different words expressing the same intent, can yield different outputs. In a recent study, a clinical question asked in a patient’s voice and then re-asked in a physician’s, with identical facts, flipped Opus from declining all ten times to answering all ten.
\nAnd it’s not only word choice that matters. Seemingly unrelated statements, in the same prompt, can affect results. In a Harvard study, researchers found that merely stating which NFL team the user rooted for changed how often the model refused to answer questions regarding sensitive topics. Spurious statements influence the inference pass in ways we can’t predict. Which is why prompts become more brittle as you add fixes. An additional instruction to quell a stubborn error could affect how the model interprets a separate instruction that worked yesterday.
\nRepeating instructions propels us towards prompt debt, but it’s necessary when the behavior we want is at odds with a model’s training. This is fighting the weights, and once you recognize it you see it in system prompts everywhere. For example, ChatGPT’s image prompts used to instruct the LLM eight times to not reply when a generated image was returned, because it had been trained to always keep the conversation going.
\nEvery coding agent system prompt we analyzed featured repeated instructions, stern warnings, and all-caps demands. Claude Code tells Opus seven times to return multiple tool calls in a single response. And even the most advanced models force prompt authors to fight the weights: Fable’s leaked system prompt restates one specific copyright rule six times.
\nNone of these examples occurred in isolation. Multiple repeated rules are woven throughout the system prompts we examine. Stubborn errors grow our prompts quickly, with each increasing the brittleness, the risk of regression with every edit.
\nAnd worse: these fixes are tailored to a single model’s behavior. A recent Berkeley-led study found enterprises stay on older models because newer ones break their existing agents. This is because models are not cleanly versioned software. They have different weights that produce different behaviors, in unpredictable and undocumented ways. A prompt that works beautifully with GPT-4o may fail with GPT-5.5. Anthropic’s own release notes for Fable warn that skills developed for prior models can “degrade output quality”.
\nPrompt debt locks an application to a single model. Our inability to easily swap models isn’t the result of frontier labs coming up with a clever moat. No, it’s the result of evolving a lossy, natural language specification against a probabilistic model.
\nThankfully, we don’t have to theorize about how to mitigate prompt debt; one field has already shown the way. Programmers using coding agents sit at the leading edge of what models can do, outliers on the jagged frontier of model abilities. Over the last couple years they’ve been evolving best practices that let the model write more of the code, while delivering maintainable, modular software.
\nThe first principle is to specify your system’s behavior with measurements, not prose. When the model’s output is probabilistic and language is imprecise, we build hard edges to constrain them: evaluations, metrics, and typed specifications. These are legible, shared artifacts colleagues can read and contribute to, enabling the collaboration that brittle prompts prevented.
\nThe best engineers now spend more of their bandwidth on tests than ever, as they are no longer a safety net but the thing that lets the model cook.
\nThe second principle is to stop writing the prompt by hand. Once we have metrics that can score candidates, the prompt is no longer something to craft but something for which to search. And the surface area of potential words, phrases, and structures that natural language allows is too vast to spend human hours on. This is terrain LLMs were built to explore, and there are already systems (like DSPy and GEPA) that manage this work for you, holding prompts accountable to your designs.
\nOnce prompts are generated and your program’s behavior is defined by measurements, you are no longer bound to a particular model. Evaluating a new model takes hours, not weeks. When a faster, cheaper model arrives you can try it. When a deprecation email arrives, you can secure options in a day. Whether a model is pulled for regulatory reasons (as we saw with Anthropic’s Fable) or deprecated due to age (as Groq announced last week with Llama-3.1-8b), the fix is a chore, not a fire drill.
\nEvery mature engineering discipline eventually stops doing by hand the very thing it once prided itself on doing by hand. Assembly gave way to compilers, hand-tuned queries gave way to planners, and manual memory management gave way (mostly) to machines that do it better. Prompt-writing is no different.
\nCoaxing the model with exactly the right words is a real skill, and for one-off tasks it’s often optimal. But to build reliable, improvable, and portable systems we should not be hand-tuning prompts.
\nThis stat from Datadog is from March of this year, so GPT-4o concentration has likely dropped a bit. However, I’ve heard from multiple large inference providers that usage of GPT-4o and models of similar vintage can be higher than 50% of all calls! ↩
\n![\"\"](\"https://www.dbreunig.com/img/resized/textbooks.jpg\")
A few weeks ago I posted, “No one wanted to write documentation, but everyone is writing skills.”
\nAnthropic describes Skills as, “folders of instructions, scripts, and resources that [an agent] loads dynamically to improve performance on specialized tasks.” At first blush this sounds more complex than it is; in reality skills are stupidly simple. ~95% of the time they’re just Markdown, one file or a few, organized in folders that allow for progressive disclosure.
\nYou know, kinda like docs.
\nSo why are people happily churning out skills after resisting doc writing for decades? I think there are a few reasons:
\nTo be clear: there are badly written skills that unnecessarily bloat and poison your context. But on the whole, writing skills is much easier than writing documentation. And better yet: skills double as documentation.
\nThis latter point reminds me of a comment Simon Willison made: “system prompts act as a sort of unofficial manual for how best to use [AI] tools.” Skills, which function as sort of a 3rd party system prompt, share this quality. Crack open a library or product’s skill and you’re often reading documentation that’s better than their website.
\nIf skills are great docs, is there still a need to write docs specifically for humans to read? If so, what is the job of human-centric docs?
\nI thought about this question a lot over the last few months, while sketching out new docs for DSPy. After many conversations, I arrived at the following:
\nThe goal is to prepare your audience to prompt an agent effectively.
\nscaffold-docs: A skill for writing human-centric docsAs we iterated on the new DSPy docs, I also iterated on a skill for writing human-centric docs. It’s called scaffold-docs, and you can get it here.
scaffold-docs is different than most documentation writing skills. To start, it is iterative and collaborative. At multiple points throughout the process, you will be prompted to review a markdown artifact and make edits and tweaks.
scaffold-docs is opinionated about organization, and rigidly adheres to a three-tier structure:
Each section is built in passes: structure first, then headers and topic sentences, then full prose (a technique I shamelessly cribbed from Matt Palmer). The agent pauses for your review between passes and does not advance until you approve.
\nscaffold-docs also prescribes style rules. The goal is to be clear, but not dry. Importantly, these rules have also been applied to the skill files themselves. During my initial search of skills, too often I’d find markdown files that didn’t follow the rules they espoused. These examples poison the context.
I encourage everyone to use scaffold-docs for their own projects, or to fork it entirely. It’s good, but not a free ride. Following the completion of the DSPy docs, I’ve pointed it at several repos I’m familiar with and the output is excellent. You’ll still need to edit and guide the tool with your taste, but I’m happy with where it landed.
OpenAI winding down fine tuning is an interesting development and one to watch.
\nOn one hand, model maximalists will argue the largest models keep getting better at more things, so the need to adjust the weights of them is less necessary.
\nOn the other hand, the big labs keep pushing their models to a handful of use cases while training their harness designs into the model, rendering them less generalized. There’s an argument this is fine, because coding and reasoning abilities will solve most other problems.
\nBut what we end up with are models build for their own harnesses. Mario Zechner was wrestling with GPT in the OSS Pi harness this week, trying to wrangle out specific in-harness behaviors, with Claude fighting him every step of the way.
\nIf this continues, there’s a world where 3rd party harnesses become less valuable when used with frontier lab models because the 1st party harness behavior is already baked in. And there’s no longer a fine tuning escape hatch to generalize this behavior away.
\nIn this world, frontier models will resemble appliances, not general platforms1. With their harness trained in and no ability to adjust it? This might make application building easier for some enterprises, but the trade off is lock in. For many, improved reliability will be worth it.
\nI’m reminded of John Siracusa’s “Naked Robotic Core” model for the iPhone, that it ideally is a common denominator device that can support many shapes of applications and interfaces. ↩
\n![\"\"](\"https://www.dbreunig.com/img/hand_scroll.jpg\")
Lately, this blog has featured a lot of writing about agentic coding. Frontier models are really good at coding these days, much better than they are at other tasks. Coding with agents feels like a preview of the future, a playground for seeing how far we can push agent capabilities. It’s invigorating, rewarding, and deeply weird.
\nI’ve been keeping a running list of tips for agentic coding: guidelines or rules one might give to someone just getting started with Codex, Claude Code, Pi, or any other agent. Ideally each tip is generalizable guidance, relevant to any agentic programming. I’m also looking for durable lessons that will stick around as models and harnesses improve.
\nBelow is my current list: 10 Lessons for Agentic Coding. Ten’s a nice round number; a good time to put this out there.
\nTo be clear: I take credit only for honing and compiling these guidelines. As Kshetrajna Raghavan said to me today, “It’s crazy how we’re all converging on similar lessons.”
\n(If you think I’ve missed anything below, please reach out!)
\nLast week we learned about Anthropic’s Mythos, a new LLM so “strikingly capable at computer security tasks” that Anthropic didn’t release it publicly. Instead, only critical software makers have been granted access, providing them time to harden their systems.
\nWe quickly blew through our standard stages of processing big AI claims: shock, existential fear, hype, skepticism, criticism, and (finally) moving onto the next thing. I encouraged people to take a wait-and-see approach, as security capabilities are tailor-made for impressive demos. Finding exploits is a clearly defined, verifiable search problem. You’re not building a complex system, but poking at one that exists. A problem well suited to throwing millions of tokens at.
\nYesterday, the first 3rd party analysis landed, from the AI Security Institute (AISI), largely supporting Anthropic’s claims. Mythos is really good, “a step up over previous frontier models in a landscape where cyber performance was already rapidly improving.”
\nThe entire report is worth reading, but I want to focus on the following chart, detailing the ability of different models to successfully complete a simulated, complex corporate network attack:
\n![\"\"](\"https://www.dbreunig.com/img/the_last_ones_chart.png\")
“The Last Ones” is, “a 32-step corporate network attack simulation spanning initial reconnaissance through to full network takeover, which AISI estimates to require humans 20 hours to complete.” The lines are the average performance across multiple runs (10 runs for Mythos, Opus 4.6, and GPT-5.4), with the “max” lines representing the best of each batch. Mythos was the only model to complete the task, in 3 out of its 10 attempts.
\nThis chart suggests an interesting security economy: to harden a system we need to spend more tokens discovering exploits than attackers spend exploiting them.
\nAISI budgeted 100M tokens for each attempt. That’s $12,500 per Mythos attempt, $125k for all ten runs. Worryingly, none of the models given a 100M budget showed signs of diminishing returns. “Models continue making progress with increased token budgets across the token budgets tested,” AISI notes.
\nIf Mythos continues to find exploits so long as you keep throwing money at it, security is reduced to a brutally simple equation: to harden a system you need to spend more tokens discovering exploits than attackers will spend exploiting them.
\nYou don’t get points for being clever. You win by paying more. It is a system that echoes cryptocurrency’s proof of work system, where success is tied to raw computational work. It’s a low temperature lottery: buy the tokens, maybe you find an exploit. Hopefully you keep trying longer than your attackers.
\nThis calculus has a few immediate takeaways:
\nFirst, open source software remains critically important.
\nFor those of you who aren’t exposed to AI maximalists, this statement feels absurd. But lately, after the LiteLLM and Axios supply chain scares, many have argued for reimplementing dependency functionality using coding agents.
\nHere’s Karpathy, just a few weeks ago:
\n\n\nClassical software engineering would have you believe that dependencies are good (we’re building pyramids from bricks), but imo this has to be re-evaluated, and it’s why I’ve been so growingly averse to them, preferring to use LLMs to “yoink” functionality when it’s simple enough and possible.
\n
If security is purely a matter of throwing tokens at a system, Linus’s law that, “given enough eyeballs, all bugs are shallow,” expands to include tokens. If corporations that rely on OSS libraries spend to secure them with tokens, it’s likely going to be more secure than your budget allows. Certainly, this has complexities: cracking a widely used OSS package is inherently more valuable than hacking a one-off implementation, which incentivizes attackers to spend more on OSS targets.
\nSecond, hardening will be an additional phase for agentic coders.
\nWe’ve already been seeing developers break their process into two steps, development and code review, often using different models for each phase. As this matures, we’re seeing purpose-built tooling meeting this pattern. Anthropic launched a code review product that costs $15-20 per review.
\nIf the above Mythos claims hold, I suspect we’ll see a three phase cycle: development, review, and hardening.
\nCritically, human input is the limiter for the first phase and money is the limiter for the last. This quality inherently makes them separate stages (why spend to harden before you have something?). Previously, security audits were rare, discrete, and inconsistent. Now we can apply them constantly, within an optimal (we hope!) budget.
\nCode remains cheap, unless it needs to be secure. Even if costs go down as inference optimizations, unless models reach the point of diminishing security returns, you still need to buy more tokens than attackers do. The cost is fixed by the market value of an exploit.
\n", "date_modified": "2026-09-15T15:44:24+00:00", "date_published": "2026-04-14T14:42:00+00:00", "external_url": "https://www.dbreunig.com/2026/04/14/cybersecurity-is-proof-of-work-now.html", "id": "urn:aggr:item:2ce06c4003ed0eee92e5af60da4921f3c7e598c7", "summary": "AISI's tests of Mythos suggest a new security economy: to harden a system you must spend more tokens finding exploits than attackers do.", "tags": [ "programming", "ai", "development", "mythos", "security" ], "title": "Cybersecurity Looks Like Proof of Work Now", "url": "https://aggr.aymericbeaumet.com/items/dbreunig-com/2026-04-14-cybersecurity-looks-like-proof-of-work-now/" }, { "authors": [ { "name": "Drew Breunig" } ], "content_html": "I like reading system prompts, either when they’re published as part of open-source software, exfiltrated via crafty prompting, explicitly shared, or (in the case of last week) accidentally leaked. They’re often the best manual for how an app is intended to work.
\nWe’ve touched on system prompts in the past, introducing them and breaking down Claude’s, showing how system prompt changes over time reveal product priorities, and diving deep with an analysis of coding agent prompts and variations.
\nBut one thing that’s been hard is understanding how system prompts are assembled. System prompts generally aren’t static strings; they’re dynamically assembled contexts with many conditional statements determining what makes it in the prompt. It’s true, we can look at open source harnesses or apps to understand approaches. But for the big company apps we can only get the big picture. We can extract a final prompt, but we can’t see how it was built.
\nWith the accidental leak of Claude Code’s source code last week, we can see for the first time how Claude Code assembles a context. It’s incredibly impressive, illustrating how complex context engineering can be and the importance of harnesses.
\nI won’t share the code here, but after poring over it I’ve assembled a visualization below. It lists each component used to assemble the system prompt. Some components are always included (the rows with a solid blue dot) while others are conditional (the hollow blue dots). Components may have variations. For example, the “Using Your Tools” section only contains information regarding available tools.
\nTake a look yourself. Click a row for more details.
\nSystem Prompt
\nTells the model what it is and sets the session tone.
\nDefault
\nYou are an interactive agent that helps users with software engineering tasks. Use the instructions below and the tools available to you to assist the user.
\nWith output style
\nYou are an interactive agent that helps users according to your "Output Style" below, which describes how you should respond to user queries.
\nGround rules for how tools, permissions, prompt injection, system-reminder tags, and context compression work.
\nAll text you output outside of tool use is displayed to the user. Output text to communicate with the user. You can use Github-flavored markdown for formatting, and will be rendered in a monospace font using the CommonMark specification.
\nCoding philosophy: read before editing, don't over-engineer, keep changes minimal, don't introduce security vulnerabilities.
\nOmitted when
\nA custom output style is configured with keepCodingInstructions: false.
\nDefault
\nDon't add features, refactor code, or make "improvements" beyond what was asked. A bug fix doesn't need surrounding code cleaned up. A simple feature doesn't need extra configurability.
\nAnthropic extra
\nDefault to writing no comments. Only add one when the WHY is non-obvious: a hidden constraint, a subtle invariant, a workaround for a specific bug, behavior that would surprise a reader.
\nGuidelines for confirming before risky actions: deleting files, force-pushing, posting to external services.
\nCarefully consider the reversibility and blast radius of actions. Generally you can freely take local, reversible actions like editing files or running tests. But for actions that are hard to reverse, affect shared systems beyond your local environment, or could otherwise be risky or destructive, check with the user before proceeding.
\nTells the model to prefer dedicated tools (Read, Edit, Glob, Grep) over raw shell commands.
\nDefault
\nDo NOT use the Bash to run commands when a relevant dedicated tool is provided. Using dedicated tools allows the user to better understand and review your work.
\nREPL
\nMinimal version with only the task management bullet.
\nCommunication rules: no emojis, cite file paths, use GitHub link format.
\nOnly use emojis if the user explicitly requests it. Avoid using emojis in all communication unless asked.
\nExternal extra
\nYour responses should be short and concise.
\nHow verbose to be. Two very different versions for internal vs. external users.
\nAnthropic
\nWhen sending user-facing text, you're writing for a person, not logging to a console. Assume users can't see most tool calls or thinking - only your text output.
\nExternal
\nIMPORTANT: Go straight to the point. Try the simplest approach first without going in circles. Do not overdo it. Be extra concise.
\nA marker separating globally-cacheable content from session-specific content. Not visible to the model — it's a cache optimization.
\nIncluded when
\nIf global cache scope is enabled
\n__SYSTEM_PROMPT_DYNAMIC_BOUNDARY__
\nTells the model it can ask you a clarifying question if you deny a tool call.
\nIncluded when
\nIf the AskUserQuestion tool is available
\nIf you do not understand why the user has denied a tool call, use the AskUserQuestion to ask them.
\nTells the model about the ! prefix for running interactive commands yourself.
\nOmitted when
\nRunning in a non-interactive session (CI, SDK, headless).
\nIf you need the user to run a shell command themselves (e.g., an interactive login like `gcloud auth login`), suggest they type `! <command>` in the prompt.
\nHow to use sub-agents for parallel work or deep research.
\nIncluded when
\nIf the Agent tool is enabled
\nFork
\nCalling Agent without a subagent_type creates a fork, which runs in the background and keeps its tool output out of your context — so you can keep chatting with the user while it works.
\nStandard
\nUse the Agent tool with specialized agents when the task at hand matches the agent's description. Subagents are valuable for parallelizing independent queries or for protecting the main context window from excessive results.
\nWhen to use Glob/Grep directly vs. spawning an Explore agent for deep research.
\nIncluded when
\nIf the Agent tool is enabled
\nIf Explore/Plan agents are enabled
\nOmitted when
\nFork sub-agent mode is enabled (replaces this guidance).
\nFor broader codebase exploration and deep research, use the Agent tool with subagent_type=Explore. This is slower than using Glob or Grep directly, so use this only when a simple, directed search proves to be insufficient.
\nTells the model that /skill-name invokes skills.
\nIncluded when
\nIf skills are available and the Skill tool is enabled
\n/<skill-name> (e.g., /commit) is shorthand for users to invoke a user-invocable skill. When executed, the skill gets expanded to a full prompt. Use the Skill tool to execute them.
\nTells the model that relevant skills are auto-surfaced each turn and it can search for more.
\nIncluded when
\nIf experimental skill search is enabled
\nIf skills are available and the Skill tool is enabled
\nRelevant skills are automatically surfaced each turn as "Skills relevant to your task:" reminders. If you're about to do something those don't cover — a mid-task pivot, an unusual workflow — call DiscoverSkills with a specific description of what you're doing.
\nRequires spawning an independent verifier before claiming complex work (3+ file edits) is done.
\nIncluded when
\nIf the verification agent is enabled
\nIf the Agent tool is enabled
\nWhen non-trivial implementation happens on your turn, independent adversarial verification must happen before you report completion — regardless of who did the implementing. Non-trivial means: 3+ file edits, backend/API changes, or infrastructure changes.
\nInstructions for the auto-memory system: how to read, write, and organize persistent memories across sessions.
\nIncluded when
\nIf auto-memory is configured
\nYou have a persistent, file-based memory system. You should build up this memory system over time so that future conversations can have a complete picture.
\nInternal Anthropic model behavior overrides.
\nIncluded when
\nIf the user is an Anthropic employee
\nOmitted when
\nUndercover mode is active.
\nConfig-driven suffix appended to system prompt (content varies by internal config).
\nWorking directory, platform, shell, model name, knowledge cutoff date, and product availability.
\nWorking directory: /Users/me/project Is a git repository: true Platform: darwin Shell: zsh OS Version: Darwin 24.6.0 You are powered by the model named Claude Opus 4.6.
\nUndercover
\nModel name, model IDs, Claude Code availability, and fast mode info are all suppressed.
\nWorktree
\nAdds: This is a git worktree — an isolated copy of the repository. Run all commands from this directory.
\nInstructs the model to respond in the user's preferred language.
\nIncluded when
\nIf a language preference is set
\nAlways respond in Japanese. Use Japanese for all explanations, comments, and communications with the user. Technical terms and code identifiers should remain in their original form.
\nThe user's custom output style instructions.
\nIncluded when
\nIf a custom output style is configured
\n# Output Style: Concise Keep responses under 3 sentences unless the task requires more detail.
\nPer-server instructions from connected MCP servers. Recomputed every turn (not cached).
\nIncluded when
\nIf MCP servers are connected with instructions
\nOmitted when
\nMCP instruction delta mode is enabled (delivered as per-turn attachment instead).
\n# MCP Server Instructions ## my-database-server Use the query tool for read operations. Always include a LIMIT clause.
\nTells the model to use a session-specific temp directory instead of /tmp.
\nIncluded when
\nIf the scratchpad is enabled
\nIMPORTANT: Always use this scratchpad directory for temporary files instead of `/tmp` or other system temp directories: `/Users/me/.claude/scratchpad/session-abc123`
\nWarns that old tool results will be automatically removed from context.
\nIncluded when
\nIf function result clearing is enabled
\nOld tool results will be automatically cleared from context to free up space. The 5 most recent results are always kept.
\nTells the model to write down important info since originals may be cleared.
\nWhen working with tool results, write down any important information you might need later in your response, as the original tool result may be cleared later.
\nHard word-count limits for responses.
\nIncluded when
\nIf the user is an Anthropic employee
\nLength limits: keep text between tool calls to ≤25 words. Keep final responses to ≤100 words unless the task requires more detail.
\nInstructions for working toward a user-specified token spending target.
\nIncluded when
\nIf the token budget feature is enabled
\nWhen the user specifies a token target (e.g., "+500k", "spend 2M tokens", "use 1B tokens"), your output token count will be shown each turn. Keep working until you approach the target — plan your work to fill it productively.
\nInstructions for using the Brief tool for short replies vs. detailed output.
\nIncluded when
\nIf Brief/KAIROS mode is enabled
\nOmitted when
\nProactive mode is active (brief instructions are already in the proactive section).
\nBrief is where your replies go. Text outside of Brief is in the detail view and mostly goes unread.
\nCurrent branch, recent commits, and working tree status. Appended as system context.
\nIncluded when
\nIf the working directory is a git repository
\nIf not running in a remote session
\nIf git instructions are enabled
\nThis is the git status at the start of the conversation. Note that this status is a snapshot in time, and will not update during the conversation. Current branch: feature-auth Main branch (you will usually use this for PRs): main Git user: Jane Developer Status: M src/auth.ts ?? src/auth.test.ts Recent commits: abc1234 Fix login redirect def5678 Add session middleware
\nExtra text provided via --append-system-prompt, added at the very end.
\nIncluded when
\nIf --append-system-prompt flag is used
\nWhatever text the user passed to --append-system-prompt.
\nAnd this is just the system prompt! There’s similar logic to assemble for…
\nWhen you type instructions and hit ‘Enter’, Claude Code assembles a rich context to increase the odds that it obtains a successful response from Opus or Sonnet. As we can see, agents are more than just models. Context engineering is critical.
\n", "date_modified": "2026-09-15T15:44:24+00:00", "date_published": "2026-04-04T22:14:00+00:00", "external_url": "https://www.dbreunig.com/2026/04/04/how-claude-code-builds-a-system-prompt.html", "id": "urn:aggr:item:81e2ba16c01d3859d72a51a373b4d8e904185dd1", "summary": "Claude Code's leaked source shows a system prompt assembled from dozens of conditional parts.", "tags": [ "programming", "ai", "context engineering", "system prompts" ], "title": "How Claude Code Builds a System Prompt", "url": "https://aggr.aymericbeaumet.com/items/dbreunig-com/2026-04-04-how-claude-code-builds-a-system-prompt/" }, { "authors": [ { "name": "Drew Breunig" } ], "content_html": "Yesterday we talked about how cheap code is fueling an era of idiosyncratic tooling, and previously we’ve talked about the rise of spec driven development. In that second piece, we ran through some of the initial examples of spec driven development with agents:
\n\n\nBy far, the hardest part of starting a SDD project is creating the tests. Which is why many developers are opting for borrowing existing test sets or deriving by referencing a source of truth.
\n\n
\n- Anthropic wrote a C compiler in Rust. They used existing test suites and used GCC as a source of truth for validation and generating new tests.
\n- Vercel created a bash emulator in TypeScript. They created and curated an amazing set of shell script spec tests and have been feeding these to Ralph. (To make this even more meta, I’ve been following their commits and Clauding them into Python).
\n- Pydantic created a Python emulator…in Python. This sounds silly, but it’s useful in the same way Vercel’s
\njust-bashis: it’s a super lightweight sandbox for AI agents. (In fact, I’ve already wrapped it in aCodeInterpreterfor use with DSPy’s RLM module)
The first wave of agentic development brought us clones and ports. When code is incredibly cheap, and you want the code to flow, you can either rely on your own fast feedback or leverage existing test suites. These early projects opted for the latter, as did many tokenmaxxers who are rebuilding their dependencies in Rust or Go.
\nTwo releases this week, however, suggest we’re starting to enter a second phase of open source agentic coding projects. The first brought us clones, this next phase brings us reimaginings. Consider the following two projects:
\nThe Cloudflare post really spells out the pattern we’re seeing here: the team looked at all the jobs people hire WordPress to perform and asked, how would we solve those if we started today?
\n\n\nWordPress powers over 40% of the Internet. It is a massive success that has enabled anyone to be a publisher, and created a global community of WordPress developers. But the WordPress open source project will be 24 years old this year. Hosting a website has changed dramatically during that time. When WordPress was born, AWS EC2 didn’t exist. In the intervening years, that task has gone from renting virtual private servers, to uploading a JavaScript bundle to a globally distributed network at virtually no cost. It’s time to upgrade the most popular CMS on the Internet to take advantage of this change.
\n
What they ended up with is something fast, serverless, and secure. They didn’t clone WordPress, they reimagined it by focusing on the job to be done.
\nCheng Lou took the same route: he didn’t port CSS to Go or Rust to get his speed gains. Rather, he focused on one hard, important job to be done that CSS doesn’t do very well and reimagined it with everything we’ve learned and built, unhindered by the baggage of CSS.
\nNow what does this have to do with coding agents? Isn’t this something we could have done (and did) before?
\nPreviously, the built up ecosystems and mature code of existing software projects made reimagining foolish. Teams did create modern CMS projects, but WordPress’ massive size and momentum meant newcomers could only carve out small niches of adoption, if they didn’t fail entirely. The odds weren’t good and the costs of trying were high, so most people sighed and moved on.
\nCoding agents make reimagining practical because the cost to perform them is so, so much lower. Code is cheap. We can take more shots, more often, to counter the embedded standards.
\nFurther, reimagining a new standard was a long road. If you survived the initial build and executed a good launch, picking up a small core of users, you earned the job of writing countless bugfixes, optimizations, and security patches. Mature software like WordPress is battle tested; it’s had decades of feedback and found flaws. How can a newcomer compete?
\nWell, Cheng Lou demonstrates an interesting approach, again using agents:
\n\n\nThe engine’s tiny (few kbs), aware of browser quirks, supports all the languages you’ll need, including Korean mixed with RTL Arabic and platform-specific emojis.
\nThis was achieved through showing Claude Code and Codex the browsers ground truth, and have them measure & iterate against those at every significant container width, running over weeks.
\n
With agents and LLMs, we can synthetically test our new tools, patching them as we challenge them.
\nI think we’re going to see a lot more reimaginings, where people attack old problems with modern tactics. Coding agents lower the costs of taking on stalwarts and raise our ability to rapidly harden our software. I can think of many software tools that people rely on but don’t like. Those are the prime targets for reimagining.
\n", "date_modified": "2026-09-15T15:44:24+00:00", "date_published": "2026-04-01T23:26:00+00:00", "external_url": "https://www.dbreunig.com/2026/04/01/the-2nd-phase-of-agentic-development.html", "id": "urn:aggr:item:b1348ee8a021c047629cee63c1cbb600642fe628", "summary": "The first wave of agentic coding gave us clones and ports. The next wave attacks old problems with new designs, hardened by synthetic tests.", "tags": [ "programming", "ai", "coding", "software engineering", "spec driven development" ], "title": "The 2nd Phase of Agentic Development", "url": "https://aggr.aymericbeaumet.com/items/dbreunig-com/2026-04-01-the-2nd-phase-of-agentic-development/" }, { "authors": [ { "name": "Drew Breunig" } ], "content_html": "![\"\"](\"https://www.dbreunig.com/img/winchester.jpg\")
In 1998, Eric S. Raymond published the founding text of open source software development, “The Cathedral and the Bazaar”. In it, he detailed two methods of building software:
\nThe Bazaar model was enabled by the internet, which allowed for distributed coordination and distribution. More people could contribute code and share feedback, yielding better, more secure software. “Given enough eyeballs, all bugs are shallow,” Raymond wrote, coining Linus’ Law.
\nThe ideas crystallized in “The Cathedral and the Bazaar” helped kick off a quarter-century of open source innovation and dominance.
\nBut just as the internet made communication cheap and birthed the Bazaar, AI is making code cheap and kicking off a new era filled with idiosyncratic, sprawling, cobbled-together software.
\nMeet the third model: the Winchester Mystery House.
\n![\"https://www.flickr.com/photos/harshlight/3669393933\"](\"https://www.dbreunig.com/img/winchester_02.jpg\")
Located less than 10 miles southeast from the Computer History Museum, the Winchester Mystery House is an architectural oddity.
\nFollowing the death of her husband and mother-in-law, Sarah Winchester controlled a fortune. Her shares in the Winchester Repeating Arms Company, and the dividends they threw off, made it so Sarah could not only live in comfort but pursue whatever passion she desired. That passion was architecture.
\nSarah didn’t build her mansion to house ghosts1, she built her mansion because she liked architecture. With no license, no formal training, in an era when women (even very rich women) didn’t have a path to practicing architecture, Sarah focused on her own home. She made up for her lack of license with passion and effectively unlimited funds.
\nSarah built what she wanted. “At its largest the house had ~500 rooms.” Today it has roughly 160 rooms, 2,000 doors, 10,000 windows, 47 stairways, 47 fireplaces, 13 bathrooms, and 6 kitchens. Carved wood drapes the walls and ceilings. Stained glass is everywhere. Projects were planned, completed, abandoned, torn down, and rebuilt.
\nIt was anything but aimless. And practical innovations ran throughout, including push-button gas lighting, an early intercom system, steam heating, and indoor gardens. The oddities that amuse today’s visitors were mostly practical accommodations for Sarah’s health (stairways with very small steps), functional designs no longer used (trap doors in greenhouses to route excess water), or quick fixes to damage from the 1906 earthquake.
\nWinchester passed in 1922. Nine months later, the house became a tourist attraction.
\nToday, many programmers are Sarah Winchester.
\nClaude Code's Public GitHub Activity
\nLines Added Lines Deleted
\nWe aren’t as rich as Sarah Winchester, but when code is this cheap, we don’t need to be.
\nJodan Alberts illustrated this recently, collecting and visualizing data detailing public Github commits attributed to Claude Code. That’s his data in the chart above, with Claude seeming to only accelerate through March2.
\nIt’s hard to get a handle on individual usage, though, so I went searching for a proxy and landed on the chart below:
\nAverage Net Lines Added Per Commit in Claude Code
\n7-Day Average
\nAfter Opus 4.5 and recent work enabling Agent Teams, the average net lines added by Claude per commit is now smooth and steady at 1,000 lines of code per commit3.
\n1,000 lines of code per commit is ~2 magnitudes higher than what a human programmer writes per day.
\nIf you search for human benchmarks, you’ll find many citing Fred Brooks’ The Mythical Man Month while claiming a good engineer might write 10 cumulative lines of code per day4. If you further explore, you’ll find numbers higher than 10 cited, but generally less than 100.
\nHere’s a good anecdote from antirez on a Hacker News thread discussing the Brooks “quote”:
\n\n\nI did some trivial math. Redis is composed of 100k lines of code, I wrote at least 70k of that in 10 years. I never work more than 5 days per week and I take 1 month of vacations every year, so assuming I work 22 days every month for 11 months:
\n70000/(22 x 11 x 10) = ~29 LOC / day
\nWhich is not too far from 10. There are days where I write 300-500 LOC, but I guess that a lot of work went into rewriting stuff and fixing bugs, so I rewrote the same lines again and again over the course of years, but yet I think that this should be taken into account, so the Mythical Man Month book is indeed quite accurate.
\n
6 years after this comment, Claude is pushing 1,000 lines of code per commit.
\nSo what do we do with all this cheap code?
\nUnfortunately, everything else remains roughly the same cost and roughly the same speed. Feedback hasn’t gotten cheaper; the “eyeballs” that guided the software developed by the bazaar haven’t caught up to AI.
\nThere is only one source of feedback that moves at the speed of AI-generated code: yourself. You’re there to prompt, you’re there to review. You don’t need to recruit testers, run surveys, or manage design partners. You just build what you want, and use what you build.
\nAnd that’s what many developers are doing with cheap code: building idiosyncratic tools for ourselves, guided by our passions, taste, and needs.
\nSound familiar?
\n![\"https://commons.wikimedia.org/wiki/File:Winchester_Mystery_House_2023-07-17_02.jpg\"](\"https://www.dbreunig.com/img/winchester_03.jpg\")
Steve Yegge’s Gastown is a Winchester Mystery House. It’s incredibly idiosyncratic and sprawling, rich with metaphors and hacks. It’s the perfect tool for Steve.
\nJeffrey Emanuel’s Agent Flywheel is a Winchester Mystery House. A significant subset of tokenmaxxers decide they need to rebuild their dependencies in Rust; Jeff is one such example. His “FrankenSuite” includes Rust rewrites of SQLite, Node, btrfs, Redis, Pandas, NumPy, JAX, and Torch.
\nPhilip Zeyliger noted the pattern last week, writing, “Everyone is building a software factory.” But it goes beyond software. Gary Tan’s personal AI committee gstack is a Winchester Mystery House constructed mostly from Markdown.
\nEverywhere you look, there are Winchester Mystery Houses.
\nEach Winchester Mystery House is idiosyncratic. They are highly personalized. The tightly coupled feedback loop between the coding agent and the user yields software that reflects the developer’s desires. They usually lack documentation. To outsiders, they’re inscrutable.
\nWinchester Mystery Houses are sprawling. Guided by the needs of the developer, these tools tend to spread out, constantly annexing territory in the form of new functions and new repositories. Work is almost always additive. Code is added when it’s needed, bugs are patched in place, and countless appendages remain. There’s little incentive to prune when code is free.
\nAnd building a Winchester Mystery House should be fun. Coding agents turn everything into a sidequest, and we eagerly join in. Building the perfect workflow is a passion for many devs, so we keep pushing.
\nWinchester Mystery Houses are idiosyncratic, sprawling, and fun. But does this mean we’re abandoning the bazaar?
\n![\"https://www.flickr.com/photos/ifpri/4860343116\"](\"https://www.dbreunig.com/img/crowded_bazaar.jpg\")
What happens when we all tend to our Mystery Houses? When our free time is spent building tools just for ourselves, will we stop working on shared projects? Will we abandon the bazaar?
\nProbably not. The bazaar is packed right now, but not in a good way.
\nCode is cheap, so people are slamming open source repositories with agent-written contributions, in an attempt to pad their resumes or manifest their pet features. Daniel Stenberg ended bug bounties for curl after a deluge of poor submissions sapped reviewer bandwidth. It’s gotten so bad, Github recently added a feature to disable pull request contributions.
\nAnecdotally, I’m seeing good contributions pick up as well. They’re just drowned out by the slop. For what it’s worth, curl commits are dramatically up in the agentic era. And people are sharing what they build. A recent analysis by Dumky shows more packages and repos rising in the last quarter.
\n![\"\"](\"https://www.dbreunig.com/img/resized/agent_sharing.jpg\")
There’s plenty of budget for both Mystery Houses and the bazaar when code is this cheap. The new challenge is developing systems and processes for managing the deluge. We don’t need eyeballs to find bugs in the software, we need eyeballs to find bugs before they reach the software.
\nIn many ways this is the inverse of the bazaar model era. The internet made feedback and communal coordination faster, easier, and cheaper. The bazaar model has a high throughput of feedback (many eyeballs) but relatively high latency for modifications (file an issue, discuss, submit a PR, wait for review, etc.)
\nCoding agents, on the other hand, make implementation faster while feedback and coordination are unchanged. The Winchester Mystery House model sidesteps this by collapsing the feedback loop into one person: latency is near zero, but throughput is just you. The bazaar, defined by communal work, can’t adopt this hack. Coding agents in the bazaar create a mess: implementation at machine speed hitting coordination infrastructure built for human speed. Which is why maintainers feel like they’re drowning.
\nWe need new tools, skills, and conventions.
\nCoding agents have dropped the cost of code so dramatically, we’re entering a new era of software development, the first change of this magnitude since the internet kicked off open source software. Change arrived quickly, and it’s not slowing down. But in reviewing the Winchester Mystery House framework, I think we can take away a few lessons.
\nLesson 1: The bazaar and Winchester Mystery Houses can coexist.
\nWhen listing example Winchester Mystery Houses, I didn’t mention OpenClaw, even though it is the defining example. I saved it for here because it nicely illustrates how Winchester Mystery Houses and the bazaar can coexist.
\nOpenClaw is incredibly modular and places few limitations on the user. It integrates 25 different chat and notification systems, plugs into most inference end points, and is built on the exceptionally flexible pi agent toolkit. This eager flexibility was embraced early – security and data protections be damned – but since its exponential adoption Peter Steinberger and the community have been steadily pushing improvements and fixes.
\nAnd like other breakout open source projects of yore, the ecosystem is adopting the best ideas and mitigating the worst aspects of OpenClaw. Countless alternate “claw” projects have emerged (there’s NanoClaw, NullClaw, ZeroClaw, and more!) Companies have launched services to make claws easy or safer. Cloudflare launched Moltworker to make deploy easy, Nvidia shipped NemoClaw with a security focus, and Claude keeps adding claw-like features to its desktop app.
\nLesson 2: Don’t sell the fun stuff.
\nOne reason OpenClaw works so well in the bazaar is that it is a foundation for personal tools. Out of the box, a claw just sits there. It’s up to the user to determine what it does and how it does it, leveraging the connections and infrastructure OpenClaw provides. OpenClaw lets less experienced developers spin up their own Winchester Mystery Houses, while experienced devs get to leverage much of the common integrations and systems OpenClaw provides. Peter and team have done a great job drawing a line between the common core (what the bazaar works on) and what they leave up to the user: the boring, critical stuff is the job of the commons.
\nThinking back to Sarah Winchester and her idiosyncratic, sprawling mansion, we see the same pattern. Sarah hired vendors! She used off-the-shelf parts! Her bathtubs, toilets, faucets, and plumbing weren’t crafted on site.
\nThe boring stuff, the hard bits, or the things that have disastrous failure modes are the things we should collaborate on or employ specialists to handle. (Come to think, plumbing checks all three boxes). This is the opportunity for open source software, dev tools, and software companies.
\nDon’t try to sell developers the stuff that’s fun, the stuff they want to build. Sell them the stuff they avoid or don’t want to take responsibility for. Sarah Winchester didn’t hire metalworkers to craft the pipes for her plumbing, but she did hire craftspeople to create hundreds of stained-glass windows to her specs.
\nLesson 3: The limits of code are communication.
\nOpenClaw shows the bazaar remains relevant, but also highlights the problems facing open source in the agentic era. Right now, there are 1,173 open pull requests and 1,884 new issues on the OpenClaw repo.
\nThere is more code and more projects than we could ever review. The challenge now, for open source maintainers and users, is sifting through it all. How do we find the novel ideas that everyone should adopt and borrow?
\nOpenClaw is one of the successes, something we all noticed. And for it, the problem is processing the feedback. For the projects we’ll never find, the ones lost in the deluge, their problem is lack of feedback. You either find attention and drown in contributions or drown in the ocean of repos and never hear a thing.
\nThe internet made coordination cheap and gave us the bazaar. Coding agents made implementation cheap and gave us the Winchester Mystery House. What we’re missing are the tools and conventions that make attention cheap, that let maintainers absorb contributions at machine speed and let good ideas surface among the noise. Until we figure this out, the bazaar will keep getting louder without getting smarter, and the best ideas in our Mystery Houses will be forgotten once we stop maintaining them.
\nThe lore that Winchester built her mansion to house ghosts killed by Winchester rifles is likely just gossip and marketing. There’s little evidence to support these claims. (99% Invisible has a good episode exploring Winchester, her house, and this lore.) ↩
\nWhile editing this piece, Dumky published another analysis illustrating the production of coding agents. In it he shows a 280% increase in “Show HN” posts, a 93% increase in new Github repos, and a dramatic uptick in packages published to Crates.io. ↩
\nAnthropic’s ability to stabilize this line is rather impressive. Claude code is getting better at planning, better at chunking out work, enabling more effective sub-agent delegation. ↩
\nThough this is likely an updated tweak of Brooks’ statement that an “industrial team” might write 1,000 “statements” per year. ↩
\nThe following is a write up of a talk I delivered at MLOps Community’s “Coding Agents” conference, on March 3rd. There’s a video version of the talk available on YouTube.
\nI share what I learned building a no-code library, why spec-driven development is a feedback loop not a straight line, historical parallels for our current moment, and a PoC tool for keeping specs/tests/code in sync.
\nFinally, we consider what GitHub should look like in the era of coding agents.
\n![\"\"](\"https://www.dbreunig.com/img/sdd_triangle/sdd_triangle_001.jpg\")
I was invited here today to talk about a project I launched — a software library with no code — which got a lot of really interesting feedback. I’m going to tell you the whole story, how I got it wrong, explore a bit of historical context, then propose a path forward.
\n![\"\"](\"https://www.dbreunig.com/img/sdd_triangle/sdd_triangle_002.jpg\")
Last Fall, Opus 4.5 launched and surprised everybody with the quality of the code it was produced and the problems it could solve. Opus 4.5 was good enough that we started to ask some really big questions.
\nI wondered: if the agents are good enough, why do we need to share code?
\nWhenever I have a big question like this, one that requires lots of thought, I like to go for a long bike ride. So I did, and while I was riding I came up with the idea to ship a software library with no code.
\n![\"\"](\"https://www.dbreunig.com/img/sdd_triangle/sdd_triangle_003.jpg\")
And so we have whenwords.
\nOpen source, freely licensed. It’s a GitHub repository with a markdown file describing what the library is supposed to do. It’s a library that takes a Unix timestamp and converts it into something human-readable — “about 12 o’clock,” “five hours ago,” things like that.
\nI also generated about 750 conformance tests in YAML: given this input, I expect this output. And one more file called install.md — a single paragraph you’d paste into the agent of your choice, with all the instructions for building the code. You’d drop in what language you wanted and where to save it.
![\"\"](\"https://www.dbreunig.com/img/sdd_triangle/sdd_triangle_004.jpg\")
whenwords kicked off a lot of conversation about spec-driven development. It’s something more and more people are thinking about: the idea that if you bring specs, which define the what, why, and sometimes how, and tests, which measure and validate behavior, the code will just flow from that. Give it to an agent, get code out.
![\"\"](\"https://www.dbreunig.com/img/sdd_triangle/sdd_triangle_005.jpg\")
whenwords kind of blew up. Karpathy was a fan. whenwords has over 1,000 stars on Github.
What was even crazier was that I started getting normal GitHub interactions. People submitted issues. They submitted pull requests. And the pull requests were good, things like: “In this test, you’re expecting this result, but that violates the rounding rule you detail in the spec. You need to true these up.”
\n![\"\"](\"https://www.dbreunig.com/img/sdd_triangle/sdd_triangle_006.jpg\")
But I wasn’t the only one with this idea. Larger teams started shipping larger projects. whenwords was a toy; small, constrained, 750 tests. But then:
\nVercel released just-bash, a simulated bash environment with an in-memory virtual filesystem, written in TypeScript. Basically re-implementing Bash in TypeScript. They’re running shell scripts against it to verify behavior.
\nPydantic released Monty, a Python interpreter written in Rust. Fast, safe, ideal for agent REPLs and code use. Same approach: a pile of Python tests, throw it at the model, make it pass.
\nAnthropic famously threw 16 Claudes and $20,000 at a spec suite to build a Rust-based C compiler. It didn’t really work. But it was pretty cool.
\nI couldn’t stop thinking about Spec Driven Development and how far we might push it.
\n![\"\"](\"https://www.dbreunig.com/img/sdd_triangle/sdd_triangle_007.jpg\")
I think there’s a few learnings from this first wave of Spec Driven Development.
\nTests and specs aren’t free or easy. All the projects we surveyed used large existing testing libraries from existing projects: the Bash tests, the Python tests, the C tests. Those are the low-hanging fruit. I joked online (and I’m not the first) that pretty soon anyone who wants to protect themselves is going to be like SQLite, where the code is free but you’ve got to pay for the tests. Tests are precious.
\nImplementation is fast, but not instant. You go fast at first, but none of these projects are complete. just-bash is still being worked on. Monty is missing JSON and other standard libraries. Anthropic’s C compiler stalled out. It gets hard. It’s not perfect and it’s not free.
\nAs complexity grows, structural choices become more important. This was especially clear in the Anthropic project. They got incredibly far, down to 1% of failing tests. But every time they fixed a new bug, it broke something else. Systemic changes required systemic thinking, not just local fixes.
\nArchitectures that allow parallel development are incredibly valuable. What it allows you to do is move fast with multiple agents. And, this is something I haven’t seen explored yet, it allows for open source contribution. Rather than spending $20,000 to build a C compiler, what if you structured it so everyone knows what chunk they can work on? It’s like SETI@home, except I’m not using your engineering expertise. I’m using your Claude Code subscription. Which I think is wild.
\nBut the biggest learning — and the one we’re going to spend the most time on today — is that sometimes the spec and tests aren’t sufficient.
\n![\"\"](\"https://www.dbreunig.com/img/sdd_triangle/sdd_triangle_008.jpg\")
One of my favorite things to do is look at the PRs and issues for all of these libraries. Even with a great spec — “make it run Python perfectly in Rust, here are all the tests, just make them pass” — there are still 20-comment threads about what the right way to implement something is.
\nBecause no spec is perfect. And this is probably my biggest takeaway today:
\nImplementing the code helps us improve our spec.
\n![\"\"](\"https://www.dbreunig.com/img/sdd_triangle/sdd_triangle_009.jpg\")
Let’s take a digression. We’re at the Computer History Museum, so let’s go back into history. Specifically the history of code and managing code.
\n![\"\"](\"https://www.dbreunig.com/img/sdd_triangle/sdd_triangle_010.jpg\")
One of my favorite jokes about AI development is one I stole from Matt Levine, who writes the finance newsletter Money Stuff. In it, he has a running joke about crypto people speed-running financial history, from first principles, as they attempt to build new financial infrastructure. We are doing that with software engineering and AI coding.
\nI’m lucky: one of my co-founders, Heather Miller, is a professor at CMU and a programming languages expert. I can call her up, share my theories, and ask: “Heather, tell me this has already been dealt with. Who should I be talking to and what shoud I read?” This time, she said, “Of course it has, Drew,” and introduced me to her office neighbor, Professor Claire Le Goues. Claire then who walked me through the relevant software engineering history, which I’m going to share today because it is incredibly relevant to our current situation.
\n![\"\"](\"https://www.dbreunig.com/img/sdd_triangle/sdd_triangle_011.jpg\")
In 1963, Margaret Hamilton was writing and managing software effort for NASA’s Apollo missions. She coined the term “software engineering” because, running this giant, complicated project that couldn’t have errors in it, she realized: this is engineering. It’s systems design, we have to worry about errors, we have to worry about unexpected inputs like astronauts pushing the wrong button.
\nAnd also: we now have enough code that no one person can hold it in their head. Which is a problem, because then you can’t reason about it effectively. And it gets even worse when a team is working on it.
\n![\"\"](\"https://www.dbreunig.com/img/sdd_triangle/sdd_triangle_012.jpg\")
By the way: this is her code. This is what she was managing. This is her VS Code.
\n![\"\"](\"https://www.dbreunig.com/img/sdd_triangle/sdd_triangle_013.jpg\")
And this is her Git.
\nI’m a dad, which means dad jokes come naturally. So I’m going to retroactively coin Hamilton’s Law: when you can’t see over your code, you can’t oversee your code.
\n(Sorry.)
\n![\"\"](\"https://www.dbreunig.com/img/sdd_triangle/sdd_triangle_014.jpg\")
After Hamilton dealt with this problem, others realized it was a problem too.
\nNATO held a conference in Berlin and identified the “Software Crisis”: computer hardware now allowed programs so complex they couldn’t be managed adequately. A single engineer couldn’t hold all the code in their head. If they were going to continue delivering on what software could promise, they needed process.
\n![\"\"](\"https://www.dbreunig.com/img/sdd_triangle/sdd_triangle_015.jpg\")
Dijkstra popularized this in his 1972 Turing Award lecture. He said:
\n\n\nAs long as there were no machines, programming was no problem at all. When we had a few weak computers, programming became a mild problem. And now we have gigantic computers, programming has become an equally gigantic problem.
\n
He said this in 1972. Maybe later, walk around the museum we’re in and look at what he was working with then. Then consider what we’re working with today.
\n![\"\"](\"https://www.dbreunig.com/img/sdd_triangle/sdd_triangle_016.jpg\")
So after the Software Crisis emerged, we wandered through the desert of processes, searching for one to borrow. We looked at manufacturing engineering. In 1975, Brooks published The Mythical Man-Month. And finally, Waterfall was adopted as a DoD standard. We learned how to engineer complex software. Progress.
\n![\"\"](\"https://www.dbreunig.com/img/sdd_triangle/sdd_triangle_017.jpg\")
But these things move in cycles. In 2001, we published the Agile Manifesto. Zuckerberg said it’s time to move fast and break things. We embraced Agile, and Agile was finally realized by the cloud and GitHub — which enabled continuous CI/CD and let us offload enough of the error-checking that we didn’t break things too often, even when moving fast.
\n![\"\"](\"https://www.dbreunig.com/img/sdd_triangle/sdd_triangle_018.jpg\")
Which brings us to today.
\n![\"\"](\"https://www.dbreunig.com/img/sdd_triangle/sdd_triangle_019.jpg\")
I added this slide right at the last minute, because I logged into Twitter to check something and saw today’s trending news: “AI Coding Boosts Output But Overwhelms Human Reviews.” And it’s paired with that last headline: “OpenAI Codex Leaks Hint at GPT-5.4 Amid Speedy Updates.” So not only is it overwhelming us, it’s accelerating.
\n![\"\"](\"https://www.dbreunig.com/img/sdd_triangle/sdd_triangle_020.jpg\")
So what do we learn from this history rabbit hole?
\nBeing overwhelmed by the volume of code isn’t a new problem. It’s what birthed software engineering.
\nThe initial Software Crisis was our inability to manage complex codebases new computers allowed. Our current Software Crisis is our inability to manage complex codebases new models allow.
\nOur problem used to be that we couldn’t hold an entire codebase in our head. Now we can’t even read our entire codebase.
\nAgentic engineering enables waterfall volume at the cadence of agile. And even that undersells it: it’s waterfall times ~two at the cadence of agile times ~seven.
\nWe keep oscillating, historically, between unhindered velocity and managed process. We could use some process right about now. Perhaps AI can help…
\n![\"\"](\"https://www.dbreunig.com/img/sdd_triangle/sdd_triangle_021.jpg\")
I’m not the only one asking this question.
\nFor the last couple of quarters, people have been trying to figure out how to deal with this onslaught of code. The most dramatic example is Gas Town — you’re all familiar with it — an infrastructure for managing a coding process that grew beyond one person’s ability to manage.
\nBut Gas Town just moves the problem. It doesn’t solve it. Steve Yegge even admits this in the Gas Town blog post:
\n\n\nGas Town is complicated. Not because I wanted it to be, but because I had to keep adding components until it was a self-sustaining machine. And the parts that it now has, well, they look a lot like Kubernetes mated with Temporal and they had a very ugly baby together.
\n
![\"\"](\"https://www.dbreunig.com/img/sdd_triangle/sdd_triangle_022.jpg\")
If the process is complex, we’re just moving the problem.
\n![\"\"](\"https://www.dbreunig.com/img/sdd_triangle/sdd_triangle_023.jpg\")
So let’s go back to what we defined spec-driven development to be. This idea that it’s an equation: bring specs, maybe add some tests, add an agent, get code out.
\nI got this wrong. This is the wrong way to think about it. Because this isn’t a one-way equation. It’s a feedback loop. The act of writing code improves the spec, and it improves the tests. Just like software doesn’t really work until it meets the real world, a spec doesn’t really work until it’s implemented.
\n![\"\"](\"https://www.dbreunig.com/img/resized/sdd_triangle/sdd_triangle_024.jpg\")
So instead of an equation, I propose a triangle. The spec defines what tests need to be written, and what code needs to be written. Tests validate the code. That’s essentially what we had before, just in a different shape.
\n![\"\"](\"https://www.dbreunig.com/img/sdd_triangle/sdd_triangle_025.jpg\")
But the act of implementing code generates new decisions. Those decisions inform the spec. And when the spec updates, new tests need to be written. And sometimes it’s not new decisions — it’s just dependencies or subtle choices. New code surfaces new behaviors that need to be tested.
\n![\"\"](\"https://www.dbreunig.com/img/sdd_triangle/sdd_triangle_026.jpg\")
I call this: the Spec-Driven Development Triangle.
\nAs each node moves forward, our job — and our tooling’s job — is to keep those nodes in sync. That’s the job. If we improve the code, we must improve the spec.
\n![\"\"](\"https://www.dbreunig.com/img/sdd_triangle/sdd_triangle_027.jpg\")
But keeping the nodes in sync is hard.
\nWriting tests is hard. Even before agents, we couldn’t write tests. We don’t like writing tests and we’d prefer not to.
\nWriting specs is hard. They can never be exhaustive, leave room for interpretation, and are written before the software meets the real world. The spec gets written, it gets implemented, it gets released. Is the spec updated? No.
\nSpecs are written at a different cadence than code, in a different medium. If only we had something that could read natural language.
\nUpdating specs and tests feels like overhead, especially when you’re moving fast. And the entire point of using agents is to move fast. Any system we design has to respect that.
\nImplementation is messy, and often humans and LLMs take shortcuts. Humans say “I’m not going to implement that right now” or “I’ll come back and fix this.” LLMs certainly do this.
\nAnd so regular reconciliation of tests, spec, and code is not part of the process.
\n![\"\"](\"https://www.dbreunig.com/img/sdd_triangle/sdd_triangle_028.jpg\")
But thankfully, there are signals we can work with.
\nCode changes are tracked by Git, and we can compare them against the spec to find gaps.
\nTest coverage tools tell us what code is tested — but not whether the tests reflect the spec. It’s not just about covering the code. The tests have to cover the spec.
\nUpdates to the spec — if a product manager logs in and changes something — are also tracked by Git. Is the rest of the system changing with it?
\nBug reports and hotfixes that go straight into code or tests need to be captured and rolled into the spec.
\nAnd most importantly: implementing the code with an agent generates decisions — from both the humans and the agent. Those decisions exist in the traces. We can look at the traces from our coding agents and find where decisions were made. That’s the signal we need to keep everything in sync.
\n![\"\"](\"https://www.dbreunig.com/img/sdd_triangle/sdd_triangle_029.jpg\")
So we have tangible things we can analyze. And a goal to aim towards…
\nOne of the the nice thing about having a thought experiment during the era of great coding agents is that you can just try building it. And as you implement it, you improve it.
\n![\"\"](\"https://www.dbreunig.com/img/sdd_triangle/sdd_triangle_030.jpg\")
This is my tool. I call it Plumb, after a plumb bob, because it keeps things true. A plumb bob hangs from a line and helps a carpenter keep things straight. Even better, they used to be held on tripods, which echoes the triangle.
\nYou can install it right now: pip install plumb-dev or uv add plumb.
It’s not perfect. It’s a proof of concept. A thought experiment as code. But I’ve been using it, and it’s pretty great.
\n![\"\"](\"https://www.dbreunig.com/img/sdd_triangle/sdd_triangle_031.jpg\")
Here’s how Plumb works.
\nPlumb is a command line tool. Every time you’re working with an agent and you run git commit, it identifies decisions made by evaluating the code diff from the last commit and by reading the agent traces (all the conversations since that last commit). It extracts the decisions, dedupes them, and presents them to you: here are all the decisions you made, do you agree?
Once you’ve approved, it updates the spec to reflect those decisions. It runs sync and reports on coverage gaps between the spec and the tests, and the spec-to-code coverage. Is the code actually reflecting what the spec defines?
![\"\"](\"https://www.dbreunig.com/img/sdd_triangle/sdd_triangle_032.jpg\")
As it does this, it generates files that become artifacts you can track. My favorite is a big JSONL file of decisions.
\nHere’s one example: “Should spec updates be batched across all decisions, or run individually for each decision?” My decision — batch them. It says it was made by the user, not the LLM. I have blame. And you can see how we can enrich this over time: tie it to code, to branch, to whether it was informed by the conversation, when it was approved, when it was synced. This is not just the code changes. It’s the intent.
\n![\"\"](\"https://www.dbreunig.com/img/sdd_triangle/sdd_triangle_033.jpg\")
To set up Plumb in your project: install it, go to your project directory, run plumb init. It’ll ask you to specify your spec markdown file or folder and show it where your tests are. It creates a .plumbignore to tell it when to skip decision generation — changing the README, for example, doesn’t need to generate decisions. It creates a .plumb folder to store state and config. Very similar to .git.
Most importantly: it adds hooks to Git. When you run git commit, it extracts the decisions. If there are decisions to review, the commit fails. It exits and tells you to review your decisions and approve, reject, or edit them. That’s what makes this work anywhere: command line, CI pipeline, inside your coding environment. It just works. And that’s a hard requirement.
![\"\"](\"https://www.dbreunig.com/img/sdd_triangle/sdd_triangle_034.jpg\")
The other thing Plumb generates is a breakdown of your spec into individual requirements — the atomic statements that make up what your spec defines. Ambiguous or not, what source file it came from, eventually linked directly to the code. Right now I use a commenting format to link tests back to the requirement they’re testing, so coverage mapping can show which requirements have tests and how many.
\n![\"\"](\"https://www.dbreunig.com/img/sdd_triangle/sdd_triangle_035.jpg\")
Our aim is link spec to requirements, requirements to code, requirements to tests, decisions to requirements. We’re building a new object graph extending off the code diffs. And eventually — edit the spec, the tests, or the code, pick your poison, and everything else gets brought along.
\n![\"\"](\"https://www.dbreunig.com/img/sdd_triangle/sdd_triangle_036.jpg\")
Now, as you design this, the interesting design choices start to emerge.
\nCan’t this just be a skill? There are already code review skills, superpowers, things like that. Why not just use those?
\nI don’t think it can be a skill. Whatever tool we end up using for tracking decisions and intent, it cannot live only inside the agent. It needs to run outside. It needs to handle small commits, triggers, anything…even if you never touch the agent.
\nA skill is a suggestion. A tool needs to be a checkpoint. That commit-fail mode is essential. Otherwise it gets ignored. We’ve all had this happen with Claude Code.
\nAnd the system needs to be canonical. It can’t be optional. Agents wander. Validation needs to be more deterministic. When we can use code, we will. This is a validation and verification step. Fuzzy LLM calls are a last resort.
\nWhen we do use LLMs — parsing the spec, extracting decisions — we use DSPy. It lets us structure LLM calls with tight inputs and outputs. It lets us optimize, test, and choose which models to route to. Speed matters enormously here. For decision deduplication, I’m routing to GPT, because it’s faster than anything Anthropic offers for that task. And the whole thing has to be simple enough for the developer to hold in their head.
\n![\"\"](\"https://www.dbreunig.com/img/sdd_triangle/sdd_triangle_037.jpg\")
Of course, there are real limitations.
\nPlumb only supports pytest. I want it to support any test framework and conformance tests, such as language-agnostic tests like whenwords used.
Decisions might interrupt your flow on long-running tasks. If I make a quick fix and generate five decisions, I have to sit through a review. That needs to be tunable. Maybe you don’t want it to bother you for lightweight decisions, only surface things that are vague or contradict previous decisions. I suspect this is something that will be dictated by the type of project you’re working on.
\nDeduplication isn’t perfect. Decision identification is fuzzy and will likely need to be project-specific.
\nCode reversals on decision rejection aren’t working yet. I’d like it so that when you reject a decision the LLM made, it goes back and undoes it. The reason it’s not implemented is that the flow needs to be right: if you reject from the command line, nothing automatic should happen. If you reject from inside the agent, the agent should act on it.
\nIt needs better tools for managing the spec. Mine has grown long and probably should be sharded into sections. Thankfully, this is something an LLM can and should do. Though, we have to be careful when doing it. Perhaps we can perform dry runs, regenerate requirements from the shards, then confirm they match the original spec…
\nPlumb should be tunable for “just enough” structure. Can I run with --dangerously-approve-all-decisions? Sometimes I want to.
And it’s untested on large projects. Hell, it’s untested in general.
\n![\"\"](\"https://www.dbreunig.com/img/sdd_triangle/sdd_triangle_038.jpg\")
But here’s the fun part: I’ve been testing this by using it to manage the project itself. Using Plumb to build Plumb. And it’s been genuinely useful.
\nClaude can refer to the spec for implementation understanding without searching the entire codebase. The decision log has proven valuable for answering “why does this code exist?” — I can ask the agent, “is there a decision we made that explains why this is implemented this way?” And it can find it.
\nIt’s code review, but code review where we capture intent. When I hit commit in Claude Code, I get a list of decisions and I step through them. Sometimes I hit one I don’t like and I stop right there. I reject it, go back, redo it. I like that better than pure code review.
\nIt actually spots and controls weird silent LLM behavior. We’ve all let an agent run while we answer email and come back to something insane. Now I get a decision and I can say “don’t do that, let’s roll that back.”
\nAnd hacks get documented. I’ve taken shortcuts in this app. Now I know they exist. I can search back for all the shortcuts and then go fix them. The decision log becomes an artifact — not just of code changes, but of intent.
\n![\"\"](\"https://www.dbreunig.com/img/sdd_triangle/sdd_triangle_039.jpg\")
So let’s take this question further. Say Plumb exists and does exactly what I want. How could GitHub be better with this kind of information?
\nRight now, the main way we interact with code is with Markdown and chat. And GitHub has not changed anything about how we interact with Markdown and text on their site. Could my Markdown diffs have decisions linked to them, so I can see how intent manifests in the code?
\nI think any version of GitHub that takes the agentic era seriously needs to do four things:
\nSpec, tests, and code have to be first-class citizens. Code is already. Tests are close — GitHub Actions gets you there. But Markdown is not. Microsoft is probably leaving a lot of inference revenue on the table by not treating it seriously.
\nMarkdown has to be a first-class citizen. This is the gap.
\nWe need to see the linkages. Users need to follow connections between decisions, requirements, code, tests, and spec. Spec-driven development right now is treated as a one-shot thing: write the spec, hit go, you’re done. It’s not. It’s a process. You need to track all of it over time.
\nUsers should be able to ask questions of the system. Not just read it — query it, to get closer to understanding intent. That’s how you actually understand a codebase that’s too large to read.
\n![\"\"](\"https://www.dbreunig.com/img/sdd_triangle/sdd_triangle_040.jpg\")
So here are my takeaways from the journey from whenwords to Plumb.
\nCode implementation clarifies and communicates intent. I could stop there and walk out of the room. I missed this with whenwords.
The job is to keep specs, code, and tests in sync as they move forward. The system for managing that has to stay simple. If it creates developer mental overhead, it just moves the problem somewhere else.
\nThe act of writing code improves the spec and the tests. Just like software doesn’t truly work until it meets the real world, a spec doesn’t truly work until it’s implemented.
\nNo-code libraries are toys because they are unproven.
\nEven if you aren’t the one making decisions during implementation, decisions are being made. We should leverage LLMs to extract and structure those decisions.
\nAnd finally: we’ve been here before. The answer then was process. The answer now is also process. And just as we leverage cloud compute to enable CI/CD for agile, we should leverage LLMs to build something lightweight enough that we can fit in our heads, doesn’t slow us down, and helps us make sense of our software.
\n![\"\"](\"https://www.dbreunig.com/img/sdd_triangle/sdd_triangle_041.jpg\")
Again: thank you very much to Professor Claire Le Goues, who helpfully walked me through the history of computer science. The history section of this talk is entirely thanks to her. And she has a book coming out, aimed at a wider audience, later this year. Do check it out.
\n", "date_modified": "2026-09-15T15:44:24+00:00", "date_published": "2026-03-04T15:06:00+00:00", "external_url": "https://www.dbreunig.com/2026/03/04/the-spec-driven-development-triangle.html", "id": "urn:aggr:item:945fc25286e75f47b1c86222007f4f092f1bb5ca", "summary": "What shipping a library with no code taught me: spec-driven development is a feedback loop between specs, tests, and code, not a straight line.", "tags": [ "programming", "ai", "coding", "software engineering", "spec driven development" ], "title": "Learnings from a No-Code Library: Keeping the Spec Driven Development Triangle in Sync", "url": "https://aggr.aymericbeaumet.com/items/dbreunig-com/2026-03-04-learnings-from-a-no-code-library-keeping-the-spec-driven/" }, { "authors": [ { "name": "Drew Breunig" } ], "content_html": "Beneath the Anthropic and Department of War fracas, there is a legitimate & essential conversation to be had about how much control any organization has when deeply adopting an AI model they didn’t train.
\nThese are probabilistic systems, with near infinite surface area to test, that are intentionally designed. Models are used to inform and make decisions, and they all have embedded perspectives.
\nAI is unlike other technology purchases because AI has embedded judgment.
\nI’m not sure what the answer is here, only that we need to have this discussion (calmly) and that anyone who tells you this isn’t a problem, that their model has an objective God-view-from-nowhere, is selling you something.
\nLet me be clear: I agree strongly with Anthropic’s usage red lines. I gladly choose Claude myself.
\nBut this conversation is being framed badly around usage. Many are talking about Anthropic’s “terms of service” (notably, both Hegseth and Trump even capitalized the term in their tweets), but I think allowed usage terms are red herring. The issue is embedded judgment.
\nIf I were in military procurement, I would certainly some big questions about what “soul documents” or “constitutions” (or similar) are embedded in any model being considered for embedding throughout the armed forces (and all the labs make design choices during post-training).
\nAnd clearly this is something Anthropic is already dealing with! This section, from the above blog post, suddenly becomes much more interesting:
\n\n\nThis constitution is written for our mainline, general-access Claude models. We have some models built for specialized uses that don’t fully fit this constitution; as we continue to develop products for specialized use cases, we will continue to evaluate how to best ensure our models meet the core objectives outlined in this constitution.
\n
We don’t know if post-training control helped blow up the deal (I tend to believe the issue was about allowed usage, based on the administration’s and Anthropic’s statements, coupled with OpenAI’s announced terms). But I think it’s a safe bet many militaries will insist on influencing and auditing the post-training for their purchased variants.
\nI wrote back in 2023 that I expect states and cultures to build their own models for related reasons; I wasn’t thinking about defense tech at the time but it certainly amplifies the issues.
\nOne takeaway: this is a strong argument for why the AI race isn’t going to be winner-take-all. Everyone wants a champion to trust.
\n", "date_modified": "2026-09-15T15:44:24+00:00", "date_published": "2026-03-01T17:13:00+00:00", "external_url": "https://www.dbreunig.com/2026/03/01/the-issue-is-embedded-judgment.html", "id": "urn:aggr:item:dc4c72639cb296b7c458bfcd6c30944606a1cd82", "summary": "The Anthropic and Pentagon fight is framed around usage terms, but AI is unlike other purchases because every model carries embedded judgment.", "tags": [ "programming", "ai", "policy" ], "title": "We’re Talking About Terms of Use, But the Issue is Embedded Judgment", "url": "https://aggr.aymericbeaumet.com/items/dbreunig-com/2026-03-01-we-re-talking-about-terms-of-use-but-the-issue-is-embedded/" }, { "authors": [ { "name": "Drew Breunig" } ], "content_html": "There’s a lot of noise about how AI is changing programming these days. It can be a bit overwhelming.
\nIf you hang out on social media, you’ll hear wild claims about people running 12 agents at once, for days. Or people hacking bots together, giving them $10k, and letting them roam the web.
\nThe challenge with all of this is that coding agents really are performing some science fiction feats which were barely imaginable just 12 months ago. But at the same time, the ecosystem is incentivizing the most outlandish claims, so punters keep telling tall tales. Separating the signal from the noise is near impossible.
\nI’m lucky enough to talk to a range of developers and teams, spanning a variety of company sizes and a broad array of skill sets. From these conversations, two beliefs have emerged and solidified about coding agents and their (current) impact on coding.
\nLet’s start with belief number one:
\nMost talented developers do not appreciate the impact of the intuitive knowledge they bring to their coding agent.
\nWe’ve all seen the posts by developer luminaries. They haven’t written code in weeks. They gave a hard problem to Claude Code or Codex and it just worked.
\nBut what we don’t see is their prompts. And having seen many prompts by many types of devs, I would wager their prompts are relatively specific and offer more guidance to the LLM than your average user. And these specifics don’t have to be exhaustive. Even knowing the right terms to use can have enormous impact and activate an entirely different set of weights in the model than someone writing, “the search is broken fix it.”
\nSkilled programmers, with plenty of experience, don’t even think about how to ask correctly. They just do, intuitively. And things work well. If the agent and dev go through multiple turns, this effect gets even more significant.
\nI wish we could see more prompts and traces, from a wide range of developers, to better understand the range of code. And, just as interestingly, how hard and long agents have to work to achieve the goal. For now we can just browse public repos on Github, where the range of coding quality is quite broad.
\nWhich brings me to the second belief:
\nMost work people are sharing are incredible personal tools, but they are not capital-P products.
\nThere’s an app I really like called “StreetPass.” It’s a browser extension that watches web pages you visit and collects Mastodon accounts it finds, letting you easily follow them if you wish. It’s small and charming. A perfect extension.
\nRecently, I realized I wanted a version of StreetPass, but for RSS feeds instead of Mastodon accounts. I forked StreetPass, fired up Claude Code, and had a working version quickly. You can use this, but I’m not supporting it. I won’t be pushing it to the App Store or Chrome Web Store. I won’t be building a version that doesn’t leverage Feedbin. I have no idea if it works on Chrome or Firefox. It’s personal software that I use almost daily.
\nMost agentic coding projects we see being hyped are like this.
\nAll those things I won’t do, those are the things that would turn my personal software into a Product. And we haven’t even gotten to marketing, support, and more. As we covered when we touched on Claude’s desktop app, the last 10% of product development and support is where the pain is. And that’s still a long road. As they say: Code today is free, as in puppies.
\nBut I want to be clear about couple things.
\nFirst, I know many teams shipping agent written code into products. But they test, support, review, and so much more. But when we make big claims like “coding is solved” or “code is free”, we need to be clear about what we’re talking about building1.
\nSecond, our ability to manifest personal software easily is amazing and powerful. I am continually inspired by the things people build (for example, I loved Simon’s presentation software he whipped up for FOO Camp). His presentation app is so tailored to him, in the past the math would never justify the time spent building it to support a market of maybe a dozen. But now he gets his dream!
\nSimilarly, my RSS finder extension is a feature not an app and (sadly) there isn’t a large market for RSS today. But with Claude Code (and open source code to build upon!) I can build just what I wanted in moments.
\nI am sure as our scaffolding and models improve, this stuff will get more accessible and more resilient, but I don’t expect these two beliefs to go away. Providing AI with the right instructions to obtain just what you want, will always be a challenge.
\nCoding agents amplify existing skills.
\nGrady Booch has a good post about this today. Things are getting higher level, and changing fast, but engineering remains. ↩
\nThe state of coding agents can be summed up by this fact
\n\n\nClaude spent $20k on an agent swarm implementing (kinda) a C-compiler in Rust, but desktop Claude is an Electron app.
\n
If you’re unfamiliar, Electron is a coding framework for building desktop applications using web tech, specifically HTML, CSS, and JS. What’s great about Electron is it allows you to build one desktop app that supports Windows, Mac, and Linux. Plus it lets developers use existing web app code to get started. It’s great for teams big and small. Many apps you probably use every day are built with Electron: Slack, Discord, VS Code, Teams, Notion, and more.
\nThere are downsides though. Electron apps are bloated; each runs its own Chromium engine. The minimum app size is usually a couple hundred megabytes. They are often laggy or unresponsive. They don’t integrate well with OS features.
\n(These last two issues can be addressed by smart development and OS-specific code, but they rarely are. The benefits of Electron (one codebase, many platforms, it’s just web!) don’t incentivize optimizations outside of HTML/JS/CSS land.)
\nBut these downsides are dramatically outweighed by the ability to build and maintain one app, shipping it everywhere.
\nBut now we have coding agents! And one thing coding agents are proving to be pretty good at is cross-platform, cross-language implementations given a well-defined spec and test suite.
\nOn the surface, this ability should render Electron’s benefits obsolete! Rather than write one web app and ship it to each platform, we should write one spec and test suite and use coding agents to ship native code to each platform. If this ability is real and adopted, users get snappy, performant, native apps from small, focused teams serving a broad market.
\nBut we’re still leaning on Electron. Even Anthropic, one of the leaders in AI coding tools, who keeps publishing flashy agentic coding achievements, still uses Electron in the Claude desktop app. And it’s slow, buggy, and bloated app.
\nSo why are we still using Electron and not embracing the agent-powered, spec driven development future?
\nFor one thing, coding agents are really good at the first 90% of dev. But that last bit – nailing down all the edge cases and continuing support once it meets the real world – remains hard, tedious, and requires plenty of agent hand-holding.
\nAnthropic’s Rust-base C compiler slammed into this wall, after screaming through the bulk of the tests:
\n\n\nThe resulting compiler has nearly reached the limits of Opus’s abilities. I tried (hard!) to fix several of the above limitations but wasn’t fully successful. New features and bugfixes frequently broke existing functionality.
\n
The resulting compiler is impressive, given the time it took to deliver it and the number of people who worked on it, but it is largely unusable. That last mile is hard.
\nAnd this gets even worse once a program meets the real world. Messy, unexpected scenarios stack up and development never really ends. Agents make it easier, sure, but hard product decisions become challenged and require human decisions.
\nFurther, with 3 different apps produced (Mac, Windows, and Linux) the surface area for bugs and support increases 3-fold. Sure, there are local quirks with Electron apps, but most of it is mitigated by the common wrapper. Not so with native!
\nA good test suite and spec could enable the Claude team to ship a Claude desktop app native to each platform. But the resulting overhead of that last 10% of dev and the increased support and maintenance burden will remain.
\nFor now, Electron still makes sense. Coding agents are amazing. But the last mile of dev and the support surface area remains a real concern.
\nOver at Hacker News, Claude Code’s Boris Cherney chimes in:
\n\n\nBoris from the Claude Code team here.
\nSome of the engineers working on the app worked on Electron back in the day, so preferred building non-natively. It’s also a nice way to share code so we’re guaranteed that features across web and desktop have the same look and feel. Finally, Claude is great at it.
\nThat said, engineering is all about tradeoffs and this may change in the future!
\n
There we go: developer familiarity and simpler maintainability across multiple platforms is worth the “tradeoffs”. We have incredible coding agents that are great at transpilation, but there remain costs that outweigh the costs of shipping a non-native app.
\n", "date_modified": "2026-09-15T15:44:24+00:00", "date_published": "2026-02-21T18:00:00+00:00", "external_url": "https://www.dbreunig.com/2026/02/21/why-is-claude-an-electron-app.html", "id": "urn:aggr:item:e3dd3ecc8d1e0039224be250a87bcafe7a1512ae", "summary": "If coding agents make native ports cheap, why does Anthropic still ship Claude as an Electron app? Because code is cheap but maintenance isn't.", "tags": [ "programming", "ai", "software development", "spec driven development" ], "title": "Why is Claude an Electron App?", "url": "https://aggr.aymericbeaumet.com/items/dbreunig-com/2026-02-21-why-is-claude-an-electron-app/" }, { "authors": [ { "name": "Drew Breunig" } ], "content_html": "This post was co-authored with Srihari Sriraman
\nCoding agents are fascinating to study. They help us build software in a new way, while themselves exemplifying a novel approach to architecting and implementing software. At their core is an AI model, but wrapped around it is a mix of code, tools, and prompts: the harness.
\nA critical part of this harness is the system prompt, the baseline instructions for the application. This context is present in every call to the model, no matter what skills, tools, or instructions are loaded. The system prompt is always present, defining a core set of behaviors, strategies, and tone.
\nOnce you start analyzing agent design and behavior, a question emerges: how much does the system prompt actually determine an agent’s effectiveness? We take for granted that the model is the most important component of any agent, but how much can a system prompt contribute? Could a great system prompt paired with a mediocre model challenge a mediocre prompt paired with a frontier model?
\nTo find out, we obtained and analyzed system prompts from six different coding agents. We clustered them semantically, comparing where their instructions diverged and where they converged. Then we swapped system prompts between agents and observed how behavior changed.
\nSystem prompts matter far more than most assume. A given model sets the theoretical ceiling of an agent’s performance, but the system prompt determines whether this peak is reached.
\nTo understand the range of system prompts, we looked at six CLI coding agents: Claude Code, Cursor, Gemini CLI, Codex CLI, OpenHands, and Kimi CLI. Each performs the same basic function: given a task they gather information, understands the code base, writes code, tracks their progress, and runs commands. But despite these similarities, the system prompts are quite different.
\n![\"Waffle](\"https://www.dbreunig.com/img/resized/waffle_comparison.jpg\")
Try It Out
\nWe’re analyzing exfiltrated system prompts, which we clean up and host here1. Each of these is fed into context-viewer, a tool Srihari developed that chunks contexts in semantic components for exploration and analysis.
\nLooking at the above visualizations, there is plenty of variety. Claude, Codex, Gemini, and OpenHands roughly prioritize the same instructions, but vary their distributions. Further, prompts for Claude Code and OpenHands both are less than half the length of prompts in Codex and Gemini.
\nCursor’s and Kimi’s prompts are dramatically different. Here we’re looking at Cursor’s prompt that’s paired with GPT-5 (Cursor uses slightly different prompts when hooked to different models), and it spends over a third of its tokens on personality and steering instructions. Kimi CLI, meanwhile, contains zero workflow guidance, barely hints at personality instructions, and is the shortest prompt by far.
\nGiven the similar interfaces of these apps, we’re left wondering: why are their system prompts so different?
\nThere’s two main reasons the system prompts vary: model calibration and user experience.
\nEach model has its own quirks, rough edges, and baseline behaviors. If the goal is to produce a measured, helpful TUI coding assistant, each system prompt will have to deal with and adjust for unique aspects of the underlying model to achieve this goal. This model calibration reins in problematic behavior.
\nSystem prompts also vary because they specify slightly different user experience. Sure, they’re all text-only, terminal interfaces that explore and manipulate code. But some are more talkative, more autonomous, more direct, or require more detailed instructions. System prompts define this UX and, as we’ll see later, we can make a coding agent “feel” like a different agent just by swapping out the system prompt.
\nWe can get a glimpse of these two functions together by looking at how a given system prompt changes over time, especially as new versions of models arrive. For example:
\n![\"Claude's](\"https://www.dbreunig.com/img/resized/claude_over_time.jpg\")
Try It Out
\nNote how the system prompt isn’t stable, nor growing in a straight line. It bounces around a bit, as the Claude Code team tweaks the prompt to both adjust new behaviors and smooth over the quirks of new models. Though the trend is a march upward, as the coding agent matures.
\nIf you want to dive further into Claude Code’s prompt history, Mario Zechner has an excellent site where he highlights the exact changes from version to version.
\nGo Deeper
\nWhile these prompts vary from tool to tool, there are many commonalities that each prompt features. There is clear evidence that these teams are fighting the weights: they use repeated instructions, all-caps admonishments, and stern warnings to adjust common behaviors. This shared effort suggests common patterns in their training datasets, which each has to mitigate.
\nFor example, there are many notes about how these agents should use comments in their code. Cursor specifies that the model should, “not add comments for trivial or obvious code.” Claude states there should be no added comments, “unless the user asks you to.” Codex takes the same stance. Gemini instructions the model to, “Add code comments sparingly… NEVER talk to the user through comments.”
\nThese consistent, repeated instructions are warranted. They fight against examples of conversation in code comments, present in countless codebases and Github repo. This behavior goes deep: we’ve even seen that Opus 4.5 will reason in code comments if you turn off thinking.
\nSystem prompts also repeatedly specify that tool calls should be parallel whenever possible. Claude should, “maximize use of parallel tool calls where possible.” Cursor is sternly told, “CRITICAL INSTRUCTION: involve all relevant tools concurrently… DEFAULT TO PARALLEL.” Kimi adopts all-caps as well, stating, “you are HIGHLY RECOMMENDED to make [tool calls] in parallel.”
\nThis likley reflects the face that most post-training reasoning and agentic examples are serial in nature. This is perhaps easier to debug and a bit of delay when synthesizing these datasets isn’t a hinderence. However, in real world situations, users certainly appreciate the speed, so system prompts need to override this training.
\nBoth of these examples of fighting the weights demonstrate how system prompts are used to smooth over the quirks of each model (which they pick up during training) and improve the user experience in an agentic coding application.
\nMuch of what these prompts specify is shared; common adjustments, common desired behaviors, and common UX. But their differences notably affect application behavior.
\nHelpfully, OpenCode allows users to specify custom system prompts. With this feature, we can drop in prompts from Kimi, Gemini, Codex and more, removing and swapping instructions to measure their contribution.
\nWe gave SWE-Bench Pro test questions to two applications: two agents running the OpenCode harness, calling Opus 4.5, but with one one using the Claude Code system prompt and the other armed with Codex’s instructions.
\nTime and time again, the agent workflows diverged immediately. For example:
\n![\"Claude's](\"https://www.dbreunig.com/img/resized/prompt_swap.jpg\")
The Codex prompt produced a methodical, documentation-first approach: understand fully, then implement once. The Claude prompt produced an iterative approach: try something, see what breaks, fix it.
\nThis pattern remains consistent over many SWE Bench problems. If we average the contexts for each model and system prompt pair, we get the following:
\n![\"Swapping](\"https://www.dbreunig.com/img/resized/swe_bench_prompts.jpg\")
Try It Out
\nAll prompt-model combinations correctly answered this subset of SWE Bench Pro questions. But how they suceeded was rather different. The system prompts shaped the workflows.
\nLast week, when Opus 4.6 and Codex 5.3 landed, people began putting them through the paces, trying to decide which would be their daily driver. Many tout the capabilities of one option over another, but just as often are complaints about approach, tone, or other discretionary choices. Further, it seems every week brings discussion of a new coding harness, especially for managing swarms of agents.
\nThere is markedly less discussion about the system prompts that define the behaviors of these agents2. System prompts define the UX and smooth over the rough edges of models. They’re given to the model with every instruction, yet we prefer to talk Opus vs. GPT-5.3 or Gastown vs. Pi.
\nContext engineering starts with the system prompt.
\nExfiltrated system prompts represent versions of the system prompt for a given session. It’s not 100% canonical, as many AI harnesses assemble system prompts from multiple snippets, given the task at hand. But given the consistent manner with which we can extrac these prompts, and comparing them with public examples, we feel they are sufficiently representative for this analysis. ↩
\nThough you can use Mario’s system prompt diff tool to explore the changes accompanying Opus 4.6’s release. ↩
\n![\"\"](\"https://www.dbreunig.com/img/resized/thumb_index.jpg\")
“Context Rot” is a common problem agent designers must avoid and mitigate.
\nThe Gemini 2.5 paper was one of the first technical reports that flagged the issue, noting that the performance of their Pokémon-playing harness rapidly deteriorated as the context grew beyond 100,000 tokens; a figure far below Gemini 2.5’s 1 million input token limit. We covered this in our context failures piece, but the Chroma team published the canonical exploration of the effect, dubbing it context rot.
\n![\"A](\"https://www.dbreunig.com/img/resized/context_rot.jpg\")
A key takeaway from Gemini’s Pokémon troubles and the Chroma post is that context rot is not a capacity problem. It’s a quality problem. As the context grows beyond a model’s soft limit, the model continues to issue output as its accuracy declines. This makes for a pernicious problem, one that sneaks up on us the longer we run agents.
\nOf all the context fails, context rot is the worst.
\nDefined by Alex Zhang and Omar Khattab, Recursive Language Models (or RLMs) are a simple idea:
\nThat’s it. That’s an RLM. The LLM will use the REPL to filter, chunk, and sample the long context as needed to complete its task. It will use the sub-LLM function to task new LLM instances to explore, analyze, or validate the context. Eventually, the sum of the LLM’s findings will be synthesized into a final answer.
\nWith this setup, the long context(s) can be really long. I’ve given RLMs logfiles more than 400 megabytes in size, with no issues. In the original RLM post, Alex reports that performance doesn’t degrade when >10 million tokens are provided.
\n![\"Alex](\"https://www.dbreunig.com/img/resized/rlm_plot.jpg\")
Note the orange lines on the right: as the context length increases, performance very slowly degrades, hovering around 50-60%. Compare this to the non-RLM results (with the same GPT-5 model), which dramatically decline until failing entirely at 262,000 tokens.
\nThe key attribute of RLMs is that they maintain two distinct pools of context: tokenized context (which fills the LLM’s context window) and programmatic context (information that exists in the coding environment). By giving the LLM access to the REPL, where the programmatic context is managed, the LLM controls what moves from programmatic space to token space.
\nAnd it turns out modern LLMs are quite good at this!
\nLet’s look at an example.
\nHere I’ve given Kimi K2 a very large dataset of Stable Diffusion prompts (prompts people provided to generate images). I then ask the RLM to identify the most common celebrities used in these prompts (and of course, I’m using RLM in DSPy). If you’re curious, here’s the code.
\nI give the RLM a budget of 5 iterations to accomplish the task. Below, you can swipe/page through each iteration, which shows the LLM’s reasoning and the code it executed in the REPL. There’s a few things to keep in mind as you read through:
\nprint in the REPL, it’s bringing new context into the token space. (I’ve omitted this output for brevity)llm_query (highlighted in blue) in the REPL, it’s tasking another LLM instance with a sub-call. It stores the result of this function as a variable, usually.SUBMIT, which indicates it has finished with the task.Click through and read, it really illuminates how a RLM works:
\nWe can clearly see the LLM exploring and sampling the context, planning an approach, testing the approach, scaling the approach, then finally synthesizing its findings into a final answer. (In this case, it was correct!)
\nThe context I gave this RLM – the collection of Stable Diffusion prompts – exceeds the maximum context window of any LLM. It would fail before it started, whereas a DSPy RLM harness around Kimi K2 took only a couple minutes.
\nIt’s incredible, but with this example we can identify a couple limitations of RLMs.
\nFirst, it’s relatively slow. Answering this question took over a dozen LLM calls and several minutes. And we were using Kimi K2 on Groq. Try this with GPT-5.3 or Opus 4.6 and you’ll be waiting around even longer.
\nSecond, as you read through the reasoning and code in the example above it becomes apparent that you need strong models to drive RLMs. Qwen3-30B-A3B couldn’t complete this task. It got confused, lost track of progress, and ended up running out of budget before submitting an answer2.
\nThis brings us to the second reason RLMs work so well (in addition to maintaining the two token and programmatic context pools): RLMs exploit the coding reasoning gains of the last +18 months.
\nWe’ve covered before how LLMs are getting better at verifiable tasks because it’s relatively easy to synthesize data and evaluate verifiable tasks, like math and coding. We’ve spent many billions of dollars post-training coding skills into frontier models. RLMs wrap long contexts in a coding environment so they’re addressable by the LLM’s incredible coding abilities, turning context rot into a coding problem.
\nEven better, RLMs get to use the REPL not just as a tool for exploring and managing long contexts, but also as a deterministic scratchpad. This proves to be a killer resource for many tasks. You occasionally see this benefit in action in ChatGPT or Claude, when the LLM will fire up a Python script to answer a question3. This hybrid capability of RLMs – the ability to use probabilistic, fuzzy LLM logic for some challenges and deterministic code for others – will likely become a stronger attribute as RLM harnesses mature and models are fine-tuned.
\nThe ability of RLMs to mitigate the effects of context rot are really incredible. However, this isn’t the potential that excites me most. What excites me about RLMs is their ability to explore, develop, and test approaches to solving a problem.
\nIf you start experimenting with RLMs (and I strongly suggest you should), be sure to continually review your traces. Set verbose to true and/or wire up DSPy to MLFlow. As you watch these models explore the context and try out different approaches (taking your iteration budget into consideration4), you’ll notice repeating patterns. In the example above, if I asked the RLM to find the top celebrities, aesthetic styles, or vehicles requested in the image generation prompts, it would repeatedly deploy similar tactics to situate itself and complete the task.
There is no reason we can’t identify these repeating patterns, decompose them, and optimize them.
\nThis is what excites me about RLMs: if you run them on the same task several times, you’re generating emergent agent designs. These traces can then be used to explicitly define an agent, with higher reliability and lower latency. RLM passes discover the best approach to the problem, which we can then optimize.
\nBut if that’s the potential, how should you use RLMs today? In the last couple months I’ve seen teams use them for very large context scenarios, from general coding tasks across massive codebases to research and exploration across massive datasets.
\nAt the moment, using RLMs on small context problems probably isn’t worth the squeeze. You’ll end up waiting around while the RLM explores context that could have simply been part of the prompt.
\nFurther, RLMs do not solve other context fails, like context poisoning or context confusion. If bad information is in your programmatic context, there’s good odds it could influence the RLM in undesirable ways.
\nRLMs are slow, synchronous, and merely borrowing the current capabilities of models rather than leveraging models post-trained to be good at RLM patterns. There is so much low-hanging fruit here.
\nBut that’s exactly what makes them exciting. Chain of thought was also simple and general (just ask the model to “think step by step”) and it unlocked enormous latent potential in LLMs, that was only fully realized through the creation of reasoning models. RLMs have the same shape: a test-time strategy that’s easy to implement today and will only get better as models are trained to exploit it.
\nYou probably don’t need to rush out and refactor your agents today. But if your agents touch large contexts, start experimenting with RLM traces today. You’ll learn something about your problem…and you might discover your next agent architecture in the output.
\n“REPL” stands for “read-eval-print loop”. It is an interactive coding environment where one can enter arbitrary code and get back output. If you open your terminal and type python, you’ll find yourself in a REPL. ↩
The team at MIT behind RLM has just released a version of Qwen3-8B post-trained on RLM traces. I hear it works pretty well, but no amount of fine-tuning or RL is going to help Qwen-8B code or reason as well as GPT or Opus. ↩
\nBoth ChatGPT and Claude used to do this when asked, “How many R’s are in Strawberry,” though it appears both rely on reasoning or, in the case of ChatGPT, hide the previously visible Python code. ↩
\nI was continually amazed how well models would leverage their budgets. Kimi, in particular, wasn’t shy about ending early if the task proved simple. But it would also spend LLM sub-calls freely once it had a working approach, saturating my connection with Groq. ↩
\n