AddyOsmani.com

Brownfield Agentic Engineering

Mon, 14 Sep 2026 00:00:00 +0000

Agentic engineering in an old codebase is about making hidden constraints visible and cheap changes trustworthy. Let’s talk what to do in brownfield codebases.

During my career I’ve worked on teams whose codebases had been around a long time. Those are brownfield systems: the repository is no longer a complete description of how the thing actually behaves. Institutional knowledge, duct tape, legacy services, and expectations other teams depend on live outside the tree. You have to learn those constraints before you write new code, and you have to prove a change didn’t break them. I love coding with agents, but throw them at an older brownfield codebase unsupervised and you may end up with something that “works” but with the wrong system design and brittle tests.

Even in teams that wanted to do modernization efforts pre-AI, you often had to take things very, very piecemeal, with strong testing in place, a strong layer of confidence to make sure that you weren’t breaking things. You kind of knew that on top of actual user journey testing, any migrations you were making had to keep things working as intended via a barrage of repeatable tests. These days some folks may say that as soon as an agent drops code you didn’t author decision-by-decision, you’re already in a brownfield project. Regardless, you want to optimize for cheap changes being made safely.

And these days, especially in the last, I would say, maybe five to ten years, this idea of caring more about testing, caring more about verification, caring more about how you make changes in a way that is not going to break things, I feel has gotten more attention. But that doesn’t change the fact that if you’re doing a lot of work trying to introduce agentic engineering, and then software factories and all of these other kinds of patterns for autonomously working through these large codebases, you have to put quite a bit of additional mindfulness in place otherwise you risk signing up for a world of technical debt.

Before we dive in, let’s assume that the code should be the source of truth. Anything we add on top to help brownfield is what can’t be easily inferred. I want to talk about this in terms of zones, blast radius and a few other patterns I think will help.

Zones

If I’m going into an older codebase that’s been around for a while, I probably want to get a sense of what code shouldn’t I be touching. You can consider these zones. E.g. Green zone = safe/good tests/isolated, yellow = mixed quality, red = sensitive/auth/billing/permissions.

What are the parts of the codebase that are very, very sensitive, or that not everybody understands well? And maybe you would draw those with different zones. Maybe you have a green area that’s got very good test coverage, and is using modern conventions that are current, and has good isolation. And for those parts of the system, agents can go off and work on that in a tight loop.

There are sites, especially commerce sites that I’ve worked with, where you could easily have five or six departments all with their own microsites, when the entire experience to the end user is going to feel like a single thing. And there’s actually a lot of inherent complexity underneath the surface. One team might have really good test coverage for their stuff; maybe it was built in the last couple of years. Other teams may not. So you have this green zone.

Maybe you have yellow, which is mixed quality, maybe it’s a mix of things, and agents can change code there after characterization tests have been written.

And then you can have red areas, where you’ve got sensitive stuff like authentication, billing, permissions, payroll, anything that you wouldn’t normally touch and make some hasty changes to. For example, if only a small number of people understand how it all works. You don’t want unsupervised rewrites in that kind of system.

Three rules make the zones an operating procedure instead of a metaphor. A person draws the map, not the agent; left to choose, the agent starts in the scariest file, because the scariest file has the most interesting names. Zones only move when it’s earned: yellow becomes green once characterization tests exist and the module’s owner has reviewed the agent’s first changes. And the zone sets the verbs: green is a tight loop, yellow is tests first, red is a human pairing on every step or the work not happening.

Write down what the code can’t say

Autonomy should follow blast radius, observability, and recoverability. A model’s confidence is a poor guide.

So I think it makes sense to have at least a sense of, how do you think about the map of the world, and what can the agent infer itself from the codebase? Agents can actually infer quite a lot from the code itself. There was this period of time when people would try to include markdown files for absolutely everything, and then they’d stuff them in their context windows. Agents are actually pretty good at understanding the map of the system. What you want to give them is the stuff that is not obvious from the code itself. Are there conventions? Are there patterns? Are there nuances that are not in there? I think that’s important.

Concretely, that means: business or team specific nuance, trade-offs that explain why the system is structured a certain way, guidelines that aren’t explicitly enforced by static analysis or tooling, domain-specific domain rules, external constraints and historical context behind counter-intuitive implementations and so on.

Write down what the code can’t say, and nothing else.

Make the research survive the session

If your agent’s exploration produces no durable artifact, the next agent pays for the same archaeology again.

One piece I would add to that map is a durable research artifact. For yellow and red work, I like a separate read-only pass that produces a short comprehension memo: entry points, owners, callers, existing abstractions, tests, production signals, relevant history, and open questions. Claims should cite a file, issue, ownership record, or dashboard.

The default loop otherwise wastes its research. The agent works out how the auth flow behaves, completes the task, and loses that model when the session ends. Chat history isn’t a great system of record, especially after compaction.

After research, I would start planning with a clean context. Ask which files the plausible approaches touch, which invariants they preserve, and how you would reverse them. A human picks the path. Implementation should stop if it discovers the map was wrong. Review starts fresh and works backward from the acceptance criteria. A clean reviewer is more likely to notice when a test proves the implementation while missing the requirement.

When instructions become a harness

Every repeated correction is a missing piece of the harness.

It is useful to be precise about where the pieces fit. Instructions record unusual facts about a repository. Skills package reusable procedures such as checking blast radius or verifying a schema change. Plugins can provide governed access to the ownership catalog, incident archive, or dashboards.

The harness is the working environment around the agent: context, tools, permissions, tests, logs, and recovery. A factory schedules many dependable loops, keeps durable state, and hands novel cases back to people.

The practical test is what happens when the agent gets something wrong. If you quietly repair the diff, the next session can repeat it. When the same review comment appears again, move it into a lint rule, hook, type, test, or skill. Keep prose for constraints that cannot be enforced mechanically.

A deny rule, scoped credential, or CI check doesn’t have to remember. Over time the harness becomes a record of failures the team has decided not to pay for twice.

Start with zero-risk work

Lock today’s behavior before you let anything improve it.

If you’re bringing agents into an existing codebase, it’s very similar to other kinds of modernization efforts. Maybe you begin with zero-risk work. It shouldn’t be like, hey, let’s rewrite this monolith in Rust or something like that. Maybe it’s, first explain how the things work.

Generate characterization tests that can lock that current behavior.

Characterization tests are automated tests used to document a system’s actual current behavior so you can safely refactor or change legacy code

By characterization tests I mean tests that pin down what the module does today, ugly parts included, because in an old system some of that ugly behavior is what the business runs on, and an agent will happily “fix” it behind a green suite. The machinery is old because the problem is old. Netflix used the same idea at production scale in its GraphQL cutover - replay and shadow traffic against the old and new paths, diff the payloads, promote only when they match. That is the promotion path when a homepage-class surface has no honest unit suite: don’t guess; run both and compare.

When an agent is the one making them pass, don’t let that same session be the only author of the tests. Pin the behavior first, in a separate pass or by a person; then let the agent work. Otherwise you get a green suite that encodes the implementation you just invented.

And then you start down the path of doing mechanical transforms. You can do dead code and unused export inventories. You don’t want to start with the trickiest or hairiest parts of the system. And ultimately you want to have that confidence with any of these migrations.

I remember working on a number of different kinds of migrations over my time on large codebases, and people exercising a great deal of care, even when fixing things that were broken.

One of the older codebases I worked on was at AOL. There was a day when I was supposed to be off, and I was visiting a comic book store near the office, and as it so happened, my boss dropped me a text and asked if there was any way I could swing by. The AOL.com homepage was completely broken, and we didn’t have enough JavaScript experts around to go and figure it out. So I said, okay, sure, I’ll come in and take a look. And you would think these days, oh, a homepage, how complicated can it be? But when you have dozens and dozens of departments of people that can own lots of different components, lots of different criteria, lots of different scripts, A/B tests, all of these things, you want to avoid breaking the world for everybody else, because you’re not necessarily going to have test coverage all over the place in the same way that you would like. In that case I was able to get it fixed, but we basically had to at least user-test the things that didn’t have their own unit tests. How well were things working, without breaking for everybody? So that was kind of important.

That’s still the job. Agents don’t remove the dozens-of-departments problem; they make it cheaper to attempt a change against it. A surface that only production traffic really understands is a red zone by definition, and until you’ve built a stand-in for that traffic, the user-testing I did on my day off is still the gate.

Migrate in complete units

A migration is complete when the new path works and the old dependency is demonstrably gone.

Half-finished migrations are particularly confusing to agents. Search returns the old approach in forty files, the replacement in twelve, and a shim that presents both as current. The agent sees contradictory precedent.

I would rather finish one route end to end, including removing the old path, than convert thirty files and leave both patterns alive. If deletion is a future cleanup ticket, the migration unit is not complete.

Tests can stay green while a replacement still calls the legacy implementation. SWE Refactor Bench calls this migration “Blindness.” Across 520 agent runs, only 28 passed its migration audit, behavioral tests, and independent verification.

If a codemod can make the routine change, use the agent to help write and check it. Give agents the exception queue. Stripe’s migration is useful here precisely because no agents were involved: the durable artifact was the migration machine.

The lessons from bigger migrations

Bun’s Zig-to-Rust port ran about 50 workflows over 11 days from a 535,000-line codebase, with two adversarial reviewers on every generated unit and the entire pre-existing test suite as the merge gate; the part worth copying is that hours went into a porting guide mapping Zig idioms to Rust before any agent ran. Anthropic’s own migration process stress-tests its rulebook on a disposable mini-migration and throws the trial output away before the broad run begins.

A controlled VB6-to-C# study measured 92% behavioral equivalence on simple features and 47% on complex ones: unit size is the lever. The shape predates agents entirely: Stripe moved 3.7 million lines to TypeScript in one PR through months of codemod work, with no agents involved, and Google’s large-scale-changes chapter explains why atomic changes shrink as codebases grow. Spotify now reports 650-plus agent PRs merged monthly on rails Backstage built years earlier.

Asana cleared a multi-year Enzyme backlog in two calendar weeks for about $12,000 in model and infrastructure cost. That $12,000 is just a token bill but not a substitute for the five-year staffing estimate they had on the books; treat it as a vendor-reported cost of generation, not a controlled savings study. The transferable part is the same as Bun: a narrow mechanical migration, a pre-existing suite, humans still reviewing every change

What transfers between companies is the structure around the agents.

What’s actually changed

Agents have changed the price of trying several plausible implementations. They haven’t changed the evidence required to choose one.

And then I think you’ve probably seen, this year we’re beginning to read more and more cases of well-established companies who are using agents to do big rewrites. I’ve talked to CTOs who are allowing teams to have agents try multiple rewrites in different languages or frameworks because its now feasible to do so more cheaply and evaluate the trade-offs.

Shopify rebuilt the Shop consumer app from React Native to native Swift and Kotlin in twelve weeks with a small team and agent-gated, screen-sized checkpoints. The much larger merchant app is still the brownfield problem: hundreds of screens, deep platform integration, same gates, longer clock.

You’ve seen other examples of rewrites to Rust. You’ve seen people do framework-level migrations. There have been all kinds of migrations that have been done. And in many cases, these are migrations people would have done on a much longer timeframe. These days, if you have enough tokens, you can just actually have agents go and attempt to complete a migration across a range of different stacks or languages.

You can try to have your agents actually implement something in a number of different competing options. Rather than having one team choose a single option that you go all in on, what you do is you have them implement all of them. They can all check against your unit tests. You can performance profile all of them, and then make a decision, which is much, much cheaper in some cases than it otherwise would have been. And that’s a completely different ball game, I think, for teams these days.

Parallelize last

More generated code should lead to more selective human review, not less human ownership. Seriously consider what will setup your brownfield project for success before you go down the path of thinking about the loops/goals/parallelization.

Software factories can run many changes at once. I would copy that part only after one unit has a dependable judge, recovery path, and review format people can absorb.

Parallelism multiplies the bottleneck you already have. Automated verification can handle five checked changes. One senior reading every line gets a queue, fragmented attention, and eventually ceremonial approval.

I prefer automated review to lead with intent, changed invariants, test results, parity mismatches, and the rollback route. The complete diff remains available. Human attention goes first to the largest blast radius and weakest oracle.

Worktrees isolate changes, not behavior. They may share Git metadata, credentials, local services, and network access. Trusted work may accept that tradeoff. Unattended agents consuming untrusted content need stronger sandboxes and scoped credentials.

Agents put a price on ambiguity

Lines generated don’t tell you whether the codebase improved. I would track lead time, review minutes, human interventions, escaped defects, rollbacks, oracle mismatches, and suppressions left behind.

For a migration, track remaining old imports, traffic served by the new path, parity mismatches, and legacy dependencies removed. A green suite with all traffic still taking the old path is busywork.

Agents put a visible price on ambiguity. Tribal conventions become recurring review comments.

That cost was always there, paid during onboarding, review, and incident recovery. Agents make more of it countable. That gives us a stronger argument for maintenance work teams already knew was valuable.

The next time an agent works on the homepage equivalent, I would want it to leave behind more than the repair: a synthetic user journey, an ownership record, and a regression test.

What the next engineer and agent inherits matters too.

Agentic Skill Decay

Mon, 31 Aug 2026 00:00:00 +0000

Mastery still comes from doing the reps.

Before agents, I got my reps as part of writing code: try different approaches out, debug what went wrong, review other’s code, read a lot. Agents can skip much of that work, so building your reps has to be deliberate.

If I was new to the industry, I’d try to form a hypothesis before prompting. Ask “why” a lot, read the diffs, try to predict what might fail. Occasionally try to work through the problem myself manually.

In my experience, good agent work depends on two abilities:

Deep expertise: you understand the problem domain well enough to define a good outcome. Understanding your user/product/business is part of this.
Applied judgment: use your taste to turn this into a clear, testable plan by choosing the right context, constraints, tests and verification.

To build these the skills I’d practice are decision making, specifying, steering and verifying.

The reps I actually did

When I got started in software engineering, I very much felt like I had no idea what I was doing. I was having a lot of fun building, a lot of fun trying things out, failing, learning from my mistakes, and each time getting a little further and further in my journey. And every time I failed, I tried to take that as another building block in becoming a better engineer. And so all of this work was me doing the reps. It’s the way that I learned JavaScript. It’s the way that I learned how to program in C++ and build desktop applications. It’s how I learned how to tune the performance of graphics intensive applications, all of these types of things.

Building up the reps, very often I would go into a task with some sort of hypothesis or an idea, even if it was, like, super wrong about how things might work. I would try out what I thought could work, and when it didn’t work, I would then go to Stack Overflow or search the web for different documentation and absorb some knowledge. Maybe read some books if it was a very esoteric topic. I would then continue on my journey. And you do that enough times and you start to build out expertise, especially once you start doing this for real and trying it out on real world projects that go beyond hobbyist stuff that you might be doing at the weekends.

Most of the judgment I use today came from thousands of small reps like these: debugging failures, reviewing other people’s code, and living with abstractions that looked good until a real system pushed back. Agents can now skip much of that work. If you’re three years into your career, plausible code may arrive faster than your ability to judge it.

The short-circuit

Now, I think that for many people who are getting started with AI, you can short-circuit a lot of the learning journey. You can go very quickly from, hey, here’s a problem, to, well, hey, here’s the solution, or here’s the outcome of the overall task, while skipping all of those things that would otherwise have built up your knowledge base, or helped educate you about, you know, don’t do that thing, this is why you don’t do that thing, do this thing, and help you reason about the trade-offs. And I think that this is one of those areas where it’s going to require junior engineers especially to be proactive about their educational journey.

I’ve also talked to a few different AI labs. Many of the main players in AI right now are very focused on helping you accomplish an outcome or get an answer as quickly as possible, and they don’t necessarily help you in your education journey unless you’re specific about that being one of your goals. Like, there’s a difference in me saying, hey, help me build an app for scheduling, and, help me build an app for scheduling and teach me how to do it as we’re going, going one step to another. Most people don’t do the second one of those things. And part of it is not knowing that that is an option. Part of it is perhaps thinking, well, hey, these days there are all these velocity expectations and there’s this pressure to ship fast and just move on to the next thing. But I do think that in order to become better engineers, to continue having this expertise that improves our taste and our judgment, you do have to go out of your way to build up mastery.

A completed task is not necessarily a rep

I think that with any kind of critical thinking, with any type of problem that you have, it’s useful to have a hypothesis about the solution, an idea about what it might look like: the shape of it if we’re just talking about logic and code, how it might look and feel and interact if it’s a piece of UI. When a task is finished, it doesn’t necessarily mean that you have learned something. It just means that the task has been finished. You have to almost look out for those learning opportunities, or you can ask your agent to summarize as it’s building or at the very end: what are the key learnings from this that would help me as an intermediate developer, or as a junior developer, increase my knowledge base or improve how I think about problems? And you can keep doing that. You just have to be proactive about your learning journey.

For me, there are several things that I’ve been able to use AI for these days, and more complex 3D graphics programming is definitely one of those. I’m not an expert. And there are definitely times when I try to make sure I’m asking the AI, okay, so can you explain how this thing works? Can you teach me about this concept you just implemented? Can you help me reason about how these different elements connect? And I think that because I want to learn, and I have that desire to learn, I am pairing with my agent in order to do that. If you’re not necessarily trying to learn, you lose opportunities there.

A completed task not being a rep is also something that happens when there aren’t mistakes in the process. When there are mistakes, you start to think, okay, well, why did it go wrong? What could be better? What am I not thinking about? And it forces you to reflect. When things go right, there’s not really a teaching moment there. You just think, okay, well, the work’s done, I’m just going to move on to the next task. And so, especially if you’re junior, you want to be looking for those opportunities to keep leveling up.

There was a 2026 study by Anthropic looking at junior engineers learning a particular Python library, Trio. People who used AI assistants scored 50% on a follow-up quiz against 67% for the group who were working by hand. And within the AI group, the strong results came from those who asked conceptual questions and requested explanations rather than treating the model as a code vending machine. This ties back to what I was saying: if you are just using AI to generate output and generate outcomes, but you’re not using it as a pair, you’re not using it to try improving your critical thinking skills, your knowledge skills, your understanding of how things work, you can end up in this situation where you largely don’t understand how things work, but you’re just good at prompting. And that means you’re perhaps not really going to be so good at the verification side of things. It doesn’t surprise me too much that people who asked questions and requested explanations did better. Those people probably had a lot more reflection on how things worked, how it connects to other things that they know. They pattern match, they start to build up residue about, okay, well, this is how this thing works, this is how I reason about it, these are the gaps in my knowledge. And so seeing that 17% difference kind of makes sense to me. Of course, this was a short-term study of just one Python library, so I wouldn’t say it’s conclusive necessarily, but it was still very interesting.

This is still how I work when I’m learning something unfamiliar. I try to keep myself in the loop. I form a hypothesis before prompting. I ask why, inspect the diff, predict what might fail, and give the agent a concrete way to verify its work. Occasionally I work through a small problem by hand. I want the agent to close the task while my mental model still moves.

Use them aggressively anyway

I don’t know how long code-level expertise will remain as valuable as it is today. Models are improving too quickly for much certainty. I also don’t think the answer is to avoid agents or romanticize typing every line. I use them aggressively. On some days I have five or ten sessions running, and I once caught myself asking the wrong project to add dark mode. That mistake clarified the constraint: agent throughput scales faster than my attention.

A thousand hours in the performance panel

I can think of things that I had to spend thousands of hours to get right. Performance optimization is one of those areas where, back in the day, you didn’t always have a whole lot of great blog posts or books that you could consult. There was some decent, very classic literature on these topics that would maybe touch on memory or how to think about hardware and constraints. But you take something like web performance optimization, JavaScript optimization, heap optimization, all of these things, there weren’t always great articles about these things. And so you would build your reps by going into the Chrome developer tools, using the performance panel to run a trace of a page or an application, interact with it, try to find, like, where is the slowness? And then trying to drill down and come up with a hypothesis of, okay, well, it looks like this is the area of the flame graph where most of the problem seems to be. Or this is where maybe, in the memory panel, I’m not allowing garbage to be collected, or anything like that. In my time, you had to have gone through the gauntlet of making enough mistakes, attempting to find out the root cause, that you built up this knowledge, this esoteric at times knowledge, about what worked and what didn’t.

These days, a similar flow would be one where you’d have the DevTools MCP go and do the performance profiling for you with your agent, and figure things out, and then come up with the fix for you itself. And so you don’t necessarily then build up that expertise in performance quite as much.

You can only prompt what you can imagine

When I scroll through Twitter these days, I am always impressed with how much imagination and creativity is in my feed. So many designers, creative people sharing amazing shaders, amazing games, UI, immersive experiences that they are building that is now even more so possible. Like, the tech was there, but imagination is now the ceiling. It’s much, much more accessible for you to build these things much more quickly. But you have to have that imagination in order to have the idea in the first place and tell your agent to build it. And then you have to have that expertise to verify it. So verification is the floor and imagination is the ceiling.

I remember, for an upcoming album site (I do music), I wanted some of the homepage to be these 3D objects that were interactive, that are part of the experience. Things like 3D CD players, and I think I had a vinyl record player in there as well, maybe a tape player, some 90s nostalgia. Now, the initial versions not only didn’t look amazing, but they didn’t follow the right interaction pattern. They didn’t perform as well on mobile. And so I had to first of all have the expertise to notice that it was buggy in some way. Maybe any user would notice that. But then I had a hypothesis about why that might be. And I could then go and either profile it myself or ask my agent to profile it and figure out what happened, what went wrong. Maybe there was just some way in which the interaction logic was written that wasn’t great. And so I think that your imagination is really important, but then so is your expertise. Both of these things are important. If you can think it, you can make it.

Does the next generation need the expertise?

There is a valid question about, like, hey, if an agent can do these tasks, and increasingly well, do humans need to build up that expertise? Does the next generation need to build up expertise in some of these esoteric areas? And I think that, at least today, where that still becomes useful is places where the agents don’t do a perfect job, where their work does need to be checked. Where you ask something to optimize a particular loop, an animation, a scheduling routine, or anything like that, and maybe it does that at the cost of something else. And if you don’t know what to spot, or you don’t know how to read the implementation and understand what was done, you can end up shipping something that actually doesn’t do what you want.

Skills and MCPs can encode a useful workflow. They cannot tell you when its assumptions no longer fit your system.

There was an Anthropic study of around 400,000 Claude Code sessions that looked at expertise as being this task-specific thing. And it found that having even intermediate expertise about the task that you were trying to complete increased the chances of you reaching verified success with that task, rather than someone who is a little bit more novice. It doesn’t mean that you have to have a decade of experience across the stack, but it does mean that you need to understand the problem domain enough to recognize what good means. We sometimes talk about that these days in terms of taste, and I’ve written about this before. This is also one reason, when I read the Claude Code best practices guide, I’m very happy to see that it starts off talking about verification: testing, using screenshots, other signals that give your agent something that it can continue to iterate against, and gives you evidence to review instead of just some simple summary saying that the task is complete. Having expertise helps you shape clay much better than someone who doesn’t have a lot of expertise but can maybe shape something that looks okay.

And this all comes back to having that expertise to be able to verify the work, to be able to judge the agent’s work. And so I’m hopeful that we can continue to invest in mastery and invest in craftsmanship, even as software engineering continues to rise in the abstractions that we’re using to build software.

The return on expertise is going up

I feel like software engineering fundamentals are going to continue to be important. Expertise is going to continue to be important. And now that the floor has been raised, AI is also increasing the return on the skills that people have, on the expertise that people have. People who are junior stop being junior by shipping real things and making mistakes, learning, building the reps. Experts kind of have an intuition about what to build, how to verify it, how to make sure that you know it’s good, it’s not broken, it’s going to be maintainable, it’s going to scale, it’s going to work in the different contexts or platforms. And especially now that so many people are able to just prompt and bring an idea into being, making it high quality and good enough to ship, delightful, and something that is maintainable and isn’t going to break in production, those skills are going to continue being important.

I run into this at least a couple of times every week. It’s so easy now to prompt any kind of app, any kind of feature. For example, I’m building a text editor at the moment, not from scratch. The idea for this is to be sort of a writing aid that highlights opportunities for your grammar to be better, or to not be using AI style writing, that type of thing. And a frontier model was able to generate me, with a lot of back and forth, a nice and okay looking UI. It wasn’t amazing. And it had a bunch of issues, such as it didn’t have the optimal use of screen real estate. It didn’t have good color contrast. It didn’t have a good scrolling model. All of these things that I know because I’ve made these mistakes before, I’ve built up the expertise. But if you don’t have that expertise, you might just prompt something, put it out into the world, and then stop. And you don’t know what’s better, because you haven’t put in the time to build up that expertise.

Put the lesson where the next agent can find it

One of the things that I tell people I mentor is that when you work with an agent, you should be making it better, and it should be making you better. And what that means is that every day, there should be some sort of cycle where you’re getting things added to lessons or to memory or something so that it’s able to improve. Because otherwise, every time that you’re starting a new session, it can feel like you’re onboarding a new hire that has amnesia. They’re not necessarily going to remember the subtleties of your business, your product, your team, your users, or any of that stuff. And so this is why we end up capturing so much in not just skills, but context and all the stuff that we try to give our agents. And we need to be careful about things that are actually useful and actually specific to problems versus things that we just think are making things better. And so I always encourage people to see, how can you make sure that you are teaching your agent more, and making sure that every day it’s getting better and you’re getting better?

If you are in a chat window and you happen to be solving a problem, like, let’s say that you discover some subtle scrolling bug in a UI component that you’re working on. You work with your agent, you go back and forth, and there’s a lesson somewhere in there that you could potentially use in the future. Now, maybe that lesson will get added to memory. Maybe it won’t. And especially if it’s a long session that has compacting, that full lesson may not necessarily go in there. So that lesson could disappear when the chat window dies. While if you instead try to codify things like specific lessons, tests, lint rules, anything, especially that is small enough that it can be codified in your repo, it can teach future agents. And I found that personally very helpful. If I learn a lesson, I take a few minutes to review and see, is this worth adding to my lessons.md, or asking my agent to add it to its memory, or something that’s just going to keep it sticky? Because I don’t want lessons to disappear. I’m going to forget personally, I’m going to move on to the next problem. This comes up all the time for me. It can be everything from, hey, I have a particular preference for how I approach UI, to how I approach writing components, to how I approach performance, all kinds of things. And if there’s a subtle way in which I address a problem, I want my agent to remember that, or have a way to remember it, rather than me having to continue restating this every single time.

Very often we treat our agents as something that’s going to remember everything that we do, and that’s not necessarily the case. Even if the agent has got a memory system, you can’t necessarily fully rely on it to recall all of the interesting things that you were maybe trying to learn, or the way that you like working, or the way that you would approach verification. And so it is okay to start capturing more of these things in markdown files. Just be very, very careful and cautious that you’re not over investing in that as a strategy. I always liked this idea of a dual loop. A good rep where you learn should sharpen you, and it should sharpen your agent. And when you have some hypothesis that was maybe corrected, you consider if that correction warrants becoming a linting rule, some type constraint, a documentation convention or a test, just so that it can stick around and benefit you in the future.

The outer loop

So I think what all of this means with respect to mastery is: invest in your expertise and in your craftsmanship. Do the reps, make mistakes, learn from them. You will over time be able to figure out what deserves to exist. You’ll be able to start writing up plans, refining plans, coming up with some definition for what done means, and also planning out for those places where humans are going to stay in the loop to check on correctness, safety, or user impact. That is going to be largely the outer loop I think engineers are going to need to own today. We’re going to keep seeing AI moving engineering further up the abstraction layers. And the more agents that I can run, the more care I need to choose where my limited time, taste, and judgment goes.

Audit your Agent files

Thu, 27 Aug 2026 00:00:00 +0000

TL;DR: Your coding agent’s configuration has a half-life. Models improve, harnesses add capabilities, codebases change, and the instructions we wrote for an older version stay behind. Recent research finds inconsistent value from personalized skills. I now run Claude’s /doctor every few weeks, review memory separately, and ask each instruction to earn its place again.

I feel like there’s been a lot of confusion about skill files and what to do with your CLAUDE.md and AGENTS.md files, especially as I’ve been reading developer discourse on Twitter over the last few months. People have been saying things like, hey, keeping your skills and CLAUDE.md/AGENTS.md files up to date is a big pain point. Very often, people are trying to get them to steer their agents, but are having a hard time keeping them under 200 lines long, even if that’s an official target. People are finding it hard to keep them lean. They’re raising token costs. They can make the agent worse as you keep adding and adding and adding stuff to them.

And the official guidance that I’ve read is that you should be periodically deleting your CLAUDE.md, your skills and your hooks, like every couple of months, rebuilding only what matters. But my own experience with that is that people have this fear of, hey, I don’t know if that’s going to actually make things significantly worse. I’m worried that if I do, that quality is going to drop very heavily. And I don’t have an easy way to just quickly restore things or try this out. There’s so many different configurations I can use models with. And so it can feel like this advice comes across as easy to try out when it’s not always going to be that way. And sometimes all of these markdown files and practices can become outdated and they can hurt more than they help. But I do think that there is something in this advice about how models and harnesses do actually get much better.

I’m a fan of skills. The critiques are still fair.

I’m personally a big fan of Agent Skills. I and a few of my friends maintain some Agent Skill packages that have gotten a little bit of traction. I maintain Agent Skills, which is an SDLC-focused pack. My friend Paul Bakaus maintains Impeccable, which is a design-focused pack. And sentiment has been generally pretty positive with developers using skills with coding agents. They’re seen as a good abstraction for turning a generalist into a specialist, right? And they work pretty well with a lot of different coding agents and harnesses. So people like them.

One of the sets of critiques has been fair. Of course, you have many different people writing them for very different domains. There isn’t really a great playbook for how to do this. So we’re all really playing it by ear. And we try to learn from each other. We take on feedback from the community and we’re constantly iterating on these things. But skill hygiene continues to be very important. Sometimes you’ll find skill packs that have thin descriptions, or they’re a little bit on the vaguer side for their workflows. And so I still think that agent skills are something that I believe in, and I think that they have a lot of value. But as people have really leaned into them over the last couple of months, auditing skills has also become increasingly important.

You could be doing a bunch of different parallel projects or parallel tasks on any given day. For some of them, maybe you’ll try out new community skills, or maybe you’ll try putting together your own skills for them. And if you can imagine, over the course of a couple of months, those skills locally can build up, and you’re probably not going to use all of those. You’re probably actually going to use a fraction of them.

When I’ve gone and I’ve read Hacker News discussion threads on public skills, the debate is very much, you know, there are people who find value in skills. There are people who call them net negative. There are people who feel like there isn’t enough evidence presented about the value. There are folks who feel like they just add a lot of noise. There’s heavy token costs. They’re unreliable. I certainly think that there’s a lot of valid feedback in here. It is sometimes challenging to come up with enough evidence to show, for everyone’s workflows, that these are actually a net positive. But there are plenty of people who say they find these things beneficial, and then plenty of cases where the feedback is valid. Like, hey, show me that this is actually going to be useful enough for me to consider for my project.

Why agent configuration rots

People keep adding rules to their CLAUDE.md files and their skills every time that they see their agent steering them in the wrong direction. I’ve certainly done that over time. And then the file balloons, adherence drops, you add more and more rules and quality can end up getting worse. And you almost end up treating it as this full knowledge base instead of a short decision guide. And that’s a very classic mistake.

AGENTS.md and CLAUDE.md bloat is also another big problem. I think it’s pretty widely acknowledged at this point. There have been a number of different research exercises done on real repos, and they found that there were a lot of configuration smells that were pretty common. Context bloat is pretty common, skill leakage, lint leakage. And most of the agent files that these research exercises have tried out had at least one issue. Files generally do grow past the Anthropic guidance of 200 lines. Some even reach hundreds or thousands of lines, wasting tokens on every session.

And I’m to blame as well for this. When I try looking at some of the CLAUDE.md files that I’ve put together in the past, not for sharing with people but just in my own setup, I’ve also gone past 200 lines. And I found that there were a few common failure modes that I’ve seen in my own files. Things like overly long examples. Redundant content that may have been present in readmes or package manifests or skill files. Add a rule every time the agent errors. That kind of growth can keep compounding. And so I think that you kind of have to think about it in a very, very focused way. And I’ve also seen that being over-specific in your CLAUDE.md file, in AGENTS.md, can sometimes not actually lead to the outcomes that you want.

For the numbers: a June study of 100 popular repositories found lint-related leakage in 62%, context bloat in 42%, and skill leakage in 35%. And in The new rules of context engineering, Anthropic says it removed more than 80% of Claude Code’s system prompt for its Claude 5 generation models with no measurable loss on internal coding evaluations. That result is not a target; the evaluations are not public, and it covers specific models in a specific harness. The lesson is that instruction value can expire, so archive first, and if a rule must always hold, encode it in a test, hook, or permission rather than leaving it as prose the model might lose.

I think there’s been some really good research I’ve been reading over the last couple of months, some good empirical studies that will maybe help with the discourse. And I wanted to make sure that I was covering some of this in an article for people, as I think some people haven’t had time to read some of these papers as well.

Do personalized skills help coding agents?

I’ve been wondering how useful it would be for Claude Code or Codex to gradually learn how I like to work. Maybe I prefer small changes, want tests run a certain way, or don’t want the agent refactoring unrelated code. This paper tries to turn that interaction history into a reusable personal skill. The surprising result was that personalization didn’t help very much. A skill based on one developer’s history performed about as well as a skill borrowed from somebody else. A generic skill built from lots of developers was more useful overall.

For many of us, we think that having a bunch of skills for our specific workflow can actually make a huge difference. But some of the research actually says that that’s not necessarily the case, and that having skills based on broader engineering, broader community best practices, can actually make more sense and actually lead to more value. And, I’m sorry, I should also say, where they can add value is if you have skills that include more examples for specific tasks. And sometimes those broader community skills will have this. For example, if I’m trying to tackle a problem related to, let’s say, scheduling, and there’s a bunch of different ways to approach scheduling, a bunch of quirks around it. If my skills, whatever ones I have, have a bunch of very concrete examples and specificity around it, maybe those can help guide the agent in a certain way. Otherwise, just having some details about, oh, I’d prefer the formatting of my scheduling primitives to look this way, that’s not actually all that helpful.

Personalization did look more promising when the same preference recurred across several similar tasks, though the experiments used an LLM-based developer simulator, so treat this as promising rather than final. My takeaway is to begin with a strong generic skill and add personal rules gradually. I wouldn’t promote a preference into permanent agent memory because I mentioned it once.

Skills once again, when I talk to companies and I’ve talked to enterprises, they see it as high value. They think that individual engineer skills are useful, but then when you have a set of skills for a team or an org, they think that that can lead to compounding value, which is kind of cool. So you’ve got your engineering culture captured in there, your compliance rules, your brand, your internal tooling quirks, how you want to approach consistency across people and different coding agents, how you want to approach the review process and provisioning and any of those types of things.

Do context files help coding agents?

I’ve been using files like AGENTS.md and CLAUDE.md as a kind of operating manual for coding agents. This paper asks whether those files actually help Claude Code and Codex solve more tasks. Across 288 runs on 17 real tasks, they didn’t make a clear difference to correctness.

Context files did change how the agents worked, though. In one repository, the guide warned that the full test suite was very slow. Claude responded by running more targeted tests and wasting less time. It didn’t become better at implementing the feature, but it followed the repository’s workflow more efficiently. I think that’s the useful distinction. A context file can tell an agent about expensive commands, generated files, architectural boundaries, or project-specific safety rules. It can’t necessarily teach the agent how to make a subtle design decision; the near misses usually came down to implementation judgment, and more repository prose wouldn’t have solved those problems.

My takeaway is to keep repository context files focused on things the model can’t easily infer from the code: how to run the right checks, which operations are expensive, what must remain untouched, and where the unusual project conventions live. I wouldn’t fill them with generic advice about writing clean code. A related study points the same way: prose summaries answered 4 of 45 behavioral questions about code while the source itself answered 27 of 45, because summaries smooth over the small details that matter. Point the agent at real code, not descriptions of it.

What I found when I audited my own setup

I was really happy to see Claude put out the doctor command in Claude Code. It’s a good hygiene command, and it basically runs a checkup covering unused skills and MCP servers and plugins relative to their context cost, whether you’ve got an over-specified CLAUDE.md file, slow hooks, or cruft, or things like that. And when I’ve run doctor on my own setup, I’ve been just shocked at things that were still hanging around that I’d completely forgotten about. Like, if you’d asked me, I wouldn’t have guessed that they were still there. I’d completely forgotten that I’d even experimented with them.

I’ll give you one example. At a point in time, maybe four or five months ago, I was curious about different writing skills that people were checking out. There were a lot of different anti-slop skills that people were experimenting with. And so I had tried out a bunch of different ones of those. And I was shocked because I’d completely forgotten how many of those I’d had installed. I had no idea. Like, are these things being triggered together? Or is one taking priority over another? Are they all being ignored? I’d just not realized that those things were still hanging around at all. And so auditing that was very useful. There were some design skills that some friends had written that I was trying out that I’d forgotten about. And now, as we’ve seen the community in some cases converge on some high-quality skills, I would prefer to lean on those than some of the other experimental ones that I tried from a few months ago.

I remember recently seeing a tweet where somebody was saying, yeah, I started auditing my skills and I went down from 250 to 25. And I was just like, how do you end up with 250 skills? That’s crazy. But experimenting with a lot of community skills, these can easily compound over time. And you don’t want to confuse your agent, right? So you’ve got to periodically lint and clean up your agent environment.

Installing a useful skill and keeping it forever are separate decisions.

Auditing quality, not just quantity

I think another thing about skills is just making sure that they’re high quality. Anthropic’s own Skill Creator now includes evals and a benchmark mode for trying to check on quality. There are good community tools for checking on SKILL.md quality. Like, do they have good descriptions, good triggers, clear steps and examples? There are more security-focused auditors and best practices for skills as well.

One naming trap worth knowing: in Claude Code, /doctor inside a session is the configuration audit, while claude doctor in a shell only prints installation diagnostics, which is why some people report that doctor “only shows a status check.” And an installed skill does not dump its whole body into every prompt: names and descriptions load for discovery within a listing budget defaulting to 1% of the context window, and the body loads on invocation. I also review memory separately with /memory, since auto-memory can hold stale preferences even after the project files are tidy.

Audit on a cadence, then test removal

And so I think there’s high value personally in, like maybe every couple of weeks, maybe at once a month even, just running doctor on your skills, on your setup, and auditing what you’re doing and seeing, okay, well, what still holds true? And then if you have the time, actually going and seeing, like, if you were to delete your skills, or you were to instruct your agent, well, don’t use any local skills at all. Only try to complete this task using the raw model and harness. And see, is it actually okay without using any of these skills? And then you can ask yourself, okay, well, actually, maybe it’s okay for me to delete these skills. Maybe that’s fine.

But I feel like sometimes we lean on skills and all of these things we’ve installed as a crutch, because we feel like it’s unsafe to remove them, because we don’t trust that the model and harness have actually gotten better. So I think that there’s a lot that we can experiment with and learn.

Continue to have hygiene around them. Audit the quality, the security. Do run doctor commands regularly, and try to just make sure that you’re keeping your local setup as lean, but as specific, as is needed.

Human judgment doesn't leave the software factory. It relocates.

Fri, 21 Aug 2026 00:00:00 +0000

A software factory is a repeatable loop around software work. If you’re building a software factory, code good enough to ship still needs human taste and ownership. We’ll discuss this including whether you need a factory just yet.

If so:

You’ll likely need humans in the loop upfront for deciding on product intent, system design (if you care) and your quality bar.
Do review code (lights-on factory) but be intentional with where it’s needed the most. I’ve found you want to watch out for where automated back-pressure breaks. Or where maintainability trade-offs need to be made.
Aim for quality checks to happen as early and continuously as possible. Not all of them have to, but this includes type systems, automated tests, mutation testing, security scanners and linting for architecture rules.
Number of checks != quality. You’ll likely need to experiment with what checks give you the best signal to noise ratio. Be ready to tighten or relax your constraints deliberately.

You want to build your factory so some aspects of human taste get encoded in the environment, the agent gives you evidence of its work being right and where a human still “owns” what ships to production.

Do you really need a software factory?

In my experience, you can get surprisingly far with your stock coding harness! i.e. Claude Code or Codex, multiple sessions, good SPECs with verification baked in and constraints. You can even throw a batch of GitHub issues at them with implementation and human-involvement criteria, but it’s when this system needs to be repeatable and event-driven that a factory is helpful.

So I started off by saying a software factory is a repeatable loop around software work. We can actually look at a prompt that demonstrates a very small factory loop here:

Read GitHub issue #123 and the repository instructions before changing code.

Implement only the stated acceptance criteria. Do not modify authentication,
billing, migrations, or existing test assertions. Work in a branch and keep the
diff reviewable.

Run npm run lint, npm test, and npm run build. If a required check cannot run,
stop and explain why. Open a draft pull request with the checks you ran, the
remaining risks, and any decision a human still needs to make. Do not merge.

A goal can keep this moving until the checks pass and we can poll GitHub issues for any specific labels or review open pull requests each morning. Branch protection could enforce a merge boundary and the human can stay in the loop by choosing what becomes ready, reviewing and making the final merge calls etc.

Add a software factory when you need an event-driven queue of work (e.g. Slack triggers, GitHub issues, Linear, a backlog) to run in an isolated cloud environment to handle triage, implementation and testing with some explicit human babysitting. Some end their loop with a monitor agent watching production and filing issues which triage again.

In my experience, the factory becomes useful when the hard part is making your different runs behave consistently, handing work off between agents and avoiding different sessions from claiming the same issue, preserving evidence and stopping production when human review is falling behind.

What solves this might sound a little boring. For example, Warp mention triaging every incoming issue into one of four states - ready-to-implement, ready-to-spec, needs-info, wait-to-implement - and the label is what fires the next agent.

This label does a few jobs in one go: it’s the queue, the lock and since a session only picks up what’s marked ready, it’s where a human can park stuff without saying no permanently.

Workflow wise, there are a few similarities and differences to just using Claude/Codex:

Steering: agent needs course-correction, you can give input and redirect it
Notifications: how the factory says its blocked. This can be because a requirement was ambiguous, it started something risky or it needs human input (steering)
Handoff: move the task, its state and context between the cloud factory/another agent/human reviewer. Good handoffs will keep track of what happened, whats left to be done and why the handoff is needed.

In a good factory, the human isn’t limited to just reviewing and approving the final diff at the very end. They can shape the work early on, steer it during implementation, get it through a handoff or stop it shipping to production.

Verification is where a responsible factory spends a lot of its time. We’ll cover this more later.

If you decide you do need a software factory, building it isn’t the only option. Standing up the infra to scale a factory can be a lot of work and you may want to consider buying vs. building. Factory, Warp and HumanLayer are all working on this.

What my day looks like now

My day-to-day experience of software development has changed a lot over the past year. I’ve been talking about increasingly doing a lot of parallel work with agents, moving towards having a lights-on software factory. And a lot of people have been asking me, like, what do these things actually mean? What are you building? What are the kinds of projects that you’re using these things on? It’s a lot of this:

So on a very average day, I have a simpler lights-on software factory. I can have tasks that are running in the cloud. Half of these tasks might be working on production client applications with smaller companies that I’m working with. They’re going to have real users. They’re going to have real authentication, payments, subscriptions, real beefy risks that you need to be careful with. You can’t just say, oh, agent, just go and do this stuff without having tests and constraints and quality checks in place.

I could work on my open source projects. I could be building out companion sites for my books. I could be working on tools. I could be building apps of my own. And these are all very, very different kinds of applications that I’m working on. And sometimes all they have in common is the tool that I’m using to work on them, right? Maybe I’m working on a migration. Others, I might be doing actual beefy feature work. And the blast radius of the work might also be very, very different.

So as you begin to think about getting to a place where we’re increasingly doing a lot of parallel work, we’re trying to improve velocity, we’re trying to improve productivity, and we’re trying to improve autonomy, which means getting the system to a place where we trust it more, you do have to think about what are the places that absolutely require human code review, human input.

And a lot of that is going to be required up front, right? When you’re defining your specification, your requirements, what is the design of the product going to look like? What is the intent of the product going to look like? And then how are you verifying that the agents have actually gotten the work done right? How are you verifying that they haven’t broken the existing system that has been in place? How are you making sure that it’s meeting your quality bar?

And so generating code is not necessarily the part that you need to worry about the most. Given enough context, agents can write the implementation and run the tests and inspect failure and revise code for us. We need to get to a place where we feel like there is enough of human taste encoded in the environment that we can trust what is being built, so that our human attention can be focused on the places where it’s needed most.

Now, there is pushback from folks saying, hey, well, I don’t buy that you can just automate away a lot of this stuff. It’s not to say that we’re automating away all of it, right? But given the volume of code that’s being generated, I don’t think that it’s realistic for humans to be reading all of it, especially when we’re not building rockets a lot of the time, right? We’re building UI, we’re building full stack applications.

Our judgment, our taste is best focused on the places where it’s needed the most. Like, what are the riskiest parts of the systems? Where do we need to apply human taste? And that can be in the front end. That can be in how the system works. It doesn’t have to be 100% of it.

My cognitive bandwidth does not scale with the agents

The reality is, yes, we can now fire up dozens, hundreds, thousands of agents in parallel, but your own cognitive bandwidth does not scale in the same way. This can feed into cognitive or comprehension debt which I’ve talked about before.

If you remember back to just five, ten years ago, there was a lot of discussion in the engineering community about context switching and the cost of it. We would talk about how people hated when a colleague or someone would walk up to your desk when you were in the middle of a task. It would then take you so long to get back into your flow state because you had to catch back up in terms of like, where was I? What was I doing? Even if you had a little bit of residue there, it still took you time.

We’re now context switching even more than we did before. On any given day, if I’m working outside of a software factory, I can be working on five or ten different projects with agents at a single time, or five or ten different features on a single project at a time. I can have five or ten different sessions, you can effectively say.

That means that I have to be able to stay on top of at least a few of those. It is possible that I’m going to be able to increase how much autonomy I give some tasks if I have trust that I’ve defined the task well enough, I’ve defined the outcome, how it’s going to verify that it’s done well enough. But then there are going to be tasks where maybe I don’t necessarily feel that way and there’s more risk involved or more nuance. I’m going to have to pay attention.

Consider optimising the software factory for your reviewer. Given every one of those approaches still routes its output to one person’s attention, you should ask how much cheaper the factory is making the decisions you still have to make.

A wrong-project mistake

I remember when I’ve been working on multiple parallel projects with my agents, and there have been times when I’ve accidentally done things like, maybe I was working on a web app where I wanted to add in a dark mode, and so I had in my head, okay, well, this is what the shape of this needs to look like. But I accidentally went to the session for a different project, and I started putting in that same prompt.

So I began implementing dark mode for something that absolutely didn’t need it. And so I can make that mistake. I don’t want my software factory making that kind of mistake.

You need to think about this really in terms of a system. You are effectively trying to encode a software engineering culture, a team culture, into a system so that it has those same kinds of behaviors, so that it has ownership that belongs somewhere, so that someone is still on the hook for what happens, and you’re being very explicit about how you think about those things.

When green is misleading

Even in these systems, you want to be very careful, right? Many of us have seen that when you have asked AI to help you pass a test, like we’re talking about a programming test, a unit test, it can change the unit test to satisfy that condition, or it can change the logic of the code to pass that condition. That doesn’t mean that it’s actually followed your intent in order to align both the functional behavior and what the test was supposed to be testing, right?

Just because a software factory is showing that everything is green doesn’t mean that it’s actually green, especially at the start when you’re setting these things up. You need to pay a lot of attention to make sure that your checks, your verifications, all of those are shaped the right way. They’re doing what you expect them to be doing. You don’t want them to be misleading.

You don’t want a situation where you had tests that said, hey, actually, I have gone and changed what authentication providers are supported. You asked me to add GitHub for example, as an authentication provider, but hey, my UI only had space for three, so I’ve gone and I’ve dropped one of the other ones. And hey, by the way, that happened to be one that your customers actually wanted. So you just need to be very explicit about how you want these systems to work.

Btw security is super important too and if your factory reads untrusted input like a GitHub issue/Slack message it might be adversarial and include problems like supply chain attacks. So some explorations into software factories, like Vercel, run their agents in isolated sandboxes holding just the secrets a task needs. That way a compromised run can’t reach what the job doesn’t need. Your defense ends up being layered.

Which old projects deserve another life?

I also think that a big part of how we work these days is deciding what should exist. If you remember back to many years ago before AI, there were so many abandoned software engineering projects, so many abandoned weekend projects, personal projects where they just wouldn’t launch because we didn’t have the time to finish them. We didn’t have the bandwidth to prioritize getting them out the door because they just weren’t that important to us or we couldn’t find the time.

Now it’s fairly trivial for us to complete those projects, but the same human judgment question comes in. Do those projects deserve to exist? Should they be launched? Because you put them out into the world and even if it has just five users, maybe you have to maintain it. Maybe you have a quality bar now that you want to maintain.

I know that I’ve had so many GitHub projects from over the years where now that I have an agent, the first thing I do is get the thing building. Because, of course, you clone it and now it doesn’t build because all the dependencies have changed. Half the things are out of date or have security vulnerabilities all over them, so you have to update that.

Then you have to add tests if you didn’t have tests so that you know that behavior is at least going to be there if you’re upgrading the project in some way, or if you’re migrating it to a more modern language or framework or thing like that.

Then you start to ask yourself, well, maybe, a silly example, but maybe I used Twitter Bootstrap back in the day for this, but now everybody is using Tailwind and shadcn, so I have to re-implement the UI. And what you’ll notice is that suddenly this is taking you more time, right? Yes, the agent can get a lot of this done quicker, but you’re now having to factor in product sense and taste and all of these things.

You still question, well, who is this for? Does it have a market? Is it for myself? Is it for other people? If I’m putting it out into the world, is it still going to be as interesting given that now anybody can spin these things up as quickly?

So I think that human question of do these things deserve to exist? How do we factor in our taste and judgment? I feel like those things continue to be extremely important. That’s where that scarce resource of human attention still really comes in. Back in the day, we only had a finite number of hours in the day. We had meetings. We had to budget in time for design and coding and so on.

Now that we have agents to help us, I think that you have to really just be very explicit about where you’re spending your time and why.

What happened when I built a sample one

So I’m going to talk about the 82-minute factory run. People have been asking me for quite some time, you know, “How do I build a software factory?” Or, “I’m used to using Claude Code or Codex. How do I evolve my setup to using a software factory?”

So the first thing that I’ve been saying is, “You may be fine. Your work may actually be totally fine without needing a factory.” But I did want to give people a reference setup that they can check out. So what I put together is a repository called Factory that you can go and check out. I also put together a demo application and workshop.

Now, for the last couple of years, my go-to demo application for a lot of things has been a movies app. I’m a big movies fan. I love watching movies. I watch movies all the time, and so I have a demo application, which really starts off as a very simple movies app. And what I want the factory to be able to do is go ahead and implement a number of features. There’s a few different features. I want a favorites feature. I want it to be able to maybe do search, and maybe also want a dark theme in there as well, those types of things.

So I have my factory go and begin working with these things. You can check out the implementation. One of the benefits of it was actually catching real problems. These problems may not have been things that I would have caught if I had just asked it to do a one-shot implementation.

Maybe around the 60-minute point, I was feeling like, “Wow, this is going unusually slow.” I asked my harness using the factory, “Why are things going slow?” It said, “This is actually totally fine. All the verifiers are still running.”

You might have expected individual tasks to take 10 minutes, 15 minutes, 20 minutes, but they can take two to four times as long once you begin to include verification, retries, browser checks, human review, any of those extra delays.

I do think that these can add up to better quality and better trust in the system. From a measurement perspective, you might look at metrics like cost per merged PR and code shelf life as comprehension debt metrics.

You also need to think about what is useful delay versus factory overhead. The verifiers, in my case, caught some real problems. A little bit of the time was maybe sunk into producing evidence that I wanted. Some of it was overhead in the factory running. I didn’t really spend any time optimizing it, but a factory that just runs a lot of checks that you’re not finding valuable does not mean it’s a high quality one.

You want to study how, for any repeated checks, are they irrelevant? Are they noisy? Are they actually making the system safer?

A verification budget

The way that I think about the budget for verification, this is basically what we’re talking about. We’re talking about a verification budget. I think about it in the same way as I’ve historically thought about performance budgets.

There are going to be certain kinds of checks that you can run early on in your software development lifecycle, and there are going to be some things that are so heavy, but they offer so much value that you will want to run them later on. There are some kinds of fast checks, linting, for example, type checking. These are relatively fast checks that you can run early on.

Our full suite of tests can be run closer to right before a draft PR is being put together or after that. That can include mutation testing, browser testing, security checks, anything like that.

I think that you don’t necessarily want to replace these with just summaries. You want real tests, but you just need to make sure that you’re budgeting for them in the right places because you don’t want to slow down your development loop. I certainly never want to slow down my development loop. Having a fast iteration loop is important to me, but I also want to still have those checks and balances.

When a run doesn’t ship

A lot of what I’ve written above concerns the checks.

In their software factory, Vercel marks every agent run as “success”, “flawed”, “blocked” or “manual” and only “success” ships to production. The rest re-enter the system. I’ve been thinking about runs in similar terms.

“Flawed” here means the wrong thing was implemented or maybe it didn’t have full context, so that has to be fixed. Blocked means the environment may have been missing a credential so you have to provide it. Manual is a boundary the factory may not be allowed to cross it yet.

Two of the three things here may have mechanical fixes and the last one is about trust.

While this is great, what sorting doesn’t show you is cost.

Back to my factory implementation with the TMDB app, the quick finder with no rejections took 7 minutes. Favorites, with two rejections and a human decision in the middle, took 56. Same factory. So I’d pair the taxonomy with per-stage timing, otherwise you know a run came back flawed without knowing what finding out cost you. The other thing I’d fix is the handoff at the boundary: my sample factory stopped issue the first issue and moved it to factory:needs-info, which was right, but I didn’t know where to put my answer. A manual run isn’t finished when the factory stops but when the human knows what to do next.

Autonomy is not a single setting

I wrote a couple of weeks ago an article about agentic autonomy and how to think about autonomy because autonomy is not going to be a single setting for every single project.

Verification buys you trust, and it buys the ability to grant more autonomy to your agents. So if, for example, I am working on a non-trivial change, but I have a number of checks in place, everything gets verified correctly, and maybe I’ve hand checked it myself. The next time I’m going to do a task like that in the same project, maybe I’ll feel comfortable giving the agent a little bit more autonomy.

That’s the thing that you think about when you’re building these software factories. Your verification is going to change with risk. Your goal is the best signal to noise ratio. You don’t just want to have some large checklist that you’re running.

The feature I had to relearn

There was a feature that I’ve been putting off on a day when I’ve been using multiple sessions with Claude, and I was working on a few different projects at a time, a few different features at a time per project. And so Claude had implemented the feature that I was working on. It looked like the tests were passing. I hadn’t put a lot of thought into verification, but the tests passed, and so I thought it worked. I merged it.

And so this was a favoriting feature. I thought that this was actually pretty good. I tried to check it out in the browser. It seemed like it was okay, but a couple of days later, I actually returned to the code because there were some tweaks that I thought I might make to this.

I didn’t want to just ask my agent to make the changes, because it was just a subtle way that it worked. You tap on the icon, and it would not show the right effect on tap, and so I wanted to just tweak it. I wanted to understand how it worked so I could guide my agent correctly.

I returned to the code, and I couldn’t explain to you how the feature worked. This repository was mine, right? I’d approved the change. I understood how a lot of it worked, a lot of the repo worked, but my understanding hadn’t kept up pace with all of the code that had been building up.

What I failed to absorb was how this feature that had been added actually worked, how the UI worked, how the effect on it worked. I had to redo this feature and actually go step-by-step, “How does this work? How can I understand it?”

What parallel work does to understanding

When you’re doing parallel work, it amplifies this overall problem, and it gets even more amplified when you’re doing it in a software factory. When you’re doing five or 10 sessions, they create much more than just a review volume problem. They create several mental models that can end up going pretty cold while you’re working elsewhere.

We’ve historically talked about the challenges with context switching, and as soon as chat compacts, you reject some approaches, you try out different things, you’re pairing with the agent, you’re going to have a difficult time remembering everything that happened in your session.

You can scroll up, and as compaction has been happening, you’re not going to have everything there, and you’re not going to be able to store it all in your head. Code often preserves a decision that was made, but not why the decision was made.

This is something that I think can be a useful learning for you, where it’s important, consider asking your agent to actually store information about its trajectory, or interesting lessons about how it approached a problem so that you can go back to it later.

This can or can’t be something that you decide to commit to a repo. You can keep it local if you want, you can share it with a team if you want, but that can be something that can then be consulted later on. Rather than you relying on it maybe being in a session, or you maybe remembering about it later.

Ownership doesn’t disappear

There is a broader principle underneath all of this.

The percentage of code physically typed by humans may fall dramatically. I don’t think human ownership needs to fall with it.

Someone still chooses the problem.
Someone still chooses the architecture.
Someone still sets the quality bar.
Someone decides which verification signals deserve trust.
Someone decides when the evidence is sufficient to ship.

And when the resulting system fails, “the agent wrote it” doesn’t cut it. This is why I don’t think the future of software engineering is best described as humans leaving the loop. Instead, human judgment is being relocated.

We should remove people from the parts of the loop where machines can produce stronger, faster, more deterministic signals. At the same time, we should concentrate people around the places where context, taste, risk, and long-term ownership matter most.

The best software factories will not be defined by how completely they eliminate human involvement.

They will be defined by how intelligently they place it.

Keep human judgment upstream on intent, system shape, and the quality bar. Review code where automated back-pressure becomes weak or the consequences become subjective. Push every deterministic signal as early and continuously into the loop as possible. Tighten and relax constraints deliberately as the system earns or loses trust.

A human still has to own what code ultimately ships. Code good enough to ship still starts there.

This post was originally published on my Substack.

Practical Loop Engineering

Fri, 14 Aug 2026 00:00:00 +0000

The way that I typically work is I have anywhere between five and ten agents working at the same time in parallel. There are going to be some tasks that I’m very happy to delegate fully to agents, as long as I have a very clear idea of the stopping conditions and the constraints around them. And then there are going to be some tasks where I am going to want to keep a closer eye and code-review what the agent is doing.

Now within that, you’ve probably heard about loop engineering. I talked about it a couple of months ago when I wrote a big blog post about it.

A loop is an autonomous, self-correcting feedback cycle where an AI agent repeatedly acts, tests its results and adjusts its approach until a specific goal is met

There are now basically two core primitives you can think about. In Claude Code you have a goal primitive, which can drive a single bounded task forward until you’ve got a particular goal, like a measurable finish line that’s been met. And then loop reruns on a timer or a fixed interval, so you can use it to kind of schedule changes.

Before the primitives were primitives

I remember back before we had primitives baked into Claude Code and Codex, loop engineering was heavily about setting up your own bash loop, a hand-rolled thing. That’s how I approached it. And you might remember earlier in the year, a number of us were playing around with the Ralph loop by Geoff Huntley. We were experimenting, we were sharing our workflows, we were sharing what worked and what didn’t, but it was largely on some of our personal projects where, if we ran into a wall, it didn’t really have a big cost to us because it was on personal projects.

And as we’ve tried to see what patterns, what aspects of loop engineering are now a little bit more baked, I think we have a clearer idea of how you babysit it. I can now largely rely on the output of the primitives in Claude Code and Codex. We’ve come a little way. But at the same time, you need to be very diligent, because loop engineering where you kind of leave it sitting alone, and you haven’t really thought about whether the end goal or the constraints have been well defined, can leave you in a problematic state. This is why there’s nuance when deciding to use it for an evergreen codebase without users or as much historical complexity vs. say a brownfield bank codebase.

How the Claude Code team frames loops

The Claude Code team published their take on four kinds of loops and it lines up with how I use the primitives (their write-up). Before we get to it, here’s my quick summary:

On the Claude Code team, we define loops as agents repeating cycles of work until a stop condition is met. We categorize a few different types of loops based on: how they are triggered, how they are stopped, what Claude Code primitive is used, what type of task is most appropriate for each. Not all tasks require complex loops; start with the simplest solution and use these patterns selectively.

Every prompt you send starts a manual loop with you directing each turn. Claude gathers context, takes action, checks its work, repeats if needed, and responds. We call this the agentic loop. For example, ask Claude to create a like button. It reads your code, makes the edit, runs the tests, and hands back something it believes works. You then manually check the work, and write the next prompt.

Their write-up walks each rung.

On goal-based loops:

Sometimes, a single turn is not enough, especially for more complex tasks. Agents do better when they can iterate. You can extend how long Claude keeps iterating by defining what done looks like with /goal. When you define the success criteria, Claude doesn’t have to make a determination on what is “good enough” and end the loop early. Each time Claude tries to stop, an evaluator model checks your condition and sends it back to work until the goal is met or a number of turns you define is reached. This is why deterministic criteria, such as number of tests passed or clearing a certain score threshold, are so effective. For example: /goal get the homepage Lighthouse score to 90 or above, stop after 5 tries.

On time-based loops:

Some agentic work is recurring: the task stays the same and only the inputs change. For example, summarizing Slack messages every morning. Other work depends on external systems, and a simple way to interface with one is to check it on an interval and react to what changed. For example, a PR which may receive code reviews or fail CI. For these, you can trigger when Claude runs with /loop, which re-runs a prompt on an interval. For example: /loop 5m check my PR, address review comments, and fix failing CI. /loop runs on your computer, so if you turn it off, it stops. You can move the loop to the cloud by creating a routine with /schedule.

And on proactive loops, the top rung:

Triggered by: an event or schedule, with no human in real time. Stop criteria: each task exits when its goal is met. The routine itself runs until you turn it off. Best used for: recurring streams of well-defined work: bug reports, issue triage, migrations, dependency upgrades. Managed usage by: routing routines to smaller, faster models and using the most capable model for judgment calls.

Their verification advice is worth lifting whole, because it moves the manual checking into something Claude applies itself:

---
name: verify-frontend-change
description: Verify any UI change end-to-end before declaring it done.
---
# Verifying frontend changes
Never report a UI change as complete based on a successful edit alone.
Verify it the way a human reviewer would:
1. Start the dev server and open the edited page in the browser.
2. Interact with the change directly. For a new control (button, input,
   toggle): click it, confirm the expected state change, and screenshot
   before/after.
3. Check the browser console: zero new errors or warnings.
4. Use the Chrome Devtools MCP, run a performance trace and audit
   Core Web Vitals.
If any step fails, fix the issue and rerun from step 1 - do not hand
back partially verified work.

Goal

The way that I use goal is I use it for building any specific piece of work until it’s provably done. Like for example, you can use a goal to say: make sure that this experience loads in five seconds, keep going until it’s done. And it will continue to use an independent evaluation check to keep checking if the completion criteria has been met. What’s better is to be even more specific about what tooling is being used.

As for real goals that I’ve run to a number, there’s a few things that I’ve done. I’ve used goals for running through GitHub issues: review and close the last 10 issues, or review and move the last 10 issues forward, something like that. And that’s semi-open-ended, right? Or: let’s make this page load 50% faster. Sometimes that works well, sometimes it doesn’t, but it’s really about the experimentation.

Loop

Loop is a little bit more of a scheduler, so it keeps an eye on something or repeatedly executes a pattern on some cadence. So think of it a little bit like a cron. It’s best for doing things like polling logs or monitoring external states. You could use it for potentially checking in. If there are repetitive tasks you find yourself doing on some cadence, loop is pretty good for that.

What I delegate, and what I watch

For me, I use probably between five and ten agents every day. Very typically I’ll max out at about five concurrently. And some of those tasks might be ones that are a little bit safer. So maybe it’s, hey, I implemented this feature, go write the documentation for it. Or go double check that we have sufficient test coverage. Or something in that vein. If I’m working on a more complex problem, or something where I know that even if I’ve given it a good spec, or what I think was a good spec, and I’ve tried to give it some stopping conditions, there is still a reasonable chance it may not get everything right, I would try to watch it more closely. If the task involves anything just a little bit sensitive, whether it is I’ve given this access to a system, or whether the feature happens to touch authentication, or something related to security or finance, I’ll definitely be watching that closely.

Generally speaking, I do think we’re going to get to a place where folks are increasingly comfortable with delegation, as long as they’re able to have these clear ways of verifying that their goal or their stopping condition has been met. But you do still need to take a look at the code, at the thing that’s being generated, to make sure that it’s meeting your mark.

The other habit that matters here is not letting the agent that did the work decide the work is good. One sub-agent drafts the change. A separate one verifies it.

Sometimes an agent can be confident about something, and a verifying agent can catch things that they weren’t necessarily expecting. Like if it thinks that the baseline performance of an experience it’s generated is actually fine, and it’s only evaluating performance based on desktop, but you’re actually caring about the experience on mobile. That might mean the agent is being very confident about one dimension of the problem, but not the other.

And I learned this one the hard way. I was curious to figure out: is there something that we’re missing that we haven’t gotten direct feedback on from our users, either in an issue tracker or in a comment somewhere? And so I asked it to go and take a look at some of our competitors, and put together a list, and some PRs, not pushed, but some PRs locally, of what solving some of those gaps might look like. And I almost pushed some of those changes. But I didn’t actually look at them closely enough. I read through its research, but I didn’t look at the implementations closely enough. So I delegated the task, but I was close to delegating the judgment as well. Now, when I actually looked through the changes, what I realized is that it would introduce a lot of additional complexity for our users, for, I think personally, not all that much gain. And so I feel like you need to sometimes check yourself, that you are not delegating the taste and the judgment to your agent. You’re delegating the task, and then you are actually checking back that it’s meeting your bar.

The evaluator sitting behind goal is not that checker, by the way. It doesn’t look at the content to see if it’s good or bad in any way, shape, or form. All it does is examine the conversation transcript to see if the hard rules you specified have been met.

/goal Refactor the data-fetching layer in Dashboard.tsx until Lighthouse performance score is >= 92 and LCP is under 1.8s as shown by the Lighthouse CLI output. Do not change the public API of any hooks. Each turn must improve at least one reported metric; abort if two consecutive turns show no improvement. Stop after 10 turns.

The workflow I run every day

One of the workflows that I do every day: I have a popular open source repository called Agent Skills. We’ve got over 80,000 stars, and up until recently we were getting anywhere up to like 80 or 90 pull requests that we had to review a day. And so every day I would go and I would spend some time checking on this. Now with loop, what you can do is say: well, every 24 hours or every 12 hours, check the GitHub repository for any new open issues and provide a summary of their urgency, or provide a first pass review, or anything like that.

/loop every 1h "Check the GitHub repository for any new open issues. Provide a bulleted summary of their urgency."

Combining loops and goals

Now you can also sort of combine loops and goals. So you use loop to schedule a check, and then goal to solve the problem. So you can say something like: loop for every 24 hours, check GitHub for issues labeled bug. If one exists, use goal to implement a fix until all local tests pass and push the branch.

/loop every 24h "Check GitHub for issues labeled 'bug'. If one exists, use /goal to implement a fix until all local tests pass and push the branch."

But also keep in mind that goal has got some limitations around just how much you can sort of cram in there.

Their write-up also has a composed example that shows where this is all heading:

The primitives above, along with other Claude Code features like auto mode and dynamic workflows (research preview) can be composed into a loop for long-running work. For example, to handle incoming feedback, you can use: /schedule (research preview) to run a routine that checks for new reports, /goal to define what done looks like, and skills to document how to verify it. Dynamic workflows to orchestrate agents that triage each report, fix it, and review the fix. Auto mode so the routine runs without stopping to ask for permission. Putting it together, a prompt could look like this: /schedule every hour: check the project-feedback channel for bug reports. /goal: don’t stop until every report found this run is triaged, actioned, and responded to. When fixing a bug, use a workflow to explore three solutions in parallel worktrees and have a judge adversarially review them.

What the triage system actually does

Inside the PR triage, when I have loops and goals working together for me, what I end up having is a system that allows me to really continue accepting PRs and issues, being able to stay on top of what is coming on my plate every day, and especially being able to cross-reference. That’s been a really big deal for me. If I want to work on a particular goal of, hey, we’re going to be redoing this part of the system, and I need to make sure that any issues that happen to touch it are closed as a result of this rework, or we’re not stepping on anyone else’s toes, I can make sure that that’s well defined.

Scheduled tasks are really useful for regularly just reviewing new PRs and closing things that clearly don’t fit. One good stopping condition, for example: we have a set of contribution guidelines, and our contribution guidelines include things like, hey, we currently don’t accept translations.

And it’s not that we don’t care, it’s that they’re difficult for us to maintain, because we don’t speak all the languages often coming in. And so if we tell it, close any issues or close any PRs which happen to touch that aspect of our contribution guidelines, that’s something that can do really well when it’s on that schedule. And so it reduces the overall batch of things that we need to review.

What loops don’t buy you

I’m often asked what loop engineering is not a great fit for. Generally speaking if you don’t have a clear idea of what the end-state/done/good means for your completion, it may not be the right pattern for your work. For example, a vague goal would be “keep going until this UI design is good”. What does that mean? Good to who? How is it being evaluated? Tasks that require human taste, subjective design, or open-ended creative exploration aren’t a good fit. When you have a pretty clear idea of the goal, I think loops are a good option to consider.

The fine print

One classic sign that you’ve got a loop spinning in place is the same command being tried over and over without any change in the result. Give the same command a third time with no change from the second and it’s probably time to stop.

One bit of fine print worth knowing. Recurring loops expire seven days after creation. I’d been telling people this was three days. It’s seven. And loops are session-scoped, so they stop when you start a new conversation - though resuming that session with –resume or –continue brings back any recurring task still inside its seven-day window. If you need something that outlives your session, /schedule runs it in the cloud.

If there’s a check you already run every morning by hand, that’s your first loop. Mine was the pull request pile.

This post was originally published on my Substack.

Agentic Code Quality

Sat, 8 Aug 2026 00:00:00 +0000

For much of human history, we’ve evaluated code quality via code review: someone reads what you wrote and makes sure it’s clean, thoughtful, fast, understandable, and tests well. For agents, that approach doesn’t scale well; there’s just too much code for anyone to read. As a result, more and more of our quality checks have to happen in the harness, environment, and operating system around the agent. I still read and review code, but am very intentional about where I am comfortable with constraints as the check.

Software quality now depends on the constraints you set around your agents.

Constraints define what the system is allowed to do by throwing tests and deterministic constraints at an agent’s proposals. It’s by setting and maintaining these constraints that we build loops that reliably deliver high-quality production software, even when agents are creating hundreds of thousands or millions of changes every single day.

We call these constraints quality gates, and they take many forms. They include conventional unit tests, property tests, and acceptance tests. They include mutation testing, where we generate variations of code, run it against the same tests, and make sure that people aren’t sneaking bugs in that we’re missing. They’re metrics around code quality, such as cyclomatic complexity and line length, that help keep things readable. Constraints also play an important role in what proposals the system will accept and apply as code changes. By the time a change proposal moves from the interpreter running the agent to the agent controller and out to production, we’ve done enough checks on it to be confident that it’s safe to ship and the impact of its change is well within the scope of the agent.

Guillermo's list is a good test for whether you can afford to skip reading. Notice that every "yes" is really a statement about how low the stakes are - no users, throwaway code, prototype. Once the stakes go up, something has to read the code. If it isn't you on every diff then it has to be the constraints.

An agent can propose anything. Your constraints decide whether a proposal is safe enough, correct, scoped, and useful, for you and your team to ship.

This model offers a lot, but also leaves out many pieces, and those omissions are worth thinking about today. One issue is autonomy; agents might apply their intentions well, but may fail when there’s missing information or when what they try to do is ambiguous. That applies both to the task itself and to how it’s parameterized by the harness, environment, and other components. Many of the reasons that humans fail to ship great code are shared with what agents might do: brittle environments that don’t hold up under script-driven stress, nondeterministic builds, missing permissions, and weak tests. This motivates a better environment that gives agents trustworthy feedback, allows for low-damage failure modes, and makes it easier to progressively build up success.

The environment we’re after is one where an agent can do real work, get feedback it can trust, and fail without doing much damage.

The other important issue is trust. We can’t credulously hand off intent to something even as smart and robust as a modern agent without checking for correctness. We start with trust, but it has to be hard-earned.

Some constraints shape work before it begins. Others give feedback while the agent is working. Others decide whether its output can cross the production boundary at all.

There are a bunch of ways to model how we put a verification structure around a system.

In my experience it helps to have a broader, but intentionally chosen, set of checks for your constraints instead of solely relying on unit tests. The idea is that each check has a distinct responsibility and that can range from type safety and performance to late-stage security scanning. Folks can define their own constraints too, including architecture rules that linting tools like ESLint can enforce. Many of these tools have built-in hooks that can be used to pull in agents, or humans, when things break.

For now, much of the difference between useful agent output and slop still comes down to the skill of the team operating the loop.

AI gives us high volume code generation and velocity, but this can also mean it gets harder for humans to review every single change. You have to instead be intentional with where their attention is going. If you put a human check into a system that otherwise moves at machine speed, don’t be surprised if that impacts productivity. Human attention is scarce and valuable so we should proactively direct it to those most nuanced problems that require our judgment. Downstream humans should only be pulled in when the automated guardrails for constraints break.

Human “code review” in the future is going to look very different

Correctness is one important dimension, but you may care about others as well, like maintainability, performance, security, efficiency, and comprehensibility. Just as correctness decomposes into many signal types, so does the rest of quality. And while it matters how many constraints we have in place, it matters more whether they’re challenging enough to meet our bar for quality and production readiness.

Software quality isn’t a single metric. Think of it as a collection of signals of varying importance to you and your team.

Back-pressure can be implemented through many tools: compilers rejecting invalid code, tests failing, security policies blocking bad practices, CI declining to deploy. Ideally it exists throughout the loop, not as a single review at the very end of all the work.

Constraints and back-pressure let agents catch bad work before its a problem

What happens if we can’t apply the constraint because the volume of changes is higher than our tools can consume? We end up building a queue and relying on a verification system that moves at human speed. To scale, we want to push as much as we can into the verification loop throughout and not wait until the end. If we can scale within our automated checks, we can increase the speed and throughput of our entire delivery system. If we run out of room in the verification loop, we need to do one of several things. First, we can scale our verification system and create more capacity to constrain and push back on changes that come in. Second, we can reduce the rate at which agents generate new changes so that verification can catch up to the volume of work. Third, we can lower our quality bar so that verification doesn’t push back as hard as it otherwise might. From a scaling perspective, we need to be ready to do all of these things. At the same time, we should not stop short of realizing that we could actually get more done by un-constraining in some directions. Maybe we can increase the speed of agent-generated changes by providing swarms of agent developers or automated software factories to create changes without waiting for us to review each of them.

And in some places we might want to give them more freedom as long as we keep tighter constraints in others. By providing tighter constraints where we care the most, we can maximize our throughput without sacrificing quality. Throughout these decisions there are many options. Most obviously, we have to trade between different dimensions of quality. As we’ve emphasized, security is very important, but we’ve also had to trade between delivering security and delivering a product on time. There is a spectrum from innovation-focused at one end to quality-focused at the other. Somewhere along the way, we have to make choices about where we want to be on that spectrum.

We want to send clear feedback from the environment and the system back to our agents or teams, so that people can focus on the more subjective concerns of taste, intent, and architecture. If we can help humans stay inside the safe range of constraints, we can avoid the need for them to work hard trying to figure out where things went wrong.

Software quality includes more than just correctness. Software quality also means maintainability, good performance, security, efficiency, and being easy to understand. All the constraints that help us meet these standards and keep our production flowing create back-pressure in our delivery pipeline.

We need to make deliberate decisions about where to apply strong constraints and where to remove or relax them. Apply strong constraints where they’re serving both of these goals. Don’t support them if they’re not serving one or both. Be ready to raise or lower standards as the case may require. And remember these constraints at different points in the software system are what make software quality enforceable.

We should apply strong constraints where they’ll serve this dual purpose best and consider removing or relaxing constraints that aren’t serving either purpose well. We should also be ready to shift quality bars up or down as needed. In effect, these constraints at various points in our software system are what give quality its teeth. In many cases, we can create more back-pressure and more constraints by deploying new tools or strengthening tools that are already in place. All of these things can be used to push back on most change requests. We want to build them throughout the pipeline. We don’t want to wait until the end of the pipeline when our CI system will simply tell us that we’re not allowed to deploy without fixing problems. We want to use these signals as early as we possibly can, through every possible pathway. The ultimate constraint in this system is the one we place on ourselves to stand behind the decisions and actions we’ve taken to build the system and to operate it. But like all other constraints, we need to make thoughtful trade-offs about how much we want our own judgment to restrain, to back-pressure, and to act as a final check.

Quality is in the constraints that we place around our agents. So as you’re thinking about quality for your own apps, take this problem statement and come up with your own constraint-driven plan.

This article was rated 100% human written by Pangram 4.

This post was originally published on my Substack.

Software Factories, Light and Dark

Mon, 20 Jul 2026 00:00:00 +0000

A software factory is harnessing loops at scale. You can run the loop with humans in it (light factory): trading judgment and concentration against speed and breakage. Or you can ignore the humans (dark factory) and let those agents scope, build and ship code, without anyone really reading the details. But if people stop reading, they’ll stop understanding your software. Your hardest job now is knowing which checks to build and how much autonomy to delegate.

This idea of the software factory is a term that dates back to Bob Bemer’s paper, “The economics of program production,” given in 1968. For half a century, many have dreamed of a world in which software is a repeatable and instrumentable production process (analogous to stamping out car parts in a factory) rather than the isolated craft of individuals. Historically, this dream has generally (although not universally) fallen flat, in part because of the difficulty of stamping out ideas.

But in the last two years, things have changed dramatically enough that now it makes sense to take a fresh look at the old dream. And since some subtleties can easily be glossed over, it’s worthwhile to be somewhat precise about exactly what is really new and different, and what may be recurring traps, dressed up as new opportunities.

Dex Horthy, co-founder of HumanLayer recently gave a great talk at AI Engineer World’s Fair called “Harness Engineering is not Enough: Why Software Factories Fail.” worth checking out on this topic.

The loop is the atom. The factory is the loop at scale.

Structure is everything, and it all starts with small units. The whole stack is really three concepts layered on top of each other: the loop, the harness, and the factory.

A loop is one agent doing a single job on repeat: gather context, take an action, check the result, and go again until some condition is met. It is the smallest unit of agentic work, and everything above it is just loops stacked on loops.

The point of loop engineering is that you stop prompting the agent turn by turn and instead design the small system that prompts it for you.

A harness is the walls around a loop: the sandbox it runs in, the tools it can reach, the memory that survives between runs, and the gates that decide what “done” means. The loop is the behavior; the harness is the environment that behavior runs inside.

Hand a raw model no harness and it will happily spin forever. The harness is everything around it that makes it useful and safe to run.

A software factory is many harnessed loops running at once, fed by a queue of work and drained through a review gate into production, with humans owning the whole thing from above. It is not a bigger agent; it is an org chart made of loops.

The final paradigm shift is moving from writing code to building and running the factory that writes it. The unit of work shifts up a level, to the loop, the harness, and the flow between them, rather than the individual code diff.

Loop → harness → factory. A factory isn't a smarter agent; it's many harnessed loops feeding one review gate, with a human owning the outer loop.

The factory, drawn

The central slide Dex spent most time on was brilliant because it’s a clarifying wiring diagram that visualizes what otherwise is an obvious loop. Here’s my take on it:

The factory is a closed loop: intent and production signals feed a queue, the harness builds, automated checks and review gate it, deploy ships it, monitoring turns prod back into signals.

Intent flows from the vision of engineering leadership, and directly from engineers, into a queue of things to be done. Signals driven by incidents and user requests drive the same queue. The harness is just the thing that picks an item from the queue and builds a change for it. Beyond the harness, we can see all the automated checks required to make changes safe enough to let into production. These automated checks run at once, effortlessly, without any conscious involvement from engineers, thanks to CI, tests, static analysis, and scanning of all kinds. The only decision point here is the review gate. After approval, changes are deployed and monitored in production, with monitoring data feeding back into the signals that kicked the loop into motion to begin with.

By and large, every box in this diagram is almost zero cost: generation, tests, scanning. They all run at scale for negligible cost. There is only one expensive box that proves stubbornly resistant to scaling, and that’s the review gate. That shiny amber box is “judgment”, and where the crux of the argument about whether we can make development faster and more frequent resides.

Why we call it “dark”

A dark factory runs with the lights physically off, because the only things on the floor are machines and machines don’t need light to see. A dark software factory is the same move: code ships that no human has read, verified only by other machines.

The image is borrowed from manufacturing. Its origins are physical rather than digital, rooted in facilities where the lights are turned off and the work is carried out by robots. FANUC in Japan has been running lights-out factories of this sort since 2001; Xiaomi, in 2024, opened a heavily automated dark factory of its own. What these have in common is a product assembled and shipped without a single human having read any of it. The “dark” comes in when that act of reading is removed from the process.

I’m not borrowing the concept for its vibe or as an insult. For all its creepy buzz, “dark” here is a simple physical claim: the original factory floor, but without light. In software, the floor is the diff. Whoever wrote the diff, whoever reviewed it, whoever shipped it, those humans are gone, and what remains is a diff verified only by the machines that built it.

This is a surprisingly easy thing to do, at least at first. It’s easy because that missing review step gets in the way of everything. Its absence makes your perception of your team’s vertical throughput seem suddenly and radically higher. It feels as if you’ve broken the sound barrier. For all its apparent ease, it’s harder than it seems to survive those dark workflows, with all their buried costs.

Harness engineering is not enough

The harness of orchestration, sandboxed prototyping, and tool calling as models interact with the world and each other will become increasingly powerful and effective. However, there’s an inherent in-model failure in trying to keep up with codebase quality over the long game and through additive changes, and I think there’s good reason to believe that models alone will ultimately lose that battle against comprehension debt.

Comprehension debt is the widening gap between how much code exists and how much any human still understands. A dark factory doesn’t pay it down; it takes it on as fast as it can, with the tests green the whole way.

This is an important distinction because models do well at some tasks. But for anything that isn’t an immediate change to a small part of a codebase, especially in a complex brownfield system, model-only automated coding faces an insurmountable obstacle. Weekend toys and side projects are alike in that a few months of development cycles is usually enough to get things in working order, or at least close enough. But an enterprise system that has been under development for a decade or more is a different beast; it has to be maintained, in a professional environment at a professional pace. Three to six months into a project, you’re already drowning in unread code. That kind of environment, and especially the constraints enforced by production code, would make even a powerful agent do poorly, all of it in contrast to the vibe-coding enjoyed by developers working on weekend toys.

Dex reports from experience that this is a major failure, so much so that it required painstaking manual debugging to pinpoint. This came from running a fully automated code factory for about four months, during which no human looked at the code that was written. Underlying the experience is a tradeoff between two conflicting metrics. One is maximizing token utilization, the number we currently treat as progress. The other, which it quietly minimizes, is the amount of the system any human participant still understands at any moment.

Where the dark factory truly shines is in its ability to burn through pristine code while the tests stay green. The ultimate reckoning, when it comes, will not be a dramatic “it all goes sideways” moment. It will be quiet and late.

Dark and lit are the same pipeline with the lights in different places. The lit version doesn't just re-add review at the end - it moves human judgment upstream to design and architecture, too.

The bottleneck was never generation

The fundamental constraint in a software factory isn’t how much code we can churn out: it’s how quickly we can verify it.

Back pressure is the rule that you can only hand a loop as much autonomy as you can cheaply and reliably verify, and not one inch more. Verification, not generation, is the real constraint on a factory.

Because unbounded generation capacity is in perpetual tension with the finite, non-scaling resource of human attention, the core problem is the gap between cheap generation and bounded review. Look at the funnel: as long as the neck representing verification doesn’t widen, it’s going to back up. As Dex points out, volume alone isn’t the problem: what we’re really suffering from is a surplus of bad PRs. When you’ve got high volume without trustworthy gates, manufactured defects are unavoidable. This is just back pressure again: autonomy can’t expand beyond what can be cheaply and reliably verified.

The second-order problem is why improving the model shouldn’t automatically close the gap between what it can generate and what can be verified. Training on well-architected systems is an arguably more difficult proposition than passing simple tests: remember, the cost functions measuring architectural excellence aren’t measured in seconds or even minutes, but in months and years. Tidy gradients are functionally impossible to compute, so a system expecting crisp, instant evaluation of complex design decisions isn’t going to be trained on good examples.

Generation is a wide mouth; verification is the narrow neck. Speeding up the mouth just deepens the pile at the neck.

Turning the lights back on

A lit factory is the same pipeline with the lights left on where judgment lives. The agents still do most of the building, but a human reads what comes out before it ships, and the lights stay on wherever a wrong call is expensive.

The lit version doesn’t tack review onto the end but moves the point of human judgment upstream, to the product, the design, and the architecture before an agent starts a loop.

One great thing about that upfront hour is that it leads to fewer implementation hours. It turns a long, frustrating code review into a quick read of a two-hundred-line plan. You get to review a decision before it’s built, so later you aren’t chasing through two thousand lines of generated code to find out what the decision even was. Some decisions are expensive and long-lived enough that you’d want a person in on them early, before the cost compounds. Of course, there are still times you look at diffs, even when you’ve spent time up front.

You might be thinking that all sounds unglamorous. You’re right. The safety net is made up of perfectly ordinary architectural practices we’ve always known about and mostly ignored: good types and method signatures so that mistakes are caught by the compiler instead of in production; test seams where we can pin behavior and make change observable; laying out the code so the next reader, human or model, knows where to find the thing they care about; keeping call stacks short and legible; keeping component boundaries well defined so a change doesn’t have a huge blast radius; and dependency injection so we can swap out one piece for another. None of it is new. We’ve always said we care about good architecture. But now that we’re using automated coding agents, that architecture is finally doing a second job as a cheap and hard-to-fake safety net against the mistakes the agent will make.

That safety net has to live outside the model, because the model won’t supply it. The coding agents that feel most capable, Claude Code and Codex among them, are reinforcement-trained against their own harness and tools: fluent with all the tools and idioms of the trade, but not with things like long-term maintainability. The deliberate architecture we’ve always talked about is the tool that catches that debt, and the investment we make in it is us buying back our autonomy. Put that together with safe infrastructure, and there are some tight, low-risk loops you can run unattended. Horthy described one in a recent post: a nightly GitHub Actions cron that fixes exactly one anti-pattern, a lint violation or a needlessly optional prop, commits, and opens one small pull request, all on its own, so the team wakes up to a slightly better codebase and a diff short enough to read. But for loops with high enough stakes, you don’t want to risk waking up to a broken auth system, billing engine, or public API contract. Keep the lights on there, and trust that a person with judgment and a real working knowledge of the system will catch the mistake.

What earns a loop the dark

This rule applies whether you call it back pressure, verification, or the light switch.

A loop can earn itself fully automated status only if the check is cheap, runs at high frequency, and relies on something that can’t be easily faked out. Green-or-red oracles, type gates, property tests, and a review agent coupled with a real rubric all fit. You also need the oracle to answer immediately and not drift over time. When done can be proven not just by you but by a machine, you’ve reached automation.

Short loops are easier to verify than long ones. Dex’s rule of thumb: an agent holds up for three to ten steps, then starts losing the thread past twenty. The reason is context accumulation, the more the agent drags along, the more likely it is to wander off. When a loop is short, verifying it is cheap. Sprawling loops hide mistakes in the corners, which is another way of saying they never earned lights-out status.

Keeping the lights on is the opposite case. A loop needs to be reviewed if a wrong answer is expensive and only a person can catch it. Subtle production bugs that can’t be caught by tests, large blast radii, and a decision that’s going to shape the work of a year or more all qualify. In those cases, human judgment does not leave the software; your attention is the costly, essential part.

The danger is forgetting to flip each switch and just setting all of them to the same mode. All dark, and you’re stuck tearing everything down four months later. All lit, and no one can get reviews done in time and you’re stuck in a gigantic bottleneck. The hard, skilled job is deciding where to put each switch.

Loops, graphs or state machines?

When you hand an agent a task, you’re probably going to build a graph around it, whether you call that graph a finite state machine or a set of conditionally-linked service calls. It’s a framing where the software isn’t just following some abstract rules but a structured workflow: every node is an explicit step, and every edge between nodes is an explicit condition. That sounds like a lot of structure, but most of it is already there in any software, since any code can be expressed as a control-flow graph. So the only real novelty is that an agent insisting on autonomy is really just walking around a particular graph, and its freedom is constrained to the inside of a node. And here’s the part people forget, which Dex wrote down a year ago: software was always going to have that structure. There’s a reason we used to draw programs as flow charts. The genuinely new move was trying to throw the diagram away, leaning on a loop where the model picks the path tool call by tool call, until it declares itself done. That felt like liberation, right up until it met a ten-year-old codebase, and the discipline everyone is now rediscovering, owning your control flow, is really just walking the graph back around the loop. So the question of whether we should shift from loops back to graphs is almost an admission that we needed the flowchart all along.

Here’s what it looks like in practice. Take a bug to fix. As a pure loop, you sit down and think: figure out what’s wrong, change some code, run the tests, see what happens, and if that round doesn’t kill the run, loop back and start again. The whole journey is decided as you go, which problem you chase, the exact code you change, which tests you run and in what order, whether you run tests at all, and whether you try again or declare victory. As a graph, the first thing you do is map out what should happen. Reproduce the bug or go ask for more information, find the cause, try a fix, run the tests, and let a failing run route back to the fix while a passing one goes on to review, where only an approval reaches done. The agent is still clever inside each box; it just can’t wander off the paths you sanctioned. Santi laid this out with a diagram that makes the difference obvious.

The real appeal of that graph, of course, is that it’s back pressure drawn as a diagram. You give up some of the agent’s freedom and get mandatory checks and legible failure points in return, so when a run dies you can point at the node that killed it. It’s the same instinct behind Dex’s blunt line that most so-called agents aren’t very agentic at all, “mostly deterministic code, with LLM steps sprinkled in at just the right points.” And this isn’t just an artifact of how people happen to be building things right now: you can see the pattern in LangGraph and LlamaIndex Workflows, in Jerry Liu’s hybrid workflow-graph-over-agents with an outer loop that grows parts of the graph as it runs, and in David Khourshid’s reminder that this is really just state machines and the actor model turning up in new clothes.

One clarification, because the term is badly overloaded: when I keep calling this a graph, I don’t mean a knowledge graph. I mean a predefined directed graph of how the work should flow, conditional edges and all, giving the loop a shape you can actually trust.

Where the human actually goes

Notice that the person never left the factory. They moved.

I think engineers need to increasingly own the outer loop. The agents can investigate a bug, write up the diagnosis, implement a fix, run the tests, and write up a report. That’s the execution of the inner loop, and they can do it as efficiently as anyone. But that was never the job. The bits you own are what I’d call the outer loop: decide whether it’s the right way to address the problem, verify that the diagnosis and implementation are sound, approve the change, and carry the consequences of being wrong. The boundary between the two loops is evidence, the diffs, the tests, the logs, and a brief explanation that connects them. Types, seams, and rubrics make it possible to oversee all this without doing a lot of work for every change.

I think it’s useful to put it this way: you’re not down on the line writing changes any more; you’re up at the end of the production line designing it and guarding the gate. There’s a lot you can do to make the model better and the harness more capable, but I’ve observed that identifying problems that are expensive in the long term is not typically something you can automate away. The core thing that’s still the job is to exercise human judgment better than any flow of paper and computing power.

Robots are fine operating in the dark, but humans need to see what they’re doing. If everything on the factory floor is dark, and you can’t see anything, and you can’t even find the light switch, that’s where the danger is.

Pangram scored this article as 100% human written.

Own the Outer Loop

Wed, 15 Jul 2026 00:00:00 +0000

In the past year, the conversation around agentic engineering has moved to harnesses and loops, fleets and software factories. My 2c is engineers need to own the outer loop - the accountability for these systems. This only gets more true as powerful models like Fable and GPT-5.6 become available.

A written version of my AI Engineer World’s Fair 2026 closing keynote. Here is the talk:

www.youtube.com

Agents have leverage, and leverage creates obligations. Someone must be able to explain exactly what changed, why it was safe, and what will happen if they’re wrong. Otherwise, their actions can’t be justified. Which makes it unlikely their organization will ask for them in the first place.

And so I want to talk about three terms. The first, Quality, refers to all the checks we install before we let the system loose. Those checks produce evidence, and from that evidence we derive a Verdict.

The second, Verdict, refers to the final decision we make before work enters our dependent system: I’m the line-producer of this content. I run the team whose work is shipped under my name. The model may write the line, but the Verdict is mine. The work of my team will not enter our dependent systems without my decision. A Verdict is the production decision: should we ship, block, redirect, narrow the response, add a guardrail, or reject outright?

The third, Answerability, refers to the guarantee that if someone asks, I can explain why.

To say this another way: our agent (which I define as a model plus a harness of files, tools, memory, skills, sandboxes, permissions, observability, and recovery) is what runs our loop (which I define as investigation, implementation, verification, and repeat). And it’s what creates our software factory.

The model is just the engine. The harness - tools, memory, permissions, sandboxes, tests - is the car you build around it so it can do real work safely.

Wrap that harness in a repeatable cycle: investigate, implement, verify, repeat. The loop is how one good run becomes a process you can trust to run again. Wrap that harness in a repeatable cycle - investigate, implement, verify, repeat - where an independent check, not the model’s own say-so, decides when the work is done.

Now run many loops at once. A factory is loops at scale - the agents ship the work inside, while humans own the decisions at the boundary.

And at the heart of that factory is a careful boundary between what’s inside the system and what’s outside it. Inside the system: we collect inputs (from the product team’s intent, or knowledge of previously shipped work, or of recent incidents, or of specific feedback from users). The agent loop investigates the task, implements a plan, and verifies the result. Then, evidence crosses that boundary. A human, who owns the dependent system, sees the evidence and decides whether to proceed.

And that, friends, is the shift we’re trying to make. Before, our agents were doing the inner loop of the execution loop. Now they run the inner execution loop. Engineers own the outer loop.

Inside the system, there’s really just one kind of thing our agents are doing: capability. The capability to investigate tasks, implement plans, test their results, and report back. That’s the capability of a model. And as we’ve said, that future is already here.

Outside the system, there’s a single kind of thing: agency. The agency to decide, verify, approve, and own.

We’re still talking about code, you see. It just needs to live in a place and be performed by people who know what they’re doing.

The potential for AI code is no longer marginal. In a Sonar 2026 survey, we asked teams about the share of their commits that were AI-assisted. It was small but non-trivial. And several of the respondents said they expect the share of AI-assisted commits to grow substantially. Sonar’s 2026 State of Code report found that 42% of committed code was AI-generated or significantly AI-assisted, with expectations for that share to keep growing rather than plateauing.

Creation, in other words, is getting cheaper. Scarcer resources are review, validation, understanding, and maintenance.

We moved the speed of generation faster than we moved the speed of control. And so we have a trust-verification gap. A lot of people we talk to still express some degree of distrust in AI code. Yet fewer of them seem to consistently build that distrust into their verification processes.

And that’s a dangerous place to be in. We’re going to need cheaper, clearer ways to verify the trustworthiness of AI code.

If you look at the GitLab June 2026 report, you’ll see that governance questions have shifted. GitLab’s June 2026 AI accountability research shows that review and validation are the current bottlenecks when using AI and, more worryingly, that governance usually happens after code creation, after we’ve accepted the risk and lost control over ownership. Today, it’s not just about control. It’s about what constraints we set on the system. It’s about how we’ll check the work with evidence, and how we’ll hold teams accountable. It’s about who will own what part of the AI lifecycle.

So the final distinction in this series is between process and quality. Quality is the concept of back pressure. We mean it literally. We don’t want to grant our agents as much autonomy as they can possibly exercise. We want to grant them just enough autonomy that we have enough back pressure to stop them, regulate them, check their work, and ensure our humanity.

Ordinary engineering holds up a lot of signals that indicate that the work being done is doing the right thing. Type checks, tests, hooks, sandbox limits, audit logs, monitors. Our engineering systems are full of these kinds of signals, and they’re designed to provide enough back pressure to keep the system honest.

And so as long as our agents are emitting these same signals, we can trust our ordinary engineering to provide appropriate back pressure.

Trusting our systems doesn’t mean we don’t want a human in the loop. It just means that the human doesn’t need to be in the inner loop. We want them in the constraints loop (what inputs, architectures, instructions, or invariants should we set?), the sampling loop (how much output should we sample and review?), the audit loop (what evidence should we keep and how do we make sure our audit log is effective?), and the ownership loop (what part of the production boundary should we own).

But the human doesn’t need to be in the inner loop.

The agent can ship more than you can review.

And the scarce resource is your own core human judgment, informed by quality signals like logs or tests.

The AI June 2026 report shows that, in the experimental setting, agentic delegation along hour-scale time horizons is essentially here. The work by OpenAI this year on agents and the future of work was a great source for these ideas. So we need to start thinking about how to establish this ownership boundary, as our systems start shipping more than we can review.

And that’s where the answerability comes in.

Because with long-horizon agents, the decisions made over hour-scale time horizons are just that - decisions. And not all the decisions are going to be recorded. You can’t trace them all back to input tokens. If all you’re doing is trusting that the output you get is the correct choice for the problem at hand, the hundreds or even thousands of human hours of work you’re going to need to reconstruct the chain of decisions that lead to it become impossible. And so, again, answerability becomes something that must be at the core of our system design.

And there are three hidden costs:

Cognitive surrender ~ blindly accepting what AI gives you. When you delegate work to an agent, the work itself may appear to be the work of the agent. But it’s actually your work. It’s your reputation. It’s your responsibility. And it’s your software that suffered the defects in the output. And it’s your software that needs to be changed to reflect that output. So the agent’s output becomes your answer. And with it comes all the accountability. The Wharton study that put this together is reassuring when the AI is right. But when it’s wrong, the news isn’t great. When the AI was wrong, nearly three-quarters of people accepted it anyway, and felt more confident than they would have without the AI.

Cognitive debt ~ erosion of your understanding and memory of how to solve problems. When you delegate work to an agent, you’re offloading all the thought work to the agent. And while thinking it all out yourself takes time and energy, thinking it out on a massive codebase takes resources that aren’t available when you’re trying to run up the learning curve. So the output you get is often unattainable by you. And the longer the time horizon of the agentic planning, the bigger the gap between the code the agent produces and your understanding of it becomes. The gap compounds. The debt accumulates. And the cost of climbing the learning curve grows almost exponentially. There’s a randomized controlled trial from Anthropic looking at whether engineers who lean on AI to write code understand it as well as engineers who write it themselves. The conclusion was gloomy: on a comprehension quiz, the engineers who worked through AI scored seventeen percentage points lower than those who didn’t, 50 percent versus 67 percent.

And then there’s the orchestration tax ~ its easy to spin up lots of agents now, but your cognitive bandwidth doesn’t parallelize in the same way. Steering your agent away from the worst behaviors, sorting the work the agent produces to identify the ones that need your attention, directing it to focus on the work you care about first, verifying your most important constraints and your most dangerous assumptions before you let it run…

All of that takes work, and it can’t be automated.

There’s no substitute for human judgment.

Brownfield systems are especially dangerous here, because the system behavior you have to audit doesn’t live in the code. It lives in the scars.

Fixes? Make attention the priority in your architectural decisions. Use worktrees, scopes, and evidence to reduce the coupling between your initial plan and the work that emerges from it. Time-box the effort to resolve unactionable steps. And make change in your software strictly an opt-in permission.

Alpha, decay, and taste: these are the three core patterns that shape careers and performances across domains.

Alpha is the lead part taken up by the highest achiever in the competition, when you’re playing your highest-value game move. Decays are established patterns that everyone learns through repetition and watching others (plateaus, if you like). Taste is the earliest we can sense the lead in an alpha or the change in a decay. It’s our judgment of what’s coming before we have any evidence that anything is happening. Paul Graham’s point is that when anyone can make anything, choosing what to make matters more, and Mitchell Hashimoto’s definition is the operational one: making high-quality qualitative judgments where no objective metric exists yet. From now on, taste drives everything: alpha shifts are taste changes. And decays fade out because we start to taste something different.

Next step? Operationalize your taste. How? Give it a name that reflects what you’re trying to move from limbic to conscious. Practice it in critique and examples. Make its rationale explicit.

And keep making the move that delivers the most durable competitive advantage in your industry. What’s that? Keep moving the edge up from just doing the task to teaching it, systematizing it, deciding when it should be done, and owning the result.

Everyone is a developer, but not everyone is an engineer. Engineering is what a developer turns into when they embrace a work discipline that is more strict: thorough and logically sound reasoning, consideration of constraints and tradeoffs, recognition of risk and exposure, and practical accountability.

In the future, people will leave the administrative work of engineering and embrace new roles that emerge as engineering becomes more demanding. Roles that are un-bundled from the spirit of craft but make clear what each person does. There will be those who prototype. Those who build. Those who sweep. Those who grow. Those who maintain.

The humans hold the edge of the system in the other direction, too. Increasing the alpha in the other direction: choosing what is worth doing, defining the constraints within which it should be done, deciding if the evidence is sufficient to proceed, and caring for the result. Whether it’s a single team or a hundred teams, this is the edge that only humans can hold.

Accountability will scale the factory. Like attention and taste, accountability is also one of the three dualities that makes everything work. Without accountability, there are no rules. No wrangling with questioners. No trade-offs. No risks. No safety nets. If nobody owns the consequence of a decision, then high agency can only bring chaos.

The half-life of an edge is one release, but the half-life of a signature is a career. A signature is your name on the work, such that you feel you can stand behind what was shipped. Skills get you leverage; accountability turns leverage into trust.

Only people can choose. Only people inherit consequence. Agents can be asked to choose, route, merge, and escalate safely inside a policy, but they cannot inherit the consequences.

Every codebase should perhaps come with some kind of accountability contract that explicitly states the checklist that was understood when the change was accepted, the evidence that went into the decision, who was accountable for the change, and the system status after the change was blocked. Just like:

Your attention and taste
Your evidence, verdict, and ownership
Your alpha, decay, and taste

High agency

In a typical agentic workflow, high agency is the art of knowing when to delegate, when to inspect, when to stop, and when to own the result of a process. The ladder of agency runs from low to high: flag a potential problem, investigate it, execute against it, diagnose it, propose solutions, recommend fixes, and resolve the issue. A high rung on the agency ladder is discernment: found it, it’s not worth fixing, moving on.

The twelve pillars that hold up the software factory

Brownfield is the frontier for factories that hope to scale. All those clever little innovations may not feel like much yet, but the production environment is a lot. When building an entirely new system, it’s much easier to plan and implement sufficient back pressure mechanisms because you have full control. When you’re adding intelligent agents to a legacy system, however, it’s another matter entirely.

Legacy systems include the entirety of production behavior, future expectations from customers, migration histories, release and budget cycle durations, unspoken assumptions, edge cases, data weirdness, runbook procedurals, and all the scars that accumulated without the will to care for the system.

To be a steward of brownfield requires a form of durable engineering. Work has to be done to turn implicit knowledge into explicit constraints, keep it coherent across teams and through generations, formalize that knowledge into test procedures and functional specifications, and tie that knowledge to objective evidence. All while ratcheting failure into more learning. Because if the system doesn’t get the care it has always received, everything will come crashing down.

New Work is Real Work

The work will get more interesting as you scale. Because when everything else is built, people will want to build new things. They’ll want to employ the alpha and taste they have developed through their craft to design new loops that can be grafted onto the software factory. Or they’ll want to build greenfield systems that employ all the knowledge of the software factory to one elegant, well meaning, principled effort. They’ll want to design and implement new forms of evidence that will rise to the level of verification for the new systems. They’ll want to take care of brownfield systems that are now so complex they need dedicated attention. They’ll want to design and manage new back-pressure mechanisms. They’ll want to design new agents. And they’ll want to build agency.

And, as they do, they’ll come to see that all this is real work. That’s a good thing.

Automation creates bottlenecks. Bottlenecks in production that are worth owning. Because automation gives us control over industrial scale. But there’s also new bottlenecks that arise from industrial scale. The bottleneck moves from “can we build this?” to “should this exist, can we answer for it?”

What I’m suggesting is a practical operating model for scaling agentic engineering. There’s inner and outer loops. The inner loop is where the work is done. Loops are designed to be as independent as possible. Put all quality assurances and verification inside the loop. Once you’ve designed and validated the loop itself, the only thing you have left to do is to grant autonomy by putting in place a back-pressure mechanism that acts to control the rate at which the loop is run and its scope of operation. And put humans in their rightful place, on the right decisions. Don’t treat understanding as a hand-off or a release gate, but rather as a point of decision where humans are primed to provide their insight. And then for every artifact that exists and is fed back into production and into new teams and engineers, leave behind better artifacts.

Build the factory; keep the lights on; make work legible, verifiable, owned.

An agent can write it. But before it reaches users, someone must explain why it should exist, why it’s safe enough to be part of production, and what they will do when it is wrong.

That is agentic engineering at the outer loop - that is the work now.

Pangram has scored this article as 100% human written: pangram.com/history/ae6caccc-b70f-4336-a019-5c3411516871

Earning taste and judgment

Tue, 14 Jul 2026 00:00:00 +0000

Taste used to be a byproduct of the reps. Agents took the reps. So if you’re junior you now have to go get the taste (and judgment) on purpose.

As I was building my career, I learned to think about code in fundamentally different ways. The first chunk of my learning came from thousands of reps of boilerplate, bug fixes, and solving problems. Almost all of the taste and judgment I have came from those reps, not from talent. Over time, I learned to handle increasingly complex abstractions and tradeoffs. That’s how a junior developer matures into a senior one.

But agents will automate all of these reps. This will fundamentally change the path from junior to senior.

Spoiler: The thing that stays durable and ungradeable is not getting better at solving problems that already have a known answer; it’s choosing what to build and judging whether it’s any good.

The entry-level is weakening. As of March 2026, unemployment among recent college graduates sat at a staggering 5.6%, with underemployment at 41.5% - and recent graduates are now more likely to be unemployed than the workforce as a whole, an inversion of the historical norm (Federal Reserve Bank of New York). Forbes, citing the St. Louis Federal Reserve, notes recent computer-engineering graduates had an unemployment rate of 7.5% and computer-science graduates 6.1%, both elevated above many non-technical majors.

The demand side reshaped underneath them: the Indeed Hiring Lab found junior and standard tech titles down 34% since early 2020 against just 19% for senior and manager roles, while the share of tech postings demanding five or more years of experience climbed from 37% to 42% - even as the number of computer-science graduates kept growing. The most AI-exposed early-career workers (ages 22–25) showed a 13% relative employment decline in the original Stanford Digital Economy Lab study; a February 2026 follow-up revised that to 16% through October 2025 (Brynjolfsson, Chandar & Chen) - and in many of these occupations that age group had accounted for roughly half of all hiring.

Should we panic? Well, entry-level software-developer jobs are more than employment; they’re a training system.

Microsoft’s Mark Russinovich and Scott Hanselman have warned that, far from taking the bread of all software developers, agents actually help senior developers while robbing juniors of theirs and narrowing the pyramid. Hanselman’s telling example: inserting a sleep() to paper over a race condition. The senior developer spots the error and helps you learn from it. The junior developer skips the code review, pushes it to production, and the race later crashes the site - then does it again. Their real point is that the agents aren’t doing this on their own; it’s less an AI story than a story about how people choose to use the products.

Worse, a 2026 MIT Technology Review piece warned that automating the learning stage buys output now but leaves society gradually less capable later, and the World Economic Forum warned in mid-2026 that cutting entry-level recruitment could weaken the talent pipeline going forward. A word of caution, though: this isn’t purely an AI story. We’re partly correcting a hiring binge that started in 2022 and ran into 2025. And even the Stanford authors were careful early on - the early signal was muddy, and only after 2024 does the data get clean enough to isolate an AI-attributable decline. Too many vendor “synthetic labor” announcements are marketing prose, not prophecy.

And of course, all of this sits right next to a genuinely optimistic story from the rest of the market. The World Economic Forum’s Future of Jobs 2025, published in January 2025, projects that by 2030 some 170 million new roles will emerge against just 92 million displaced - a net gain of 78 million. Apollo Global Management’s chief economist Torsten Sløk caused a stir with the claim that there is “zero evidence of job losses because of AI” - arguing that, if anything, the spending is creating work for people with AI skills.

So how can there be contradictory reports in the same year?

Because the net and the entry level are two different questions. The optimists aren’t wrong. Senior roles requiring judgment and critical thinking will grow substantially while junior roles will likely shrink. Indeed’s numbers say the same. And Goldman Sachs, looking at 2026 payrolls, estimated a net loss of about 16,000 US jobs a month with Gen Z and entry-level roles hit hardest - the empirical answer to “zero evidence.”

Aggregate growth and a vanishing first rung are both true at once - and if you’re the one trying to climb onto that rung, the net number is cold comfort.

So if junior developers are at risk in this phase, the real question is how we should proceed? If the AI layer gets good at anything, it will be anything that has an answer key. School used to be answer keys all the way down, all about getting the right answer.

So I propose we look less at the availability of jobs and focus more on developing taste: compressed experience in the form of earned pattern-matching. Kent Beck notes that it’s unlikely AI agents will ever possess taste, by which he means judgment, so we’ll be left to supply it in these intersections.

We now owe two debts: cognitive surrender and comprehension. When you first learn to code, a wise teacher told you not to copy and paste from Stack Overflow. Unfortunately, I think we’ve all violated that advice at some point, especially mid-career. The problem gets worse when we offload the copy-paste decision to an AI agent. Shaw and Nave surveyed 1,372 participants over ~10,000 trials and found that they tended to accept incorrect AI output nearly 80% of the time. Their accuracy dropped ~15 points below the no-AI baseline, while confidence rose ~12%.

The agent runs a loop of activities (investigate, implement, test, report).

You, the human, own the outer loop of deciding whether the result is worth your attention; verifying that the result is worthy of approval (diffs, test results, logs, and a short why); approving or blocking; carrying the consequence. The boundary is evidence (my psalm). The gap between the loops is growing rapidly, and you must manage it.

When agents write code, it’s easy to think they’re doing all the hard work. It’s hard to see what a human must do to maintain quality when agents are getting it mostly right. Let me now turn from managing agents to building your taste and judgment, starting with seven concrete things you can do.

Read far more code than you generate. Hunt for logic errors, security holes, simple or subtle edge cases. On a regular basis, ask yourself about the code you read: Did I consider the right things?
Keep a wrong log. Every mistake an agent makes gets one sentence. After thirty days you get a sense of patterns.
Do a few things the hard way on purpose. Build a parser manually, or a CRM, or something meaningful from scratch. Protect your collateral learning. Karpathy emphasizes fundamentals like memory, views (how the world appears to the system), and storage that agents get wrong.
Go deep on one system end to end. Push it all the way to failure. Then learn what real depth feels like.
Learn to specify and verify separately. Spec-writing is clear thinking. Verification is evidence. Specification quality is the biggest lever.
Build an eval, a test framework around a rubric of correctness, maintainability, efficiency, security, style. Run it on fifty real PRs generated by AI agents. Note surprising test failures and fixes. Calibrate to make your internal quality function explicit.
Calibrate autonomy per task. Turn it way up on cheap, reversible tasks. Down on expensive failures. Learning to calibrate is a senior developer instinct worth exercising daily.

What do people who are building these agents tell juniors?

“People are still going to need to know the craft underlying software engineering, including languages, compilers, runtimes, and system design” Boris Cherny of Claude Code says. His ‘coding is largely solved’ statement is about the workflow he personally finds productive, not about forgoing your fundamentals. As an engineer, you really need to know what’s going on under the hood.
Beware the paradox of supervision. A report from Anthropic says they only use AI to help when they already know the answer, and warn that supervising an agent requires exactly the skills that atrophy when you over-rely on one. For a junior developer, growing into that state of mind deliberately, rather than accepting answers blindly, takes effort. Use the tool carefully while deliberately building deeper understanding of the systems you work on.
Think carefully about which muscles you are willing to let atrophy. Gergely Orosz uses AI for coding but zero AI for writing. Be mindful of trade-offs and decide which ones you are choosing to keep sharp.

Where does durable value concentrate? Start by optimizing for scarce resources.

Capital is abundant. Time is abundant. Real relationships, and especially a track record of doing good work, are still scarce: I can raise money in a couple of weeks, but I can’t raise a reputation. When vibe-coding makes earning a quick buck trivial, that quick buck is worth very little; the scarce move is choosing something worth shipping. With that lens, four principles.

Finish the last mile. Automation covers the easy 80-90% of software engineering. The last mile - edge cases, architecture, taste - is the whole game. As first drafts become free, the finish is the product, and it’s where people distinguish themselves.
Solve the hard version. Richard Sutton’s famed bitter lesson of the last three decades is not just career advice: The easy version is already solved; durable value comes from solving the hard one.
Build in public near hard problems. The same constraint that drove junior developers to abandon open source - a well-conceived project takes one person a year, easy to space-out or clone in bits - is less important than the scarcity of an open-source track record of really good work. Think of it like expected goals in soccer: your reputation and public work decide how many chances you get in front of goal, and judgment is whether you convert them. You can’t script which chances arrive, only whether you’re standing where they land; almost every real opportunity I’ve had came from work I did in public, never from a job I applied for. So get a foot in the door wherever there is a hard problem you want to solve. That means taking an internship at a small or midsize company, getting on a software engineering track somewhere, or building side projects that matter with production code. Those who try to cut corners will find that it’s really hard.
Be a T-shaped generalist. Developers with the deepest knowledge in one or two areas while maintaining broad literacy tend to deliver the best results. A good dose of AI assistance means developers will be able to accomplish more as single contributors working on fewer different areas than they could before. Teams that lean into small cross-functional teams and end-to-end engineers will thrive.

To put it plainly: the world isn’t short on opportunity; it’s short on people who can find the right problem, tell whether the machine solved it, and finish past where the machine stopped.

We talk about the “last mile” as the biggest piece of the puzzle, but in the world of agents the last few feet are effectively infinite - agents scale output infinitely; you don’t. Your attention is your most precious asset, and it doesn’t refill, so protect it. Anything gradeable by someone else is getting automated.

The career is the ungradeable part: choosing what matters, judging honestly when you’ve got it, and answering for it. Do that. In public. Near the hard problems. The rest tends to follow.

Pangram scored this article as 100% human authored

The Agent-Era Career

Mon, 6 Jul 2026 00:00:00 +0000

AI gets good at anything with an answer key. Your career is everything that doesn’t have one.

If the AI layer gets good at anything, it will be anything that has an answer key. School used to be answer keys all the way down. School is the ultimate anchoring of success, because it’s all about getting the right answer. The thing that makes work durable and ungradeable in the age of AI is not getting any better at solving problems. It’s not being able to build systems, or understanding people, or making cool new things. It’s choosing what to build and judge if it’s good. The rest will all be done better and faster by AI.

I started in engineering at 16, building a browser in rural Ireland. I was at Google for over 14 years, where I led engineering teams working on Chrome, Gemini and Cloud AI, and written a number of O’Reilly books. I’ve turned down offers from frontier labs and FAANG companies when the fit wasn’t right. Good people are always needed, so we each have an obligation to try our hardest and make the best thing we can.

Most career advice still holds up. Get on the rocket ship, don’t over-optimize your seat. The specifics have changed a little because of agentic coding, but here’s what I wish I’d known for ambitious engineers out there now.

Optimize for scarce resources. Almost nothing I’m known for came from chasing the highest pay. The years I spent in open source had almost zero direct payoff. But they led to reputation and relationships that very efficiently compounded into opportunities later. I would have spent the comp I got from any single job. My reputation kept paying.

Many resources are abundant. Capital is abundant. Time is abundant. Real relationships, and especially track record of doing good work, are still scarce. I can raise money in a couple weeks, but I can’t raise a reputation. So here’s the plan: do good work, and make sure the people who like good work see it. In a world where vibe-coding makes earning a quick buck trivial, I think that quick buck is worth very little. When shipping stuff is so easy, the scarce move is choosing something worth shipping.

Learn to find problems, not just solve them. The first time I ever felt the burden of selection rather than solution, LeetCode seemed a measure of skill. But as agents absorbed all that work, solving problems went cheap while selecting them became scarce. My origin story: I noticed dial-up was slow, created chunked multi-connection fetching, realized I’d never solve that problem in my life, and quickly moved on to whatever absorbingly complex one I could find next. Finding problems predated solving them.

I’ve watched students who were wildly good fall flat on their face when an agent ran through their problem set (like watching the wrong microwave number on the clock). The same agent. The same problem set. Wildly different token and time budgets. Why? Because at the end of the day, the strong ones bring judgment and intuition to the work; the rest bring a prompt.

I used to build that judgment by grinding out boilerplate and fixing bugs. I got to see and deeply feel the worst abstractions humans could devise. I approached each commit with the awe of someone who’d just seen the fever dream of previous authors. Each commit brought hindsight and judgment. The agents automate those reps. Taste is pattern-matching, but all that pattern-matching has to be earned by doing the work.

The real risk isn’t agents writing bad code. We’ve been there before. It’s losing the ability to tell. Judgment will atrophy. Output will look a lot like working code.

Good practitioners don’t put agents in front of everything. They engage in deliberate practice. Pick a few problems that really matter. Do them the hard way, without the agent, building deep mental models of how systems and languages work. Read a thousand times more code than you ever write. Treat every diff from an agent like a human review you need to carefully justify. Go deep on at least one system end to end, from intake to output. On a daily basis, keep a private log of every time you see an agent suggest something that looks wrong and confidently flag it. That’s where taste accumulates.

The real thriving engineers won’t be the fastest at getting suggestions. They’ll be the ones who know instantly when to say no.

Shift from doing to directing. Just like you’d delegate to a person, you need to learn to delegate to an agent. Scope the task, define done, calibrate trust, and verify the result.

Autonomy is a setting, not a rank; it’s a per-task switch. Turn it up to the maximum on something small and reversible and cheap to check. Turn it down on anything where mistakes will be hard to undo.

Specification and verification are two distinct, complementary skills. The agent isn’t as good as the intent you hand it. The best engineers are those who know how to write precise specs; clear thinking made legible.

It’s verification, not evidence. Not evidence in the form of an agent grading its own homework. There’s nothing more demoralizing than delegation without verification at scale.

Own what you ship. If the agent wrote it and it breaks in production, “the AI did it” is not a defense. Your name is on the change. Adopt the posture of an accountable human who understands what went out the door and how to fix it.

Most ambitious version of the problem. Rich Sutton’s bitter lesson: in almost every field general methods that scale with additional compute beat out hand-tuned equivalents. As a career lesson, there’s no point in solving an easy version of the problem, it’s worth almost nothing. The value ends up concentrated in the hard version.

Sprint the last mile. No turnkey agent writes a whole system from end to end. As a rule, you’ll get 70% of a feature quickly from an agent, and the last 30%—debugging the gnarly edge cases, figuring out the right architecture, cultivating the right taste—will be the whole game. The median output today is whatever the agent produces from some lazy prompt. The only personal value you can bring to the table is getting as far as you possibly can past that median. When first drafts come free, finish is the product. To sprint the last mile, here’s my tactic: every few months I completely rebuild from scratch using the latest, sharp-end-of-the-sword model. It’s less exhausting than nursing half-hearted old code to health.

My job as a software engineer has been to finish strong. The difference between finish strong and finish okay is the polish: spending an extra hour, which shows instantly to everyone who matters.

Increase both your xG and your finishing

If soccer had a stock ticker, it would be xG. xG measures the number of chances your play should produce. Finishing measures whether you convert them. You can’t plan the number of chances you get, but you can hope your play produces enough, and over your career you can get better at finishing them.

The same is true of careers: your reputation gets you in front of goal, and you convert them with good judgment. Chances arrive whether you’re ready for them or not; how many you get, and which ones you finish, is up to you. I’ve only ever had big opportunities as a result of work I’ve done in public, never from a job I’ve applied for. You can’t script which chances arrive, only whether you’re standing where they land. You have to create the opening as much as you can, and then be ready to take it.

One easy mistake is anchoring on whatever product your company has right now. It’s true that your work has to exist somewhere, but a good team quickly mutates their current offering into something unrecognizable. So bet on the team and the market opportunity, not the demo. It’s just a snapshot. The team is the trajectory. On superintelligence: it’s possible (I believe) that future models will eventually come to replace much of what we do as knowledge workers. It won’t erase it overnight, it won’t replace all of it, and won’t be able to do many of the tasks we do. New kinds of jobs will be created. Verification will always be a bottleneck. Someone has to make the call on which problems are worth solving and allocate the correct amount of judgment to each, and that someone can be you.

But importantly, you can do frontier work right now, from where you are. The gate to AI research is smaller than it looks, and you don’t need a lab to build intuition. Just use models hard, and turn what you notice into evaluations. Evals and benchmarks are where understanding lives.

To summarize: the world isn’t short on opportunity; it’s short on people who can find the right problem, tell whether the machine solved it, and finish past where the machine stopped.

We sometimes talk about the “last mile” as the biggest piece of the puzzle. But in the world of agents, the last few feet are infinite (agents scale output infinitely; you don’t). Your attention is your most precious asset, and it doesn’t refill. You can’t afford not to protect it. Anything which is gradeable by someone else is getting automated. The career is the ungradeable part: choosing what matters, judging honestly when you’ve got it, and answering for it. Do that. In public. Near the hard problems. The rest tends to follow.

This piece grew out of Phil Chen’s original, which is well worth reading in full.