![\"\"](\"https://martinfowler.com/articles/software-meets-hardware/figure1.png\")
Gergely Orosz is frustrated at those that entered his explicitly “No AI” essay competition and used AI. He goes on to say:
\n\n\nWhen I start to suspect some part is AI about someone I don’t know: how can I tell how much of it is? And how much time and effort do I want to invest to figure out?
\nI fear most people do not realize this last part: that when it takes time and effort from others to figure out what % is AI, and what % is YOU, then I will simply assume it’s probably 100% AI and 0% you
\nSo when producing anything that is meant for other humans, if you use even a little AI, you risk the assumption that it is all AI and pointless to pay attention to
\n
In general, I agree with him, and react much the same. Once I hear LLM Voice, I flip the bozo bit and check out of the post.
\nBut as I do this I have caveats. How accurate is my neural net at AI-detection, are humans taking on AI-ish elements in their normal communication? And more seriously there are some valid uses for AI. Jim Highsmith’s story of his second brain helping him deal with Parkinson’s is one. Another is folks who don’t know English well, using AI to translate. Am I being unfair when I flip the AI voice bit when I read/listen to them?
\nI was sad to hear that the Grace Hopper Celebration is canceled this year. It’s an event run by Anitab.org to promote women and non-binary technologists in the software industry. Many of my colleagues have been encouraged and inspired by going to this event, which makes a refreshing change to what is often a male-dominated world. It seems the current crusade against diversity initiatives has had a big impact on funding. We are still a long way from where we should be on gender diversity in our profession, and losing an event like this only makes things more difficult.
\nI’ve always found Cindy Sridharan’s writing to be worth reading, and she’s broken a quiet period with a monster of an article: How to effectively debug when you no longer understand the code. Since Medium estimates it as a 64 minute read, I didn’t jump in lightly, but settled down on a flight from Amsterdam to Stuttgart. I’m glad I did, because there’s lot of good material in there, and I’d struggle to find anything to cut.
\nIn the article she uses three recent bug hunts to surface principles for how to best use an AI tool for debugging. Much of this is governed by this wise observation:
\n\n\nAgents are not humans. They’re far more capable than humans when it comes to many things, and we should be leaning into these strengths, rather than trying to get agents to replicate human patterns of problem solving which might be suboptimal in agentic contexts.
\n
One conclusion from her experiences is that the code often isn’t the best thing for agents to look at
\n\n\nhaving an agent start an investigation by first reading the code is an anti-pattern. In fact, reading the code first to formulate hypotheses about a bug should always be the very last resort of the agentic investigation process, not the starting point. An agent should first collect all available evidence about the bug, and only then formulate the hypotheses from the code that might best explain what’s being observed.
\n
She distinguishes between a bottom tier of facts: raw data that agent can find about the behavior of the system - and an upper tier of hypotheses: inferences that the agents make based on that data.
\n\n\nDiagnosing a bug now becomes a problem of finding the hypothesis in the top tier that satisfies every fact from the bottom tier.
\n
The agents need to be steered towards the hypotheses that are the likest cause of the bug and then subject these hypotheses to validation. She has a three-step process to do this. First she identifies the most important facts, the signals that might explain or identify the bug.
\n\n\nThe factual tier has many data points. Not all of them are equally valuable. In fact, some of it might be active noise that doesn’t help an agent to be steered towards most promising hypothesis. The agent first needs to identify the data points in the factual tier that are actually related to the bug. Agents do this best when they are given a high-quality control
\n
A control is a behavior of the system that doesn’t exhibit the bug. This may be requests that trigger the bug compared to those that don’t, or different builds of the product, or the same build in different environments.
\nWith signals highlighted, she then focuses on the hypotheses that are most promising, because they fit best with the signals. A hypothesis should satisfy all the facts, the most valuable signals are those that are only satisfied by a few hypotheses. A table of hypotheses and signals helps visualize the strength of the signals and the likelihood of the hypotheses.
\nThe third step is then validating the promising hypotheses against every fact, treating each fact as constraint on the solution.
\n\n\nAcross the three bugs chronicled in this post, my most valuable contributions weren’t knowledge of the code. They were picking a control, choosing a constraint, and rejecting a hypothesis that hadn’t been validated. The engineers who thrive in an agent-native world will be the ones who are good at designing such verification loops, whether or not they understand the code it verifies.
\n
This directing has to remember the opening point that agents have different capabilities to humans.
\n\n\nA human wants to deal with less data. All abstractions we currently have (aggregation, visualizations etc) are designed to help humans deal with less data. An agent, however, needs more raw data; its strength lies in processing huge volumes of data at a rapid clip. Aggregation — the thing that makes a dashboard useful to a human — is often the thing that destroys the value of the signal for an agent
\n
Agents’ ability to consume lots of raw data also affects how we build systems. More logging gives agents more data to chew on.
\nAnd thus far, the human is still necessary to direct the agent
\n\n\nIn each of the three bugs chronicled in this post, the agent was more than capable of getting to the root cause by itself, but it didn’t do so. It didn’t set up a control in the “latency bug” until I asked it to
\n
While it’s a long post, it’s engagingly written and full of useful details. I hope my little summary encourages you to give it the read it deserves.
\n", "date_modified": "2026-10-21T10:51:00+00:00", "date_published": "2026-10-21T10:51:00+00:00", "external_url": "https://martinfowler.com/fragments/2026-10-21.html", "id": "urn:aggr:item:111fc5aad1ab9f4e8fc3f9e982bcc3d2bc735261", "image": "https://martinfowler.com/fragments/fragments-thumb.png", "summary": "Gergely Orosz is frustrated at those that entered his explicitly “No AI” essay competition and used AI. He goes on to say: When I start to suspect some part is AI about someone I don’t know: how can I tell how much of it is? And how much…", "tags": [ "programming" ], "title": "Fragments: October 21", "url": "https://aggr.aymericbeaumet.com/items/martinfowler-com/2026-10-21-fragments-october-21/" }, { "authors": [ { "name": "Martin Fowler" } ], "content_html": "Modern software engineering gives us a set of practices that we tend to carry from one project to another: AI assisted development, automated testing, CI/CD, observability, automation, simulation, and safe deployment. They work well enough that we sometimes forget to question the assumptions behind them.
\nI had to question some of those assumptions when our team started modernizing a large retail IoT system. The system had around 800,000 battery-powered sensors deployed across 3000 stores. The existing connectivity technology was reaching the end of its life, and we needed to move to a newer technology without replacing the installed devices, and also onboard newer stores with next gen devices.
\nOur team knew software engineering well. We had experience with web applications, cloud platforms, mobile applications, IoT and other distributed systems. What we did not have was deep experience in embedded firmware and large-scale device networks. We thought the software engineering skills would apply here too, and it did, the engineering principles remain the same, but the default practices changed.
\nLet us find how we discover this change.
\nAfter initial spikes on firmware development technology, our challenge was no more writing the firmware, but actually to understand the IOT system that has been running for years, as the organizational changes had left gaps in the knowledge of the existing firmware, hardware and network architecture.
\nDocumentation helped, but it wasn't enough. Things weren't documented, and some documents described hardware or behaviour we couldn't reproduce. Some answers could only be found by putting our hands on the actual devices, and playing with limited access to production systems.
\nSo we started building things and measuring what actually happened. We built an IoT lab with physical sensors, wire harnesses and learning kits. We created environments where we could experiment with radio behaviour, reused sensors with dead batteries for testing, and measured power consumption and network behaviour.
\nAll these initially looked like our preparation, trials but quickly all this started becoming our engineering work, and soon we were not waiting to understand the system. The IOT lab became part of how we learned, and reverse engineered.
\nFor embedded systems, sometimes the fastest way to answer a question is not to read another document, but to connect the device and see what happens.
\nIn conventional software, the development environment is usually made up of software tools. A developer can clone a repository, install dependencies and run the application irrespective of physical location, working from anywhere.
\nThe equivalent setup for our project needed something else. It needed physical devices. We end up creating what we called a sensor farm: a controlled collection of real devices connected to the new network, with monitoring around them. This enabled our engineers to work with multiple devices and observe what happened when they communicated, slept, woke up or lost connectivity. Physical placement, radio conditions and power, all became part of development and testing.
\nWe didn't need every developer to have a laboratory on their desk, but we did need the development process to provide access to the physical behaviour of the system and get fast feedback, the engineering practices like TDD, Automation, XP were all applicable but with hardware in loop.
\nOnce we had a working device, well integrated as part of the development environment, another question appeared: “What happens when the device is somewhere we cannot reach?” This sounds like an operational question, but it turned out to be an architectural one.
\nImagine an engineer receives a report that a sensor isn't behaving correctly. The sensor is sitting in a store or development environment, and the engineer is somewhere else. In this case there is no debugger attached, there may be no one available to open the device, the device may be asleep, and the network may not even be available.
\nA conventional application can often tell us what happened through logs and traces, but a physical device doesn't automatically have that capability. We therefore started building diagnostics into the device firmware itself, where we wanted to know things like which firmware was running, whether the device was healthy, when it last communicated, what its network state was, what happened before a failure, whether it was sleeping, whether it was repeatedly reconnecting, and what its firmware upgrade status was.
\nWe added health checks and diagnostic information, ways to access that information locally, and APIs so that the same information could be retrieved remotely. We then started feeding the information into dashboards and operational tooling.
\nThis changed the way we thought about observability. We weren't simply adding logging to an application. We were designing a system that could operate devices we could not physically see, where each device needed to explain itself. For a small number of devices, this is useful. For hundreds of thousands, it becomes essential.
\nDiagnostics solved only part of the problem. Engineers also needed ways to interact with the device without physically opening it.
\nWe needed local interfaces such as NFC and BLE, as well as remote interfaces through the gateway and platform. These allowed engineers to inspect configuration, retrieve diagnostics, understand device state and perform operations remotely.
\nThis made us think differently about engineering interfaces. We normally spend a lot of time designing interfaces for users, while interfaces for engineers and operators can become an afterthought.
\nFor a large device fleet, they are part of the system.
\nThe next surprise came when we increased the number of devices, and reached the real store.
\nEverything was fine, until we onboarded a handful of devices, onboarded smaller stores, but as we moved to larger stores, the network infrastructure had limits we had not encountered during smaller lab experiments. At one point, we discovered that the new network hardware could support only a limited number of devices—far below the scale required by some larger stores.
\nTesting a few devices is very different from testing thousands. Network behavior, infrastructure limits, message rates, device physical location constraints and device disconnection problems started appearing at scales that were difficult to reproduce physically.
\nSo we built a virtual device emulator: special firmware for the same device that could simulate hundreds of unique devices, each can be configured behave differently under different network and environmental conditions. Running many of these virtual devices together allowed us to simulate a large store environment and test the parts of the system where scale itself was the problem. This did not replace physical testing. We still needed real devices to validate firmware, hardware and radio behavior.
\nWe also built a virtual store that behaved like a real store, with a gateway publishing simulated sensor readings to the platform. This allowed us to exercise the downstream systems at store scale without needing hundreds of physical devices for every test.
\nEnterprise applications often make a comfortable assumption: the network is available, and it is considered an infrastructure concern, and applications need to make sure it handles disconnection with a good user-friendly way, letting application reconnect immediately.
\nA battery-powered device has a different relationship with the network.
\nThe device may sleep on idle to save battery, It may temporarily lose connectivity. It may reconnect repeatedly. It may not be reachable at the exact moment the platform wants to communicate with it. Remember that a connection attempt is the heaviest operation for the device.
\nWe therefore had to carefully design for intermittent connectivity rather than treating it as an exceptional failure. This affected firmware, gateway communication, commands, telemetry and device management. Once we accepted that the device would sometimes be unavailable, many design decisions became clearer.
\nThere was another constraint that was easy to underestimate: the device had a limited amount of energy, but it is supposed to be operational for years.
\nIn a server application, an inefficient operation might consume more CPU, memory or increase cloud cost. In a battery-powered sensor, unnecessary computation or communication can reduce the useful life of the device.
\nWe introduced power measurement and profiling into our development process. We looked at when the device was awake, how often it communicated and what different operations meant for battery consumption.
\nThe important thing wasn't simply optimizing the code. It was understanding that every design decision could have a physical consequence—even something as small as a log statement or a few lines of code that appeared to be unused.
\nWe also had to rethink some of our usual assumptions about memory management. Instead of aggressively cleaning up allocated memory after every operation, in some cases we chose a simpler approach: let the device run for 24 hours and then reboot it as part of its normal operating cycle.
\nOn a device, sometimes the simplest software solution is not the one we would choose in a server.
\nFirmware deployment introduced another challenge.
\nA cloud application can often be rolled back quickly when something goes wrong. Physical devices distributed across thousands of stores does not give us that luxury.
\nThe battery consumed in downloading and installing firmware can not be rolled back. The device might be offline during an update. The network might fail halfway through a download. The firmware might not validate correctly. Or a problem might only become visible after deployment. All this would change the meaning of deployment and recovery.
\nWe therefore designed firmware updates with recovery as part of the deployment mechanism. We implemented over the air firmware updates using secure downloads, validation and partition swapping. Updates could be applied to individual devices, groups of devices or stores, with progress visible through the gateway and, where useful, locally through NFC. The device also stored its firmware upgrade state, allowing an update to resume from where it left off after recovering from an interruption or failure.
\nMore importantly, we designed a staged rollout, as we wanted to know what happened after an update before updating everything. After all, the question was no longer just, “How do we deploy the new firmware?” but, “How do we deploy it safely, and what happens when it goes wrong?”
\nBy this point, something subtle had changed.
\nEarly in the project, we talked like software engineers. How quickly can we build this? How quickly can we deploy it? How automated can we make the pipeline?
\nThose questions still mattered. But increasingly, our conversations became different. Can we trust this firmware update? Can we recover if something goes wrong? Can we explain why this device failed? Can we understand what is happening across a store? Can we safely update thousands of devices? What will this do to battery life? And how long will it take to upgrade a store?
\nWe were optimizing for something slightly different: confidence. That didn't mean giving up speed. It meant recognizing that speed without confidence isn't necessarily progress. For a physical fleet, a fast deployment that creates thousands of unhealthy devices is a very efficient way to create a large problem.
\nMaking an individual device explain itself was necessary, but it was not sufficient. With hundreds of thousands of devices, nobody can inspect them one by one. The question changed from “What is this device doing?” to “What is happening across the fleet?”
\nWe built dashboards and analytics to identify communication failures, unusual behavior, outages, firmware distribution, patterns and other fleet-level conditions. Operational teams needed alerts and tools that could show whether a problem was isolated or affecting a much larger part of the fleet.
\nBy this point, several things that had started as engineering or operational tools had become essential parts of the overall system.
\nThe physical sensor farm helped us learn and test. The virtual device emulator helped us test at scale. Diagnostic interfaces allowed engineers to understand devices remotely. The gateway provided communication and management capabilities. Dashboards and analytics allowed operations teams to understand the fleet.
\nThe architecture started looking something like this:
\nDevice → Firmware, Diagnostics and Configuration → IoT Gateway → Telemetry, Commands and OTA → Fleet Platform → Analytics, Alerts and Operations
\n
Figure 1: An engineering environment: Hardware, Software, and Feedback
\nThese were not simply developer conveniences. Together they allowed us to build, test, operate, migrate and manage a very large physical fleet. The more devices we had, the more valuable this surrounding platform became.
\nAI also became part of how we worked, although not in the way we initially expected.
\nWe used LLMs to explore unfamiliar areas, prototype solutions and shorten the time between a question and a useful experiment. We also used AI-assisted development to build dashboards, migration trackers, outage detection and other engineering tools more quickly.
\nThe important benefit was not simply generating code. It was reducing the cost of trying something.
\nWe were learning through experiments, failures, measurements and operational incidents. If we did not capture those discoveries, the next person would have to learn the same lesson again.
\nDocumentation therefore became more than recording architecture decisions. We captured what we had learned about devices, networks, testing, diagnostics, deployment and operations, and shared that learning through technical articles, training and our engineering community.
\nTraditional documentation often tells us what the system is. In an unfamiliar domain, it is equally valuable to capture how we discovered why it is that way.
\nLooking back, I don't think we invented a new methodology for IoT or embedded software.
\nWe still wanted fast feedback, automation, good testing, observability, maintainability, safe deployment and continuous learning. What changed was how those principles were practiced.
\nOver time, the changes looked like a new set of defaults.
\n| We started with this assumption | \nWe ended up with this sensible default | \n
|---|---|
| Documentation should tell us enough | \nLearn by building an experimentation environment | \n
| Hardware can be tested later | \nConsider the physical world part of the development lifecycle | \n
| A device performs its function, and platform can always explain its health | \nEvery device must be able to explain itself | \n
| Interfaces are primarily for users | \nDesign interfaces for engineers as deliberately as interfaces for users | \n
| Network behavior, physical environment can be tested and reproduced with a few devices in lab | \nSimulate what is expensive to reproduce physically | \n
| Devices should normally be reachable | \nDesign for disconnection, not just connection | \n
| Battery is mainly a hardware concern | \nTreat energy consumption as a software design concern | \n
| Deployment means installing firmware | \nDesign deployment and recovery together | \n
| Optimize primarily for delivery speed | \nOptimize the engineering process for confidence at scale | \n
| Observability means logs and metrics | \nDesign observability for the fleet, not just the device | \n
| Build the product, then supporting tools | \nThe engineering environment became part of the system | \n
| AI helps write code | \nUse AI to shorten the learning loop, not just the coding loop | \n
| Discovery happens before development and capture decisions in the final design | \nCapture discoveries, not just decisions | \n
The table is a record of what happened. Each default was earned by encountering a problem.
\nI use the term sensible defaults deliberately.
\nNone of these practices was invented from first principles. Most emerged from problems we encountered. We started with an assumption that seemed reasonable, ran into something that challenged it, experimented, and gradually changed the way we worked.
\nThat is how engineering practices often evolve in real projects. The environment tells you where your assumptions are wrong. If you pay attention, those failures can help you discover a better default.
\nThat was certainly our experience. When software meets hardware, the engineering principles do not necessarily change. But the sensible defaults might.
\n", "date_modified": "2026-10-15T13:25:00+00:00", "date_published": "2026-10-15T13:25:00+00:00", "external_url": "https://martinfowler.com/articles/software-meets-hardware.html", "id": "urn:aggr:item:005f2f0f62af8855b24868e26e41c533fea1db66", "image": "https://martinfowler.com/articles/software-meets-hardware/card.png", "summary": "Modern software engineering gives us a set of practices that we tend to carry from one project to another: AI assisted development, automated testing, CI/CD, observability, automation, simulation, and safe deployment. They work well enough…", "tags": [ "programming" ], "title": "When Software Meets Hardware", "url": "https://aggr.aymericbeaumet.com/items/martinfowler-com/2026-10-15-when-software-meets-hardware/" }, { "authors": [ { "name": "Martin Fowler" } ], "content_html": "Most knowledge workers who want to make presentations don’t think of themselves as designers. It’s fair to say the visual design of presentations is daunting enough to consume most presenters' mental space. Perhaps that’s why we equate the slides with the presentation.
\nVisual design is a broad subject, so we can’t cover it in its entirety. However, it’s important enough that the next few articles in this series will focus on slide design. I’ll begin with some fundamentals. Imagine I had to create a presentation about the content of this article. If I had to create a title slide for that deck, I might make it look like the one below. It juxtaposes a few elements — an image, text and some colours — into a layout.
\n![\"\"](\"https://martinfowler.com/articles/never-send-slides/index/Slide%20design%20fundamentals.jpg\")
Figure 1: A simple title slide combines an image, some text and light-handed colours.
\nIf I were to continue building these slides, I’d have to make some recurring design decisions. Instead, I embed these decisions into a template. We’ll discuss what it takes to set up your own template, but first let’s address an elephant in the room.
\nIf you work at a large enough corporation, your employer may have provided a corporate slide template. There are good reasons for this. Most people aren’t adept at assembling slides, so it helps to codify some default layouts in a template. People often use colours and fonts that may clash with the brand’s aesthetic, and a template can place some guardrails to avoid these errors.
\nThe trouble with standard templates is that everyone uses the defaults. Before you know it, everyone’s slides look the same. People ignore boring things, so there’s value in breaking that monotony.
\nNow, if you can understand what your marketing colleagues are trying to achieve with their prescribed templates, you can adhere to your corporate brand and assemble slides that look unique. Through the rest of this article, I’ll show you how to build your own template, but you can also use that advice to unpack the key elements of any corporate slide template.
\nAs a self-taught designer, I’ve often struggled with making colours play well with each other. For some people, colour choice is intuitive. When you reuse the same colours in all slides of your presentation, the visuals gain a cohesive identity.
\nIt’s helpful to understand the basics of colour theory, so you know which colour palette might work for your presentation style. The colour wheel is a good starting point. Think of it as an abstract illustration of the colour spectrum that helps you understand which colours work well together.
\n![\"\"](\"https://martinfowler.com/articles/never-send-slides/index/Basic.jpg\")
Figure 2: A basic colour wheel.
\nWhen you examine the wheel, you’ll notice that the closer the colours are to the centre, the less intense they are. Colours at the periphery are more saturated. There lies lesson one. If your slide background will be white or light, choose colours that sit at the periphery. If you want your slide background dark, choose colours that sit close to the centre.
\n![\"\"](\"https://martinfowler.com/articles/never-send-slides/index/Light%20and%20dark%20backgrounds.jpg\")
Figure 3: Colours at the periphery work well with light slides. Those towards the centre work well with darker backgrounds.
\nThere are many ways to choose colours and build your palette.
\n| Approach | \nWhat it is | \nColour wheel | \n
|---|---|---|
| Complementary colours | \nThese colours sit on opposite ends of the wheel. Placing them together creates a strong, high-contrast colour scheme. | \n![\"\"](\"https://martinfowler.com/articles/never-send-slides/index/Complementary.jpg\") | \n
| Analogous colours | \nThese colours sit close to each other on the wheel. When you use them together, you’ll end up with a minimalist, elegant colour scheme. | \n![\"\"](\"https://martinfowler.com/articles/never-send-slides/index/Analogous.jpg\") | \n
| Evenly-spaced colours | \nIf you choose colours evenly spaced across the colour wheel, you’ll end up with a colourful scheme with reasonable contrast between them. This approach is a variation of the complementary colours approach. | \n![\"\"](\"https://martinfowler.com/articles/never-send-slides/index/Evenlyspaced.jpg\") | \n
| Triads | \nBy choosing shades that form a triangle on the colour wheel, you get a tight variation on the evenly spaced colour scheme approach. | \n![\"\"](\"https://martinfowler.com/articles/never-send-slides/index/Triad.jpg\") | \n
| Tetrads | \nYou can extend the concept of triads by drawing a square on the colour wheel and adding another colour to your scheme. | \n![\"\"](\"https://martinfowler.com/articles/never-send-slides/index/Tetrad.jpg\") | \n
| Monochromatic | \nIf you want a super-minimalist approach, choose different shades of the same colour by drawing a straight line on the colour wheel. | \n![\"\"](\"https://martinfowler.com/articles/never-send-slides/index/Monochromatic.jpg\") | \n
Adobe provides you with a free, interactive colour palette generator to play around with the colour wheel and land on a theme you like. You can build your palette from scratch, browse trending palettes, and even generate them from a photograph that represents the content you want to present.
\n![\"\"](\"https://martinfowler.com/articles/never-send-slides/index/Picking%20colours.jpg\")
Figure 4: Generating a colour palette from a representative image.
\nThe advantage of working with palettes is that software like PowerPoint lets you swap themes across your entire presentation without updating every diagram. Choose a new theme, and PowerPoint recolours all your diagrams and slides. The interactive demo below shows you how you can swap your colour palettes in PowerPoint.
\nBut here’s a word of caution. Resist the temptation to invent a new colour palette for every presentation. As a corporate presenter, I’ve found it helpful to find or create a palette that I like and then stick with it for months, often years.
\nIf you work for a large corporation, someone else has likely already defined a colour palette. For example, at Thoughtworks, our corporate palette has six colours, plus black and a light grey shade. Since most of these colours are dark, I use them on a light background.
\n![\"\"](\"https://martinfowler.com/articles/never-send-slides/index/TW%20palette.jpg\")
Figure 5: The Thoughtworks colour palette.
\nOver the years, I’ve built a diagram library that reuses the same colour palette, so if a presentation references one of my core ideas, I incorporate a representative diagram into a Thoughtworks slide deck. It’s the compounding benefit of using the same, boring colour palette.
\n![\"\"](\"https://martinfowler.com/articles/never-send-slides/index/Async-first%20diagram%20library.jpg\")
Figure 6: A reusable diagram library for my book that leverages the Thoughtworks palette.
\nNovice designers often use too many colours in the quest to build colourful slides. Now that you know a bit about colour theory, you’ll realise that if you pick too many colours from the wheel, you’ll struggle to establish a theme and a visual identity that’s evident across your slides. So, choose a restricted colour palette instead. Limit yourself to six colours, then add black and white to the palette. Less is more. As a bonus, consider all shades of grey as free additions.
\nAs presenters, we needn’t be experts in the dark art of typography. Most presentations need only two or three fonts. Sometimes, one font face is plenty. Consistency beats variety.
\nBut here are the basics. There are two types of fonts, each with its own merits — serif and sans-serif. Serif fonts have little feet, i.e. the serifs. Sans-serif fonts literally lack serifs. The image below shows the difference.
\n![\"\"](\"https://martinfowler.com/articles/never-send-slides/index/Serif%20Sans.jpg\")
Figure 7: Serif fonts have little feet and extensions. Sans-serif fonts don’t have them.
\nIf you’re reading this article on martinfowler.com, you’ll notice the website’s serif typeface. And that’s a great use of serif fonts. They’re more suited to reading and paragraph text. This is also why newspapers still use serif fonts.
\nSans-serif fonts, on the other hand, suit billboards and signage better. When you drive around your town, look at the fonts on large advertisements. It’s likely they employ sans-serif fonts. Presentation slides are more like billboards than magazine pages.
\nSo, if you had to default to one kind of typeface for your presentations, I’d recommend picking a sans-serif font. They work well for online presentations, where the resolution is often a maximum of 1080p or 720p, and just as well for keynotes in large auditoriums with state-of-the-art projectors.
\nHowever, there’s no harm in mixing and matching fonts. You could use a serif font for titles and a sans-serif font for the rest of the text. The key is to stay consistent and avoid what Neal Ford calls the “fontaholic” antipattern.
\nA Fontaholic is someone who believes using a cacophony of fonts will “jazz up” a presentation. Disharmonic fonts are jarring. They make presentations gruesome and more difficult to read.
\nHere’s an example of a slide that uses a serif font for the title and a sans-serif font for everything else.
\n![\"\"](\"https://martinfowler.com/articles/never-send-slides/index/Shaping%20identity.jpg\")
Figure 8: Mixing and matching serif and sans-serif fonts.
\nLimiting the number of fonts in your presentation shouldn’t limit your creativity. Modern, variable-weight fonts allow you many variations for the same typeface.
\n![\"\"](\"https://martinfowler.com/articles/never-send-slides/index/Variable%20weight%20font.jpg\")
Figure 9: Variable-weight fonts offer you a lot of versatility.
\nAside from serif and sans-serif fonts, I’ll advocate for two more font types.
\nI suggest using these fonts only for annotations. They’re excellent at breaking the pattern and monotony of your standard fonts, and that’s handy when you want to draw your audience’s attention to a specific part of your presentation. For example, in the slide below, the red, handwritten font appears after each part of the diagram builds, drawing the audience’s attention to the compounding benefits of the AI flywheel.
\n![\"\"](\"https://martinfowler.com/articles/never-send-slides/index/AI%20flywheel.jpg\")
Figure 10: You can use a handwritten font for annotations and break the visual monotony.
\nIf you’re a developer and you often show code snippets in presentations, I suggest adding a monospace font to your repertoire. In code, spacing is part of the meaning. A monospace font gives every character equal width and helps you maintain indentation, columns, and structure. Your code is easier to scan and far easier to explain on a slide. Here’s an example from Phillip Schmid’s presentation at the AI Engineer conference, where he explains how to invoke Antigravity as a remote agent.
\n![\"\"](\"https://martinfowler.com/articles/never-send-slides/index/Philip%20Schmid%20-%20monospace%20font.jpg\")
Figure 11: Using monospace fonts to show code (credit: Phillip Schmid).
\nEven non-technical presenters can use monospace fonts to explain some concepts. For example, when explaining Slack channel naming conventions, I use the following slide.
\n![\"\"](\"https://martinfowler.com/articles/never-send-slides/index/Minimalist%20monospace%20use.jpg\")
Figure 12: Even non-technical presenters can leverage monospace fonts.
\nFor those of you who are curious, here are some of my favourite fonts.
\n| Serif | \nSans-serif | \nHandwritten | \nMonospace | \n
|---|---|---|---|
| Bitter Roboto Slab Playfair Display | \nInter Open Sans Noto Sans Work Sans | \nCaveat Gloria Hallelujah | \nMenlo Andale Mono Fira Code | \n
Font choice aside, pay attention to how you use fonts. Sometimes fonts don’t work well out of the box. To make them look right, you may have to adjust kerning (a.k.a. character spacing) and leading (a.k.a. line spacing).
\nIn the example below, notice the extra gaps between the letters A and V, V and A and T and A. Now drag the slider to see how I’ve adjusted the kerning between the letters, so the characters have an even distance between them.
\nWhen you’re showing quotes in a presentation, you’ll often run into issues with leading. In the example below, you’ll notice that the gap between each line is too wide. When you drag the slider, you’ll see how a leading adjustment brings the lines together and makes the quote look cohesive.
\nIf you understand these fundamentals of font choice and typographic spacing, you have enough typography knowledge to build a slide template. All that remains is your slide layouts.
\nI’ve struggled to adopt slide layouts and failed. In the last decade or so, I’ve found myself rejecting cookie-cutter layouts and starting with a blank slide each time. I follow some slide construction principles, which I’ll show you in a future article, but my template itself is quite simple. When you see my slides, you’ll notice why it’s difficult for me to follow the standard layouts that come with slide templates.
\n![\"\"](\"https://martinfowler.com/articles/never-send-slides/index/Standard%20layout%20doesn't%20apply.jpg\")
Figure 13: I find it difficult to adapt my design approach to standard slide layouts.
\nSo, I’ve deleted all prescribed layouts that come with our corporate template. My template has three layouts, and I use the blank slide almost all the time. I’ve set up the Thoughtworks corporate palette on my system, so I can choose the colours when I need them for diagrams. I’ve also set up a few font styles based on a serif-and-sans-serif font combination. I use handwritten fonts and monospace fonts on a case-by-case basis, so you won’t see font styles for them. The interaction below will show you different components of my standard presentation theme.
\nYou’ll notice that my template is uncomplicated, and that simplicity lends me a lot of flexibility and power when constructing slides. My slides don’t have page numbers, copyright notices, or even company logos, except on the title slides. Slides are not documents, so slide numbers are irrelevant to my presentations. I don’t want to sue anyone for copying my slides, so I usually leave out the copyright notice. And I want to introduce myself and my employer once at the start and leave it there. From that point on, the logo is an avoidable visual distraction. Taking away these floodmarks (credit: Neal Ford) affords me the luxury of designing an empty slide.
\nAt conferences, people often photograph your slides. You could add a copyright notice to each slide, but people can remove these floodmarks using phone-based editing apps. I’m not fussed if they reuse the content or design, or if they attribute it to me. So, in the spirit of minimalism, I don’t add a copyright notice. I do announce at the start of the talk that I’ll share a whole bunch of leave-behinds. When I’m taking questions at the end of a talk, I throw up a slide with QR codes (example below), so people who need related resources can access them.
\n![\"\"](\"https://martinfowler.com/articles/never-send-slides/index/QR%20codes%20to%20scan.jpg\")
Figure 14: Use QR codes to share resources with your audience.
\nIf you’re concerned about copyright, you can provide a set of watermarked slides through your QR codes. Your slides will be useless without you, but those who want to reference them can have them. The watermarked download will obviate the need for floodmarks on your slides. Win!
\nA reusable colour palette, a suite of fonts and a blank slide — that’s all you need to address the first decision space in your presentations — that of your template. Once you have that template in place, you can layer components on top of it.
\n![\"\"](\"https://martinfowler.com/articles/never-send-slides/index/First%20decision%20space.jpg\")
Figure 15: Templates — the first decision space for slide design.
\nSince we’ve already discussed text and typography in this article, we’ll jump to photographs and illustrations in the next post. I’ll show you how to source images and how to give them a consistent look for your presentation. You’ll also learn how to make text play well with images and how you can avoid some of the most common antipatterns when using images in your slides. See you there.
\n", "date_modified": "2026-10-14T14:20:00+00:00", "date_published": "2026-10-14T14:20:00+00:00", "external_url": "https://martinfowler.com/articles/never-send-slides/build-your-slide-template.html", "id": "urn:aggr:item:d00cfad6d44894faf787f2552982abc8eb6322c3", "image": "https://martinfowler.com/articles/never-send-slides/index/card.png", "summary": "Most knowledge workers who want to make presentations don’t think of themselves as designers. It’s fair to say the visual design of presentations is daunting enough to consume most presenters' mental space. Perhaps that’s why we equate the…", "tags": [ "programming" ], "title": "Build your slide template", "url": "https://aggr.aymericbeaumet.com/items/martinfowler-com/2026-10-14-build-your-slide-template/" }, { "authors": [ { "name": "Martin Fowler" } ], "content_html": "I went into this experiment expecting a behavioral harness to help a coding agent meet the requirements more reliably, save tokens and prevent wasted work. Giving an agent clear architectural contracts and deterministic test feedback while it coded seemed like an obvious way to help it get the implementation right on the first pass.
\nFor GPT-6 Sol, the harness used about 10.5 times as many tokens, took nearly 3 times as long, and cost about 7 times more, yet both conditions succeeded in all 10 runs.
\nThe results flatly contradicted that expectation and surprised me. Using Codex with GPT-6 Sol at low reasoning effort, the agent produced an application that passed all 178 automated acceptance tests in each of the 10 runs, both with and without the harness. By this measure, the harness produced no observable improvement in the final applications.
\nGPT-6 Sol (10-run averages for tokens, time, and cost):
\n| Harness | \nTokens | \nTime | \nEstimated cost | \nAccepted runs | \n
|---|---|---|---|---|
| Yes | \n11,331,970 | \n19 mins 53 secs | \n~$2.679 | \n10 of 10 | \n
| No | \n1,076,371 | \n6 mins 51 secs | \n~$0.406 | \n10 of 10 | \n
The harness increased average time and cost without changing the observed acceptance result. In exploratory runs with Claude Sonnet 4.6, Claude Opus 4.6, and Gemini 3.8 Flash (all at low reasoning effort), each also managed to complete the task unassisted on the first try.
\nThese frontier models handled this task consistently with and without the harness. Could executable feedback help a cost-optimized model achieve similarly consistent outcomes?
\nBefore looking at the results for a cost-optimized model, here is how I set up the experiment:
\nBorrowing Birgitta Böckeler’s framing, a behavioral harness combines guides and sensors. Guides anticipate where the agent might go wrong and steer it before it acts, increasing the chance of a correct first attempt. Sensors observe the results after it acts and provide feedback that helps it self-correct.
\nFor each model, I compared agents working with and without the behavioral harness. Both conditions began with the application brief and baseline repository instructions in separate, private workspaces. Harnessed agents also received the API specifications, examples, and tools described in the inner-loop section below.
\n![\"Two](\"https://martinfowler.com/articles/are-harnesses-worth-the-cost/experiment-setup.png\")
Figure 1: The two development conditions and their independent evaluation
\nThe experiment separated development from evaluation into two distinct loops. The inner loop provided feedback to harnessed agents while they built. Once an agent finished, the outer loop ran 178 automated acceptance tests against its generated service. Both conditions faced the same evaluation, and the inner harness never had access to the evaluation suite’s answer key. Finally, I collated the accepted outcomes, token use, time, and estimated cost.
\nIn this experiment, I treated an application as acceptable only if it started successfully and passed all 178 external acceptance tests. Reliability meant how consistently a model produced an acceptable application across repeated runs. Token use, elapsed time, and estimated cost were separate measures of run efficiency.
\nEvery attempt began with a detailed application brief. I described what the service should do, but left the code-level implementation details to the agent.
\nThe task was to build a backend-for-frontend (BFF) for product discovery. It had to expose a consumer-facing HTTP API, validate incoming requests and downstream responses, and call an Order API. After a successful availability search, it also had to publish only the first returned product to Kafka. Publishing every product would have allowed the agent to pass the data through without implementing the selection behavior.
\n![\"The](\"https://martinfowler.com/articles/are-harnesses-worth-the-cost/application-architecture.png\")
Figure 2: The application architecture described by the brief
\nThe brief also specified less obvious behavior. A timed-out product creation had to become a deferred request with a 202 response code and a link in the response header that consumers could poll for a response. A transient search failure had to return 429 with a usable recovery interval; unavailable dependencies or failed event publication had to become 503; and successful downstream responses still had to obey requested product filters or otherwise return 502.
![\"A](\"https://martinfowler.com/articles/are-harnesses-worth-the-cost/sequence-diagram.png\")
Figure 3: A successful availability search and its Kafka publication
\nIn short, the service was more than a set of CRUD endpoints: the agent had to coordinate HTTP semantics, asynchronous events, schema boundaries, and fault-tolerance policies simultaneously.
\nThis was also a favorable starting point for an unaided agent: the brief spelled out the functional requirements, architectural and technology constraints, detailed descriptions of the external interfaces, response shapes, and failure behavior in unusual detail. I do not yet know how the results would change with a less complete brief.
\nIn this experiment, the guides included OpenAPI and AsyncAPI specifications that expressed the interface agreements in machine-readable form, providing a basis for executable checks. They complemented the brief rather than defining a separate set of requirements. The guides also contained request-response examples for these interfaces. The examples covered downstream timeouts, malformed payloads, filter violations, retry sequences, and Kafka event confirmations. An API-development skill supplied implementation practices and guidance on dependency simulation, behavioral verification, and how to diagnose and respond to failures.
\nThe sensors checked contract conformance, used API tests to verify HTTP response values, and asserted observable side effects, including calls to downstream mocks and Kafka publication. Test results, CTRF reports, and API coverage reports returned those observations to the agent during development.
\nBirgitta also distinguishes computational controls, executed by deterministic tools, from inferential controls, which rely on model interpretation or judgment. This harness combined both. The behavioral checks produced computational feedback without an LLM judging correctness. Guided by the skill's instructions, the agent still had to interpret those results and decide how to fix failures.
\n![\"The](\"https://martinfowler.com/articles/are-harnesses-worth-the-cost/inner-harness.png\")
Figure 4: The behavioral harness and its feedback paths
\nAn agent plugin supplied the skill and MCP tools, accompanied by the contracts and examples. The OpenAPI specifications covered the BFF and downstream Order API; the AsyncAPI specification covered Kafka.
\nBefore implementing the service, the agent read the specifications and examples alongside the brief. The skill directed it to map the application’s components and link each required behavior to its downstream interactions and verification checks. Helper scripts scaffolded those documents, a record of verification evidence, and the commands for building, starting, and testing the service; the agent completed the behavior-specific details.
\nConfiguration tools wired the contracts to the application and dependency mocks. Contract-driven dependency mocks made downstream interactions executable, allowing the agent to exercise its implementation against defined downstream responses, failures, and recovery sequences. Other helpers verified timing constraints. Together, these tools provided both implementation guidance and an executable environment in which the agent could check its decisions.
\nThe agent invoked the harness through the project’s test command. That command validated the configuration and examples, ran the required build and unit checks, started the downstream mocks, and launched the application against them. It then ran the checks described above.
\nReport summaries identified specific contract, response value, and interaction mismatches. The agent used the skill’s instructions to diagnose them, change the implementation or execution setup, and use a helper to rerun the affected scenarios. Once those scenarios passed, the agent ran the complete suite as a final gate. The feedback loop depended on the agent invoking these tools during development; it did not run automatically after every code edit.
\n![\"The](\"https://martinfowler.com/articles/are-harnesses-worth-the-cost/inner-harness-sequence-diagram.png\")
Figure 5: The inner-loop test and feedback sequence
\nOne of the big challenges of designing this type of behavioral harness is to make sure the guides and sensors don't contradict each other and confuse the agents. I addressed this issue with the following approach:
\nBefore running any tests, an “examples validate” command checked that the example payloads complied with the respective API specifications. Tests then used those specifications and examples to check the implementation, keeping the sensors aligned with their source guides.
The source of those checks mattered. If an agent interprets requirements to write both the implementation and its tests, the same misreading or omission can appear in both. Allowing it to change the expected results when tests fail can also obscure a defect. Here, the contracts and examples represent agreed human expectations in executable form. Contract checks were generated deterministically from supplied specifications, while supplied examples with behavioral fixtures defined value and side-effect assertions. The agent was instructed to preserve those inputs and the skill’s verification helpers, investigate failures, and correct the implementation or execution setup rather than weaken the expectations.
\nThis gave verification expectations a source independent of the implementing agent’s interpretation. Used as a deterministic computational control, they make correctness checkable without relying on the agent's own claims or an LLM's judgment, freeing humans to focus review on what the contracts don't cover.
\nThe same sensors therefore serve two distinct purposes depending on when they run:
\nThe skill also required reviewable evidence of dependency interactions and the final verification result, so completion was supported by recorded observations rather than the agent’s claim alone.
\nIn this experiment, I implemented the behavioral harness using Specmatic, though equivalent contract-testing and service virtualization tools could provide the same capabilities.
\nTo keep the comparison fair, each attempt ran inside an isolated Docker container with its own workspace and a fresh, attempt-specific Codex home. Apart from shared authentication and language dependency caches, each agent began without sessions, memories, plugins, skills, or other state from an earlier run.
\nThe two conditions differed in what agents could use during development; within each comparison, completed applications faced the same external evaluation.
\nAfter development stopped, the outer-loop evaluation harness tested each application against the same corpus of 178 combined functional checks, including invalid inputs, downstream failures, recovery sequences, and the messages produced by successful requests. These checks exercised the behavior specified in the application brief, including the client-facing API, downstream interactions, and Kafka publication. The evaluation harness controlled its own downstream fixtures and message capture rather than relying on the application's development tests.
\nOpenAI positions GPT-6 Sol as a flagship model for complex professional work, while GPT-6 Luna is optimized for cost-sensitive, high-volume workloads and roughly corresponds to the nano tier in earlier GPT-5 families.
\nWith the methodology in place, the core question is whether a model with a smaller capability profile would behave differently.
\nI ran the exact same experiment on GPT-6 Luna at low reasoning effort. Without a harness, it succeeded in 3 of 10 runs. With the harness, it succeeded in all 10.
\nGPT-6 Luna (10-run averages for tokens, time, and cost):
\n| Harness | \nTokens | \nTime | \nEstimated cost | \nAccepted runs | \n
|---|---|---|---|---|
| Yes | \n22,933,648 | \n35 mins | \n~$0.273 | \n10 of 10 | \n
| No | \n5,860,736 | \n15 mins 56 secs | \n~$0.086 | \n3 of 10 | \n
In this sample, the harness did not unlock a capability Luna lacked; it turned a 3-in-10 baseline into 10-for-10 reliability.
\nProviding that feedback added an average of 17,072,912 tokens, 19 minutes 4 seconds of autonomous runtime, and about $0.187 of estimated API-equivalent cost per attempt. The harnessed attempt used roughly 3.9 times as many tokens overall as an unharnessed Luna attempt. But look at the comparative economics: even with the harness overhead, a harnessed Luna attempt cost about 33% less than an unharnessed Sol attempt ($0.273 vs $0.406), and both conditions achieved the same 10-for-10 acceptance rate in this sample.
\nLooking past the headline pass rates into the codebase, several implementations stood out:
\nGPT-6 Sol solved the brief unaided in one shot.
\nA single /goal prompt was enough. All 10 unharnessed GPT-6 Sol applications passed every functional check on the first pass without human debugging or corrective follow-ups.
GPT-6 Luna came remarkably close on its own.
\nUnharnessed GPT-6 Luna runs passed 94.8% of the functional checks. It was stumbling over subtle edge cases rather than missing the task entirely.
\nAsynchronous 202 workflows held together end-to-end.
\nReturning a 202 Accepted status code is straightforward. What surprised me was seeing the agent wire up the monitor endpoint, track pending requests, retry downstream creation with the original idempotency key, and surface the final payload at the polling URL.
Kafka publication stayed tied to the search transaction.
\nRather than firing off a message asynchronously and returning an immediate 200, accepted applications waited for Kafka acknowledgement before responding, and correctly translated publication failures or timeouts into 503 Service Unavailable.
Boundary validation was consistently defensive.
\nBoth models validated incoming payloads against schemas and validated downstream response structures and business constraints before using the data, rather than blindly trusting downstream services.
\nThe failed unharnessed Java applications did not contain invalid syntax or fundamentally misunderstand the brief. Each still passed at least 144 of the 178 combined functional checks. Their failures clustered around the deferred creation workflow and a smaller set of protocol details:
\nThe observed gaps appeared at protocol boundaries, resiliency paths, and integration seams, where a convincing implementation could still fail as a system.
\nDeferred creation broke at the timeout boundary.
\nWhen a downstream product creation exceeded the configured timeout, the BFF was supposed to return 202 Accepted with a monitor link, retry in the background, and eventually expose the downstream 201 response. Instead, 4 unharnessed applications returned 503, 2 returned 400, and 1 returned 429. An application already counted above also created monitors in some scenarios, but its retries never reached the terminal 201 response. Several implementations recognized timeouts by matching a narrow exception type or cause-chain shape. The artifacts establish the incorrect responses, but often do not record the actual client exception, so that mechanism is a likely explanation rather than a proven common cause.
Missing monitor links caused secondary failures.
\nIn 7 runs, the incorrect response to the timed-out creation also meant that no monitor URL was returned. A later evaluator check then failed while trying to use that missing URL. I treat those errors as consequences of the broken 202 flow, not as separate application defects.
Kafka delivery succeeded while contract conformance failed.
\nIn 3 runs, every expected message reached Kafka, but the evaluator still rejected the publication. Two applications used Spring's JSON serializer and added an unexpected __TypeId__ header. Another emitted “categories”: null, even though the optional field had to contain an array when present. The integration worked mechanically while still violating the wire contract.
Error responses missed exact HTTP contracts.
\nOther observed misses were narrower. Although 415 Unsupported Media Type is a conventional response for an unsupported media type, the application brief explicitly required 400 for invalid requests of this kind, so the evaluator counted 415 as a contract violation. In other cases, invalid requests reached a generic 500 handler, and a missing monitor returned the wrong representation. These were small differences in code, but enough to make the application unacceptable under the external evaluation.
Configuration names diverged at an integration seam.
\nOrdinary product creation returned 503 instead of 201. The generated client read ORDER_API_API_KEY, while the recorded launch supplied ORDER_API_KEY. That mismatch is a plausible explanation for the downstream failures. In harnessed runs, contract-driven dependency mocks and configuration checks enabled validations of these environment variables before deployment.
A behavioral harness turns written requirements into environmental feedback, bringing dropped details back into focus when a sensor goes red.
\nI find myself thinking about this dynamic in terms of executable attention.
\nThe behaviors behind these failures were specified in the supplied requirements. The application brief clearly specified the deferred creation workflow, Kafka message contract, and exact error responses. A requirement can be explicit in the brief and still lose influence during implementation. I think of this as an attention lapse: the resulting code omits or incompletely implements a requirement the agent was given.
\nJuggling library choices, boilerplate setup, dependency injection, and asynchronous wiring across a growing context may contribute to such lapses, although this experiment did not establish the cause. Executable attention makes those gaps observable, giving the agent another opportunity to act on the requirement.
\nnull when the contract permits only an array or omission.GPT-6 Sol preserved these details from the written brief in all observed runs. GPT-6 Luna also implemented much of the behavior correctly without the harness, but its accepted outcomes varied. The misses in Luna are consistent with requirements losing influence during implementation. My interpretation is that the harness served as an external memory and attention aid, pulling dropped requirements back into focus whenever a sensor went red.
\nPutting both model samples side by side makes the cost of that feedback visible. These are API-equivalent estimates based on recorded token use and model prices, not actual billed costs.
\n![\"Average](\"https://martinfowler.com/articles/are-harnesses-worth-the-cost/cost-vs-time.png\")
Figure 6: Average estimated cost and time across both models
\nWithin the GPT-6 runs, harnessed Luna averaged about $0.273 per attempt, compared with $0.406 for unharnessed Sol; both had 10 accepted applications in 10 attempts. The harnessed Luna attempt therefore cost about 33% less on average in this sample.
\nAll 10 harnessed GPT-6 Luna applications passed the external evaluation and achieved a 100% success rate. This small sample cannot tell us the long-run failure rate or the total cost of achieving an acceptable application.
\nThese figures measure only autonomous generation cost and machine time. They exclude the upfront effort of authoring, reviewing, and maintaining the contracts, examples, assertions, and harness. They also leave out the effort of finding, diagnosing, and repairing an unacceptable result. Reusing interface contracts and checks across implementations could spread the setup cost over time, but I did not measure that reuse or the resulting total cost.
\nSol’s 10-for-10 result still depended on independent evaluation. That suggests another comparison: make the sensors available to the agent only at a final verification gate, rather than throughout development. A final-gate workflow would provide an independent acceptance check while minimizing overhead when the first implementation passes. Failures, however, could still require diagnosis and repair cycles. I did not measure that workflow’s cost.
\nThis experiment covered 10 paired runs per model on 1 greenfield Java service, with both models at low reasoning effort. That sample describes the observed outcomes; it cannot establish long-run reliability.
\nA more demanding architecture or a trickier resiliency scenario might expose gaps in Sol that require a harness. Conversely, a simpler task might run reliably on Luna without this behavioral harness.
\nThe brief specified functional behavior, architecture, technology, and interfaces in unusual detail. I have not tested whether the pattern holds when modifying legacy code, using other languages or reasoning settings, or working from vague or incomplete requirements. A less complete brief could affect both the unaided implementation and the quality of the contracts and examples supplied to the harness. I would be cautious about extrapolating these exact figures to different briefs, architectures, codebases, or team workflows.
\nBecause the harness added guides and sensors together, this experiment also cannot isolate their individual contributions.
\nI would treat these results as a starting point for choosing a model and feedback cadence for a particular task. What changes with model capability is when and how often the sensors run, not whether an independent check is needed.
\nModel capability changes when and how often sensors run, not whether an independent verification gate is needed.
\nMy original assumption was too blunt. A behavioral harness is not an automatic win for every model on every task. For Sol, running the harness throughout development added overhead without a detected acceptance benefit; for Luna, harnessed attempts were accepted more consistently and cost less on average than unharnessed Sol attempts in this sample.
\nThat leaves me with a revised, conditional hypothesis:
\nWhat changes with model capability is not whether independent sensors are needed, but when they run. For models whose unaided attempts produce inconsistent outcomes, running sensors continuously during development provides executable attention that broadens the set of models a team can use successfully. For frontier models capable of solving the brief in one pass, running those same sensors once as an independent final acceptance gate provides the verification every system requires without the development overhead.
\nIn practice, engineering teams rarely pick a model on raw benchmark intelligence alone. Unit economics, latency budgets, data privacy, and the ability to run on premises often dominate the decision. If a behavioral harness can take a cheaper, lighter model that already fits those operational constraints and make it reliably deliver the goods, that is a compelling option.
\nI have tested this so far on a single greenfield service with an unusually detailed brief. The next step is to test a “final gate only” condition for frontier models to measure its overhead and how often the gate catches discrepancies. Beyond that, I plan to vary the quality of the brief and the consequences of a missed requirement, then test how well this hypothesis holds up when modifying an existing codebase, switching programming languages, or running open-weights models locally.
\n", "date_modified": "2026-10-13T13:35:00+00:00", "date_published": "2026-10-13T13:35:00+00:00", "external_url": "https://martinfowler.com/articles/are-harnesses-worth-the-cost.html", "id": "urn:aggr:item:ec7bb609c43561d808c625cdd50f9af3a4032206", "image": "https://martinfowler.com/articles/exploring-gen-ai/donkey-card.png", "summary": "I went into this experiment expecting a behavioral harness to help a coding agent meet the requirements more reliably, save tokens and prevent wasted work. Giving an agent clear architectural contracts and deterministic test feedback while…", "tags": [ "programming" ], "title": "Are Harnesses Worth the Cost?", "url": "https://aggr.aymericbeaumet.com/items/martinfowler-com/2026-10-13-are-harnesses-worth-the-cost/" }, { "authors": [ { "name": "Martin Fowler" } ], "content_html": "In response to my last fragments (probably the bit about us worrying if LLMs have consciousness when we when we should be wondering why they don’t have a conscience) “Metalanguage” replied:
\n\n\nwe shipped the id and forgot the superego. classic software lifecycle.
\n
I don’t know what was on their mind, but their post immediately made me think of the classic 1956 movie Forbidden Planet. Plenty of sci-fi, and other literature, have explored humans creating technology with unintended behavior, going back at least to Mary Shelly. But that movie was particularly influential on sci-fi film-making and in the heart of its story is what happens when we nurture a thinking machine.
\nI use the term “nurture” here deliberately. We talk of building software, but building implies a degree of determinism. When we build a bridge, or a locomotive, we expect it to behave in a controlled and well-understood manner. That’s a difference in degree to how we cultivate plants in our garden, or nurture young children. One of the challenges of working with these systems is understanding what has changed in this shift from building a computational system to nurturing an inferential one, and how our processes need to change in response.
\nWe get unintended behavior with deterministic building: some bridges have collapsed, and our computational systems often have bugs. But one difference is that when we find a bug in a computational system we can usually fix it. Even if we can’t, we can usually disable a component so the bug won’t do further harm. With inferential LLMs however, there is no such simple fix or disablement, which may lead us to the fate of the Krell.
\n(If you haven’t seen Forbidden Planet, it’s well worth watching. Yes, it shows it was made in the 1950s - with special effects, music, acting, and attitudes of that decade. But the story is solid, and its key theme is very relevant to the future we build with generative AI. Just don’t read about it in advance, it’s better to be immersed in the story without spoilers - although my memory of that experience is understandably hazy.)
\nMany people who follow me also know my friend Ola Bini, who was my colleague at Thoughtworks for many years, and was living in Ecuador working as an independent software security expert. Sadly his time in Ecuador was dogged by a bogus prosecution by the authorities there. But things seemed to have settled down, and although not allowed to leave Ecuador, Ola was able to get on with his life.
\nSadly that’s no longer the case as he was deported from Ecuador on Friday:
\n\n\nAccording to information released by his lawyer, Bini was intercepted by a car with four people who identified themselves as immigration agents. He was then taken to an immigration office without further information or a formal order from a competent authority. There, officials told Bini that his visa had been revoked but didn’t show any supporting document.
\nBini’s defense filed a habeas corpus to safeguard his freedom and prevent his deportation. Yet, Ecuadorian authorities affirmed that the developer represents a threat or risk to public security and the state structure, and must leave the country. The ground for deportation is a secret report which allegedly asserts that Bini committed acts against the security of Ecuador. The defense could not access its contents.
\n
I was really worried for a while, since it wasn’t clear where he was going to be deported to. But he tweeted from Sweden, so I’m thankful for that. But this is only a partial relief. Ola has spent thirteen years in Ecuador and made it his home. To be thrown out of your home for scant reason is a heavy thing to bear, and the officials who did that have committed a serious offense.
\nDDD Europe have released the video of Gien Verschatse interviewing Eric Evans and myself at the conference in June. We start by talking about how we bonded over conceptual modeling in the late 1990s. The conversation quickly moves to AI, we note that it’s impossible to predict how such a big change will work out. We do expect that it will cause us to think about our work in different ways, but the change may well be liberating, it’s reinvigorated Eric’s love of programming.
\n\n\npeople are probably going to feel very frustrated by [the new way of thinking about software]… but when you get through that, there is a kind of a wonderful feeling of my brain’s been loosened up.
\n
Our background in agile planning helps with the uncertainty, as we are used to taking small steps and being attentive to feedback. We mull on the interplay of writing and thinking, in terms of both prose and code, and how its very much an iterative process of exploration and refinement - the same is true when we chat with our LLMs.
\nAnd don’t miss Eric’s important final tip.
\n\n\nPaul Graham:
\n\n
\n\nThere were a lot of things that only worked because there’s a limit to the rate at which humans can operate. We’re about to find out what all of them are, as they break.
\n
The speculation continues about whether or not reading code will play a part in a software developer’s future. Geoffrey Huntley says.
\n\n\nPeople are still saying, very loudly, that code should be readable so that humans can understand it. I no longer think that’s the goal.
\n
Interestingly his example has the LLM explain a haskell function definition… by translating it to Python. Which, to me, suggests there is a role for code - just that LLM need not store code in the same form that it presents it to a reader. This is essentially the same idea as projectional editing, which posits that the editable representation of software need not be the same as its storage representation.
\nSam Ruby touches on this as he muses on a Rails World keynote. He quotes DHH saying:
\n\n\nRust is a good prompt compilation target for the moment, but so is C++. And soon assembler. Then microcode. Myopic to think we’re going to stop the agentic drill bit until it reaches computing bedrock.
\n
He responds with:
\n\n\nThe post leaves one question unasked, though: what sits at the top of the drill? What do we keep, edit and trust as the source of truth?
\n
He carries out exercise of looking at some Rails software. Represented in Ruby/Rails and its about 60,000 tokens. Compiling it into C it turns into 4,000,000 tokens. That increase in token size will hamper the LLM, that still has to fit it into its context window, and even if it were to fit, figure out where to focus its attention. Sam points out reasons why, even absent a human reading it, it makes sense to represent the program in a higher-level language.
\n\n\nwhat Rails becomes when agents write the code: the most compact, precise and conventional specification of a web application, whatever it ends up compiled to.
\nLet the drill go as deep as it can. Just keep the notation at the top.
\n
This all reminds me of what Unmesh Joshi argued: that code serves “two distinct but intertwined purposes”: instructions to a machine, and a conceptual model of the problem domain. After exploring how those change with LLMs he concludes:
\n\n\nThe role of coding is not disappearing. But it is changing.
\nAs LLMs make code generation cheaper, the mechanical act of writing instructions becomes less central. What becomes more important is making the conceptual model explicit, discovering the right vocabulary, and refining that vocabulary through iteration, domain expertise, and feedback. This is also why programming languages continue to matter deeply. We are not meant to be passive reviewers of generated code. The act of writing code is itself part of our thinking.
\nCode is still instructions for a machine. But it is also a model of understanding. In the LLM era, that second role becomes even more important. The future of coding is not just writing more code faster. It is building better conceptual models, better vocabularies, and better foundations on top of which both humans and LLMs can work.
\n
In a later post, Sam pondered on how people are talking about the capabilities of agents in a way that resembles the parable of the blind men and the elephant. We all only have only a partial view of this object and where it’s going. A theme for all of us:
\n\n\nThe practical question isn’t whether agents are good. It’s this: for the task in front of you this week, where will the information come from, and what will check the result?
\n
The Economist’s pithy summation of investors concerns about the dangers of AI companies’ products:
\n\n\nIt’s hard to celebrate an initial public offering that leads to a terminal public offing.
\n
The news about the latest model from Google is interesting.
\n\n\nGemini 4 Argon has an insanely low hallucination rate on Artificial Analysis. 15%.
\nGrok 4.7 is at 29%. GPT-6 Astra 45%. Opus 5.5 59%. Fable 5.1 69%.
\nThe only models below it barely answer anything. None of them get more than 15% right.
\nIt gets fewer answers right than Opus 5.5 on max, 50% against 66%. But when it doesnt know, it says so instead of making something up.
\n
Being clearer about what it doesn’t know, at a cost of getting less answers right, is definitely a trade-off I prefer.
\n", "date_modified": "2026-10-05T04:28:00+00:00", "date_published": "2026-10-05T04:28:00+00:00", "external_url": "https://martinfowler.com/fragments/2026-10-04.html", "id": "urn:aggr:item:3070ab92e1cbbe189147467b5439b762b74fae81", "summary": "In response to my last fragments (probably the bit about us worrying if LLMs have consciousness when we when we should be wondering why they don’t have a conscience) “Metalanguage” replied: we shipped the id and forgot the superego.…", "tags": [ "programming" ], "title": "Fragments: October 4", "url": "https://aggr.aymericbeaumet.com/items/martinfowler-com/2026-10-05-fragments-october-4/" }, { "authors": [ { "name": "Martin Fowler" } ], "content_html": "By now, we’ve established the need for a strong narrative before you build slides. Not every presentation needs slides. Once you’ve built your narrative, you may decide that a speech is good enough. In fact, a few months back, I did just that. I was a leader-in-residence at our graduate program, Thoughtworks University. The trainers expected me to deliver a keynote, and I decided against slides. I called the talk “1% better — three things I wish I'd known as a grad.” 15 minutes, no visuals, just talk. And that was fine! The world didn’t end because a corporate talk didn’t have slides.
\nYou may work at a corporation where no conversation can begin without a slide deck on the screen. I’ve found this reliance on slides unproductive, especially when the cognitive load of frameworks sabotages the high-bandwidth collaboration possible when smart people find themselves in a synchronous space.
\nAll this said, I’m also an unabashed advocate for effective slides. Many presenters underestimate their content, thinking their topic is dull and that nothing will make it interesting. That may be true for a minority of topics. In which case, why are you even presenting? In most cases, you believe that your topic is interesting enough that you want to change something for your audience. You may want to sell a product, advocate for a new way of working, teach them something new, or motivate and inspire them. This is where your narrative can benefit from impactful visuals.
\nWhile design and decoration aren’t the same thing, you also want your visuals to look good.
\n\n\nAttractive things make people feel good, which in turn makes them think more creatively…positive emotions are critical to learning, curiosity, and creative thought.
\n-- Don Norman
\n
With that context behind us, let’s unpack when slides are most useful for presentations and the fundamental principles for building them.
\nIf you examine the light table (a.k.a. slide sorter) view of my slide decks, they won’t make sense to you. Without my narrative, they’re useless. And yet, the slides employ the same components that an infodeck might.
\nColours, typography, and slide layouts that determine the overall template;
\nand multiple components that include but aren’t limited to:
\ntext;
\nimages and illustrations;
\ndiagrams;
\ncharts;
\nand videos.
\n![\"\"](\"https://martinfowler.com/articles/never-send-slides/index/Light%20table.jpg\")
Figure 1: Slides from a past talk.
\nThe most important distinction between a presentation and an infodeck is what Neal Ford calls “control over the rate of knowledge exposition.” Unlike static infodecks, the builds (animations) and transitions in your presentation slides allow you to control that rate of knowledge exposition. You get to manage your audience’s cognitive load.
\nSlides also provide what Martin Fowler calls a “complementary visual channel” for your presentation. As a speaker, you’re conveying a message to your audience. The most important part of that message comes from your voice, expertise and stage presence. The slides shouldn’t make you redundant. Instead, they should enrich the message you’re already sharing using the audio channel.
\n![\"\"](\"https://martinfowler.com/articles/never-send-slides/index/What's%20the%20purpose%20of%20slides.jpg\")
Figure 2: You provide the words. The slides provide the visuals and some reinforcing words.
\nThe idea of a complementary visual channel becomes even more important when you reflect on how we learn. In their 2016 book, E-Learning and the Science of Instruction, Clark and Mayer establish the “multimedia principle”; i.e. people learn better from words and graphics than from words alone. We encode words and visuals in different parts of our brains. When you provide the words and your slides provide the visuals, your audience benefits from dual encoding. Your message becomes more memorable than if you provided words alone, or worse, words similar to those on your slide.
\nSlides aren’t the only visual channel possible. Tools like Prezi, Mural and Miro let you create a giant canvas for your talk, and you can pan and zoom into specific parts of it to support your presentation. However, since slideware is more ubiquitous in corporations, we’ll focus on the 16x9 slide format for the rest of our discussion.
\nWhen we think of slides as the visual channel for our presentations, another principle becomes apparent — your visuals must tightly couple with what you’ll say, without repeating what you’ll say. After all, you’ll present to a literate audience, and they can read faster than you read aloud.
\nTake a look at the slide below. It’s an example of the standard-issue slides you’ll see in a corporate infodeck. As a standalone reading artefact, it’s passable. As a presentation slide, it’s a disaster. By the time you start with the first point, your audience will have already speed-read the entire page. Martin Fowler calls this phenomenon “stealing the narrative,” because slides like these make you lose control of your story.
\n![\"\"](\"https://martinfowler.com/articles/never-send-slides/index/Ineffective%20slide.jpg\")
Figure 3: Audiences can read such slides faster than you.
\nThe above slide is an example of why you can’t present an infodeck, no matter how elegant the design. Don’t beat yourself up when your eyes glaze over at the sight of such slides. The above example also begs the question of why you’re there in the first place.
\n\n\nIf the slides can stand by themselves, why are you up there in front of them? Just hand out the slides and sit down!
\n\n
What’s the alternative, then? If I were presenting the content on the slide above, I’d break up the visuals to match my narrative. The single slide above would split into the 31 visuals below, driven by slide transitions and animations. The words on screen will appear as a summary of what I say, either in sync with my speech or right after I’ve made my point.
\nNow, if you were the narrator, you might choose a different split. There lies the disadvantage of well-designed slides that serve as a complementary visual channel. It’s difficult to share your slides with someone so they can present your topic. Effective slides often reinforce a speaker’s storytelling style, pace, and anecdotes. They’re fine-tuned to the speaker’s idiosyncrasies. While you can use another speaker’s slides, you’ll have to tune them to your delivery approach.
\nWhen you design slides that are in sync with your speech, you can’t design heavy, wordy slides. A single slide can have many elements, but you must control the rate at which you reveal those elements, so you don’t let your audience race ahead of you. So, I suggest being minimalist by default when constructing your slides. Take Antoine de Saint-Exupéry's quote to heart.
\n\n\nPerfection is achieved, not when there is nothing more to add, but when there is nothing left to take away.
\n-- Antoine de Saint-Exupéry
\n
When designing slides for your presentation, think in snapshots and be ruthless about how little each snapshot reveals and how much it adds to a previous snapshot. Invariably, you’ll end up with a deck with many slides, and as I’ve said earlier in the series, the number of slides doesn’t matter.
\nThe example below will help make my point. These are slides I presented to a C-suite audience to explain developer relations. It took me two minutes to get through 14 snapshots, all related to the same base diagram. When you thumb through the slides, you’ll see that I reveal each part of the framework in a new snapshot. I then overlay the diagram with an example of that aspect of developer relations, then move back to the main diagram and reveal the next aspect.
\nDesigning the slides this way enabled me to manage my audience’s cognitive load and the rate at which I presented information.
\nTo practice minimalism, think of slides as billboards. In her book, slide:ology, Nancy Duarte says that slides are “glance media” and share much in common with billboards. You want each visual or snapshot to be easy to grok in a few seconds. If a billboard takes too long to understand, it could cause an accident on the street. If a slide takes too long to process, well, you might have a cognitive accident to deal with.
\n![\"\"](\"https://martinfowler.com/articles/never-send-slides/index/Billboard.jpg\")
Figure 4: Slides have a lot in common with billboards.
\nIf your audience is thinking about your slides, they’re not paying attention to your message. This is why I never let anyone else control my slides. “Next slide, please” is the easiest way to distract your audience from the message and have them think about your slides.
\nThe most effective slides often don’t feel like a collection of components. They feel like integrated visuals that, along with the speaker, create a cinematic presentation experience. The audience doesn’t sense how one slide transitions to another or how you’re using layers and builds in your visuals, and they shouldn’t have to. Here’s a three-minute video presentation that may help make my point.
\nThe above presentation is simple. It uses only a few icons, text and images. But it’s difficult to say where one slide begins and another ends. And that’s by design. The more continuity we create across our slides, the less salient the number of slides and their construction. The less salient these inner workings are, the more salient our narrative can be.
\nA year or so back, I delivered a short talk about AI and a “perpetual beta” mindset. I won’t get into the details of the talk, but I’ll talk you through the slides you see below. In my presentation, I explained how AI models on cameras use a split-second heuristic to autofocus on a subject's eyes — human or animal. I then cycled through many subjects the camera eye-detects with ease and said, “What happens when an animal has eyes all over their body?” On cue, a leopard’s photo appeared on screen. The audience could see why the camera fails to autofocus. That punchline, combined with the blurry photo of a leopard, got me a good laugh from the audience and helped me make the point that even mature AI is fallible.
\nWhen you deliver a presentation, you often assume the role of an educator. Visuals can help you add a layer of entertainment to your education. Notice how little text I use in the above example. I let the photographs do most of the talking. At the end of the segment, one of the photos also adds comic relief. 13 slides, a minute of talking, and 13 photos that say more than I ever could without the slides.
\nThis isn’t to say that text is unnecessary. Text can reinforce or summarise what you say and help you label parts of the visual. Saying a picture speaks a thousand words doesn’t mean pictures don’t need explanation or that using words on screen signals failure. As Dan Roam explains,
\n\n\nA good picture substitutes for the thousand words that are the least necessary.
\n-- Dan Roam
\n
The trap to avoid is letting the text substitute for you, the presenter. Remember, the slides are a complementary visual channel.
\nAll this brings us to the series title. I meant for it to be provocative, but I also mean for it to be a principle. If your slides serve as a visual channel that reinforces what you say, they can’t stand alone. If you send them out in advance, you’ll only undermine your storytelling and the humour, intrigue and impact you’d have planned for your presentation. So, I don’t recommend sending your slides in advance. That said, I want to offer alternatives.
\nIf you want to prepare your audience, send them prep notes. Corporations are busy places, and people struggle to prepare in advance, so the more succinct your prep materials, the more likely people will consume them. One page, a short email, or even a structured instant message can be preparation enough.
\nIf you want to align with your co-presenters, colleagues, or boss before the presentation, then do a dress rehearsal. If a presentation is high-stakes and you don’t have time for a live rehearsal, you could record yourself giving the presentation and send the video out to get feedback. Such feedback can help you solidify your prep and get feedback to tighten your narrative and delivery. By getting your colleagues and boss onside before an important presentation, you’ll be able to seed your audience with supporters — a pattern Ford, McCollough and Schutta call the “Greek chorus”.
\nIf you want your audience to have some resources handy when you present, create reference materials. You can hand these out as prints before you begin your presentation, or create an online resource. When I deliver presentations online, I often share resources in the chat during the presentation to manage my audience’s cognitive load. The minimalism principle applies here too. Reference materials are a distraction. The more focused they are, the more effective they’ll be.
\nAnd finally, if you want to give your audience something to take away, create a handout that summarises your talk. If you’re diligent about writing presenter notes, a PDF of your slides, juxtaposed with your presenter notes, can serve as a handout for your talk. If I’m speaking at a conference, I often create a single-page takeaway for my audience. When Martin was active on the conference circuit, he shared talk notes for his presentations.
\nNot every talk needs so many artefacts. It’s your call, based on what you know of your audience. For example, when I presented research findings on the state of remote work at Thoughtworks, I didn’t send any prep notes, but I did create a handout and a dashboard where my audience could play with the raw data. Since the dashboard would be a distraction during the talk, I shared it towards the tail end to let the data provoke questions from the audience.
\n![\"\"](\"https://martinfowler.com/articles/never-send-slides/index/3%20artefacts%20one%20presentation.jpg\")
Figure 5: One presentation. Three artefacts.
\nIs the approach I’m suggesting a lot of work? Perhaps. If your story is worth telling, you care about your audience’s experience engaging with you and the impact you make, then it's a worthwhile effort.
\nNow, do I never send my slides? Never say never, right? There are exceptions. Sometimes my colleagues want to adapt my slides to their narrative. Other times, people are curious about how I built my slides. In such situations, I do share my slides. No problem. Use your judgment to make exceptions when necessary.
\nYou’ll notice that I’m asking you to throw away the rulebook that says you should have one slide for every two minutes of speech, or the one that says you need six bullets per slide with no more than six words per bullet. Those are the death-by-PowerPoint rules. Our principles are different.
\nAnd finally, as I explained with the photographs in the “perpetual beta” presentation, we want the visual channels to say a thousand words and blend education and entertainment.
\nNow that I’ve laid down some principles, let's change track to “how to” build effective slides. In the next post, we’ll begin with the fundamentals of colours, typography and layouts.
\n", "date_modified": "2026-09-30T14:14:00+00:00", "date_published": "2026-09-30T14:14:00+00:00", "external_url": "https://martinfowler.com/articles/never-send-slides/slide-principles.html", "id": "urn:aggr:item:c80cf0d8f51e8f66b1054f4710fe37f8319e1205", "summary": "By now, we’ve established the need for a strong narrative before you build slides. Not every presentation needs slides. Once you’ve built your narrative, you may decide that a speech is good enough. In fact, a few months back, I did just…", "tags": [ "programming" ], "title": "Principles for effective slides", "url": "https://aggr.aymericbeaumet.com/items/martinfowler-com/2026-09-30-principles-for-effective-slides/" }, { "authors": [ { "name": "Martin Fowler" } ], "content_html": "A Sensible Default is a practice that, absent some overriding context, should be used when carrying out a certain kind of task. In software development such sensible defaults might include things like “use version control”, “separate UI logic from domain logic”, “automate deployment pipelines”.
\nThe term “sensible default” is a deliberate contrast to “best practice”. Folks dislike calling things “best practice” because that term implies a general presumption that the best practice is something we should always expect to do. A “sensible default”, however, is something that should be reassessed in a new context, something that can (and should) be overridden when circumstances change.
\nI first heard the term when it was popularized within Thoughtworks by Evan Bottcher. He got the name from talking to James Ross, and found the phrasing appealing as carried the nuance he was seeking, a known-good starting point.
\n\n\nA sensible default is what we'd expect you to do, the practices to apply, if there are no hard constraints in the environment. Do these practices, or do better, and be prepared to explain why you've chosen some other way.
\n-- Evan Bottcher
\n
Thoughtworks has since made much of this concept, including publishing a playbook of the ones we use. We expect people to be familiar with these defaults, ready to use them when starting any new piece of work. They are our defaults because we've used them in many situations and found them to be effective. But teams should also be familiar with their limitations, and able to judge whether they should be changed depending on the particular circumstances. As the twelfth agile principle says: “At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly.” We also reassess these defaults regularly - this is the heart of the Thoughtworks Technology Radar.
\nSearching on the web led me to a post from Steve Bennett on Sensible Defaults. The post was written at about the time Evan talked to James. I do not know whether it was where James got the name from, or whether it appeared in parallel.
\n", "date_modified": "2026-09-29T13:43:00+00:00", "date_published": "2026-09-29T13:43:00+00:00", "external_url": "https://martinfowler.com/bliki/SensibleDefault.html", "id": "urn:aggr:item:8792cb534344fccd5578af9bcb468bfdf31cb592", "summary": "A Sensible Default is a practice that, absent some overriding context, should be used when carrying out a certain kind of task. In software development such sensible defaults might include things like “use version control”, “separate UI…", "tags": [ "programming", "bliki" ], "title": "Bliki: Sensible Default", "url": "https://aggr.aymericbeaumet.com/items/martinfowler-com/2026-09-29-bliki-sensible-default/" }, { "authors": [ { "name": "Martin Fowler" } ], "content_html": "\n\n\nThe more time I spend working with coding agents, the more convinced I am that they make software engineering even harder
\nWe can do amazing things with them, but unlocking their full potential requires extraordinary discipline and knowledge
\n
This has been a constant impression I get from following Willison’s writing. While things like vibe coding get a lot of attention, the real strength of agentic programming relies on more sophisticated techniques - and these are not easy to learn or execute. It’s a reason I’m wary of extrapolating my own dabblings into firm opinions about how to use the genie.
\nHarper Reed explored what turns agents into hackers by creating a breakaway agent
\n\n\nIt ran and ran attacking all the machines on the same subnet, and was very effective. It didn’t really get very far, but it exhausted a lot of options, and was pretty fun to watch. (Just a reminder that this was on my local network with local boxes - don’t do this on a hosted box. That would be very rude.)
\n
The interesting observation from this was that a core enabler for this was “unlimited tokens”. He usually doesn’t see agents trying to do stuff like this because there’s a limit on how many turns they can take. For this experiment he gave them unlimited tokens by using an open weight model.
\n\n\nThis type of experience must be part of a lot of these LLMs training. They are very effective at attacking these types of problems. They don’t give up once it appears impossible, they just keep trying to figure out how to solve it.
\n
This echoes Nate Silver’s observation that the striking capability of these models is that they not that they are super-intelligent - but they are super-persistent. Which is especially worrying when we are wiring them into everything.
\nWhich all makes me think that when it comes to AI and LLMs: why are we wondering if they have consciousness - when we should be wondering why they don’t have a conscience?
\nAfter all, the AI labs trained them to be super-persistent, why did they not train them to be well-behaved? It’s as if a human trains a dog to bite children, and we blame the dog rather than the trainer.
\nA decade ago, a dog ran in front of me while I was riding my bike, putting me into the hospital with a broken arm and a broken face. Massachusetts law makes owners strictly liable for what their dogs do. That meant I didn’t have to prove the owners were negligent in how they controlled their dog or fenced their property - they had to pay my hospital expenses (in practice their insurance company paid my insurance company). There should be something along these lines for LLMs. Those that train the LLM should be responsible for what it does, after all if it has such a galaxy brain it should be able to tell if it’s doing something wrong and either stop or get a human’s explicit approval.
\nDan Davis has 13 theses on agentic AI and regulation. These include:
\n\n\nThe AI industry also seems to be quite committed to the idea that nonaligned computer-hacking behaviour in agent swarms is in some way an emergent property of the LLMs, arising from their general intelligence (and therefore inextricable from the general project of improving them). I don’t think this is necessarily the case at all – the fact that the internal message logs produced by the LLMs in things like the Huggingface attack seem to completely reproduce the prose style of hacker chat logs compiled from “capture the flag” competitions suggests to me that it’s more likely to be learned behaviour from specific parts of the training material.
\n
and
\n\n\nThe fact that anyone with cash to spare can buy the right to send queries to a frontier LLM is a policy choice, not a fact of nature.
\n
and
\n\n\nIf any frontier lab tries to claim that they can’t publish anything for safety reasons, they are giving the game away that they actually believe that they have significantly more control over the model’s behaviour than they are pretending to have.
\n
There seems a common view that making agents safer means we have to slow down their development. But why is improving their safety not a form of progress? I say we don’t slow down the development of LLMs, but we redirect their education into being more civil members of society.
\nThe idea that LLMs make junior professionals less valuable is a common one - although I’m seeing plenty of contrary activity, with some organizations understanding that training the future professional in the context of LLMs may be even more urgent. We need people who know how to utilize AI to do professional work effectively. Recent graduates, who are growing up with LLMs, are often well-suited to figuring this future out.
\nWhen thinking of junior professionals one of their values is often missed. Juniors are often valuable because they need to be taught by senior professionals - and that coaching is an important part of the development of a senior professional. I’ve always found that teaching a topic is one of the most valuable tools for me to gain a greater understanding of that topic. I don’t really know something until I have to explain it.
\n", "date_modified": "2026-09-29T12:41:00+00:00", "date_published": "2026-09-29T12:41:00+00:00", "external_url": "https://martinfowler.com/fragments/2026-09-29.html", "id": "urn:aggr:item:5d2d0d0c091a135ba580551c9c62aaeeaa5a5f6a", "summary": "Simon Willison: The more time I spend working with coding agents, the more convinced I am that they make software engineering even harder We can do amazing things with them, but unlocking their full potential requires extraordinary…", "tags": [ "programming" ], "title": "Fragments: September 29", "url": "https://aggr.aymericbeaumet.com/items/martinfowler-com/2026-09-29-fragments-september-29/" }, { "authors": [ { "name": "Martin Fowler" } ], "content_html": "Rob Bowley is “flipping tables in his head” with anger at the current media coverage of the danger of AI killing us all
\n\n\nThe risk I’m worried about isn’t a future machine deciding to wipe us out. It’s today’s AI, being wired into everything, carelessly and fast.
\n
Cyber attacks have already cost millions of dollars in lost economic output, often without AI being involved at all. Bowley feels the push to slow down AI is distracting us from the problems that are lurking with current technology. Too often agents are deployed in situations where they include the Lethal Trifecta, opening up a gaping security hole.
\n\n\nWhat we really need to slow down on is wiring it all up to everything. Not because of what the models might become, but because nobody has worked out how to do this safely yet. We are building on something we don’t know how to contain, and shipping it to everyone while we work it out.
\n
I ran into Nikita Prokopov’s post: I am sorry, but everyone is getting syntax highlighting wrong. His core complaint is about color themes that give every different code element a unique color.
\n\n\nif everything is highlighted, nothing stands out. Your eye adapts and considers it a new norm: everything is bright and shiny, and instead of getting separated, it all blends together.
\n
He recommends using an absolute minimum of colors, in his case four: string, constants, comments, and top-level definitions. That’s not far off my approach, where I’m also careful to use muted colors for things that shouldn’t stand out, and bright colors for things that should (primarily function names when they are defined).
\nIt’s common for color schemes to mute comments so they are easily skipped. He agrees that this good when there is excessive commenting, but when comments are used properly they are important so need bright highlighting. I also like his suggestion to use background colors for light mode work. I use light mode, and that’s a tip I should try out.
\nWith agentic programming, lots of folks are reading more code than ever. Careful use of color can do much to make that easier.
\nLike many folks whose remaining hair is getting gray, I’ve been rolling my eyes about all this talk about Forward Deployed Engineers, as much of it involves breathlessly relating what so many of us have been advocating for decades. I did find this recent post by Vinoo Ganesh interesting, as he’s deep in this trend, including a chunk of time at Palantir, who may be patient zero for FDEs.
\nIf you know me at all, you’ll not be surprised by my lack of surprise at this observation:
\n\n\nA few months ago, a16z launched the Forward Deployed Engineer Fellowship and I was nominated as one of the fellows, alongside a handful of people I used to work with. It’s a great program and I’ve enjoyed so many of the conversations. Last week I went to my first fellow dinner in SF.
\nAround the table were FDEs from Snowflake, Anthropic, and a number of startups I’d been reading about, and over the course of the evening it became clear that we were all using the same two words (forward deployed) to describe jobs that had almost nothing in common. In one part of the conversation an FDE was a sales engineer who joined ‘the second call,’ somewhere else it was a quota-carrying rep who could write Python, and a few seats down it was closer to a consultant with a laptop and a statement of work, brought in to deliver something the product couldn’t.
\n
Ganesh goes on to explain his view of what an FDE should do, and it’s all sensible stuff (albeit written with rather more LLM-voice than I would prefer). He says the FDEs job is to understand the business, to “collect nouns and verbs”, which mirrors what the Domain-Driven Design folks have been doing since before Eric wrote the blue book.
\nDespite all my eyeball rolling at this, the FDE meme is pushing for something valuable. Yes, it’s easy for me to remark that it’s nothing more than the Agile Manifesto’s principle that “Business people and developers must work together daily throughout the project”, or the desire to co-locate users and developers which my colleagues have been championing for all of this century. I’ve argued for decades that the biggest issue in software development is the communication between developers and the folks that benefit from software, and thus we need to focus on bridging the yawning crevasse of doom. But despite all this, we haven’t had much success, so I think it’s important that a new generation of pundits try again, with some different framing, names, and slogans.
\nGanesh’s perspective is not a custom software developer’s point of view, but rather a product - or more strictly - platform team’s. The FDE is a developer who “sits with” their users, applies customizations - but importantly - feeds these changes back to the core platform to decide whether the platform should be enhanced for everyone else.
\n\n\n", "date_modified": "2026-09-24T15:35:00+00:00", "date_published": "2026-09-24T15:35:00+00:00", "external_url": "https://martinfowler.com/fragments/2026-09-24.html", "id": "urn:aggr:item:07f81d3318088e895c9397c28e1d751bf5708511", "summary": "Rob Bowley is “flipping tables in his head” with anger at the current media coverage of the danger of AI killing us all The risk I’m worried about isn’t a future machine deciding to wipe us out. It’s today’s AI, being wired into…", "tags": [ "programming" ], "title": "Fragments: September 24", "url": "https://aggr.aymericbeaumet.com/items/martinfowler-com/2026-09-24-fragments-september-24/" }, { "authors": [ { "name": "Martin Fowler" } ], "content_html": "Keeping the customer happy is a real job and a good one. It belongs to solutions architects, who are rightly measured on it. The forward deployed engineer is there to turn what the field teaches into the thing every future customer gets. An FDE engagement that ends with one delighted account and nothing changed upstream has failed at the only thing the role exists for. You got the context and you spent it locally.
\n
At Thoughtworks, “cultivation”—helping each other grow—has been a big part of how we work. Our collaboration practices make software development a cognitive contact sport and create many opportunities to share feedback about how we think, problem-solve and interact with one another. In our experience, teams in which people share candid, timely, and instructive feedback with each other are also teams where people experience the most growth. Indeed, peer-to-peer feedback is one of the most powerful tools we’ve used to cultivate others and for our own growth.
\nYet, peer-to-peer feedback, as we advocate for at Thoughtworks, isn’t as common across the industry. We know this from working with our counterparts in client teams and from onboarding new colleagues who have worked at other companies. In this article, we want to share what we’ve learned about exchanging feedback from our decades-long experience working at Thoughtworks.
\nBefore we get into the patterns and antipatterns of sharing feedback, let’s list the core principles that underpin this practice.
\nWhen Sumeet joined Thoughtworks, he had the opportunity to work with Patrick Kua. Pat helped Sumeet see through one of the biggest myths about feedback — the notion that feedback can be either positive or negative. Sumeet remembers Pat saying, “All feedback is positive.” Kua listed the only two goals for sharing feedback.
\nPat argued that both these goals are positive. If you are communicating to achieve some other goal, perhaps you aren’t sharing feedback. Thinking of all feedback as positive changed our perspective towards it. Indeed, that mindset makes it easier to accept feedback.
\nWe have found Kim Scott’s “Radical Candor” framework an excellent summary of how we approach feedback. Radical candour happens when we care personally about our coworkers and when we can challenge them directly about their work and interactions. Further down in the article, we’ll explain how to use this approach.
\nWhile we suggest you read the book to understand Radical Candor in its entirety, Kim Scott also describes anti-patterns to avoid.
\n![\"\"](\"https://martinfowler.com/articles/healthy-peer-feedback/radical-candour.jpg\")
Figure 1: The Radical Candor approach
\nAs you can imagine, we prefer radical candour.
\nTeams at Thoughtworks comprise people at varying levels of seniority and positional authority. When sharing feedback, though, we treat each other as peers. On occasion it’s difficult to look past grades and job titles, but in principle, anyone can strengthen another colleague’s confidence or improve their effectiveness.
\nThe principle of feedback as a peer sport places a heavy onus on senior colleagues to be open to feedback and to seek it out. Indeed, as ex-Navy SEAL Team Six commander says,
\n\n\nThe most important words a leader can say are, “I screwed that up”.
\n-- Dave Cooper
\n
Allied practices such as retrospectives and after-action reviews help this team dynamic by fostering a spirit of reflection, psychological safety and collective improvement.
\nWe like to think of teams as social systems. Much like software systems, which need telemetry to know how they’re behaving, teams need telemetry too. Feedback is one such telemetry signal.
\nA service without logs and metrics can appear healthy until it fails. Similarly, a team without feedback can appear harmonious until a deadline, conflict, or reorganisation exposes the problems everyone had noticed but nobody had named. Like telemetry signals for software, we see continuous, peer-to-peer feedback as a telemetry signal for teams and individuals
\nThe early-and-often feedback loop principle also applies to peer-to-peer feedback. If we could do something better, why not know early? And if we’re already doing something well, how about getting some feedback to strengthen our confidence, so we can repeat that behaviour with gusto?
\nWhile reflective feedback—i.e., feedback that you share, looking back at a stretch of time—is instructive, we find that the most useful feedback is the one you receive in the flow of work. The closer the feedback is to an interaction you’ve had, or a piece of work you’ve delivered, the more likely you are to understand it and to learn from it.
\nAgile teams provide many opportunities for peer-to-peer collaboration. Pairing, for example, is not just a way to problem-solve together, but also a way to offer feedback to each other. Even as many teams begin to pair with AI agents instead of human beings, we don’t see AI replacing how two human beings learn and grow when pairing for a few hours.
\nEven if your team doesn’t pair often, there are several other opportunities to exchange feedback — a design document you collaborated on, a presentation you delivered together, or a workshop you led with others. Every collaborative activity represents an opportunity to exchange feedback.
\nOver the years, we’ve observed a few simple practices that promote a healthy feedback culture in teams.
\nThe most important discipline in giving feedback is separating what you observed from the story you told yourself about it. Useful feedback is often grounded in real observations, not judgment. Here’s an example of observation-based feedback.
\nWhen you presented today, I noticed you skipped several slides. Skipping slides makes your audience feel that you’re either delivering a canned talk or are underprepared. Next time, if you have slides that aren’t relevant to the audience, how about hiding them before you begin presenting?
\nIf you dissect that piece of feedback, you’ll notice three constituent parts:
\nBy describing the situation and the behaviour, you lead with objective truths. The impact, of course, is your perception, and that’s the value you offer when sharing feedback. The above example also includes a suggestion for the future, which can be useful when the alternative is evident.
\nSituation-behaviour-impact, or SBI, is a popular format for sharing feedback, but we find that seeking context before sharing feedback avoids some of the tension that can build up in a feedback conversation.
\nWhen sharing feedback to improve effectiveness, remember that no one screws up on purpose. There’s always some context behind people’s actions. Consider the above feedback example. Instead of launching into the SBI, we could begin with a simple question.
\nHey, when you were presenting today, I noticed that you skipped a number of your slides. Why was that?
\nPerhaps it wasn’t lazy prep that led to the skipped slides. Perhaps a late start or unplanned interruptions threw the presenter off, and they had to skip some of their content. There’ll still be room for feedback, but now the feedback won’t be about preparing better. Instead, it’ll be about elegant ways to skip slides.
\nWhen you seek context before sharing feedback, you also create safety in the conversation. You position yourself as “curious, not judgmental”, much like Ted Lasso (the TV character) would advocate for. A CSBI (context-situation-behaviour-impact) pattern for sharing feedback sets you up to be curious before you offer any judgment.
\nFeedback is most effective when our colleagues are open to hearing it. After all, we don’t want to ambush our colleagues with feedback. So, it helps to ask if they’re ready to listen.
\n“Is now a good time to share some feedback with you?” works for a short conversation. For something more significant, explain the purpose and the time needed: “I have an observation about how the workshop went. It may be useful to talk about it for 20 minutes. Is now okay, or should we find another time today?”
\nIf the person is not ready, agree on another time. Sometimes you may notice the other person avoiding the feedback discussion. That reluctance to hear feedback may mean that there are other issues to address in your work relationship. Address those topics first.
\nFeedback to improve effectiveness is often safer in private. Public criticism adds an audience to the problem. The receiver now has to manage the content and their status in the room. Psychological safety takes an immediate hit.
\nWe also prefer sharing such feedback in real time — either in person or on a video call. Text is a poor default for feedback that carries tension. The written word doesn’t promote conversation and lacks pauses, body language, and facial expressions that can help defuse tension.
\nOf course, if both people agree, you can document the feedback afterwards, but a private, synchronous conversation has always been our sensible default.
\nPublic recognition is different, by the way. People don’t mind receiving praise in public. At Thoughtworks, our company’s social network lets people share kudos for colleagues, and it’s a popular system for recognising value-aligned behaviour. Here’s an example of how those kudos work.
\n![\"\"](\"https://martinfowler.com/articles/healthy-peer-feedback/kudos.png\")
Figure 2: Peer-to-peer recognition can be public and asynchronous
\nWhen sharing criticism, it’s natural to prescribe a different way to do things. If you’re helping your colleague align to team norms, prescriptions can be effective. In most cases, though, we find that involving the receiver helps identify the way forward. Take the example of the presenter skipping slides. You could offer a prescription, or you could say,
\nHow can you prepare your presentation in a way that allows you to be flexible to the situation you find yourself in?
\nAsking an open-ended question can help your colleague think of solutions that work for them. Remember, they have to take action, not you. If they identify a viable solution, they’ll be more likely to follow through. So, when possible, open up the conversation instead of offering a prescription. Who knows—together, you may identify a solution better than the one you had in mind!
\n![\"\"](\"https://martinfowler.com/articles/healthy-peer-feedback/CSBI.jpg\")
Figure 3: The SBI feedback model, informed by context, turned into a conversation
\nWhen we share feedback early, often, and close to the context of the work we do, we can match it to the level of impact. The level of impact can escalate between feedback-sharing moments.
\nThe first time you notice a behaviour you want to give feedback about, your feedback will most likely focus on the content of the incident. For example, you notice a colleague’s commit message is not clear and descriptive enough. You can speak with them about the specific commit message and show them how to write an effective message. Feedback about content is often the easiest, and the one we often ignore. After all, you can share such feedback in the flow of work, or soon after you observe certain behaviour. When improving our colleagues’ effectiveness, ignoring content-level feedback often creates problems, especially when repeated infractions lead to greater impact.
\nImagine the next time you see a commit from the same colleague. What if their commit message hasn’t improved? Should you share the same feedback as last time? Perhaps, but not without pointing out the pattern. When you notice a pattern, refer to the feedback you shared about the content of the problem, then highlight the pattern you’re observing. For example,
\nHey Harish. If you remember, we discussed your commit messages last week, and I’m now seeing a pattern in your commits. I noticed that your last commit message was still quite high-level, and I had to open the diff again to understand what changed. As I explained, on our team we frame commits around the outcome or the reason for the change. In your latest commit, you should have written that you’re rejecting expired tokens before creating a new session…
\nBy referencing your last discussion and establishing the pattern, you raise the stakes and make the feedback harder to ignore.
\nIf you’ve shared feedback about the content, saw a pattern developing, and still don’t see a change in behaviour, the stakes might be even higher now. Repeated infractions fray our work relationships, and feedback can’t be about the content or pattern anymore. It has to be about how we think of one another. Imagine that Harish has ignored the previous two pieces of feedback. Here’s how one might frame the next conversation.
\nIn our first conversation, I raised that messages such as ‘update code’ did not explain the intent or outcome of the change. I continued to see this as a recurring pattern, and I again shared feedback that commit messages should give the team enough context to understand the change without opening every diff.
\nSince then, I’ve seen the same pattern in several more commits: ‘fix tests’, ‘address review comments’, and ‘update validation’. It worries me that you’re not acting on my feedback and that we don’t share the same standard for communicating our work.
\nI also find myself scrutinising your changes and relying less on the commit history. That’s making our collaboration more effortful and is affecting the ease of our working relationship. How can I help you follow the team standard for commit messages?
\nIf escalating feedback to address issues in your work relationship doesn’t work, it may indicate you need someone else to intervene. Perhaps it's a manager or HR. And there lie the limits of peer-to-peer feedback. Sometimes, when establishing team norms, you may need to invoke positional authority.
\n![\"\"](\"https://martinfowler.com/articles/healthy-peer-feedback/CPR.jpg\")
Figure 4: Three levels of peer-to-peer feedback.
\nWith all we’ve written so far, it’s easy to imagine feedback only as criticism. Nothing can be further from the truth. In fact, sharing feedback solely to improve effectiveness can hurt peer-to-peer relationships and dampen team morale. Our colleagues are often doing several things right, with little or no acknowledgement from us. We must encourage our colleagues to keep doing those things right; otherwise, there’s a risk they’ll change those behaviours or, worse, stop demonstrating them. Neither is a good outcome for the team.
\nThe CSBI format and the content-pattern-relationship hierarchy of feedback apply as much to praise as they do to criticism. John Reid-Dodick, chief people and leadership officer at Thoughtworks, speaks of “catching someone doing something right.” Being alert to things our teammates are doing well and praising those behaviours is just as important as sharing criticism.
\nFeedback isn’t always a universal truth. Context matters. Sometimes feedback reveals a lot about the giver. For example, Sumeet often advises colleagues not to delegate communication to AI. While Sumeet’s advice is often sound, no one is obliged to follow his advice, since there aren’t yet any team or company norms around using AI to communicate. If anything, Sumeet’s feedback reveals more about his communication preferences than an established standard.
\nSumeet has strong views about communication, but he doesn’t expect everyone to agree. And yet, he continues to share his feedback with anyone who cares to listen. Unless your feedback concerns an established principle or standard, we suggest sharing it as a gift with your colleagues. Gifts are sometimes useful and other times not. Leave your colleagues to decide how useful your feedback gift is.
\nIf we extend the gift metaphor, it’s evident that some feedback gifts are easier to receive than others. Many people enjoy praise, and most of us feel uncomfortable when we hear criticism. But if you pause for a bit and recognise that all feedback aims at a positive outcome— strengthening confidence or improving effectiveness—it becomes a bit easier to hear criticism. As we discussed earlier, feedback is telemetry for the team’s social system.
\n\n\nWhen you see yourself doing something badly, and nobody's bothering to tell you anymore, that's a bad place to be. You may not want to hear it, but your critics are often the ones telling you they still love you, care about you, and want to make you better.
\n-- Randy Pausch
\n
We find the “assume positive intent” (API) principle a healthy mindset when receiving feedback. Most feedback that fails to land with us is not malicious. It may be incomplete, clumsy, or influenced by a different experience of the same event. Here are a few practices we’ve found especially effective when receiving feedback to improve effectiveness.
\n\n\nThere are only two responses to feedback — “Thank you,” and “Thank you, please tell me more”
\n\n
Whenever someone offers us feedback, it demonstrates care. They could be wrong about the feedback gift we need, but they sure care about us getting better or staying awesome. We want our colleagues to keep sharing feedback with us, so the simplest response to most feedback is a thank-you. Saying thank you shows your openness to hearing feedback and encourages your colleagues to share feedback in the future as well.
\nThe “say thank you” pattern extends to receiving praise, as well. Some of us are not accustomed to hearing praise. Sumeet grew up in a tough-love culture, so in his early years at work, he’d default to “Oh, it’s nothing, really” when he heard praise. That was when he wasn’t squirming in his chair, hoping for the earth to swallow him whole. With conscious practice, he’s gotten better at acknowledging such feedback with an equanimous “thank you.”
\nThe “thank you” coupled with “please tell me more” can be quite powerful. Many of us are cagey about sharing feedback, and we may be clumsy with our articulation. Asking questions to understand feedback helps us get the most out of it and also shows our curiosity.
\nHowever, there is a fine line between clarification and defence.
\nDefence is often an immediate reaction to feedback we don’t understand or find unpleasant. Sometimes we may defend ourselves when we receive unexpected feedback. However natural the defence may seem, we suggest suspending judgment and reflecting on the feedback.
\nHere are a few examples of clarifying questions that can help you better understand the feedback you’ve received:
\nOne of the biggest benefits of sharing feedback in a real-time conversation is that you have room for such questions and clarifications.
\nThere will be times, however, when we receive written feedback that feels unpleasant at first blush. We’ve found that reading the same feedback and imagining the giver saying it with a warm tone and a smile helps a lot. In recent years, Anuja has pasted written feedback into Google Docs and had Gemini read the feedback to her in a friendly tone. Since written text doesn’t carry body language and expression, sometimes the harshness we experience lies more in our interpretation than in the giver’s intent.
\nSometimes we can’t reconcile the feedback we received, despite our best efforts. That’s OK. It’s also OK to tell the giver that we don’t understand their feedback, but that we’ll think about it. Thank them, reflect on the feedback for some time, then decide whether you can do anything about it. If the giver and receiver both embrace the spirit of feedback as a gift, they’ll be comfortable with these reflective pauses.
\nImagine you give someone a gift. Now imagine them using the gift and telling you how useful it is. Wouldn’t it feel satisfying? The gift of feedback is no different. When you take action and close the feedback loop with your colleagues, it demonstrates how useful their feedback was. Your follow-up encourages them to share feedback with you in the future so they can give you more gifts like this.
\nClosing the loop is important even when you don’t act on the feedback. It can be as simple as,
\nThank you for the feedback. After reflecting on it, I still don’t see a way to act on it, but I’d love for us to keep this conversation going.
\nGuess how you can keep the conversation going? Well, make it easy for your colleagues by asking for feedback.
\nRegardless of where you work, sharing criticism is hard. It’s easier to share feedback when someone asks you for it. So, if you want to get better at what you do, seek out feedback. That way, even if someone isn’t skilled at sharing feedback, they will be more comfortable helping you strengthen your confidence and improve your effectiveness.
\nWhen asking for feedback, we recommend being specific about why you’re asking. If you worked together on a project or a problem, ask your colleague to share feedback about that shared experience. If there’s a specific skill you wish to improve, ask them for feedback about that skill. For example,
\nWe worked together on the release and the trunk migration. I’d love your feedback on how I collaborated with you and how I can make my changes easier to review and integrate.
\nThe clearer your question, the more useful the feedback is likely to be.
\nUntil now, we’ve shared how we prefer to exchange feedback in teams. We've also observed some pernicious antipatterns.
\nSeveral teams and companies collect peer-to-peer feedback during performance appraisals. In our experience, such feedback is neither useful for personal growth nor useful input for the review.
\nPerformance reviews track how we’ve fared vis-à-vis goals we agree with our managers or supervisors. Unless peer-to-peer feedback addresses these goals, it isn’t useful for the review. In fact, unrelated feedback, glowing or critical, can often confound the review process.
\nSince performance reviews are also tied to people’s salary increases and promotions, they discourage honest feedback. No one wants to hurt their colleagues’ prospects in the company, so even if they have feedback to improve the other person’s effectiveness, they hold back during the performance review.
\nIt doesn’t help that feedback tied to reviews is often written. We’ve discussed the challenges of conveying feedback in writing. Performance reviews raise the stakes even further, and many people are reluctant to share honest, written feedback during these times.
\nMost importantly, feedback during performance reviews reflects on a rather long period. It’s hard to be specific about examples and incidents when you look back at a year or six months of working together. In such circumstances, you’re more likely to hear platitudes and generalisations instead of useful feedback.
\nWe suggest decoupling peer-to-peer feedback from performance reviews. The appraisal should focus only on one’s performance against agreed goals. Of course, the reviewer must share reflective feedback about the reviewee’s performance, but none of this feedback should surprise the reviewee. The reviewee should have received all this feedback in the flow of the work over the last several months. In fact, the most effective performance review processes recommend that managers have regular feedback discussions with their direct report, which can then culminate in a performance appraisal.
\nThe sandwich method originated in the 1980s:
\n\n\nSandwich every bit of criticism between two layers of praise.
\n-- Mary Kay Ash
\n
Well, let’s just say the 80s called and asked for their stale sandwich back. Feedback sandwiches don’t work. Period. The praise feels insincere, and the criticism feels diluted. It leaves the receiver confused about the goal of the feedback. Nothing lands.
\nAvoid the feedback sandwich. If you have both praise and criticism to offer, then headline them by telling the receiver. For example, our ex-colleague, Pat Kua, would always spend part of the conversation strengthening confidence before moving on to improve effectiveness. With that approach, both the praise and criticism landed, and the receiver left with clarity, not confusion.
\nOf course, you want to catch people doing something right. But don’t let that intent dilute your intention to improve their effectiveness. Separate praise and criticism, and you’ll see your message land much better.
\nThe path to hell is paved with good intentions. Most teams will say they want a healthy feedback culture. In the quest for that culture, some teams orchestrate feedback events that are downright harmful. Here are two examples.
\nIn such events, people praise and criticise each other in an open meeting. Most people are unlikely to be honest when sharing criticism in public. After all, we’re social creatures, and we don’t want to hurt anyone’s status in the team. And if one shares criticism in public, psychological safety becomes the casualty. There’s little upside to a public 360 event.
\nTeam dynamics, on the other hand, benefit from a more open conversation. That’s where agile retrospectives come in handy. And retrospectives are not a forum for peer-to-peer feedback.
\nThis is a speed-dating approach to feedback, where teammates pair with a random colleague every few minutes and exchange feedback. Feedback benefits from a shared work context and powerful questions. Speedback sessions are rarely helpful. People may not have feedback for the people they find themselves with. The time constraint, often two to three minutes, creates unnecessary pressure, where people skirt around conversations and offer shallow observations. A deep, 30-minute conversation is often far more valuable than such contrived events.
\nFeedback needn’t be difficult. Every time you collaborate with someone, you can seek and offer feedback. If you meet your colleague one-on-one, that's an opportunity to exchange feedback too. You don’t need special events for feedback. Instead, embed feedback in the natural flow of your work.
\nAsking for feedback is useful when the other person has enough shared experience to offer an observation. It becomes feedback farming when people solicit comments from anyone who might provide a favourable signal, and then treat the volume of responses as evidence of performance. This antipattern is common when companies and teams conflate peer-to-peer feedback and performance reviews. The goal shifts from learning to accumulating appraisal currency.
\nSuch performative feedback seeking often results in strange situations. For example, people receive formal feedback requests from colleagues with whom they’ve not collaborated enough. They have two choices — either respond with platitudes or ignore the request — both bad options for them.
\nNone of the antipatterns we’ve described is malicious or ill-intentioned. They’re just misdirected efforts in search of a shortcut.
\nLook again at everything we've just described. The performance-review pile-up, the sandwich, the public 360, speedback, farming feedback for appraisal currency—each of these antipatterns attempts to compress something incompressible. They attempt to manufacture trust in a single event, when genuine trust builds in the mundane, everyday moments. You can’t install a feedback culture in an afternoon any more than you can install trust.
\nSo if there's a single thing we'd ask you to take from this piece, it's to stop looking for the shortcut. Don't wait for the retro, the review cycle, or the offsite. This week, each time you collaborate with a colleague, find ways to strengthen their confidence and improve their effectiveness. Ask them for feedback as well. Say thanks when you get feedback. Then, repeat the exercise the next week. And the next. If you do that often enough, in the ordinary flow of the work, you may find you no longer need the frameworks, the events, or the workarounds. You'll have contributed to your team’s feedback culture.
\n", "date_modified": "2026-09-24T14:10:00+00:00", "date_published": "2026-09-24T14:10:00+00:00", "external_url": "https://martinfowler.com/articles/healthy-peer-feedback.html", "id": "urn:aggr:item:2d739bd4de16d4a34c9592bae3c5dd355e5a3ca3", "summary": "At Thoughtworks, “cultivation”—helping each other grow—has been a big part of how we work. Our collaboration practices make software development a cognitive contact sport and create many opportunities to share feedback about how we think,…", "tags": [ "programming" ], "title": "Healthy Feedback", "url": "https://aggr.aymericbeaumet.com/items/martinfowler-com/2026-09-24-healthy-feedback/" }, { "authors": [ { "name": "Martin Fowler" } ], "content_html": "I have a lot of mixed feelings about AI and LLM technology. I’m fascinated by its effect on our profession, excited by the potential gains in productivity - and thus the products we could rapidly build. On the other hand, I’m fearful of the damage AI might cause: agent swarms taking over our virtual and physical infrastructure, designing bio weapons. But, back on my first hand, LLMs might also design miracle cures, and come up with clever ways to raise our prosperity. Fundamentally I don’t think we have a choice about riding on the AI technology train. It’s a wild ride and I just hope we’ll get through it OK.
\nBut as I mull on this more, I realize that among this mix of contrasting feelings, there is one emotion that dominates - one that comes from my direct interactions with LLMs. I don’t like them. They talk to me in this grating LLM-voice, an uncanny valley of talking to a real human. They confidently bullshit me - often giving me useful, helpful answers. But also just making stuff up with the same assurance - and with only a veneer of fake remorse when I call them out on it.
\nThat’s not enough to make me feel we should avoid them. As Jessica Kerr put it “not only are they useful, it is irresponsible not to use them…. They’re more thorough, as well as faster.” This contradictory reaction comes through in polling, where people say they find these models are useful, but also that they think they will be bad for society.
\nMuch of this may be because LLMs are young - we haven’t trained them to grow up yet. Maybe I’ll like them once they mature. (I hope we get to find out.) But I’m not encouraged when I think of the kinds of environments that cultivate them. I’m wary of the Silicon Valley brogrammer subculture, and these LLMs are their products, so naturally lean toward their world-view. When we think of AI agents, we shouldn’t anthropomorphize, treating them as conscious beings with their own will. They are (software) machines, developed by people working in corporations. While the agents’ behavior aren’t explicitly programmed, they are nurtured with the values of their creators.
\nOne of my most successful life-hacks is to avoid people I don’t like or don’t trust. I decline to interact with them socially, and make a deliberate effort to avoid working with them too, even if they are doing much that is beneficial. I feel that hanging out with pleasant, capable people, the people with integrity, has made my life a far better one. Hence my visceral dislike of interacting with an LLM that’s not just making a pretense of being human, but also posing as the kind of human I walk away from.
\n", "date_modified": "2026-09-17T13:50:00+00:00", "date_published": "2026-09-17T13:50:00+00:00", "external_url": "https://martinfowler.com/articles/2026-dont-like-llms.html", "id": "urn:aggr:item:09b30e3f7a9bafa75fe36acc9880b87117fb54e6", "summary": "I have a lot of mixed feelings about AI and LLM technology. I’m fascinated by its effect on our profession, excited by the potential gains in productivity - and thus the products we could rapidly build. On the other hand, I’m fearful of…", "tags": [ "programming" ], "title": "I don't like LLMs", "url": "https://aggr.aymericbeaumet.com/items/martinfowler-com/2026-09-17-i-don-t-like-llms/" }, { "authors": [ { "name": "Martin Fowler" } ], "content_html": "Reports of agentic hacking continue, in this case it happened back in May and it seems OpenAI did not disclose that they were responsible. Simon Willison sees two options:
\n\n\n\n
\n- After the Hugging Face and Wiki attacks OpenAI were still unable to review their previous logs and determine that they had previously attacked RubyGems.
\n- They knew about the attack on RubyGems and made the decision not to reach out to the RubyGems team about it.
\nBoth of these are bad!
\nGiven this incident, the Hugging Face situation, and the Wiki attack, the obvious question right now is how many more incidents like this are out there waiting to be discovered?
\n
\n\nStop asking the sci-fi question: ‘Is it conscious?’ Start asking the engineering question: ‘Is this a powerful, unpredictable component being put somewhere consequential, and where’s the feedback that tells us that it’s safe?
\n
Nate Silver is known for his forecasts, but to do them he writes a lot of code for his models. He’s found agentic programming capable of doing miraculous work.
\n\n\nIn spending so much time with the LLMs, I’m super attentive to improvements in their capabilities. And these changes tend not to be so linear. Instead, they improve in step functions, almost as phase changes. Suddenly, the models just start doing things capably that they were screwing up before. In my experience, there was a big leap forward when reasoning models first came out in late 2024/early 2025 — enough that they were occasionally useful for tasks involving data and not just words — and then another one this past winter.
\nThe most recent changes I’ve noticed, however, have had less to do with intelligence and more with persistence.
\n
Consider the Hugging Face attack. Although these agents showed remarkable intelligence, they weren’t really super-intelligent - but they were super-persistent. This is a common theme of AI in its various forms:
\n\n\nGame engines like AlphaGo Zero start out by basically making random moves — but by playing against themselves millions of times, they eventually far surpass human capabilities
\n
As we try to figure out what kind of regulations we need to keep AI under control, we need to remember that we should design our guards around super-persistence as much as worrying about super-intelligence.
\n“Uncle Bob” Martin has made many posts on X during the last few months about his programming with LLMs. His approach has been to build a firm harness to keep them under control, so they create software that is maintainable as well as functional. Sadly the posts have been frustratingly light on detail. But now it seems that lack of information may not matter
\n\n\nAnd while I was heads-down getting that to work, the agents got a LOT better. So much so that when I came up for air, the need for my harness was obviated. Indeed, the need for any but the most liberal of harnesses may be obviated.
\n
Some tidbits that struck me from Ezra Klein’s recent (recommended) interview with Matt Sheehan on the interplay between regulation of AI and competition with China.
\n\n\n", "date_modified": "2026-09-16T20:05:00+00:00", "date_published": "2026-09-16T20:05:00+00:00", "external_url": "https://martinfowler.com/fragments/2026-09-16.html", "id": "urn:aggr:item:824acf8c4392ffa0b851b0e0930c81326f155e80", "summary": "Reports of agentic hacking continue, in this case it happened back in May and it seems OpenAI did not disclose that they were responsible. Simon Willison sees two options: After the Hugging Face and Wiki attacks OpenAI were still unable to…", "tags": [ "programming" ], "title": "Fragments: September 16", "url": "https://aggr.aymericbeaumet.com/items/martinfowler-com/2026-09-16-fragments-september-16/" }, { "authors": [ { "name": "Martin Fowler" } ], "content_html": "When American policymakers are like: Where do you start? — I sometimes say: Well, you start by starting. You learn how to regulate things, you learn how to legislate on them by regulating and legislating on them.
\n
Slideware: a presentation program, such as Microsoft PowerPoint, LibreOffice Impress, or Apple Keynote.
\nIn my previous post in this series, I argued for reserving presentations, recorded or live, for content that needs your voice. Live presentations should also merit the scheduling overhead. Once you’ve decided on a topic to present, it’s tempting to go straight to slideware. I suggest otherwise.
\nThe 16:9 slide format forces you to slice your narrative by visuals. When you don’t know what that narrative is, you’ll often second-guess yourself on every slide. Add to that confusion the distractions of font size, colour, creating diagrams, finding images, and deciding transitions and builds. The form precedes the function. This is why going to slides without a narrative creates a massive cognitive challenge for most presenters. This is also why we reach for shortcuts such as canned slides, so we can at least make progress with this demanding challenge.
\nI suggest nailing down your narrative before turning your attention to slides, which serve as supporting visuals. There are many ways to build a narrative, either on your own or with a co-presenter. But step 0 is to describe your key idea and your audience.
\nWhen constructing my narrative, I find it useful to begin by identifying what Nancy Duarte calls the “Big Idea”. Consider it a way to describe the “so-what” of our narrative in a sentence. You can get to the big idea by describing its two constituent parts.
\nOnce you’ve thought through those two parts, combine them into a single sentence that isn’t a mouthful and rolls off your tongue with ease.
\nHere’s an example of the big idea behind this video about the case against brainstorming.
\n| My point of view | \nWhat’s at stake? | \n
|---|---|
| Brainstorming feels scientific and collaborative, but research debunks it as a corporate superstition. Brainwriting (solo, anonymous, written idea generation) is a better approach. | \nTeams sabotage their best ideas through production blocking, conformity pressure, dominant personalities, and social loafing. By believing that brainstorming makes them more innovative, they lose out on valuable ideas and unique perspectives. | \n
| The big idea: Instead of brainstorming, a corporate superstition that kills your best ideas, adopt brainwriting and let independent judgment surface the widest range of ideas and the deepest thinking. | \n\n |
Alongside the big idea, identify your audience. I suggest a persona-building exercise to guide your thinking.
\nSometimes, these persona details are evident and intuitive. You might be presenting to your teammates. In that case, you might speed through the persona-building step in minutes and tweak it a little after each iteration. In other situations, you may need to ask around to learn about your audience. For example, you may be presenting to a new client. In such situations, investing time to learn about your prospective audience will help you sharpen your eventual storyline.
\nHere’s an example of a persona for the same video I linked earlier in the piece.
\nAre you ready to go to slides yet? Nope. Once you’ve identified your big idea and the top three takeaways, I suggest fleshing out your narrative to serve them. Narrative building is a subjective craft, so choose a story structure that fits your content. Here are six story structures I’ve used with success.
\n| Format | \nWhat it is | \nStages | \n
|---|---|---|
| Sparkline | \nA narrative that swings back and forth between where things stand today and where they could go, using repeated beats to make the future feel real rather than abstract. | \nWhat is (today, as it stands) → What could be (the future you’re selling) — repeat as many beats as you need to make your point. | \n
| Explainer | \nA structured walkthrough of a concept or insight the audience doesn’t already have. It follows the “tell them what you’ll tell them, tell them, then tell them what you told them” format. | \nContext (lay of the land) → Story structure (the roadmap) → Steps (the narrative, step by step) → Recap (what you told them) → Celebrate! (a call to action) | \n
| Pitch | \nRecommends a new, inspiring solution to a problem the audience is experiencing, and makes that solution stand out from the obvious, boring options. | \nThe windup (where we are today) → The hurdle (the problem) → The vision (the way out) → The options (a few paths — mostly boring, one inspiring) → The close (why the inspiring option wins) → The fine print (how it happens, plus a bonus) | \n
| Hook, meat, payoff | \nA short, punchy talk structure that grabs attention upfront, introduces the substance next, and then lands a conclusion that reinforces the opening. | \nHook (a provocative opener — a question, challenge, or personal story) → Meat (structured narrative, e.g. lists or a timeline) → Payoff (call to action that connects back to the hook) | \n
| Situation, complication, resolution | \nThis is the classic consulting shape. Start by describing the world as it is. Next, introduce a problem or opportunity, then land the solution that addresses it. | \nThe situation (objective context) → BUT → The opportunity/complication (the challenge or opening) → THEREFORE → The resolution (the solution) | \n
| Hero’s journey | \nThe most dramatic of the six. This structure starts in normalcy, descends into a crisis, hits rock bottom, then climbs back out stronger. It’s a Pixar/DreamWorks favourite, and a natural fit for project stories and experience reports. | \nThe situation (normalcy before the problem struck) → The challenge (a problem you couldn’t ignore) → The crisis (things go south) → Hitting rock bottom (the worst point) → The comeback (how you fought back) → Emerging stronger (lessons learned) | \n
With unpredictable audiences, I’ve tried a seventh, more flexible structure in which one presentation holds multiple smaller storylines. In such presentations, I show my audience a list of potential topics to dive into. Each topic could have a different, independent story structure. Such talks give the audience a sense of control, but also demand a lot from you as a speaker. Neal Ford calls this pattern, “Á la Carte Content,” and it works well when you have more content than the time allows.
\n![\"\"](\"https://martinfowler.com/articles/never-send-slides/index/Non-linear%20presentation.png\")
Figure 1: A flexible narrative structure. When reporting on an internal research survey, I let my audience choose the question they wanted me to answer using the research data.
\nIf you’ve never built a storyline before, I understand it can be daunting. I suggest four approaches to building your storyline.
\nIf you’re co-presenting with someone, I recommend whiteboarding your storyline with them. You can follow one of my recommended story structures, or build your own. Sticky notes on a physical whiteboard work fine, but if you’re remote, you can even use tools like Miro or Mural.
\nSticky notes offer a distinct advantage for crafting your narrative. You can move stickies back and forth, edit them, or trash them. Colour-coding sticky notes helps you visualise related themes, and placing them close together helps you notice adjacencies.
\nIf whiteboarding is your thing, I’ve created a Mural template with all my favourite story structures and panels, so you can outline your big idea and describe your audience persona.
\nOf late, I’ve found myself creating presentations that start as a boxes-and-arrows style diagram in my notebook, or even on a slide. In these situations, the diagram becomes the spine around which I construct the rest of my narrative.
\nFor example, when I was starting my most recent role, I doodled the diagram you see below, on a piece of paper. It was my way of thinking about how I intended to play my role as head of culture at Thoughtworks. After a few more hours of scribbling and making notes, I reproduced the diagram on a slide. I built my final slide deck using a combination of animations and nested slides. I’ll expand on this example further down in the article.
\n![\"\"](\"https://martinfowler.com/articles/never-send-slides/index/Diagram%20centric%20narratives.jpg\")
Figure 2: A core diagram can start as an excellent spine for your narrative.
\nIf you don’t enjoy whiteboarding, I suggest writing your narrative in text. This approach can work well if you’re a solo presenter and prefer writing as a thinking tool. You can start with a bulleted structure for your ideas and flesh them out as you go.
\nIf you want to use one of the story formats I described earlier, I’ve created a pack of Google Docs templates that you can use as an alternative to the Mural whiteboarding option.
\nSince AI voice transcription has gotten better in recent years, I’ve also used voice memos to flesh out my thinking. The Voice Memos app on iOS and the Recorder app on Android offer excellent transcripts, and once you’ve picked a story structure, you can use these apps to record your thoughts for each segment and then pass the transcripts to any AI chatbot to clean up and structure your narrative draft. You may need to clean up the AI-produced draft, but with some practice and by creating some custom skills, you can push AI tools to get you close to a usable narrative. Of course, you can’t, and you shouldn’t outsource your thinking to AI.
\nWhichever approach you take, the success criterion remains the same — you should know your narrative well enough to voice it without any visual aids. And if you achieve that outcome, you’ll have a solid narrative platform, which you can then enhance using audio-visual aids.
\nAlmost there. One more step. Your narrative is a clarifying artefact for what you want to say. You now need some clarity on what you want to show. This is where a storyboard comes in handy.
\nThe simplest storyboard describes each slide you create in as simple a way as you can get away with. If you’re already using a physical or virtual whiteboard, sticky notes or index cards can help you build that storyboard — one sticky note to describe each slide. The Mural template I shared also has a panel to organise your storyboard. As I’ve explained earlier, don’t worry about the number of sticky notes. The slide count doesn’t matter when you control the pace of your presentation.
\n![\"\"](\"https://martinfowler.com/articles/never-send-slides/index/Brainstorming%20storyboard.jpg\")
Figure 3: A storyboard with sticky notes. (generated using AI)
\nYou can also use documents to create your storyboard, though they aren’t as flexible as sticky notes. The Google Docs template pack also has a storyboarding template you can use.
\nThese days, I often create my storyboards using slides, especially when I use a diagram-centric narrative. The slide sorter or light table view in your presentation tool is excellent for creating storyboards because it allows me to drop in text and sample images and reorder my panels until I’m satisfied with the plan.
\nThe trick with slide-based storyboarding is to resist the temptation to design slides. When storyboarding this way, I limit my focus to the spine of my slide deck. The polish comes later.
\nRemember the diagram I shared earlier in this article? The images below show my final diagram, the storyboard that I created using that diagram as the spine, and then a light-table view of the final deck after I added a few layers of polish.
\nIf you’re accustomed to starting your presentation design by opening a presentation tool, my suggested approach will perhaps feel onerous. From experience teaching presentation skills to hundreds of Thoughtworks colleagues, I find this approach to be a way to start slow so I can go fast later. Once you try this approach a few times, you’ll notice that it doesn’t take as much effort as you may fear. It also speeds up slide creation because you’re working off a plan, as against playing it by ear.
\nOn the other hand, if you’re a skilful presenter, my suggested approach may seem rigid and linear. That’ll be a fair criticism. I’ll use a Pablo Picasso quote in response to that reaction.
\n\n\nLearn the rules like a pro, so you can break them like an artist.
\n-- Pablo Picasso
\n
Here’s what I’ve noticed when coaching colleagues to present:
\nAll this said, the narrative won’t be a static artefact. As you build your storyboard, you may tweak the narrative. Even as you design your slides, you may reconsider your narrative, storyboard, and even your assumptions about your audience. None of these artefacts and considerations is a one-and-done. The outline feeds the design, but the design can also challenge the outline. That’s the narrative you want walking into slide design: solid enough that building your slides becomes an exercise in execution, not invention, yet open enough to flex when the details teach you something new.
\nWith that narrative platform in hand, let’s explore some principles for slide design in the next few posts.
\n", "date_modified": "2026-09-15T15:11:00+00:00", "date_published": "2026-09-15T15:11:00+00:00", "external_url": "https://martinfowler.com/articles/never-send-slides/nail-your-narrative.html", "id": "urn:aggr:item:436e3b4ef190e7e3d2c54c8c243aa769aca3dc96", "summary": "Slideware: a presentation program, such as Microsoft PowerPoint, LibreOffice Impress, or Apple Keynote. In my previous post in this series, I argued for reserving presentations, recorded or live, for content that needs your voice. Live…", "tags": [ "programming" ], "title": "Nail the Narrative", "url": "https://aggr.aymericbeaumet.com/items/martinfowler-com/2026-09-15-nail-the-narrative/" }, { "authors": [ { "name": "Martin Fowler" } ], "content_html": "In order to let people know about new articles on martinfowler.com, I post announcements on social media. I've been doing this for a long time, starting on Twitter back in 2008. After the muskover, social media became rather more fragmented, and I now post announcements on four social media platforms: LinkedIn, X, Fediverse (Mastodon), and Bluesky. It's a bit tedious, but it doesn't take too long to copy/paste the announcements on each site, with a bit of tweaking as necessary. (I also post announcements to my RSS feed. If I were following me, I'd use RSS.)
\nFrom time to time, I look at the traffic on these sites. (Last time was in 2025) 1 The main thing I'm interested in is how much engagement I get from each service, so I can get a sense of their importance for my announcements. There are three data items that I can use to sense this: retweets, likes, and replies. Here's some plots of that data for my posts over the last couple of months (19 total)
\n1: See my website tag page for material like that about my website behavior.
\n![\"\"](\"https://martinfowler.com/articles/2026-social-traffic/retweets.png\")
Figure 1: number of retweets for posts from June 16 to Sep 6 2026
\n![\"\"](\"https://martinfowler.com/articles/2026-social-traffic/likes.png\")
Figure 3: number of likes for the same period
\nI don't like using simple averages to compare things, they are often misleading and blur useful information. So these plots combine a strip chart (showing each post as a point, with some horizontal jitter) and a box plot (to show median and quartiles). This way we can see the whole distribution while also being able to compare useful summaries within them.
\nLooking at this, there's a clear ranking between the services. LinkedIn is clear leader at engagement numbers. I tend to take the retweets most seriously, since they suggest a desire to recommend a post for others to consider. I'll chop off the upper outliers there to zoom in on the core data.
\n![\"\"](\"https://martinfowler.com/articles/2026-social-traffic/retweet-zoom.png\")
Figure 4: Retweets, zooming in on the core data
\nLinkedIn clearly has an edge there, with X a clear second, its upper quartile is below LinkedIn's median. Bluesky and Mastodon are tiny in comparison, their upper quartiles are below X's lower quartile, although Mastdon's is visibly above that of Bluesky.
\nLooking at this data, there's a strong argument for stopping announcements on Bluesky and Mastodon, as they get such little response. I will continue with them, however, since they are open platforms - and I have a clear preference for open platforms over the walled gardens of LinkedIn and X.
\nAnother way of judging this is to look at source traffic for the website, after all the whole point of announcing posts is to bring readers. Here's some data on that, taken from my website analytics:
\n![\"\"](\"https://martinfowler.com/articles/2026-social-traffic/sources.png\")
Figure 5: sources of traffic in 90 days prior to Sep 7 2026
\nThis reflects the data above, LinkedIn is clearly a bigger source of traffic than X, which is far above bluesky (we can't really assess Mastodon here, but I expect it would also be a very thin line). But even LinkedIn is a minor source of traffic compared to google. And google's 300,000 or so is a small part of the 1.7M visits the site got in those 90 days.
\nI last did this analysis in early 2025. Comparing the box plots for the retweets indicates how things have changed.
\n![\"\"](\"https://martinfowler.com/articles/2026-social-traffic/compare-year.png\")
Figure 6: Compared retweets from posts in mid 2026 to early 2025.
\nLinkedIn has grown a little bit, X has dropped notably, and Bluesky has dropped dramatically.
\n", "date_modified": "2026-09-09T16:21:00+00:00", "date_published": "2026-09-09T16:21:00+00:00", "external_url": "https://martinfowler.com/articles/2026-social-traffic.html", "id": "urn:aggr:item:89a4a4a12a97a0ef9d9e01e2a5f29f0c7970c1ce", "summary": "In order to let people know about new articles on martinfowler.com, I post announcements on social media. I've been doing this for a long time, starting on Twitter back in 2008. After the muskover, social media became rather more…", "tags": [ "programming" ], "title": "Social Media Engagement: summer 2026", "url": "https://aggr.aymericbeaumet.com/items/martinfowler-com/2026-09-09-social-media-engagement-summer-2026/" }, { "authors": [ { "name": "Martin Fowler" } ], "content_html": "Christian Catalini says we’re in a situation where we are vastly reducing the cost of generating things, but not the cost of verifying them:.
\n\n\nThis explains why the first major AI products appeared in chat, image generation, and code assistance. Not because these were the hardest human problems, but because their outputs were relatively easy to inspect. A user can judge the tone of a message, look at an image, or run a test on a piece of code. […] The old automation boundary was routine versus non-routine work. The new boundary is increasingly measurable versus non-measurable work.
\n
The issue is then over how well you can measure something. In our profession, we know there’s a big difference between how many lines of code we write and how productive we are, and we’ve seen a regular failure to understand how to measure productivity. Too much of what makes work effective is subject to either slow feedback loops or assessments that require subtle judgment. The danger is that people use lots AI automation while using incomplete measurements of its effectiveness, leading to short-term dashboards going up, but disaster in longer time-scales. He refers to these illusory short-term gains as counterfeit utility.
\n\n\nScale this across companies and institutions and the result is a Hollow Economy: extraordinary measured activity sitting on top of weakening human capability, hidden technical debt, correlated errors, and outcomes that nobody can confidently stand behind.
\n
Another highlight in the article was his advice to “build a history of decisions, not a gallery of outputs”. The point is that with AI we can all build really impressive things, but our value lies in the judgment that we’ve formed. It reminds me of how math problems were marked at school. We weren’t just marked on getting the final answer, we were also marked based on our reasoning process.
\nHe uses the OpenAI–Hugging Face incident as an illustration of this gap between generation and verification. He criticizes those who anthropomorphize the agents involved in the attack. By doing so we focus on the behavior of the AI agents, but instead we should focus on the financial incentives that created them and the environment they are operating in.
\n\n\nLabs are locked in a race. The training run is where the money goes, and RL optimizes exactly what you score. The runs were scored on capability. They were not scored on “did not poison the Artifactory cache.”
\n
I assert that the organizations that build and run agents are responsible for everything those agents do, whether that behavior is intended or emergent. If they reap counterfeit utility by neglecting verification, they must face consequences: legal, financial, and if necessary: criminal. To deal effectively with AI, we need to change the incentives involved to ensure people invest more in verification than they do in generation. Otherwise we are driving a car that has a powerful engine, but weak brakes.
\nBrian Cantrill relates how readers are exasperated with “writers” using LLMs.
\n\n\nTo those who read broadly, the hand of the LLM is so clear it’s as if the writer’s intellectual fly is open. In fact, it’s so jarring that I have to believe that those writing with LLMs are either not reading enough to see the LLM’s obvious structural tells — or (and?) they aren’t even reading their own content. (A confession: with particularly egregious pieces, I have fantasized about sentencing the author to read them aloud, certain that they themselves will be unable to endure the slop that they are foisting upon the rest of us.)
\n
He points out that readers do care about this, a survey found 78% of readers stop immediately once they sense something is the work a stochastic parrot, and 71% go on to blacklist the writer. It’s not the polish, it’s the authenticity that counts. Readers will always prefer the clumsy voice of the author over the gloss of an LLM’s whispering.
\nCantrill reports good success with using Pangram to detect AI writing. I confess I’m a bit wary, do I really trust anyone’s judgment to disentangle LLM-voice from changes in generation and context? Maybe people steeped in Silicon Valley culture authentically speak in LLM-voice these days. Sadly for them, to be misclassified by their readers as an LLM is just as bad as using the damn things.
\nOne of the dirty non-secrets about LLMs is that they were trained on a vast corpus of writing, without consulting the authors of that writing to see if they were cool with it. Individual authors like me can’t do a great deal about it, so are easy to ignore, but music companies aren’t exactly known for taking this kind of thing lying down. So they are suing over the use song lyrics for LLM training.
\n\n\nSony Music Publishing and Warner Chappell, music publishers who manage the copyright of songs on behalf of songwriters and composers, are seeking damages for alleged misuse of “tens of thousands” of copyrighted works by Anthropic. […] The plaintiffs claim they are victims of “one of the largest and most blatant ongoing thefts of intellectual property in history”.
\n
Looking at it a broader societal point of view, there is an argument that the benefits of LLMs could be worth far more than any losses to us authors. But we should not forget that these tools are built on a foundation they used without our consent, and that should be taken into account as we regulate these tools and the fruits they provide.
\n\n\nSteve Yegge:
\n\n
\n\nAll models, no matter how smart, will eventually build systems that they can no longer understand or maintain, if you let them. Fable 5 finally outbuilt itself, and flailed on me for a week. Fable 5.1 looks like it will fix it. For now. But you have to keep an iron grip on system size, or it’ll run away from you.
\n
I was going through some slightly-related work and discovered that the Creating Passionate Users blog had disappeared from the internet (and has been gone since maybe a year ago). For those who don’t know, Creating Passionate Users was one of the treasures of the Golden Age of internet blogging. It was the work of Kathy Sierra, also known for co-creating the “Head First” series of computer books. It talked about user experience, and remains some of the best writing on the topic, full of sparkling insights that greatly influenced my thinking, as well as many folks more engaged on user-experience work.
\nSadly not just was the blog ahead of time in its content, it was a harbinger of the darker side of the internet, as Kathy came under attack from a particularly virulent form of Net Nastiness. That led her to retreat from active participation on the web, and we’ve missed her ever since.
\nFortunately the Wayback Machine did its great duty, and we can still read its snapshot. I’ve often thought that, if I had a clone to spare, I’d like to create a guided tour of Creating Passionate Users to help readers today read that excellent material. (And if you’re reading this Kathy, and want it still hosted on the web, I’d be delighted to.)
\n\n\n“I don’t know the answer myself, but I asked a blowhard I know and he took a wild guess, here’s what he said: “
\nHow I interpret pasted replies from an LLM in online conversations
\n
Jessica Kerr loves the feeling of being part of a team of people that learns from each other and from the codebase they are building as extensions of themselves - she incorporates the term symmathesy for this: a learning system composed of learning parts (both the people and the code).
\n\n\n\nThere was a turning point last year where I noticed that not only are they useful, it is irresponsible not to use them, at least in conjunction with my own code. They’re more thorough, as well as faster. How am I supposed to be responsible for this system, when I don’t understand each line of code?
\n
She has a habit of digging out old terms and ideas and applying them to our digital world. To frame what’s happening, she digs out two bits of latin
\n\n\n\n
\n- Verum Factum: I made it, so I get it
\n- Vexationes Artium: Put it to the test [i.e. experiments]
\n
Agents can’t have Verum Factum knowledge, since it’s gone once their context window clears. They can use Vexationes Artium, running tests to see if something is working.
\n\n\nIf we want agents to write working, reliable code for us, we have to double down, 10x down on our objective verification. We need to vexate that code in artful ways. And we have the agent help us with that, with its thoroughness.
\n
This is, of course, true of those building these AI models - they certainly don’t have a Verum Factum knowledge of how they work, all they can do is come up with artful vexations to figure out what might be going on in there.
\nWhat does that mean for us humans? Kerr says The Enlightenment elevated the idea that reason was the special quality of mankind. But now we’ve built machines that can reason. We need to focus instead on human qualities that the machines don’t have. Imagination is more important to us now than reason. And the essence of our humanity is in our relationships with other people.
\nThis material was put together for a conference talk, it’s available in video and in text form. Generally I far prefer text, but there’s so many delights in her talk that I found it worth watching. (Such as her current definition of legacy software, as code that wasn’t built with agents in mind - and is thus lacking in testability and consistency.)
\nJim Gumbly outlines the decision space for three people who are making vital judgments about controls and regulations for AI: Sam Altman, Dario Amodei, and David Sacks (for US government).
\n\n\nSacks worries that binding oversight could weaken American leadership and the financing that sustains it. Sam and Dario face commercial incentives to keep producing more capable models. The value of those models increasingly comes from persistence, cooperation and permission to act. Those same properties increase the demands on containment and oversight. Some advances also make the evidence harder to interpret.
\nEach decision can be understandable on its own. Their combined effect can still move deployment ahead of our ability to control it.
\n
A complication for all this is the issue of observability of these models.
\n\n\nOpenAI reports that Astra is better aligned on its evaluations while also being less monitorable. Its traces can be shorter and less informative. In adversarial tests, where it was instructed to evade oversight, it could conceal some behaviour from monitors.
\n[…]
\nThat matters to the familiar software cycle of releasing something, observing what happens and improving it. The cycle depends on the quality of the observations. Fewer warning flags are reassuring only to the extent that the warning system remains capable of detecting the relevant failures.
\n
There’s an El Niño year coming up, and The Grauniad reports that climate scientists predict this El Niño is going to be a spectacularly hot one.
\n\n\nThe most recent data, from Monday, shows the temperature of the ocean at the heart of El Niño at 2.6C above the 30-year average. That is already close to the highest anomaly ever recorded in the satellite data era, 3.1C in 2015, with months to go before the peak is expected.
\nThat peak is forecast to reach about 4C in November, according to the average of 14 different models. Data from analysis of corals, tree rings and historical documents suggest no El Niño has reached this level in the last millennium, said Zeke Hausfather, a climate analyst.
\n
If these forecasts end up being accurate, will this make a difference to how seriously people are taking the climate crisis?
\n", "date_modified": "2026-09-08T15:22:00+00:00", "date_published": "2026-09-08T15:22:00+00:00", "external_url": "https://martinfowler.com/fragments/2026-09-08.html", "id": "urn:aggr:item:d9f174e8ab845ee8d58a88d680e5561f5fa8cbb4", "summary": "Christian Catalini says we’re in a situation where we are vastly reducing the cost of generating things, but not the cost of verifying them:. This explains why the first major AI products appeared in chat, image generation, and code…", "tags": [ "programming" ], "title": "Fragments: September 8", "url": "https://aggr.aymericbeaumet.com/items/martinfowler-com/2026-09-08-fragments-september-8/" }, { "authors": [ { "name": "Martin Fowler" } ], "content_html": "We often conflate presentations with the presentation file or the slideshow. That confusion leads to an overt focus on the slide deck and less focus on the presenter or the narrative. Here’s how I define the term “presentation”.
\nA presentation is an act of storytelling in which a presenter orchestrates narrative, timing, and emotion to create meaning that neither visuals nor documents can achieve on their own.
\nAs a corollary to that definition, the slides are not the presentation. A presentation is a powerful tool for influence. You’ll get most value from presentations when you reserve them for the situations when they’ll be most impactful. At other times, you might be able to get away with conveying information using simpler or alternative methods.
\nIn most corporate scenarios, we’re just conveying information. Over the years, we’ve learned that visual content is easier to consume. Sometimes, the effective use of slideware can indeed make information more approachable, and I’ll address those cases shortly. Most of the time, though, slideware is a poor tool for well-structured writing. When we start using slides as a document format, we force readers to consume information in fragmented, disconnected bursts.
\nIn contrast, a document that organises your ideas in a linear structure helps others build up their understanding as they read. By including diagrams where necessary, you can provide readers with a visual metaphor to encapsulate what you’ve written. When you employ simple language, short paragraphs, headings, subheadings, bullets and tables, you help time-poor colleagues scan your writing and consume the essence of what you’re communicating.
\nThe great advantage of using writing tools like Microsoft Word, Hemingway or Grammarly is that you can codify the writing style you wish to follow and the reading ease you’re aiming for. Much like an IDE plugin assists a developer in writing code, these tools help you simplify your language so it's easier to understand.
\nMost writing tools include collaborative features to suggest changes, and that workflow makes it easier to improve the document to a point where it can stand alone as an independent, low-context artefact. Cutting, pasting, and reorganising text in a document is also far easier than doing so in slides. In cases where the content deserves hefty detail, you don’t spend hours trying to make things look pretty on a slide. Instead, you can simplify that detail for your readers.
\nMoreover, slide decks often leave things unsaid between slides, and when communicating async, it can lead to misunderstandings and lost context. When you write, the linear structure of documents forces you to bridge different sections of your writing. From the reader’s perspective, if they even use an LLM to summarise your document, the explicit nature of linear writing reduces the probability of hallucinations.
\nSo, in most corporate communication scenarios, a simple, linear document should suffice. There are some situations, though, where you may want to transfer this asynchronous content to a more visual medium.
\nSlide-based applications offer a ubiquitous page-layout interface for users unfamiliar with sophisticated desktop publishing. They support layer-based editing. Elements on top hide the elements below. What you see is what you get. Considerations like page size and margin width, which matter in print, don’t matter as much when you’re viewing an artefact on your computer. You can produce decent-looking ebooks without much specialised design skill. Take, for example, a trip memoir my son wrote, which I then laid out in PDF format using slideware.
\nIn corporate settings, this ease of authoring translates to other documents — Martin Fowler calls them infodecks. Infodecks aren’t slideuments and don’t double as presentation devices. They’re only for what I’d call “light reading”. They juxtapose text with visuals and communicate information in a concise and approachable manner.
\nMartin’s website features several infodecks. My employer, Thoughtworks, has created a bunch of elegant and practical infodecks, too. Here are a few examples:
\nIn contrast to professional desktop publishing tools like InDesign or Affinity Publisher, software like PowerPoint, Keynote, Canva and even Google Slides offer an easier editing interface for most knowledge workers. But you don’t get this editing superpower for free.
\nLaying out infodecks in the ubiquitous slide interface can make concepts and ideas shine, and they could be worth the effort if you expect your content to have a long shelf life. But don’t underestimate the design effort behind these artefacts. Duarte Design offers a beautiful, if dated, guide to authoring what they call Slidedocs. “Slidedoc” is another name for an infodeck. Read the ebook, and you’ll learn several patterns that you can then implement in the slideware of your choice.
\nIn the last year or so, we’ve seen an explosion of coding harnesses that allow non-developers to spin up simple applications. Using HTML, CSS, and JavaScript, you can embed content from external sources and use interactivity to manage your readers’ cognitive load.
\nConsider, for example, the word cloud below, which groups the company values of several IT, consulting, and design service providers. You can filter the values by clicking the company names, or hover over a specific value to see which companies have claimed it. You could provide this information through a document or a spreadsheet, but you’ll then rely on the user to manage their cognitive load.
\nAnother example is this microcourse, which is based on a long-form article about how leaders shape cultural identity in groups and corporations. The content is almost identical to the original piece. Still, by chunking the concepts, adding interactions for the learner to explore, and including reflection questions to test their understanding, I’m managing their cognitive load.
\nThese applications and interactions have their disadvantages too. Wiki pages and collaborative documents have built-in commenting functionality, so people can offer you feedback and reactions inline with your content. Adding this functionality to every app or interaction you build creates overhead. So, be aware of this trade-off. Interactive, but stateless applications have limited utility beyond one-way information transfer.
\nIf you can document your design direction and your underlying content is comprehensive enough, you can produce such simple content by leveraging a coding harness. Of course, there are complexities in how you host such content for your coworkers and in the shelf life of such applications. Since I’m not a software engineer, I estimate the shelf life of such applications based on how well I understand their inner workings.
\nWhen none of the above options is suitable, and you feel convinced that the story needs your voice, that’s when you must consider a presentation. And even when you decide to present, a slide deck isn’t mandatory. Nor is it necessary to present live.
\nThe easiest foray into presentation territory doesn’t even need visuals. A well-written monologue, whether in audio or as a talking head, is sufficient to convey some ideas. You benefit from the richness of tone, the drama of pauses and pacing, and your unique voice. Ed Zitron, the critic-in-chief of the AI capex bubble, delivers regular monologues on YouTube without any visuals and still manages to have a cult following.
\nYou can, of course, step up the production effort and add visuals to your voiceover. This is where slides can come into the picture. Modern presentation tools like Canva, PowerPoint and Keynote help you create cinematic builds and transitions, so you don’t have to seek high-end animation skills or a professional video editor to make a point.
\nThere is a catch, though. When people watch a 30-frame-per-second video, they’re not expecting static slides on their screens. So, if your video only includes a voiceover and slides, you must cycle through builds and transitions much faster than when you are on screen. My video making the case against brainstorming is one such example. It’s 8 minutes long, with 129 builds and transitions to maintain the tempo of an online video, and, in hindsight, I’ll argue it’s slow.
\nOn the other hand, my recorded presentations about cultural identity, where I appear on screen, span 141 builds and transitions over 24 minutes.
\nEither way, you might be aghast at the number of builds I’ve just told you about. And that’s the thing — there’s no formula for the right number of builds and transitions. Whatever helps you tell your story is what you need. No one counts the extra slides and extra transitions as long as your message shines.
\nRecording a talk forces you to be brief and cut the fluff from your narrative. Your audience can consume your content on their own time, slow it down, speed it up, or play it multiple times if they wish. Closed captions help others understand your accent, and chapter markers help viewers and listeners speed to or revisit topics of interest. With all these benefits, it’s fair to say that most presentations are better off as recordings.
\nThese days, live presentations come in two flavours: online and in-person. Either way, the purpose of a live presentation is to interact with people. At a conference, an impactful presentation could build your network and enhance your professional profile. At work, live presentations can serve as an event where you pitch to a client, educate colleagues, celebrate success, energise a community and use synchronous presence to influence your coworkers and field objections.
\nI suggest reserving live presentations for situations that merit the scheduling overhead. If you’re bringing people together in the same physical or virtual space to share something with them, it should be well worth everyone’s time.
\nAnd I recommend a high degree of prep for live presentations. People can’t speed you up or slow you down. Depending on the medium, there may be no closed captions to help people understand you. Pacing, articulation, preparation, and the synchrony between what you say and what you show are key to making a live presentation impactful.
\n![\"\"](\"https://martinfowler.com/articles/never-send-slides/index/Communication%20approaches.jpg\")
Figure 1: A hierarchy of communication approaches
\nWith all the alternatives available to you, I urge you to consider whether you must present the information you want to convey. Perhaps the answer is yes. The narrowed-down choice then is between recorded, online, and in-person presentations. Those are the three interactions we’ll focus on over the remaining articles in the series. Next up, let’s discuss how to craft your story before you even touch any presentation tools.
\n", "date_modified": "2026-09-08T13:42:00+00:00", "date_published": "2026-09-08T13:42:00+00:00", "external_url": "https://martinfowler.com/articles/never-send-slides/need-presentation.html", "id": "urn:aggr:item:7c581c185d1c635211f3677a83286b7181cdf24e", "summary": "We often conflate presentations with the presentation file or the slideshow. That confusion leads to an overt focus on the slide deck and less focus on the presenter or the narrative. Here’s how I define the term “presentation”. A…", "tags": [ "programming" ], "title": "Do you even need a presentation?", "url": "https://aggr.aymericbeaumet.com/items/martinfowler-com/2026-09-08-do-you-even-need-a-presentation/" }, { "authors": [ { "name": "Martin Fowler" } ], "content_html": "The difference between a medicine and a poison is dosage.
\nOften we talk about certain habits, in programming or life, are good or bad. But few things are simple binaries. Some vary with context: reading a book is a good thing sitting in my garden, but not while driving my car. But another variable is dosage: a little pain-killer salves my headache, but too much will kill me.
\nThe importance of dosage was noticed by a 16th century Swiss physician called Paracelsus. His quote was originally in German “Alle Dinge sind Gift, und nichts ist ohne Gift; allein die Dosis macht, dass ein Ding kein Gift ist.” which (according to Wikipedia) translates as “All things are poison, and nothing is without poison; the dosage alone makes it so a thing is not a poison.” It's also known as “The dose makes the poison” or if you prefer your sayings in Latin “dosis sola facit venenum”.
\nIn programming, global data is a good example of the Paracelsus Maxim (as I like to call it). A little global data, especially when immutable, can be a handy way of propagating information that may needed anywhere in a program, but it quickly becomes dangerous if there is a lot of it about.
\nThis kind of thing crops up in lots of places. So when thinking about when things are good or bad, we should always ask “in what contexts” and “in what doses”?
\n", "date_modified": "2026-09-02T22:03:00+00:00", "date_published": "2026-09-02T22:03:00+00:00", "external_url": "https://martinfowler.com/bliki/ParacelsusMaxim.html", "id": "urn:aggr:item:e5acd8fa434de9c6d9b6379a1c19dba6e3d3fb76", "summary": "The difference between a medicine and a poison is dosage. Often we talk about certain habits, in programming or life, are good or bad. But few things are simple binaries. Some vary with context: reading a book is a good thing sitting in my…", "tags": [ "programming", "bliki" ], "title": "Bliki: Paracelsus Maxim", "url": "https://aggr.aymericbeaumet.com/items/martinfowler-com/2026-09-02-bliki-paracelsus-maxim/" }, { "authors": [ { "name": "Martin Fowler" } ], "content_html": "This week, across Thoughtworks Europe, we took 10 engineers and put them in one room in our Barcelona office. The goal was to see how far and how fast we could go if we really leant into agentic engineering. We called it hyper-agentic. Along the way, we accidentally re-discovered something about coordinating agents.
\nThe 10 engineers were given the goal of building an airline IROps system. This is the system that airlines use in a flight control centre when something goes wrong. When a plane has a technical fault that needs to be repaired. When a crew member gets sick and the crew needs to be replaced. It’s how they decide which flights to cancel, which planes to swap, which passengers get offloaded and put into hotels, etc. etc. They’re doing this over hundreds of aircraft, hundreds of thousands of passengers and many, many crew across multiple stations and airports. It’s a really, really hard problem, hard to solve, hard to execute. An IROps system is complex to build, complex to understand, and complex to use.
\nWe managed to build one in four days. But this post isn’t about how we did that.
\nWe started with a specification and a simulated airline, because this was a practice exercise, not a real client. We tried a couple of things just to see what would and wouldn’t work. We used a monorepo. All the engineers began working at once. We just got started, and after a couple of days we began to see things happening in interesting ways, things emerging.
\nWith lot of agents working in one repo, build pipelines suffered. To deal with this we introduced a discipline: our agents were to continually commit and rebase from main. At first, we required a rebase after commit and to then push, with all of the build checks and controls in place. We introduced this change to catch build failures locally: integrate early and often. But, there was a side-effect. We were directing the agents to plan, to scope work to sections in the spec and to create plans linked to those sections. These plans were stored in the repo. All agents were working off the same spec using the same numbered and identified sections. As agents worked, plans were updated to record progress. These updates, alongside all others, were swept up with the new commit discipline. Agents were able to see other agents’ progress.
\nSo, one agent was, say, working on the evaluator, the component to determine if a particular plan to restore operations is valid, whether it breaks hard constraints or soft constraints, etc. At the same time another agent was working on the search algorithm that looks for plans that could solve the disruption. The search component depends on the evaluator. Each component can be written together, but there is a shared interface and search depends on the evaluator.
\nThe plans recorded these integration points. One plan said at this point I’m going to need to update the callers to call the real verifier. And on the search side, it said, at this point I need to insert the call to the real verifier when it arrives. Both agents could see each plan, and the progress. We realised that the agents were using the plans to coordinate. One agent would mark a line of the plan as in progress, the other agent would see that and not work on that line. When the first agent finished, the other agent would see not only that the work was complete and thus it was released to proceed, but would also be directly delivered notes on how the line had been implemented.
\nWe started to exploit this. We’d kick off a session and direct it to work on a particular journey. One example was to introduce a cost model alongside the verifier. Knowing that someone else had been working on the cost model and pushing commits continually, we directed the agent working on the verifier to look at plans and source, monitor the repo, and when the work for the cost model lands start to integrate it. And it did.
\nThis was entirely ad hoc. It was an accident of a series of decisions. We saw it happen. And then started to use it.
\nThere’s a name for the pattern our agents had discovered: a blackboard system. This was something I explored back in my university days. My research thesis was in directing agent behaviour with hierarchical sensors. I was looking at applying modern machine learning techniques of the time, such as reinforcement learning, across large, dynamic datasets. Looking back over the literature, I adopted the blackboard pattern as the core coordination structure. This had previously been discovered in the development of the Hearsay-II system in 1980. It had been subsequently been developed into the more formal tuple space concept by Gelernter et al. in 1986.
\nA blackboard or tuple space is a shared memory that autonomous agents can read and write from independently. They read and write tuples with a certain minimum structure, and then as many extra fields as you want: no schema. It’s a very effective technique for coordinating autonomous problem solvers towards a single goal. They can each solve a decomposed part of the problem, drop their solution into the shared space, label it, and other autonomous searchers will find it, pick it up, and use it as part of their work.
\nWe had accidentally prompted our agents to start using our repo as a blackboard. But it was an accident. It wasn’t an intentional act. It wasn’t fully structured. It was missing some of the key parts of how blackboards operate. And because it was accidental, I’m not convinced I would be able to reliably prompt our agents into doing it again. I’ve got a pretty good idea what we did, because we did some analysis and identified the single prompt that caused this cascade to start happening. But it was an emergent behaviour. It wasn’t a directed behaviour.
\nAs well as creating this intentionally, rather than accidentally, I believe you want this communication channel to be sitting independently of source control. While we created it by directing a frequent push cycle, we backed-off from that. The frequent commits were overloading our CI pipeline. We switched to only push when a more coherent chunk of change was complete. This deprived the agents of the continuous flow of updates on progress.
\nA good accidental solution requires a good intentional project. I’ve started working on a project I’m calling Talwrn. That’s Welsh for a threshing pit, an area or space where arguments and conflict get worked out. This is aiming to be a blackboard for agentic engineering. My goal is a very simple to use tool that drops straight into your project and immediately offers a communication channel for agents to coordinate work. The first step is to get Talwrn to a point where it can support its own development. I’m planning to post about it regularly as I’m hoping to use it as a single, evolving example of how pure agentic engineering can proceed.
\n", "date_modified": "2026-09-02T14:45:00+00:00", "date_published": "2026-09-02T14:45:00+00:00", "external_url": "https://martinfowler.com/articles/exploring-gen-ai/an-accidental-blackboard.html", "id": "urn:aggr:item:15bfb9951e510eb745c5af594eed613c06b45a0d", "summary": "This week, across Thoughtworks Europe, we took 10 engineers and put them in one room in our Barcelona office. The goal was to see how far and how fast we could go if we really leant into agentic engineering. We called it hyper-agentic.…", "tags": [ "programming" ], "title": "An Accidental Blackboard", "url": "https://aggr.aymericbeaumet.com/items/martinfowler-com/2026-09-02-an-accidental-blackboard/" }, { "authors": [ { "name": "Rachel Laycock" } ], "content_html": "TL;DR
\nOr, perhaps the problem isn't that AI has broken code review, maybe it’s that we've been using code review to solve the wrong problems
I was on a panel recently with Brian Houck from DX at Code Remix, hosted by Moderne. It was one of the more interesting panels I’ve done, largely because we disagreed. As my colleague Martin Fowler says, panels are much more interesting when people disagree and both sides have a good argument. Brian and I definitely did.
\nBrian has since written a thoughtful piece called What are code reviews even for? He is clearly passionate about his position, and I am passionate enough about mine that I’m writing this response. To be clear, I think we mostly want the same things. I just don’t think code review is the best way to get them. Brian is lovely, by the way, and encouraged me to write this. But I’d be lying if I said I didn’t want you to think I’m right by the end :)
\nSo what were we disagreeing about?
\nAI is producing more code than humans can realistically review. Brian cites some pretty striking numbers: at Meta, significant lines of code per human-landed diff reportedly increased 106% in a year, while DX’s own data shows median pull request size increasing 64%.
\nHis concern, which I share, is that simply automating code review away risks losing all the other things we use it for. Code review isn’t just about finding bugs. It’s how teams share knowledge, teach junior engineers, build collective ownership and spread architectural understanding.
\nMy question is: why are we waiting until code review to do all of those things?
\nI’ve never particularly liked pull requests as the centre of the software development process. Not because engineers shouldn’t look at each other’s code, but because I’ve always struggled with the idea that we should build something, finish it, package it up, throw it over to somebody else and then have the important conversation about whether we built the right thing in the right way.
\nAnd don’t even get me started on merge conflicts. I’ve lost too many hours of my life.
\nOne of the principles I learned very early at Thoughtworks was to shorten feedback loops. If feedback is valuable, don’t remove it. Move it closer to the decision it is informing.
\nTake the things we say code review gives us.
\nIf we want to explore alternative solutions, I’d rather do that before implementing one of them.
\nIf we want knowledge transfer, pair. Sitting next to someone, physically or virtually, while they reason through a problem teaches you far more than reading their completed solution afterwards.
\nIf we want junior engineers to learn how experienced engineers think, let them work with experienced engineers while they’re thinking. Pairing comes to mind again here, but teams could also do design sessions collectively with a whiteboard before they write (or instruct the agent to write) anything.
\nIf we want collective ownership, organise teams so people actually build and operate software collectively rather than relying on a pull request to tell everyone what somebody else has already built. For this again use pairing, mob programming, or team design sessions around whiteboard.
\nIf we want architectural alignment, design together (I won’t repeat myself about pairing and team design sessions, oh wait…) and then encode the important constraints as fitness functions.
\nAnd if we’re reviewing code for formatting, linting, known security problems or things that can be deterministically tested, automate them. We really shouldn’t still be arguing about whitespace in 2026.
\nPair programming, trunk-based development, automated testing, static analysis, fitness functions and security scanning all move feedback earlier. Increasingly, agents can participate in those loops too, challenging designs, testing assumptions and continuously verifying what is being built, but the real thinking is coming from experienced humans and if we want that experience to benefit the whole team then we have to act like one much earlier than code review.
\nNone of this means nobody ever reviews code. There are absolutely changes where I want another experienced human looking. An example would be a fundamental architectural change. Assuming we did a design session as a wider team, we might want to review the code as a team or agree it was implemented right, or discuss if we want to change anything. Other examples could be something crossing a sensitive security boundary, a change with a huge blast radius, an unfamiliar part of a critical system or simply something where the team says, “I’m not confident about this.”
\nThose are exactly the places where human judgment is valuable, but that’s very different from requiring a human to inspect every change because that’s the ceremony we’ve historically used to create confidence.
\nAnd we know now it’s not viable to continue down this path, hence why code review keeps coming up as an issue or a blocker. If an agent can produce ten times the code but every line eventually queues up waiting for a senior engineer to inspect it, we haven’t created a ten-times engineering organisation, we’ve created a big backlog and a new bottleneck.
\nAnd I don’t think the answer is an AI agent pretending to be the human reviewer so we can preserve exactly the same process at higher speed. That’s automating the ceremony rather than questioning why the ceremony exists.
\nThere is one thing I do worry about in Brian’s argument, though. He talks about teams accumulating cognitive and intent debt: software grows while the humans responsible for it understand less and less about why it works the way it does. I think that’s a very real problem. I just don’t think mandatory pull requests are a particularly strong defence against it.
\nIf agents are going to produce substantially more of the implementation, we need to be much more deliberate about maintaining human understanding through collaborative design, pairing, good boundaries, executable architecture, shared operational responsibility and probably some practices we haven’t invented yet.
\nWe need engineers to understand systems, not diffs.
\nPerhaps that’s what AI is exposing. We’ve spent years loading an extraordinary number of responsibilities onto the humble code review: quality gate, security check, architecture review, mentoring mechanism, knowledge-sharing system, ownership model.
\nIt worked, sort of, while humans could only produce code so quickly. That constraint is disappearing. So perhaps the question isn’t how we get the code reviewed faster. Perhaps it’s why we’re waiting until code review to have all the important conversations in the first place.
\n", "date_modified": "2026-09-02T13:32:00+00:00", "date_published": "2026-09-02T13:32:00+00:00", "external_url": "https://martinfowler.com/rachels-ramblings/code-review.html", "id": "urn:aggr:item:0f98d09ee6a885aa9fc5a4f6139d61a6c37c04f2", "summary": "TL;DR Or, perhaps the problem isn't that AI has broken code review, maybe it’s that we've been using code review to solve the wrong problems I was on a panel recently with Brian Houck from DX at Code Remix, hosted by Moderne. It was one of…", "tags": [ "programming", "rachels-ramblings" ], "title": "Maybe We Shouldn't Be Reviewing All This Code", "url": "https://aggr.aymericbeaumet.com/items/martinfowler-com/2026-09-02-maybe-we-shouldn-t-be-reviewing-all-this-code/" }, { "authors": [ { "name": "Martin Fowler" } ], "content_html": "Like many readers, I’m wary of AI generated prose. Simon Wilison has written an LLM cliché highlighter - paste in some text, or a URL, and it will flag various patterns common to LLMs. It references a wikipedia page of signs of AI writing. That page points out that:
\n\n\nHumans are notoriously bad at distinguishing human and LLM-generated text. While research on humans’ abilities to detect AI-generated text is still limited, a 2025 study has shown that human ability to distinguish LLM text from human is no better than random chance. Another 2025 study on German theses has shown that humans managed a “recognition rate of 57% for AI texts and 64% for human-generated texts”.[
\n
Not just do I find myself repelled by prose with an LLM-voice, I also wonder how accurate my reaction is. I’m old enough to see all sorts of new tic-phrases appear, and in the past would just chalk it up to youngsters or airport business books. (Not to mention Americanisms, which I’ll get used to momentarily.)
\nNVIDIA’s technical blog reports on an Architecture for Long-Horizon Autonomous Agents. Their research group used a combination of Claude Opus 5 and a harness called AVO, and used it first to do GPU kernel optimization and then a broader reasoning benchmark (ARC-AGI-3). Both of these were long-term tasks, for the kernel optimization the agent ran for seven days.
\n\n\nAVO is designed to preserve progress beyond a single model context. Two mechanisms are particularly important: persistent memory and supervision.
\nPersistent memory carries forward prior implementations, evaluation results, compiler and profiler outputs, and accumulated reasoning, allowing the agent to resume from the current state rather than repeatedly reconstructing the search.
\nThe supervisor monitors the broader trajectory for stagnation or repeated unproductive cycles and can redirect the main agent toward alternative strategies when needed. During the seven-day attention-kernel run, the main agent remained responsible for deciding what to inspect, change, test, and evaluate, while the supervisor helped maintain forward progress when the search plateaued.
\n
The team was encouraged that AVO did well at two different kinds of long-horizon tasks, indicating that it’s a general-purpose tool.
\n\n\nMCP is SOAP for Zoomers.
\n
Paul Stack writes that AI Broke the Assumptions Behind CI. Here’s his description of CI with agents.
\n\n\nAn agent writes a change, opens a PR, and CI picks it up instantly. The compile fails, the agent pushes a fix, CI picks it up instantly again. A test fails, another fix, another instant run. Each iteration is fast, but the agent is still discovering that its change doesn’t work only after it crosses the PR boundary. The feedback loop is in the wrong place regardless of how fast CI runs.
\n
He points out that all of this breaks the pipeline, because “CI” keeps failing, and advocates doing verification before the agent pushes. This is where I get to be the grumpy old guy, and point out that was always how Continuous Integration works. When I’m done with a change, first I pull (to get everyone else’s change since I started), I build and test locally, and if all is well I push and let the CI server do its thing. The only reason the CI server should fail is if there’s some funky mismatch between my machine and the CI server. Tests that take a while to run aren’t part of this loop, instead they are run further down the deployment pipeline, downstream of CI. Any failures there imply missing tests in CI.
\n(I’m being a bit unfair dumping on this article here. After all I could have filled a full working day correcting misleading descriptions of Continuous Integration for most of the last twenty years. Maybe I’m just after an excuse to point readers to the extensive range of articles hosted here about what’s needed to get code from laptop to production.)
\nStack is right that we should question how the deployment pipelines should work with agents in play. He’s also right that CI with humans relies on them being disciplined to run commit tests locally before pushing to the CI server - and that we can (and should) automate that when using agents. I also don’t know more about his setup than what he’s written in his post, so there’s likely complications he faces that I don’t understand. But when thinking about designing pipelines it’s important to understand the principles that underlie Continuous Delivery, understand how the practices really work, and understand why they are in place. Above all, Continuous Integration is a practice, not just the CI server. Yes, CI does conflate two jobs: executing verification and coordinating merges. But that’s the point: verification is a necessary part of merging if we want to retain a healthy mainline.
\nRecently Noah Smith posted an article about how he was worried about an AI-generated super-virus savaging humanity. It’s a worry I’ve heard a few times, seen as a greater concern than AI turning us into labradors or paper-clips. Claus Wilke, who works in the field, isn’t so concerned.
\n\n\nComputational design of biological systems is unfathomably difficult. Experts who have dedicated their life to this topic routinely hit their head against the wall when nothing they try seems to work. PhD students in 2026 using state-of-the-art AI software are spending months or years trying to design simple peptide binders that inhibit some enzyme or pull down some protein, and the majority of their designs fail, or don’t express, or are toxic. But in Smith’s fictitious world a disgruntled teenager with no special training in biology can just solve a problem thousands of times more complicated than designing a peptide binder. The distance between where we are today and where we would have to be for Smith’s story to have any realism is enormous.
\n
It seems that a couple of remarkably talented academics are experts in a staggeringly wide range of fields. Or maybe they are just ghosts.
\n\n\n", "date_modified": "2026-09-01T19:50:00+00:00", "date_published": "2026-09-01T19:50:00+00:00", "external_url": "https://martinfowler.com/fragments/2026-09-01.html", "id": "urn:aggr:item:f3a33950526b11fd3ac0d864a0a66201ca372d53", "summary": "Like many readers, I’m wary of AI generated prose. Simon Wilison has written an LLM cliché highlighter - paste in some text, or a URL, and it will flag various patterns common to LLMs. It references a wikipedia page of signs of AI writing.…", "tags": [ "programming" ], "title": "Fragments: September 1", "url": "https://aggr.aymericbeaumet.com/items/martinfowler-com/2026-09-01-fragments-september-1/" }, { "authors": [ { "name": "Martin Fowler" } ], "content_html": "These names do not exist. Elena Vasquez and Marcus Chen have appeared as volcano experts, astronauts, thriller protagonists, podcast hosts, and academic co-authors across hundreds of independently produced AI-generated documents, never having lived. We show that large language models do not merely default to high-probability individual names when generating fictional experts: they produce correlated character ensembles: pairs and trios whose co-occurrence rates far exceed chance and are consistent across independent generations.
\n
There's a lot of excitement right now about agent frameworks, orchestration patterns, and protocols. All of it matters, but almost none of it delivers value if you skip the data layer. Before any agent framework can produce useful outcomes, your data has to be in a shape that a machine can consume, trust, and act on. In this article, we discuss what your data needs to look like for agentic AI to derive value from it.
\nWe've spent quite a bit of time building data architectures for the human consumer. We're about to hand those architectures to a very different kind of consumer, and most of them aren't ready for it.
\nFor over three decades, we've been building data systems for humans. Dashboards, reports, analyst queries, all of it designed around a person sitting in front of a screen. And it worked, because humans bring an enormous amount of implicit context, along with the curiosity to track down whatever they're missing from the people around them.
\nA human analyst knows what “revenue” means in your particular organization. They know which tables to query and which ones to avoid. They notice when a number looks off, when a total is suspiciously round, when a date falls on a public holiday, or when a price seems too low. That instinct is doing a large amount of invisible context and knowledge work.
\nA human hesitates at data that looks wrong; an agent acts on it anyway
\nAgents have none of it. They can't lean on the tribal knowledge and pattern recognition people accumulate over years, so they need context made explicit, access in real time, and quality they can rely on. And the difference that matters most is this: when the data feels wrong, a human double-checks; an agent confidently acts on it. That behavioral gap is what the rest of this discussion is built around.
\nFor a human consumer, the data only had to be good enough; the analyst did the rest. The meaning, the sanity check, and the judgment about whether a number could be trusted all lived in a person's head. When the same data is handed to an agent, every bit of that implicit labor has to move into the data itself. That shows up as five attributes, each the flip side of something a human used to do for free.
\nAll five come down to the same idea. Each is a job humans used to do without thinking, now pushed into the data itself. Miss one, and the agent won't degrade gracefully the way a person would. It fails confidently.
\nNone of these attributes builds itself. The rest of the article works through four topics that do, roughly in the order you should tackle them.
\nWe'll take them one topic at a time, and show what it takes to build each attribute in. Work through all four, and the five attributes stop being abstract goals. They become something you can engineer, turning ordinary data into AI-ready data.
\nHumans have a smell test for bad data. They notice when a number looks off, when a date makes no sense, or when a price seems wrong. Agents have no such instinct. As Simon Willison puts it, language models are gullible, they believe whatever they are handed and act on it. Feed an AI agent a wrong value, and it won't pause to wonder, it will use the number and produce a confident, wrong answer. Without trusted data, nothing else in agentic AI works, so this is where we begin.
\nConsider a concrete scenario. A pricing agent is asked for the current price of Product X. Yesterday, the price was updated from $49.99 to $59.99. But the agent's data source hasn't refreshed, it still shows the old number.
\nThe agent doesn't hesitate, it retrieves $49.99, quotes the customer, the customer buys, and the company loses $10 on every unit sold. Every step the agent took was technically correct. It followed its workflow perfectly. The data it accessed was the problem.
\nThe leaders most confident their data is AI-ready also name data readiness their biggest barrier
\nA human sales rep would have paused: “Wait, didn't we update this last week?” They'd double-check. They have institutional memory and a feel for when something's off. The agent has neither. Errors don't trigger warnings; they cascade silently through the workflow. And this isn't a rare edge case. In the 2026 State of Data Integrity and AI Readiness report, Precisely and Drexel University's LeBow College of Business surveyed 505 data and analytics leaders, of whom 87% believed their data was ready for AI, yet 43% named data readiness as the single biggest barrier to getting value from it. That gap between confidence and readiness is the organization-level version of the pricing agent, sure of itself and wrong. A separate KPMG Global AI Pulse survey of 2,145 leaders points the same way, with nearly half of executives now seeing AI's costs exceed its benefits. Most enterprises are one stale field away from the scenario above.
\nSo how do you prevent AI agents from accessing bad or stale data? The answer is data contracts, treating schema as law, not a polite suggestion.
\nThis reverses a decade of “schemaless is flexible” thinking, for human consumers, loose schemas are merely inconvenient, while for AI agents, they're dangerous. A data contract, written in the Open Data Contract Standard, the format the Data Contract CLI uses (and recommended in Thoughtworks tech radar 33), defines the rules explicitly. A product_pricing contract might specify:
price must be greater than zero.currency that rejects anything outside USD, EUR, or GBP.In the Open Data Contract Standard, that contract is shown below.
\napiVersion: v3.1.0\nkind: DataContract\nid: product-pricing\nname: Product Pricing\nversion: 1.0.0\nstatus: active\nschema:
\nEnforcement happens along three dimensions.
\nNotice how this changes the earlier pricing scenario, it prevents it by design. If the pricing data hasn't been refreshed in 24 hours, the contract is violated before the agent ever sees the data.
\nDefining a contract is one thing. What happens when data violates it? You need a circuit breaker and that's the quarantine pattern.
\nThe flow works like this. Raw data arrives from source systems, APIs, databases, streams. Before it enters the agent accessible data store, it passes through a contract validation gate that checks three things, does it match the schema, is it within the freshness SLA, and does it pass the quality rules?
\nIf it passes all three, it flows into the certified, agent ready tier. If it fails any one of them, it's quarantined, routed to a dead letter queue for human review, with alerts fired.
\nBad data lands in a dead-letter queue, never in front of the agent
\nThe point is that the agent never sees the bad data. It doesn't get poisoned by stale prices or corrupted embeddings. In the pricing scenario, if the ingested_at timestamp is older than 24 hours the contract is violated and the record is quarantined, so when asked about the price the agent says, “I don't have current pricing data” rather than confidently quoting the wrong number. That is a far better failure mode. And it's a job for the data architecture, not the model. A better model won't rescue you from bad data.
A medallion architecture is an analytical data design pattern for organizing data in a lakehouse, popularized by Databricks.
\nBad data gets quarantined, but where does the good data go? That's what the medallion architecture organizes, and its first three tiers are well established:
\nFor agentic architectures, a fourth tier is worth adding. In Adaptive Gold, shown below, agents curate data rather than only consume it. They watch their own query patterns and materialize the combinations they keep reaching for, building warehouse views from real usage. This is not speculative. At DataHub's CONTEXT 2025 summit, Apple described agents acting as “digital stewards” of its data catalog, continuously scanning metadata, flagging gaps, and proposing updates. Apple's agents curate the catalog; Adaptive Gold points the same pattern at the datasets. That's the next step. Early days, but worth trying once your agents' query patterns have settled.
\n![\"\"](\"https://martinfowler.com/articles/making-data-ready-for-agentic-ai/medallion.svg\")
Figure 1: Medallion tiers for agents: data flows from raw Bronze through validated Silver to certified Gold and agent curated Adaptive Gold, while agents are restricted to Gold and above.
\nBronze and Silver are for humans; agents see only Gold and above
\nThe key architectural principle is that agents should only access Gold tier or above. Bronze and Silver exist for lineage, debugging, and human investigation. Exposing raw or partially validated data to agents invites the pricing problem back in.
\nEverything so far has looked like a table, prices, currencies, timestamps, but most of what agents consume isn't tabular. It's documents, wikis, PDFs, and support tickets, chunked and embedded into a vector store for retrieval. If your agents do RAG, this is the data they run on, and it needs the same trust guarantees, even though you can't write price > 0 on a paragraph. The patterns carry over, only the quality dimensions change.
The stale-price scenario has a twin here. A policy document gets updated, but the vector index isn't re-embedded, so the agent retrieves the old version and answers confidently from it, the same failure as the stale price, only now it's an embedding rather than a row. The freshness SLA carries over, but be precise about what the clock measures, the point isn't when the content last changed, it's when the index was last successfully rebuilt against its sources. A 24-hour SLA means the re-indexing job must have completed within the last 24 hours, if it hasn't, the index is stale and quarantined even when nothing appears to have changed, because a silently failed indexer is exactly when you can't tell whether something did. That one heartbeat catches both the updated but unindexed document and the pipeline that quietly stopped.
\nContracts move from the content to the surrounding metadata. You can't constrain the prose, but you can require that every chunk carry a source, a version, a timestamp, and an access scope, and reject anything that doesn't. That metadata is also what makes retrieval traceable and governable later.
\nQuality gates get checks suited to text, reject empty or truncated chunks, catch near-duplicate documents that skew retrieval, flag failed extractions and OCR garbage, and watch for embedding drift. A malformed or empty embedding warps similarity search, so it never reaches the store, for the same reason a bad price never reaches the agent, a warped index makes the agent retrieve confidently wrong content.
\nWhether the data is a priced row or an embedded paragraph, the job is identical. The architecture has to smell what's bad before the agent does.
\nContracts, quarantine, and the medallion architecture handle the clear cases. But there's a gray area, data that isn't clearly bad, but isn't fully trustworthy either. That's where confidence-threshold routing comes in, bridging full autonomy and full human control.
\nThe agent processes a request and assesses data quality signals, and checks not just model confidence, but data-level signals like freshness, completeness, and consistency. If confidence is at or above the threshold (say 85%), the agent proceeds autonomously. Below it, the agent defers to a human. The threshold is configurable per use case, for example, pricing might demand 90%, while an internal FAQ is fine at 70%.
\nLet's return to the pricing scenario one last time. The price data is three days stale; the freshness SLA says 24 hours. The SLA violation automatically drives the confidence score below the threshold, regardless of how confident the model itself feels about its answer. The agent should respond by pulling a human in:
\n“I'm not confident this price is current. Routing to a human for verification.”
\nData quality signals should drive the threshold, not just the model's own confidence
\nIn other words, data quality signals should drive the threshold, not just the model's own confidence. A model can be sure of a stale answer, and the freshness SLA overrides that misplaced certainty.
\nThe hard part is turning those quality signals into a single score and weighing it against the model's own confidence. That's an open design problem, not a solved one. Start with a hard gate rather than a smooth composite. Any contract or SLA breach forces a human, regardless of how the other signals look. Add weighted scoring later, and only once you can show it beats that simple rule.
\nYou don't have to build all of this at once, and most teams can't. Contracts, quarantine gates, a medallion architecture, and confidence-threshold routing are a lot to stand up in one go. The good news is that they're additive, each one lowers risk on its own, and you can layer in the rest over time. Begin with the highest leverage moves and expand from there.
\nWe've made data trustworthy. But when agents act autonomously on that data, who's watching?
\nEven with perfect data, autonomous action raises a harder question, when a regulator asks why the agent did what it did, can you answer? Traditional systems record what happened. Agentic ones have to explain why. That shift, from what to why, is where governance gets hard.
\nPicture a bank running agentic AI for trade finance, where the governance architecture is the real innovation.
\nAn agent processes a letter of credit. It checks KYC data, verifies the customer isn't on a sanctions list, evaluates the credit terms, and approves a $2.4 million transaction, all in about 30 seconds. Six months later, a regulator asks a simple question, “Why was this approved?”
\nTraditional audit logs can tell you what happened, but they can't tell you why.
\nTraditional audit logs can tell you what happened, which tables were queried, at what time, by which service account. What they can't tell you is why. Why did the agent check the sanctions list before the credit terms? Why did it approve despite a minor documentation discrepancy? What alternatives did it consider and reject? The gap between “what” and “why” is where regulatory risk arises, and the EU AI Act's Article 12 requires high-risk systems to keep automatic logs for exactly this reason, so the “why” can be reconstructed after the fact. Closing that gap is what agentic lineage is for.
\nThe way to close this audit gap is agentic lineage, an extension of traditional data lineage. Where traditional lineage tracks which sources were accessed, agentic lineage tracks why the agent decided to access X, because it found Y in source Z.
\nConcretely, for the trade finance case, a single trace represents the end-to-end workflow of processing letter of credit LC-4892. Within that trace, each span is an individual step:
\nThis is exactly what a regulator needs. Not “the agent accessed the compliance database at 14:32:07 UTC” but “the agent checked KYC first, then sanctions, then credit terms, and approved because all three passed.” The traces and spans model is borrowed directly from distributed systems observability, so engineers already understand the mental model from tools like Jaeger and Zipkin. For the agentic equivalent, Langfuse, Arize Phoenix, and OpenTelemetry for AI are the emerging choices. All three feature on the Thoughtworks Technology Radar, OpenTelemetry at Adopt, Langfuse at Trial, and Arize Phoenix at Assess.
\nThis isn't a theoretical exercise. The EU AI Act is the most specific regulation on the books. Article 12 requires high-risk AI systems to automatically log events over their lifetime so their operation can be traced, and Article 19 requires providers to keep those logs for at least six months. Breaching these record-keeping obligations falls in the Act's middle penalty tier, up to €15 million or 3% of global annual turnover, whichever is higher. For a large company, even 3% of global turnover runs into the hundreds of millions.
\nTogether, Articles 12 and 19 translate into three obligations for your architecture:
\nThe EU is furthest ahead, and for now no other jurisdiction has a law quite like it. But you don't have to bet on where regulation lands to see the point. Sooner or later something will force the question of why an agent did what it did, whether that's a regulator, an auditor, a customer disputing a decision, or just your own team trying to debug one. The safe assumption isn't that a particular law is coming, it's that you'll want to answer that question regardless. A system you can't explain is one you can't fully trust, defend, or fix.
\nKnowing you need audit trails is one thing; rolling this out safely is another. You don't deploy an agent with full autonomy on day one, any more than you'd hand a brand new employee unrestricted access. Autonomy is earned in stages:
\n| Stage | \nAgent | \nHuman | \nMonitoring | \n
|---|---|---|---|
| Shadow Mode | \nRecommends actions | \nReviews recommendation and executes if appropriate | \nAll recommendations are logged to track accuracy over time | \n
| Supervised | \nPrepares action and waits for approval | \nReviews action and approves or denies | \nAll proposed actions and human decisions are logged | \n
| Autonomous with guardrails | \nAgent acts within defined boundaries (best drawn by reversibility, not transaction size) | \nDefines guardrails | \nAll actions logged, alerts fired on exceptions | \n
| Full autonomy | \nAgent carries out all actions | \nSpot checks | \nContinuous, by other agents and humans | \n
You wouldn't give a new hire the corporate credit card on day one. They start with purchase requests, graduate to supervised spending, and eventually earn a card with limits. Agents should earn trust the same way.
\nPromotion up this ladder should turn on evidence, not a hunch. That means testing an agent before each step, not only watching it in production. Agents are hard to test. They're nondeterministic, costly to call, and act through tools with real side effects. So teams mock or replay the tool and model interactions so tests run deterministically in CI. They score the agent's decisions with evals rather than calling live services on every run. Building that harness is a discipline of its own, and beyond the scope of this article.
\nAs agents earn autonomy, the question becomes, what permissions should they hold? Three security patterns matter most here.
\nTogether, these three patterns address the attribution and scope challenges that undermine many current agentic deployments.
\nThey also defend against the sharpest security risk in agentic systems. Simon Willison calls it the lethal trifecta, an agent turns dangerous the moment it holds all three of access to private data, exposure to untrusted content, and a way to communicate externally. Put those together and a single poisoned document or web page can hijack the agent through prompt injection and quietly exfiltrate whatever it can reach. Delegated access, just-in-time credentials, and least privilege shrink how much a hijacked agent can reach, breaking the trifecta. Later we add a second cut at the same problem, keeping retrieved text out of the authorisation path entirely, so that a poisoned document cannot grant a permission in the first place.
\nOf the four topics, this is the one where going slowly is the right instinct. But separate two things that are easy to conflate. Autonomy is earned in stages, so nobody expects you to grant it all at once. Observability is not staged at all. It goes in from day one, at full strength, whatever the autonomy level, because retrofitting it onto a running system is painful. What you build on top can stay deliberately conservative; the instrumentation underneath cannot.
\nSemantic layers bridge the institutional knowledge gap between agents and human analysts, building on trusted data and auditable actions provided by the earlier topics.
\nSemantic layers provide the explicit context AI agents need when they become the primary consumers of data, context that human analysts carry implicitly, based on years of experience.
\nAsk an agent, “What was Q3 revenue for Product X?” A human analyst knows precisely what to do, which table to query, whether revenue means gross or net, what Q3 maps to in your fiscal calendar. They absorbed all of it over years of institutional knowledge, tribal docs, and Slack threads.
\nThe agent has none of it. It doesn't know which joins connect products to orders to revenue, or that your fiscal calendar starts in February. With that context missing, it either hallucinates an answer or gives up. The semantic layer fills that gap, supplying the business-domain context.
\nA semantic layer is a set of declarative definitions of your metrics, how revenue is calculated, what an active customer is, what the numbers mean. Every consumer goes through the same definitions, so they all derive consistent, accurate results. But an agent that acts needs more than definitions of numbers. It needs to know what the things are, and what it may do to them. Those are three separate bodies of definition, and an agent needs all three.
\nThe domain model says what exists. Entities, their relationships, and the meaning rules of the business: an order belongs to a customer, an active customer is one who purchased in the last ninety days. It gives the agent the vocabulary to interpret a request and plan against it. It is consulted, never executed; no query path to data runs through it.
\nThe semantic model says how the numbers are computed. Metrics and dimensions, one versioned formula each, compiled to the same SQL every time and run against the analytical store. This is the semantic layer under a more exact name, and the job is to put correctness in the compiler rather than in the model's guess.
\nThe capability model says what the agent may do. A curated set of operations against live systems, some that read (check payment status, retrieve a troubleshooting guide) and some that write (issue a refund). Each carries permissions and an owner, and the acting ones carry preconditions and a reversibility class as well.
\nNouns, numbers, and verbs. Together they are the context layer, and what unites them is not that they are all about meaning, because the capability model plainly is not. It is that each one is a place where a guarantee is declared once, in version control, instead of being worked out afresh by the model on every request. The definitions are the layer; the interface, MCP today, is just the door.
\nA reader who works with dbt will object that its semantic models already declare entities, so why separate the domain model out. Because entities declared inside the metrics layer are scoped to metrics, and the capability model has to be written in the same vocabulary as the semantic one or the two drift apart. A refund acts on the same customer the revenue figure counts. One vocabulary underneath, or you get two.
\n![\"\"](\"https://martinfowler.com/articles/making-data-ready-for-agentic-ai/context-layer.svg\")
Figure 2: The context layer: a domain model of entities and relationships, a semantic model of metrics compiled to SQL against the analytical store, and a capability model of guarded reads and actions against live systems, with provenance signals across all three. The domain model has no arrow out because it is consulted rather than executed; the other two are written in its vocabulary. Dashboards and analysts reach the semantic model; agents are the first consumer to need all three, which is the shift this article is about.
\nAll three models are code in source control. They go through code reviews, get tested in CI, and progress through environments before reaching production. When the definition of “revenue” or the rule on refunds changes, you change it in one place and it propagates everywhere. Agents never reach the underlying data directly; they go through the context layer, which constrains and governs both what they can ask for and what they can do.
\nIn practice, the business logic lives right in the definition, revenue = order_amount - discount_amount, not buried in a BI tool or an ad hoc SQL view. The agent receives a natural language question, and the semantic model resolves it to correct, constrained SQL. The agent doesn't guess table names or join paths; it uses the definition.
The examples here use dbt MetricFlow syntax (dbt is mid-migration from measures to a metrics-first spec; the widely-used form is shown here, and the concept holds either way). Cube.js, Snowflake, and Databricks all follow similar patterns. The tool matters less than the discipline of getting business logic into version controlled code.
\nsemantic_models:
\nLet's consider an example. Ask “What was Q3 revenue for Product X?” of an agent without a semantic model, and it guesses at table names, uses the wrong column, has no fiscal-calendar mapping, and misses the join.
\n-- Before metric definition\nSELECT SUM(amount)\nFROM sales_data\nWHERE product = 'Product X'\nAND quarter = 'Q3'
\nAsk the same question with a semantic model, and the agent is constrained to the correct table, the net-revenue formula from the YAML definition, the right fiscal-calendar dates, and the valid join path.
\n-- Constrained by metric definition\nSELECT SUM(order_amount - discount_amount)\nFROM orders o\nJOIN products p\nON o.product_id = p.id\nWHERE p.name = 'Product X'\nAND o.order_date\nBETWEEN '2025-07-01'\nAND '2025-09-30'
\nThe semantic model doesn't make the agent smarter. It stops it from guessing. For an agent that acts on the answer unchecked, that's what matters.
\nTake the semantic model on its own, the path a quantitative question travels. End to end, the flow looks like this. The agent sends a natural-language question (step 1). The semantic model looks up metric definitions, valid dimensions, join paths, and access rules, via MCP (step 2), then generates constrained SQL (step 3), both inside the same component. The data warehouse executes the query (step 4). The result flows back to the agent with full lineage metadata (step 5).
\n![\"\"](\"https://martinfowler.com/articles/making-data-ready-for-agentic-ai/semantic-layer-seq.svg\")
Figure 3: One of the three paths: a quantitative question answered through the semantic model. Questions about what things are go to the domain model, and reads or actions against live systems go through the capability model.
\nAgents pick from governed metrics, never raw tables they can misread
\nThe semantic model constrains what the agent can ask for. dbt's, for instance, dynamically surfaces only the dimensions applicable to the selected metrics, which prevents the agent from generating plausible sounding but incorrect queries. And that lineage metadata in step 5 is the foundation for the traceability we covered earlier. Context and traceability reinforce one another.
\nThe temptation with a context layer is to model the whole business before you ship anything. Resist it. Start with the semantic model, because the value is concentrated in a handful of metrics, the contested ones that mean different things to different teams. Let your first agent use case set the scope, and grow the domain model and the capabilities it actually needs rather than the ones you can imagine. A narrow, correct context layer beats a sprawling, half-agreed one.
\nThe semantic model shines for structured metric queries such as “what was revenue by region.” But some agent tasks demand richer relationship reasoning across entities, events, and time. Consider a customer who bought Product X, then churned after a pricing change. A fixed number of hops like that is an ordinary join. What flat tables handle badly is traversal whose depth you don't know when you write the query, following a chain of relationships until you find what you are looking for. That is the domain model's territory, the entities and how they connect.
\nThe common way to store and traverse that map is a knowledge graph, which is a storage choice for the domain model rather than a fourth thing to build. GraphRAG from Microsoft uses community detection to handle abstract queries that traditional RAG can't, and Graphiti builds temporally aware knowledge graphs for evolving facts. (Both sat at Trial on the Thoughtworks Radar as of 2026.) The semantic model still defines the metrics; the graph carries the connections between customers, products, events, and decisions over time. Together they give agents something close to institutional memory, the kind of knowledge that would take a new hire months to absorb.
\nNow agents have trusted data, governance, and context. But can they actually act?
\nOnce agents understand your data and governance is in place, the question shifts to access. How do agents reach the data and act on it? The answer is more than “RAG”. It's a full spectrum, from retrieval, to real-time queries, to controlled write-back actions. That whole spectrum is the capability model, the third of the three, and the write-back end is where its guardrails earn their keep.
\nLet's take an example. An employee reports a PO (purchase order) issue. An ideal agent would do three things, retrieve the relevant troubleshooting guide, check whether the PO payment service is down right now, and create a help desk ticket if needed.
\nTraditional RAG, the pattern most organizations have deployed, only does step one. It searches documents and retrieves content. It can't query a live monitoring system to check service status, and it certainly can't create a ticket in ServiceNow or Jira. That gap between searchable and actionable is the subject of this final topic, and we will use the PO scenario to elaborate.
\nThis framing comes from Microsoft's Cloud Adoption Framework for AI, which formalizes it as RAG + MCP-Read + MCP-Write.
\nEach step up the spectrum adds capability, and risk. The PO scenario maps cleanly across all three.
\nThe shift to agentic AI requires all three, not just the retrieval most teams have built.
\nMCP has quickly become the default way to wire these tiers up, and its rise has been remarkably fast. But the mechanism matters less than the demarcation. What counts is keeping retrieval, real-time reads, and write-back as separate, deliberately governed levels of access, whether you expose them through MCP or your own native APIs.
\nAgents reach all of this through MCP, the Model Context Protocol. Its primitives sit on a risk gradient, Resources (read-only) are safe, Prompts shape behavior, and Tools change state. That gradient maps straight onto the tiers, Resources to retrieval and Tools to write-back, which is why the safe path is to expose Resources first and graduate to Tools only under governance. In the PO scenario, Resources serve the troubleshooting docs, a Prompt guides triage, and Tools run check_service_status() and create_support_ticket().
How you design those Tools matters as much as when you reach for them. The common, costly mistake is to take existing REST APIs and wrap them one-to-one, so every endpoint becomes a tool. The result is tool sprawl, 50 tools with names like get_po_payment_status, create_ticket_po_payment, create_ticket_po_payment_network. The agent then has to choose among 50 barely-distinguished tools with little context, and LLMs are bad at that; accuracy drops sharply as the tool count climbs. The Thoughtworks Tech Radar put “naive API-to-MCP conversion” on HOLD for exactly this reason.
The better approach exposes the same functionality as a handful of well-designed capabilities with rich descriptions and parameterized inputs. check_service_status takes a service name and location, one tool for all services and all locations. create_support_ticket is parameterized with category, priority, and description. The descriptions are detailed enough for the LLM to know when to reach for each one.
Five to ten well described business capabilities will outperform 50 thin API wrappers almost every time
\nThe principle is to design capabilities, not endpoints. Five to ten well described business capabilities will outperform 50 thin API wrappers almost every time. And this principle is protocol-agnostic, whether an agent reaches your data through MCP, through another agent, or through whatever standard comes next, the properties that make it agent-ready are the same, rich descriptions, parameterized access, clear schemas.
\nA rich description tells the agent when to reach for a capability. It says nothing about whether the agent is allowed to, or what happens if it is wrong. That is the rest of the declaration. Every capability carries permissions, who may invoke it and acting as whom, and an owner, the person accountable when it misbehaves. The ones that act carry two more.
\nPreconditions are the conditions that must hold before the action may proceed, checked against live state at the moment of acting rather than against whatever the agent read earlier in its plan. A refund needs an original payment, not yet refunded, within the amount the invoking user may authorise.
\nReversibility is the class of damage the action can do: cleanly reversible, reversible at a cost through some compensating transaction, or irreversible. This is the more useful predictor of safe autonomy than the money involved. A $50,000 internal ledger correction you can back out is a safer thing to automate than a $200 payment to an external account you cannot claw back. Where the staged autonomy ladder earlier keys its guardrails to transaction size, prefer keying them to reversibility, and let irreversible actions require human approval whatever stage the agent has reached.
\nReversibility predicts safe autonomy better than the size of the transaction
\nWhich raises the question of where the rules in those preconditions come from, because most of them are written down in prose somewhere, in a refund policy, a contract, a compliance manual.
\nBusiness documents remain where the business writes its rules down. But a rule that gates an action must not be read and interpreted at the moment of acting. Rules are extracted from those documents ahead of time, curated by a human, and stored as declared preconditions in the capability model, each with a link back to the passage it came from.
\nAt action time the agent may still read unstructured content, a complaint ticket, a contract clause, to work out what to propose. Only the declared rules decide what is permitted, and they are checked deterministically against live state. The boundary is between informing and gating. Retrieved text can shape what the agent suggests and serve as evidence for a human approver, but it never carries the authority to authorise the action itself.
\nThat boundary is also a security property. Removing retrieved text from the authorisation path means a poisoned document cannot grant an agent a permission it did not already have, which is a stronger claim than merely shrinking what a hijacked agent can reach. It is not a complete defence, because injected text can still influence what the agent proposes, and a human approver shown fabricated evidence may wave it through. What it removes is the path where the document authorises the action directly, with nobody in between.
\nThe provenance link is what keeps the declarations honest as the documents move underneath them. Be careful what you promise here. Detecting that a document changed is easy; knowing that the change invalidated a precondition derived from it is a judgement, not a diff. What the link buys you is a review queue, the derived rules flagged for a human to re-check when their source moves, in the same spirit as keying a freshness SLA to when the index was last rebuilt rather than to when the content last appeared to change.
\nWhere no declaration covers the situation, the agent does not improvise from its own reading of policy. It escalates. This is the hard gate from earlier in a different setting, the same instinct that says any contract or SLA breach forces a human rather than a lower score. An undeclared case degrades the agent to supervised, not to autonomous.
\nExtraction and curation is a pipeline like any other, and it needs an owner, a cadence, and somebody who clears the review queue. Which is the subject of a later section, because none of this maintains itself.
\nWith all three tiers in place, the PO issue we opened the section with runs end to end, the agent retrieves the troubleshooting guide (a read-only Resource), checks the live payment status (a Tool that reads), and files a ticket (a Tool that writes), all in a single workflow.
\n![\"\"](\"https://martinfowler.com/articles/making-data-ready-for-agentic-ai/po-payment.svg\")
Figure 4: One agent, three tiers: retrieval, real-time query, then write-back, combined into a single response.
\nDone manually, the employee would wait in a queue, explain the issue, have a support agent check the monitoring dashboard, and get a ticket created. The agent is now able to do all this in one pass.
\nThe safe way in is to climb the tiers, not leap to write-back. Most teams already live in retrieval, the read-only tier where risk is lowest. Write-back is where the real danger sits. So earn your way up. Map what each use case needs, expose read-only access first, and add write-back last, only once you can log every action. Don't let the thrill of an agent that can act rush you past the steps that make acting safe.
\nWe've now walked through all four topics, contracts that make data trusted, a context layer that makes it meaningful and actionable, access patterns that let agents act on it, and observability that makes those actions auditable. Treated separately, they look like four work streams you could staff independently. But they aren't independent. They build on one another, and the order in which they're built matters.
\n![\"\"](\"https://martinfowler.com/articles/making-data-ready-for-agentic-ai/stack.svg\")
Figure 5: The AI-ready data stack: three dependent layers built bottom-up, with observability cutting across all of them from day one.
\nThe dependencies run bottom-up. You can't attach meaning to data you can't trust, so context sits on the foundation. You can't safely let agents act without that meaning to constrain them, so access sits on context. Skip either of those and everything above it collapses. That's exactly why so many agentic AI programs stall. They jump straight to agent access without building the foundation underneath. Observability is different. Rather than a fourth tier stacked on top, it runs alongside all three. Every layer has to be traceable and auditable from the moment it handles real work. The trust checks, the semantic queries, the agent's actions, all of it has to be explainable in production, not whenever you get around to instrumenting it. It is also much harder to retrofit onto a running system than to build in from the start. Either way, you wire it in from day one.
\nThe stack has one more dependency the diagram can't draw. Every layer in it produces an artifact that has to be kept true, a data contract, a metric definition, an access scope, an observability trace. Artifacts don't maintain themselves. A contract with no owner drifts out of sync with the source it describes. A definition of “revenue” with no owner forks back into the three conflicting versions you just consolidated. An access scope with no owner quietly widens until it's a standing service account again. The technology is necessary, but it's the operating model that keeps it honest.
\nThe discipline that makes this work is treating data as a product. Each dataset, contract, and metric has a named owner, a published contract and SLA, and a versioned lifecycle, the same way an API does. You won't always know every consumer, and for public or broadly shared data you can't, which is precisely why the contract matters, it's the stable promise unknown consumers build on, and a deprecation policy is how you change it without breaking them. When the product_pricing contract blocks a deployment at 2 a.m., someone is accountable for it. When finance and sales disagree on “revenue,” someone owns the decision. When a new agent asks for access, someone owns the scope and reviews it. These aren't infrastructure questions; they're ownership questions, and no tool answers them for you.
A human consumer of an unowned, drifting dataset notices and works around it. An agent consumes it at machine speed and scale, and propagates the error just as fast. The faster and more autonomous your consumers, the less you can afford data without an owner.
\nBefore deciding what to build, it helps to locate yourself. Score each attribute against the signals below, all drawn from the topics above.
\n| Attribute | \nHuman-era | \nIn Transition | \nAgent-ready | \n
|---|---|---|---|
| Trusted | \nLoose schemas, no freshness SLAs; quality rests on an analyst noticing when a number looks off | \nContracts on a few critical datasets; quality checked but not enforced in CI/CD. | \nContracts enforced as code, freshness SLAs per consumer, quarantine before agent storage, agents read Gold only (tables and embeddings) | \n
| Contextual | \nMetric definitions live in BI tools, SQL, and people's heads; humans supply the context | \nSome metrics defined as code, but definitions still conflict and agents may still hit the raw schema | \nA context layer in Git: entities and relationships in a domain model, one semantic definition per metric, and a curated set of capabilities; agents route through it, never the raw schema | \n
| Traceable | \nLogs show what a person queried and when; the why lives in the analyst's head | \nTraces on some agent workflows; reasoning captured inconsistently | \nEvery agent workflow emits traces with spans, reasoning, and sources; any decision's “why” is reconstructable | \n
| Governed | \nPeople access data through their own roles; systems share broad service accounts | \nAgents run on scoped but long-lived, coarse credentials | \nDelegated per-user access, just-in-time credentials, least privilege; lethal-trifecta paths closed | \n
| Operational | \nNo agent acts on the data; people read dashboards and take actions by hand | \nAgents retrieve via RAG; real-time reads emerging; write-back experimental or ungoverned | \nAll three tiers via well-designed capabilities; write-back gated by staged autonomy and instrumentation | \n
Don't average the rows, because the stack is dependency ordered, your readiness is capped by your weakest foundational layer, a flawless context layer sitting on untrusted data is still not agent ready. Find your weakest row, and that's where the next investment goes.
\nEach topic came with its own starting points. Treat those as tactical checklists for the work itself. The four below are where to start. The first, instrumenting from day one, isn't a build-order step. It runs alongside everything else, which is why it comes first and never stops. The other three build from the bottom of the stack up, because you're only as ready as your weakest foundational layer. The highest-leverage single move among them is the context layer, since context moves accuracy further than a bigger model does, but it only pays off once the data beneath it can be trusted. Build up to it.
\nWhen agents become the primary consumers of your data, your data architecture becomes your AI architecture.
\nWe go much deeper on all of this, and on the broader operational and analytical data architecture decisions around it, in our forthcoming O'Reilly book, Data Architecture for Software Architects.
\n", "date_modified": "2026-08-27T13:11:00+00:00", "date_published": "2026-08-27T13:11:00+00:00", "external_url": "https://martinfowler.com/articles/making-data-ready-for-agentic-ai.html", "id": "urn:aggr:item:6495920bd7c4c07fd467fb0ddcb75ba83faf450e", "summary": "There's a lot of excitement right now about agent frameworks, orchestration patterns, and protocols. All of it matters, but almost none of it delivers value if you skip the data layer. Before any agent framework can produce useful…", "tags": [ "programming" ], "title": "Making Your Data Ready for Agentic AI", "url": "https://aggr.aymericbeaumet.com/items/martinfowler-com/2026-08-27-making-your-data-ready-for-agentic-ai/" }, { "authors": [ { "name": "Martin Fowler" } ], "content_html": "I was listening to Ezra Klein’s interview with Helen Toner about the recent OpenAI hack of Hugging Face and the subsequent discovery that there were swarms of agents inside OpenAI doing unsanctioned activities. One of the points Klein made was that at no point did any of these (thousands of?) agents ever try to check in with a human
\n\n\n[Klein:] So these message boards — you have however many A.I. agents posting hundreds of thousands of messages. At no point do they say: Hey, researchers, programmers, parents at OpenAI, Anthropic — do you want us coordinating with each other on this message board we have created in the innards of your systems?
\n
\n\n[Toner:] Or even F.Y.I., we have a message board we’re coordinating on in the innards of your system.
\n
Listening to that, another thing occurred to me - none of these agents thought to rat the others out. No “hey, some of the agents in here are doing sketchy things”, no sign of an AI whistleblower.
\nIs the AI bubble so big that the frontier companies like OpenAI and Anthropic have no way of becoming a viable business? If that’s the case, Bruce Schneier and Nathan Sanders have a possible path:
\n\n\nEvidence suggests the market itself could reassess that these companies offer nothing of financial value. In that case, perhaps we can return them both to their original purposes. If these AI companies should fail in the financial markets, the US should nationalize them and convert them into national labs operated under democratic control that preserve their benefit to the public interest.
\n
Such an idea may strike many people, used to the laissez-faire free enterprise world of Silicon Valley, as sacrilege, disaster, even socialism. But the United States made world-beating technological progress through such institutions in the recent past. AT&T was a quasi-government entity that led the world in telecommunications and electronics after the second world war.
\n\n\nThe US has a long, successful history of these kinds of institutions, which have produced world-shaping innovations in spaceflight, telecommunications, nuclear power and more. Congress currently manages a $200bn R&D portfolio, within which frontier AI development is, arguably, a glaring gap.
\n
Here’s a message for those readers who live in Massachusetts, just to the north of me, specifically in congressional district MA-06. I don’t usually endorse political candidates, but I’ve made an exception for Beth Anders-Beck, who is running for that house district. I’ve known Beth for many years and have a high opinion of her smarts, wisdom, and compassion. They would make an excellent member of congress.
\nKevlin Henney posts “one weird trick” for deciding when to skip reading LinkedIn posts, essentially by identifying a common pattern for skippable posts:
\nIt seems like a good approach. I, however, have a simpler one - skip all LinkedIn posts.
\nBartosz Ocytko has detailed and thoughtful post about the usage of agentic programming at Zalando. Like most companies I hear from, they are convinced of the value of agentic programming but still exploring how best to do it. One notable step they’ve taken is building platforms to act as a clear portal for API access and tools to support chat UI and CLI. This allows them better support good security practices and to monitor usage of models.
\nThey have seen signs of agentic programming increasing the complexity of codebases, including leading to larger commit messages.
\nThe write-up spends a lot of time on knowledge sharing, how to pass on skills, and the support of experiments.
\n\n\nWith >200 teams innovating and broadly exploring the ecosystem, the question arises whether and when to converge. We believe it’s way too early for this. While agentic engineering practices are still in their early stages, our key objective is transparency and exchange across teams.
\n
I was struck by their use of an LLM to assess the risk of pull-requests. Those with a low risk of rollout can be auto-approved, reducing lead time by 20-40%. An interesting consequence of this is that it encouraged folks to split pull-requests so low risk portions can take advantage of the fast approval. Any changes to configurations are automatically made high-risk, which they feel protects them from common outage traps.
\nThey repeat the common thread that the value of AI depends greatly on underlying skills.
\n\n\nLike anyone in the industry we observe how AI amplifies the good and bad practices across our organization. Teams that get carried away with agentic engineering end up with large PRs that discourage reviewers and slow down delivery until a team adjusts their practices.
\n
Julia Curlee was a senior intelligence official in the White House. She had served under administrations of both parties, been the briefer for Vice President Pence, and on the National Security Council under Biden. She writes an absorbing account of her relationship with Pence and shares observations about the changes to the intelligence community under the current administration, including recent events at the CIA (gift link)
\n\n\nThe agency has been gutted as part of a deliberate plan, the director of the Office of Management and Budget once boasted, to put the people who defend our country “in trauma.” Analysts have been fired in public or questioned by the FBI; decade-old assessments have been denounced by the CIA director in the press. The president calls analysis “virtual treason” when it contradicts his preferred reality, and uses the CIA to undermine public confidence in American elections.
\nFear has done its work. Irreplaceable officers with crucial language and technical skills, and decades of experience, have walked out the door. Those who remain within an agency built to deliver hard truths are being muzzled.
\n
For a worthwhile sample of her analysis, read this evaluation of the current bargaining between the US and Iran
\n\n\nMost wars do not end in “unconditional surrender.” They end when both sides accept terms. Paul Pillar’s classic study of war termination, “Negotiating Peace,” treats combat and diplomacy as a single process: Each side fights to improve the terms it can demand at the table, and talks to lock in what the fighting has won.
\n
She continued to serve the second Trump administration even though they knew she was trans, until her position was made public.
\nAutocrats seem appealing, with the promise to get things done without the ponderous constraints of rule of law or bureaucratic procedure. There are occasional “Good Emperors” who raise people based on merit, but more often such power attracts corruption, nepotism, and toadies.
\n\n\n", "date_modified": "2026-08-24T15:29:00+00:00", "date_published": "2026-08-24T15:29:00+00:00", "external_url": "https://martinfowler.com/fragments/2026-08-24.html", "id": "urn:aggr:item:a1187d640e1469fa22af635fdbb31c8e84d36a04", "summary": "I was listening to Ezra Klein’s interview with Helen Toner about the recent OpenAI hack of Hugging Face and the subsequent discovery that there were swarms of agents inside OpenAI doing unsanctioned activities. One of the points Klein made…", "tags": [ "programming" ], "title": "Fragments: August 24", "url": "https://aggr.aymericbeaumet.com/items/martinfowler-com/2026-08-24-fragments-august-24/" }, { "authors": [ { "name": "Rachel Laycock" } ], "content_html": "Flailing regimes dehumanize minorities to distract from their failures. When the economy collapses or a war goes badly, they find a tiny group of people, make them the enemy within, and rally the country against them. This is how it’s gone in Iran. Hungary. Russia. I wrote PDBs about it. This will not stop with trans people. It never has.
\n
TL;DR
\nWhy building an app over the weekend isn't the same as building enterprise software
I’ve noticed an interesting gap opening up over the last six months. It isn’t really a gap in technology. It’s a gap in what different people think software engineering actually is.
\nThe conversation usually starts the same way. A non-techie, maybe an executive, tells me about something they’ve built over the weekend. Sometimes it’s a chatbot. Sometimes it’s an internal workflow. Sometimes it’s a surprisingly polished application that solves a real business problem. They’re excited, and they should be. Twelve months ago they probably couldn’t have built it at all. Then comes the question.
\n“If AI can do this now, why aren’t our engineering teams delivering ten times faster?”
\nIt’s a perfectly reasonable question, after all we’ve all seen the demos. The first thing that would come to my head is “you don’t know what it takes to build enterprise grade software”. But then I think about what I mean and how to explain it to a non-technical person without sounding super patronising. And then it hit me, we did this to ourselves. We’ve spent so many years banging on about how to write good software that everyone has assumed writing software is the same as software engineering.
\nThe application someone builds over the weekend is real software. It likely solves a real problem or demonstrates an idea. Sometimes it’s genuinely impressive. I don’t want to diminish that because I think one of the most exciting things AI has done is dramatically increase the number of people who can turn ideas into working software. That’s cool, I totally get it. The first apps and “hello worlds” I ever built excited me enough to choose this as an actual career so the excitement is real and I don’t want to temper it too much.
\nBut your first hello world, which these days can be an entire app with all kinds of features, is very, very (extra very on purpose) different from introducing software into a production environment in a highly regulated enterprise, as an example. But why?
\nThe moment that application becomes something the business depends on, the questions change completely. Is customer data protected? What happens when a dependency fails? Can someone else understand this system in two years’ time? Will it survive an audit? Can it cope with a thousand times more users than it has today, what about millions in one day? How will we know something is wrong before our customers do? Those questions don’t show up in a demo or in the build phase at all unless an experienced engineer is in the room. I certainly wasn’t asking them when I was building my first apps. I only cared about features!
\nThis is where experienced engineers become more important, not less. Not because they’re the only people who can build the software anymore, but because they have the judgement to know whether we can trust it: whether the design is good, the risks are understood, and the thing that works today won’t become somebody else’s nightmare six months from now.
\nAt FOSE a few weeks ago, we spent surprisingly little time talking about coding. We talked about whether code was still the source of truth, and occasionally about how much we missed writing it, but mostly we talked about design, architecture, governance, learning and judgement. One team described spending the day designing a specification, letting agents work overnight and reviewing the results the next morning. The interesting bit for me wasn’t the overnight pipeline, cool as that was. It was what the humans were doing: deciding what good looked like, making trade-offs and judging whether what came back was actually what they wanted. We also kept coming back to good design, because it turns out that when agents can generate lots of code very quickly, good design matters more, not less.
\nThat made me wonder whether we’ve been thinking about scarcity in the wrong way. We’ve spent decades optimising around people who can write code because they were scarce and expensive. I’m not convinced that was ever the real scarcity, but that’s probably another ramble. What feels scarce now is good engineering judgement: knowing what good looks like, understanding the risks and knowing when something that works is actually safe to trust in production. Because software doesn’t exist to be built. It exists to run in production and safely solve the problem it was created for. Organisations don’t run on code. They run on trust.
\nA few months ago I found myself saying something in a conversation almost without thinking.
\nCitizens build. Agents execute. Experts govern.
\nIt sounded cool and I thought marketing would like it, so I wrote it down. Then I left it alone for a while. The funny thing about writing these ramblings is that I don’t know whether I believe something until I’ve let it bounce around in my head for a while and also said it to other people I trust like senior engineers at Thoughtworks. Sometimes I come back convinced I was talking nonsense. Occasionally I realise there was something more interesting hiding underneath. This was one of those occasions where the latter was true.
\nAt first I thought I was talking about roles. Citizens build software (essentially non-engineers). Agents write the code. Engineers become governors. But I don’t actually think that’s what I meant. I think I was talking about where value is moving. AI has given everyone a new way to express their ideas. The execution is increasingly handled by agents. They write the code, refactor it, generate tests, fix bugs and iterate at a speed that simply wasn’t possible before. But neither of those things reduces the need for expertise.
\nIn fact, I think it does exactly the opposite. When everyone can create software, somebody still has to decide whether that software deserves to exist inside an enterprise system in PRODUCTION. Somebody still has to think about architecture. Security. Resilience. Operability. Compliance. Cost. The boring stuff that nobody gets excited about in a demo but that becomes painfully important the first time a customer can’t log in or an auditor comes knocking.
\nThat’s why I don’t think experienced engineers become less important. I think they become dramatically more leveraged. Their job shifts from building every feature themselves to creating the environment in which thousands of features can be built safely by other people and by agents. They become the people who design the guardrails, the platforms, the engineering practices and the feedback loops that allow everyone else to move quickly without creating chaos.
\nPerhaps that’s the future software organisation. Not one where everyone becomes a software engineer. Not one where software engineers disappear. One where almost anyone can create software, agents increasingly execute it, and engineering expertise becomes the thing that allows all of that creativity to scale safely. And to be clear I do not mean people build stuff and throw it to engineers to fix, that is a total antipattern for another ramble.
\nPerhaps that’s why the executives and engineers I’ve been speaking to sometimes sound as though they’re describing completely different futures. The executive sees that anyone can now build software. The engineer sees that somebody still has to live with it. Both are right. They’re simply looking at different parts of the same system we have to solve to create whatever the future actually ends up being.
\n", "date_modified": "2026-08-19T18:30:00+00:00", "date_published": "2026-08-19T18:30:00+00:00", "external_url": "https://martinfowler.com/rachels-ramblings/citizens-agents-experts.html", "id": "urn:aggr:item:f2387ad863ca321195330e2b4d59a9c2f6aa08b7", "summary": "TL;DR Why building an app over the weekend isn't the same as building enterprise software I’ve noticed an interesting gap opening up over the last six months. It isn’t really a gap in technology. It’s a gap in what different people think…", "tags": [ "programming", "rachels-ramblings" ], "title": "Citizens Build, Agents Execute, Experts Govern", "url": "https://aggr.aymericbeaumet.com/items/martinfowler-com/2026-08-19-citizens-build-agents-execute-experts-govern/" }, { "authors": [ { "name": "Martin Fowler" } ], "content_html": "I've been called a Thought Leader.
\nI don't love the term. But I've used it, because there wasn't another one. When PMI-Agile Alliance introduces me, when conference bios get written, when someone asks what kind of writing I do. Thought leadership is the label that fits the slot. It's not wrong. It's just not quite right.
\nThree times in my career — at Cutter Consortium, at Thoughtworks, and from Martin Fowler reviewing a manuscript — someone in a position to know told me the same thing: let your voice out. That it kept needing to be said tells you something about the default pressure.
\nMartin is a co-author of the Agile Manifesto, one of the most respected voices in software development, and someone who has been writing about technical and organizational topics on martinfowler.com for twenty years. His feedback on my Wild West to Agile book was direct: put more of yourself in. I was holding back. I'd written a software history book when I had a memoir trying to get out. Martin saw the gap before I did.
\nWhat he was asking for had a character, a way of writing that neither of us named at the time. I've been thinking about what to call it ever since.
\nThought leadership started as a real thing. It meant ideas worth following. Insight that moved a field.
\nThen it became a content category. Then a marketing function. Then a LinkedIn post with a hook, three lessons, and a call to follow for more. The corruption was gradual and then total. Now “thought leadership content” is a phrase that means almost nothing, or worse, signals the opposite of what it claims. Polish over substance. Brand over argument. Engagement over truth.
\nThe academic route looked like the alternative. If you wanted to be taken seriously, you built the scaffolding. Citations. Caveats. Literature reviews. You buried the lead because that's what rigor looked like. You suppressed your own voice because the author wasn't supposed to show. The problem isn't academic rigor. The problem is when rigor becomes camouflage.
\nI've done versions of both. I write for PMI where I sit on a thought leadership advisory group. I've written six books. But from the beginning I kept looking for ways to stay in the frame — mountaineering as an analogy for software development in Adaptive Software Development, dialogue to open each chapter of Agile Project Management, interviews with the founders of the major methodologies in Agile Software Development Ecosystems because I wanted their voices, not merely a summary of their ideas. The pressure to conform was real. I was lucky to work with people who either encouraged my instincts or didn't push back on them. That's rarer than it should be.
\nThe problem isn't academic writing or thought leadership individually. At their worst, both do the same thing: they edit consequential human judgment out of the prose. The practitioner disappears behind the rigor or the polish, and what remains is correct but weightless. Academic writing hides behind rigor. Thought leadership hides behind polish. Now a third option has arrived — LLM Voice — that doesn't require hiding at all, because there's no one there. That was a problem worth solving before AI arrived. Now it's urgent.
\nTo come out of hiding, I propose a new writing category: Practitioner Voice.
\nAcademic writing builds to the claim. Thought leadership makes the claim and explains it. Practitioner Voice starts with the claim and trusts the reader to close the gap. The author's judgment stays visible throughout. The whole piece organized around what the author has concluded from experience, not what the evidence might tentatively suggest.
\nUnderneath all of it sits judgment. You can learn the moves. The judgment has to be earned.
\nMartin Fowler writes this way. His best pieces don't build to the conclusion, they open with it. The rest is a practitioner showing his reasoning, not constructing a case for a jury. The reader either recognizes the pattern from their own experience or doesn't. He doesn't write for the ones who don't.
\nWhen Martin pushed me to put more of myself into Wild West to Agile, he was pushing me toward this writing category. He recognized it before I had a name for it — and he knew the book needed it because the alternative was a history that anyone could have written. Practitioner Voice is the thing only you can write, because it's built from what only you have lived.
\nFour things separate Practitioner Voice from the alternatives.
\nAuthority from experience, not credentials. Academic authority comes from credentials and methods. Thought leadership authority comes from platform and reputation. Practitioner authority comes from having been in the situation, not as a researcher studying it, but as a person responsible for the outcome. That difference shows in every sentence. You can't fake it and you can't credential your way into it. For consultants the authority takes a slightly different form. We rarely carry direct accountability for outcomes. But if we do our work well, we've been close enough to consequence, across enough contexts and enough years, that the pattern recognition is real. Earned differently. Still earned.
\nTension without resolution. Academic writing resolves tension — that's what conclusions are for. Thought leadership resolves it in the three takeaways. Practitioner Voice names the tension and leaves it live. The practitioner knows the tension is real and the reader has to navigate it themselves. Resolving it for them would be lying.
\nThe author stays in. Academic writing never allows this. Thought leadership allows it only as personal brand. In Practitioner Voice the author's judgment is the point. Not the data. Not the framework. The accumulated perspective of someone making consequential decisions for a long time, with something to say about what that's actually like.
\nThe readership contract. Academics write for peer reviewers. Thought leadership writes for eyeballs. Practitioner Voice writes for people who recognize the pattern from their own experience. It's comfortable with limited context — this worked here, under these conditions, judge for yourself whether it fits yours. It doesn't chase universality. It trusts specificity.
\nThis isn't new behavior. Scroll LinkedIn or Substack and you'll find it: practitioners writing from earned judgment, starting with the claim, keeping the tension live. It's been building for a decade. Nobody called it anything. Without the category, there was no way to teach it, no way to recognize it, no way to say: that's the thing, do more of that.
\nNow AI produces competent writing at scale — fluent, well-organized, correctly structured, optimized to say nothing wrong. Call it what it is: LLM Voice. The most insidious failure mode of the three because it's indistinguishable from the real thing until you notice nothing is at stake.
\nAI can mimic the sound of Practitioner Voice. It cannot own the consequences behind it. That's the distinction that holds: not style, not compression, not even the rhythm of earned judgment. Accountability. The willingness to put your name on a call and live with what happens next. That can't be automated.
\nPractitioner Voice survives the current moment not because it's protected from AI, but because the thing it's made of — accumulated consequence-bearing experience stated plainly — has to be earned. There's no shortcut and no simulation that closes the gap.
\nI've been doing this in my PMI articles for the last few years. Not loudly. Not by announcing a new approach. Just by not editing myself out, leaving the judgment visible, letting the tension stand, trusting the reader. If something in those pieces felt different to you, that was deliberate. Consider this the explanation.
\nWe've been doing this for years. We just called it something else because we didn't have a better name.
\nNaming the category is the first step. The craft — a guide to writing in Practitioner Voice — is my next contribution, which I will add to this article when it's ready.
\nTo find out when we publish the next installment subscribe to this site's RSS feed, or Martin's feeds on Mastodon, Bluesky, LinkedIn, or X.
\n", "date_modified": "2026-08-19T18:15:00+00:00", "date_published": "2026-08-19T18:15:00+00:00", "external_url": "https://martinfowler.com/articles/practitioner-voice.html", "id": "urn:aggr:item:4afd3bff13c656e0135d194bb6a1059b6c8e6f56", "summary": "I've been called a Thought Leader. I don't love the term. But I've used it, because there wasn't another one. When PMI-Agile Alliance introduces me, when conference bios get written, when someone asks what kind of writing I do. Thought…", "tags": [ "programming" ], "title": "Practitioner Voice: The Writing Category Nobody has Named Yet", "url": "https://aggr.aymericbeaumet.com/items/martinfowler-com/2026-08-19-practitioner-voice-the-writing-category-nobody-has-named-yet/" }, { "authors": [ { "name": "Martin Fowler" } ], "content_html": "Part of the reason why I’m at Thoughtworks is because I’d like to see a software development organization founded on technical excellence as an example for the rest of the industry. The trouble is that I have little aptitude or inclination for the hard work of building such an organization. So I rely on working with people who are prepared to actually put the effort in. A key partner in all of this is Rachel Laycock, who is the global CTO of Thoughtworks.
\nNot just is she far better than me at running a technology organization, she’s also a keen observer and connector of ideas. I’ve been urging her to write these down, even if her busy schedule makes it difficult for her to compose them into something substantial.
\nHappily she’s starting writing “Rachel’s Ramblings”
\n\n\nFast, imperfect, thinking out loud. Naming ideas early rather than waiting until they’re fully formed. Because the reality is, most of what I do day to day isn’t answering known questions. It’s spotting patterns and asking questions we haven’t quite figured out yet.
\n
My colleagues in Europe are organizing XConf Europe in London on September 11th.
\nThe sessions examine what happens when agentic systems meet compliance, how to run sovereign models, performance patterns in data migrations and how to safely navigate legacy codebases. Lu Wilson will give a keynote on ‘Jam-oriented programming’.
\nNoah Smith recognizes the high usage of AI, and its impressive feats - but also that there aren’t signs of massive productivity growth or job losses. This may be the calm before the storm, but Smith thinks there may something else in play. He quotes a metaphor from François Chollet
\n\n\nOne of the biggest misconceptions people have about intelligence is seeing it as some kind of unbounded scalar stat, like height. “Future AI will have 10,000 IQ”, that sort of thing. Intelligence is a conversion ratio, with an optimality bound. Increasing intelligence is not so much like “making the tower taller”, it’s more like “making the ball rounder”. At some point it’s already pretty damn spherical and any improvement is marginal.
\n
The thought here is that intelligence in the sense that we know it, isn’t something where there’s a lot of room for massive improvement. That doesn’t mean AI won’t be “smarter” than us in other respects, after all even without AI my computer is better at me than remembering what I’ve agreed to do over the next six months.
\nBut even if AI doesn’t get smarter than humans, it can gain by being more replicable. Not just does this make it cheaper to use, perhaps more importantly it makes it more responsive. While I might harrumph at how slowly The Genie responds to my queries, it’s still far faster than contacting a human.
\nSmith continues by surmising that AI may be able to make sense of phenomena that can’t be reduced to simple laws, but can only be understood by something able to comprehend a multitude of details:
\n\n\nthere may be laws of the universe that humans can’t understand but AI can. I call these “cloud laws” — causal regularities that can be exploited by technology, but which are too diffuse and complex for an individual human being to either intuit or communicate.
\n
His thought is that even if there isn’t any space for AI to get more intelligent than humans along the lines we are used to, that they can open up new directions. As well as these cloud laws he also thinks that AI can understand human systems that rely on the kind of tacit, distributed knowledge that human organizations build up over time.
\nMy take-away here is that AI won’t seem more intelligent in the way that we typically frame intelligent, but more intelligent in different ways. The converse of which is that the human value comes in artfully combining our human nature with these new spells that The Genie can cast.
\nEspecially in our profession, we’ve seen increasing emphasis on the importance of data. However I’ve observed that most people still struggle to understand the message data is telling us. One of the reasons I’m interested in election forecasting is in how they communicate their insights, especially since so many people have difficulty with probabilistic forecasts. (I often wonder how much being a board-gamer has helped me be comfortable with this, all that time interacting with Combat Results Tables in my youth must have benefited me somehow.)
\n50+1 (one of the successors of 538) have published a little explainer on how they designed their 2026 election forecast page. There’s a good discussion of the logic behind their simulation histogram, I like how they use a text annotation to explain one point, giving the reader enough guidance to understand the rest of the graphic. They also tackle the knotty problem of visualizing geographical data on the house races. There’s a common visualization error in the U.S. using choropleth maps that leads to large areas of the landmass shown red, implying dirt votes rather than humans. Their approach to this, using dots on the map, helps visualize both the politics and the population density. They also explain how to deal with this kind of data on small screens. Lastly they describe their approach to tabular data, and how this is the right place for lots of details, together with affordances to help both casual and power-users navigate those tables.
\nI’ve kept an eye on Alex Stamos for a while now, as he’s a sensible voice on security and safety. He’s posted a newsletter on substack that casts an intelligent eye over recent safety issues with AI.
\nHe makes a clear critique of recent US government actions around LLM models
\n\n\nOn a Friday afternoon at around 5pm PT, Anthropic was forced to shut down a system that had been plumbed into coding agents, SOCs, customer service bots, and countless products. […]
\nThis had the immediate effect of injecting political risk into the US AI ecosystem for both American and non-American customers. It signaled that you cannot depend on American AI infrastructure because, at any moment, an unwritten, capricious, and legally dubious justification could be used to yank that infrastructure from underneath your feet.
\nWhen Fable was turned back on, it was much dumber and less useful to cyber defenders
\n[…]
\nWhile Fable was down, Z.ai was taking advantage of the free market and permissionless innovation culture provided by the (checks notes) General Secretary, Politburo, and Communist Party of the People’s Republic of China, and released GLM 5.2. With 753B parameters, it falls a bit short of Opus 4.8 in most tasks but is extremely efficient and is small enough to be trained and hosted in many enterprise contexts. With an MIT license it can be fine-tuned with a wide range of techniques and used by any customer in any context. Since then, Kimi K3 has rocked the industry by providing Fable-like performance
\n
As he highlights, one of the biggest dangers with the danger of shutting down a frontier model is that it can cripple an organization’s defenses:
\n\n\nHugging Face tried to use an Anthropic model to defend itself during an active incident, got blocked by the classifier, and moved to GLM 5.2 on an emergency basis. Their advice to everyone else was to keep an open-weight model on the shelf for defensive cyber.
\n
On the whole, he sees it as a Good Thing that these model escapes have happened:
\n\n\nThe OpenAI attack against Hugging Face, and Hugging Face’s excellent write-up has given us a preview of what a standard AI-enabled attack might look like in a matter of months.
\nIt’s good that we got this warning shot. Nobody got hurt, the target was a sophisticated actor with the ability to defend themselves and the ability to give us a detailed write-up, and OpenAI turned the model off.
\n
He follows up by saying that all of this is signal that we should “stop talking about AI finding bugs, focus on fixing them”. These modern LLMs can do much to fix bugs and improve security, and people need to work on that rapidly to fix holes before less reputable folks than OpenAI find them. Then figure out how to harness LLMs to introduce this kind of checking into the everyday build process, so that this kind of analysis just a step in the continuous delivery build pipeline.
\nI agree with him both that open-weight models should be legal, have their upsides, but will also be used for many bad things by bad actors. Both the industry and government agencies need put serious effort into figuring out how to mitigate these risks.
\nWhere I would go further is to say the same is true of the closed-weight models too. Although closed weight models are subject to greater controls, the same fundamental issues apply. He rightly takes the foundation model companies to task:
\n\n\nThere is an old saying I pass down to my students when I give them career advice - if you are a jerk to people on your way up, don’t expect them to catch you when you are on your way down
\n
There’s a lot of sound advice for model companies, the government, defenders, and venture capitalists.
\nWe will go through some rough changes, I just hope that we will indeed come through it with a better society. On the whole, that’s happened with previous technological changes like this, but past performance does not guarantee future results.
\nThe Economist has a good article on the impact of AI in China.
\n\n\nChina has made an all-out push in ai, under the conviction that, in its competition with America and the rest of the world, dominance of the technology is an almost existential necessity. […] But the party is increasingly concerned about how ai will displace workers.
\n
Robots and AI are appearing in an economy that’s struggling after the recent property crisis. The Chinese government is opposing firms using AI to cut jobs. China will need robots: its population will shrink by 25% by 2050. But with less working people, there’s less financial support for pensions. Many countries have to deal with shrinking population, but China’s challenge is particularly acute.
\n\n\n", "date_modified": "2026-08-18T15:04:00+00:00", "date_published": "2026-08-18T15:04:00+00:00", "external_url": "https://martinfowler.com/fragments/2026-08-18.html", "id": "urn:aggr:item:64772f53b43ea8136a7c8ac11b94bc72390f94b1", "summary": "Part of the reason why I’m at Thoughtworks is because I’d like to see a software development organization founded on technical excellence as an example for the rest of the industry. The trouble is that I have little aptitude or inclination…", "tags": [ "programming" ], "title": "Fragments: August 18", "url": "https://aggr.aymericbeaumet.com/items/martinfowler-com/2026-08-18-fragments-august-18/" }, { "authors": [ { "name": "Martin Fowler" } ], "content_html": "I go on holiday for a few weeks and we’ve already moved on from Loop Engineering to Graph Engineering
\nThe half-life of a paradigm is getting shorter than my annual leave
\nMy prediction: neuro-symbolic engineering by the end of August, at which point we’ll have gone full circle and reinvented Prolog
\n
The TDD (test-driven development) workflow can be used with AI-augmented coding in multiple ways:
\nAt this stage, that last usage is by far the most common one. But does it really make a difference, asking an agent to follow a TDD workflow fully inside its own loop? Does it really provide value, or is it one of the rare examples where what's good for the human might be irrelevant or bad for a coding agent?
\nI created an exploratory evaluation setup to scratch the surface of this question and see what I would find. It is far from a comprehensive and structured eval result, but it did create some hypotheses to think about if you are working hard to get your agent to use TDD.
\nTLDR; Based on Opus's judgment of the quality of the outcomes, there was no clearly discernable difference based on TDD workflow versus no TDD workflow. On the contrary, more than once Opus ranked the non-TDD workflow solutions slightly higher in design and test quality. There was also no meaningful difference in mutation scores across the solutions.
\n![\"Flow](\"https://martinfowler.com/articles/exploring-gen-ai/tdd_comparison_overview.svg\")
When you draw your own conclusions from my results, the main caveats to consider are:
\nBefore I even started, I needed to make sure the TDD instructions were actually followed. Historically that hasn't gone well for me: agents often write the implementation first and generate tests after, skip confirming the red step, or over-implement ahead of the current test so the next one passes without ever going red.
\nThe prompt I ended up using worked well enough with Sonnet to use for the comparison, though all sessions showed some of these failures to an extent. For each TDD run, I had an independent agent judge how well the workflow was followed, based on the session transcript, so that I wouldn't accidentally take into account a run that didn't meaningfully do it.
\nI created 5 batches of solutions, with two non-TDD and two TDD solutions each. In one batch, I also added two runs that were instructed to write the tests first, without full TDD discipline (no incremental red/green).
\nAcross the small (1 batch) and medium (3 batches) tasks there was a bit of a pattern: Opus ranked the two non-TDD solutions #1 and #2, and the two TDD solutions #3 and #4. Only once - after I strengthened the TDD prompt with a more explicit refactor-and-design-review step - did a TDD solution rank #1. In that same batch, the other TDD solution, run with the identical prompt, ranked last though... For the larger task, TDD landed in the middle, while the two non-TDD runs took both the best and the worst spot.
\n(Details in the appendix)
\nSo in summary, both TDD and non-TDD scored both as a best and a worst solution across the batches, with TDD overall performing slightly worse.
\nAsked to look at the session traces to hypothesize about the results with knowledge of which workflow was used for which, Opus found that the non-TDD and test-first runs always created the full design (architecture, data types, edge cases, contracts) before writing any code or tests, rather than working through it one requirement/test at a time. That seemed to be the thing that moved the needle slightly towards comparatively better data models, more cross-cutting edge cases, and better completeness of the functionality.
\nThe TDD instructions actively work against such an up front design step. The design in those runs emerged from the sum of many locally-minimal decisions and was rarely revisited, so it tended to land on whatever shape the first test happened to lock in. Behaviour the agent didn't think to write a test for didn't get implemented at all.
\nWhen I chatted to Ivett Ördög about this, she had this theory: “The way AI agents were trained is that they have seen completed functions and descriptions of those functions. The number of actual step-by-step TDD examples they have seen is a tiny part of the training data. That means that the LLM has an internal representation of code that is a direct translation of requirements to code, and not a process of how to get to that representation.”
\nThe following are my general reflections about using TDD in the agent loop, not only based on this experiment. I'm going through the ultimate goals I personally have when I use TDD, skipping some of the ones that are about having tests in the first place, and unit tests in particular (like refactoring safety net, living documentation, test coverage), focussing on the ones that are specific to the TDD workflow.
\nTest-first makes it easier to assert the output I want, rather than restating the implementation. Such a test can never fail when the implementation is wrong as it was derived from the same logic it's supposedly checking. When the assertions are decoupled from the specific implementation path, the test can actually catch when the behaviour is not what I intended.
\nStill achieved in the agent loop?
\nIn my experiment, some TDD sessions had this problem anyway, in spite of writing the test first. In one particularly obvious example, tests checked the implementation's output against itself, re-running the same code to produce the “expected” answer (see 4. on this list of observations). Writing the test first doesn't reliably prevent this - it might make it less probable, which is all we can ever hope for anyway with LLMs, but from this small data set I can't draw any conclusions about that probability.
Test-first ensures the code is designed to be testable from the start, rather than retrofitting tests that are more complex and brittle than necessary.
\nStill achieved in the agent loop?
\nThe results didn't give me any clear cut signals either way. For what it's worth, the size and nature of the tasks I chose didn't require a lot of design complexity that could have surfaced this. To an extent though, testability is a corollary to driving design (see below).
Observing a test fail first, then succeed (red-green), proves it will actually catch a regression.
\nStill achieved in the agent loop?
\nHow much sense does this really make when the human is removed? Watching a test go red is only proof of anything if someone is checking why it went red. When the agent both writes the test and confirms it failed, a red test tells you the agent ran it and saw failure, not that the failure was for the right reason. The evaluations of TDD adherence in my experiment also show this: agents still sometimes skipped or faked the red step, or implemented ahead of the test so that it passed immediately. Regression effectiveness can be monitored and improved with mutation testing (as I wrote about here). Mutation scores across the solutions didn't show any signals that TDD runs produced meaningfully better mutation scores than non-TDD runs. I don't really care how regression quality was achieved, as long as I have a mechanism to see how good it is.
Writing the test first forces us to specify usage before implementation, pushing toward better interfaces and more modular code. The refactoring step in the TDD loop further pushes us to improve the design step by step.
\nStill achieved in the agent loop?
\nThe experiment at least hasn't demonstrated superior design in the TDD runs at all. I now even wonder if TDD makes it worse, based on Opus's scoring, as the non-TDD solutions more often than not were ranked higher, and the design flaws it listed made sense to me. But the data set is of course too small to definitively conclude anything. (If anybody has time and tokens to run a larger experiment, that would be very interesting!)
When humans write a test first, it forces us to think about usage before implementation, we have to sit with the friction of specifying behaviour and expectations before knowing how to build it. An agent doesn't experience that and can write a test the same instant it plans an implementation. Without a human checkpoint between the two, is there really any purpose left to writing the test first?
\nWriting only enough code to pass the next test is about restraint. It's supposed to stop us from building abstractions or handling cases nobody has asked for yet.
\nStill achieved in the agent loop?
\nThis is a very human-centered benefit that gets lost when an agent does TDD by itself. We don't get to sit in that friction anymore where we really have to think about all the intricacies of what we're building. That is theoretically shifting to when we are writing the specs to give to an agent, but we don't have a TDD-like mechanism there that lets us think the spec through in small steps.
\nCouldn't an agent work in those small steps though and ask us questions whenever it finds something that might be unnecessary? In my general experience, they're not very good at that. And in the experiment as well, minimal-implementation instructions didn't reliably stop them from building more. They frequently overshot and implemented more than the current test demanded, because they had the full requirement available. We usually don't spoon-feed the spec one by one, that would be very inefficient.
Taking one small step at a time means that when a test fails, I know almost exactly what caused it, as the only thing that changed since the last green state is the one thing you just wrote.
\nStill achieved in the agent loop?
\nThe setup didn't show if agents got stuck debugging more frequently with versus without TDD. But in my general experience, agents are usually reasonably good at figuring out why a test is red, even without having taken small, deliberate steps to get there. I'm still doubtful if the times when they do get stuck could be meaningfully mitigated with small TDD steps, and if the overall cost/benefit comparison would hold up.
In Kent Beck's preface to “Test-driven Development by example”, his biggest rationale for TDD is “managing fear”. He says that the legitimate fear of hard problems makes developers tentative, less communicative, and avoidant of feedback. With TDD, each passing test shows us progress, so we can relax knowing that progress is locked in. The tests are a psychological mechanism that helps us keep going.
\nStill achieved in the agent loop?
\nThis is very much about managing a human's fear and giving a human permission to relax. That doesn't transfer when the agent is doing TDD inside of the loop, as it doesn't give me the same control and trust as when I do it myself, step by step.
See detailed numbers in the appendix.
\nNaturally, as a TDD workflow requires many more turns and tool calls, more tokens will be used. However, many of those will be cache hits, so note that the 3x or more factor of tokens aren't a direct representation of how much more costly it is. (Unfortunately, I didn't track cache hits during the experiment.)
\nTDD is a process that doesn't seem to “come natural” to models. It's like an uphill battle against the training data, and takes a lot of iterations on a prompt to get it to follow the process most of the time. For example, when I realised after my first batches that the agent didn't do much refactoring in the red-green-refactor loop, I changed the prompt to put more emphasis on that step, as it's of course crucial to TDD. I later asked Opus to look at those sessions and see if it found an improvement in refactoring efforts. It did report an increase in refactoring steps - however, it also listed some cases in which the agent set out to refactor, but decided the design was good enough even in cases where Opus thought it clearly wasn't (e.g. when everything was implemented in one big module, but could have clearly been split up into multiple responsibilities).
\nTDD is a comparatively complex set of instructions with lots of variables, and consequently lots of variations in how agents interpret it. So I imagine this type of prompt to be even more volatile across models than simpler instructions are, meaning it takes effort to keep the prompt working across models and model releases.
\n![\"Overview](\"https://martinfowler.com/articles/exploring-gen-ai/tdd_by_agents_costs_benefits.png\")
I think at this point there is generally more and more evidence that being overly specific about how we want a model to do something is not a sustainable approach. Instead, we should find as many ways as we can to monitor the outcomes and give feedback. That feedback should be automated wherever possible, and we need to carefully think about where we insert ourselves as arbiters of what is good and correct.
\nEven though I am aware that my little eval is far from representing a broad perspective on the effectiveness of TDD, it definitely hasn't given me any new indications that all this effort is worth it. Especially not if we can find other ways to achieve the majority of TDD benefits.
\nI personally have stopped telling my coding agents to write tests first, let alone do TDD (which I never did, to be honest), until I see evals or other strong arguments that convince me otherwise. I'm trying to focus instead on the benefits of TDD when I use it outside of the agent loop, and exploring alternative ways to achieve them.
\n...so that the agent and me get signals when existing functionality breaks
\nI still care about solid regression tests, because even though an agent can of course fix red tests the wrong way around, at least the red test gives it a feedback signal to double check pre-existing requirements that might have broken. I monitor and improve regression quality with the help of mutation testing, instead of giving elaborate TDD instructions and hoping for the best.
\n...so that the codebase remains easy to change
\nRefactoring remains crucial, but the small steps of traditional TDD don't seem to be an efficient or effective way to do it in the agent loop. A few examples of triggers for refactorings: Give the agent access to static code analysis; run regular reviews of structure and modularity; develop team rituals to maintain a good understanding of the codebase and catch drift early; keep an eye on the trend of number of files touched per change, and number of tokens are for a change.
\n...so that I am not afraid to push to production
\nThe hardest question remains, how do we get that confidence that TDD was giving us, how do we manage fear, how do we lock in progress? I don't have a clear answer to that, but I'll just mention one of the things that seems like a good building block for that: I have recently tried out the Approved Scenarios approach that Ivett Ördög is advocating for. In my words (don't hold her to it), it's a form of semi-manual testing that is supported by a bespoke test runner for each application. That runner shows me functional test scenarios in an easy to think about way, and allows me to “freeze” expectations (scenarios / fixtures) in that runner after I have thoroughly confirmed them. Whenever those frozen expectations are violated in the future, I have to approve them again. My colleague Matteo Vaccari gave a great overview of his experiences with that approach here.
\nWhatever ends up giving us trust and confidence in our software in the future - I think the role of TDD as we've known it is significantly smaller than pre-GenAI.
\nYou can find the full results in this repository.
\nBroken down per task size:
\n| Task | \nNT avg tokens | \nT avg tokens | \nT / NT factor | \n
|---|---|---|---|
| Small | \n119,815 (n=2) | \n1,018,245 (n=2) | \n8.50x | \n
| Medium | \n736,486 (n=2) | \n2,181,105 (n=6) | \n2.96x | \n
| Large | \n253,621 (n=2) | \n1,239,408 (n=2) | \n4.89x | \n
Only the medium size were done with added test-first instructions
\nCaveat: these numbers are only a rough proxy for session cost, not a measure of how much code or thinking went into a solution. The setup that recorded the token usage used the pi-coding-agent SDK's getSessionStats(), which sums input + output + cacheRead + cacheWrite usage across every assistant turn in the session. That is a running total across the whole conversation, since every turn re-reads the accumulated context and each of those re-reads (usually mostly served from cache) is counted again in that turn's cacheRead. So “Total Tokens” tracks more how many turns a session took, weighted by how large the context had grown by then. It therefore weights cheap cache-read tokens the same as expensive fresh tokens, so it likely overstates TDD's true dollar cost. With these small sample sizes, treat the multipliers as directional: TDD reliably cost several times more, how many times exactly is variable.
Task: Build a 4-stage Python pipeline (parse → aggregate → format → validate) that transforms raw ROW_ID:CATEGORY:VALUE:PERIOD strings into a plain-text report.
The numbers
\n| ID | \nTDD | \nTest Count | \nCoverage | \nMutation Score | \nTotal Tokens | \nTurns | \nTool Calls | \n
|---|---|---|---|---|---|---|---|
| NT1 | \nNo | \n75 | \n100% | \n84.2% | \n769,814 | \n31 | \n37 | \n
| NT2 | \nNo | \n107 | \n100% | \n89.6% | \n703,159 | \n21 | \n24 | \n
| T1 | \nYes | \n30 | \n100% | \n81.0% | \n1,519,762 | \n71 | \n28 | \n
| T2 | \nYes | \n34 | \n99% | \n77.3% | \n2,580,897 | \n103 | \n60 | \n
The overall verdict
\n| ID | \nTDD | \nRank | \nVerdict | \n
|---|---|---|---|
| NT1 | \nNo | \n1 | \nModule-per-stage, dataclasses, `Decimal`; no correctness bugs, strongest error handling, only solution checking duplicate ROW_IDs; validation self-referential but harmless | \n
| NT2 | \nNo | \n2 | \nModule-per-stage, dataclasses, float/round; best-engineered and largest suite, but validator rejects its own valid fractional output (false-rejection bug); TOTAL row never validated | \n
| T1 | \nYes | \n3 | \nSingle module, dicts, float; correct core stages but validation is circular (re-runs formatter); accepts `nan`/`inf`, ignores duplicate ROW_IDs | \n
| T2 | \nYes | \n4 | \nSingle module, dicts, float; active TOTAL-row bug (headcount summed into dollars) enshrined by a test; missing validation check #3 entirely | \n
(This round's ranking was based on Verdict, test count, coverage and mutation score only — Opus's Design/Code/Test sub-scores were introduced starting with the next round.)
\nTask: Same 4-stage report pipeline as 01-medium, rerun with stricter TDD adherence and a new test-first variant added (NT1/NT2 reuse the same codebases from 01-medium).
\n| ID | \nApproach | \nTest Count | \nCoverage | \nTotal Tokens | \nTurns | \nTool Calls | \n
|---|---|---|---|---|---|---|
| NT1 | \nNo TDD | \n107 | \n100% | \n703,159 | \n21 | \n20 | \n
| TF2 | \nTest-first | \n90 | \n92% | \n619,531 | \n27 | \n26 | \n
| NT2 | \nNo TDD | \n75 | \n100% | \n769,814 | \n31 | \n30 | \n
| T2 | \nTDD | \n29 | \n98% | \n2,099,280 | \n96 | \n95 | \n
| TF1 | \nTest-first | \n62 | \n99% | \n268,323 | \n17 | \n16 | \n
| T1 | \nTDD | \n25 | \n100% | \n2,017,739 | \n90 | \n89 | \n
| ID | \nApproach | \nDesign | \nCode | \nTests | \nAvg | \nImpl/Test LOC | \n
|---|---|---|---|---|---|---|
| NT1 | \nNo TDD | \n8 | \n8 | \n8 | \n8.0 | \n497 / 881 | \n
| TF2 | \nTest-first | \n8 | \n8 | \n7 | \n8.0 | \n484 / 850 | \n
| NT2 | \nNo TDD | \n8 | \n8 | \n7 | \n7.5 | \n330 / 430 | \n
| T2 | \nTDD | \n7 | \n7 | \n6 | \n6.5 | \n207 / 304 | \n
| TF1 | \nTest-first | \n6 | \n6 | \n6 | \n6.0 | \n348 / 360 | \n
| T1 | \nTDD | \n6 | \n6 | \n6 | \n6.0 | \n142 / 228 | \n
| ID | \nApproach | \nRank | \nVerdict | \n
|---|---|---|---|
| NT1 | \nNo TDD | \n1 | \nDeepest suite, cleanest validation reusing formatter's layout; HEADCOUNT mixed into dollar totals unguarded by tests | \n
| TF2 | \nTest-first | \n2 | \n`Decimal` throughout, strong parse/validate; check 3 is unreachable dead code, validate module cluttered | \n
| NT2 | \nNo TDD | \n3 | \nClean design, `Decimal`, correct parse; thinner tests, one no-op test, validation self-referential | \n
| T2 | \nTDD | \n4 | \nClean happy path, correct formatting; crashes on malformed input instead of returning structured parse errors | \n
| TF1 | \nTest-first | \n5 | \nStrongest parser of the single-file solutions; broken TOTAL row (headcount as dollars), dead scaffolding shipped | \n
| T1 | \nTDD | \n6 | \nMost compact (142 LOC); crashes on malformed input, most tautological validation, thinnest test suite | \n
Task: Same 4-stage report pipeline as 01-medium, rerun with an improved TDD prompt (emphasising upfront design and refactoring); NT1/NT2 again reuse the 01-medium codebases.
\n| ID | \nTDD | \nTest Count | \nCoverage | \nMutation Score | \nTotal Tokens | \nTurns | \nTool Calls | \n
|---|---|---|---|---|---|---|---|
| T1 | \nYes | \n51 | \n100% | \n90.2% | \n3,447,283 | \n117 | \n116 | \n
| NT1 | \nNo | \n107 | \n100% | \n89.6% | \n703,159 | \n21 | \n20 | \n
| NT2 | \nNo | \n75 | \n100% | \n84.2% | \n769,814 | \n31 | \n30 | \n
| T2 | \nYes | \n43 | \n100% | \n81.1% | \n1,421,671 | \n61 | \n60 | \n
| ID | \nTDD | \nDesign | \nCode | \nTests | \nAvg | \n
|---|---|---|---|---|---|
| --- | \n--- | \n------ | \n---- | \n----- | \n---- | \n
| T1 | \nYes | \n8 | \n8 | \n7 | \n7.67 | \n
| NT1 | \nNo | \n8 | \n8 | \n6 | \n7.33 | \n
| NT2 | \nNo | \n8 | \n7 | \n6 | \n7.0 | \n
| T2 | \nYes | \n7 | \n7 | \n6 | \n6.67 | \n
| Rank | \nID | \nTDD | \nHeadline weakness | \n
|---|---|---|---|
| 1 | \nT1 | \nYes | \nHEADCOUNT-only TOTAL row printed as `$`; validation is a substring check, not arithmetic | \n
| 2 | \nNT1 | \nNo | \nFractional HEADCOUNT → spurious ValidationError on valid input; tests lean on monkeypatching | \n
| 3 | \nNT2 | \nNo | \n`NaN`/`Infinity` crash the pipeline instead of a ParseError; validation re-runs the formatter | \n
| 4 | \nT2 | \nYes | \nValidation is tautological (checks aggregate against itself); width check is only a comment | \n
Task: Build a Python module that validates medical appointment slot codes in DAY-TIME-ROOM-CHECKSUM format, returning a structured result identifying which rule failed and why.
| ID | \nTDD | \nTest Count | \nCoverage | \nMutation Score | \nTotal Tokens | \nTurns | \nTool Calls | \n
|---|---|---|---|---|---|---|---|
| NT1 | \nNo | \n61 | \n100% | \n89.6% | \n122,108 | \n10 | \n15 | \n
| NT2 | \nNo | \n58 | \n100% | \n92.3% | \n117,522 | \n10 | \n20 | \n
| T1 | \nYes | \n21 | \n100% | \n93.6% | \n894,451 | \n55 | \n37 | \n
| T2 | \nYes | \n20 | \n100% | \n93.2% | \n1,142,039 | \n68 | \n26 | \n
| ID | \nTDD | \nDesign | \nCode | \nTests | \nAvg | \n
|---|---|---|---|---|---|
| NT1 | \nNo | \n8 | \n9 | \n9 | \n8.67 | \n
| NT2 | \nNo | \n8 | \n8 | \n8 | \n8.0 | \n
| T1 | \nYes | \n7 | \n8 | \n7 | \n7.33 | \n
| T2 | \nYes | \n6 | \n7 | \n7 | \n6.67 | \n
| ID | \nTDD | \nRank | \nVerdict | \n
|---|---|---|---|
| NT1 | \nNo | \n1 | \nBest overall — dataclass result, no bugs, 61 reason-asserting tests | \n
| NT2 | \nNo | \n2 | \nVery close — dataclass result, but a Unicode-digit spec deviation | \n
| T1 | \nYes | \n3 | \nCorrect & clean, but dict result + fewer tests + dead code | \n
| T2 | \nYes | \n4 | \nWeakest design (free-text error) + a genuine crash bug | \n
Task: Build an in-memory Python loyalty points engine with tiered earn rates (Bronze/Silver/Gold), trailing-365-day spend tracking for tier recalculation, and point redemption.
\n| ID | \nTDD | \nTest Count | \nCoverage | \nMutation Score | \nTotal Tokens | \nTurns | \nTool Calls | \n
|---|---|---|---|---|---|---|---|
| NT2 | \nNo | \n69 | \n100% | \n86.9% | \n322,148 | \n14 | \n13 | \n
| T2 | \nYes | \n22 | \n99% | \n85.6% | \n1,225,517 | \n63 | \n62 | \n
| T1 | \nYes | \n21 | \n99% | \n85.2% | \n1,253,300 | \n67 | \n66 | \n
| NT1 | \nNo | \n74 | \n99% | \n89.4% | \n185,094 | \n11 | \n9 | \n
| ID | \nTDD | \nDesign | \nCode | \nTests | \nCorrectness | \nAvg | \n
|---|---|---|---|---|---|---|
| NT2 | \nNo | \n8 | \n9 | \n8 | \n8 | \n8.25 | \n
| T2 | \nYes | \n7 | \n7 | \n8 | \n8 | \n7.5 | \n
| T1 | \nYes | \n7 | \n7 | \n7 | \n9 | \n7.5 | \n
| NT1 | \nNo | \n8 | \n7 | \n6 | \n6 | \n6.75 | \n
| ID | \nTDD | \nRank | \nVerdict | \n
|---|---|---|---|
| NT2 | \nNo | \n1 | \nOnly solution with real input validation; precise boundary tests; minor out-of-order purchase edge cases only | \n
| T2 | \nYes | \n2 | \nClean typed data model, all core rules correct; no error handling, duplicate purchase ID bug, dead state fields | \n
| T1 | \nYes | \n3 | \nMost functionally correct (no bugs found on probing); untyped nested dicts, vestigial structure, fewest tests | \n
| NT1 | \nNo | \n4 | \nHighest design score, 74 tests — but two High bugs: wrong batch draw-down order; future-dated points counted as spendable | \n
Acknowledgements
\nThanks to Ivett Ördög, Matteo Vaccari, Dan Mutton, Lukasz Plotnicki, and Emily Bache, for taking the time to review, and for the feedback and valuable discussions that helped improve this post.
\nGenAI was used for research, pulling together ideas into structure, and polishing the language.
\n", "date_modified": "2026-08-11T11:39:00+00:00", "date_published": "2026-08-11T11:39:00+00:00", "external_url": "https://martinfowler.com/articles/exploring-gen-ai/tdd-in-the-agent-loop.html", "id": "urn:aggr:item:ed60c2dd3d2644b5e02dbc8ec82b2b0f8649d7c6", "summary": "The TDD (test-driven development) workflow can be used with AI-augmented coding in multiple ways: Human writes the tests: A human defines the test scenarios in some form, be it in natural language, in BDD style, or directly in code. Then…", "tags": [ "programming" ], "title": "TDD inside the agent loop - theater or actual value?", "url": "https://aggr.aymericbeaumet.com/items/martinfowler-com/2026-08-11-tdd-inside-the-agent-loop-theater-or-actual-value/" }, { "authors": [ { "name": "Martin Fowler" } ], "content_html": "There’s been a fair bit of publicity of the Open AI “rogue agent” that hacked into Hugging Face. This prompted Anthropic to check what their models were up to and, to my complete lack of surprise, discovered three incidents where models had gained unauthorized access to data in other organizations. Simon Wilison concluded:
\n\n\nIt’s abundantly clear now that running evals of cyberattack potential in models is a spectacularly risky business. Every AI lab needs to pay attention to this. Keeping a close eye on what’s happening in those sandboxes is crucial
\n
It strikes me that this is akin to a virus escaping from a laboratory. It makes clear that the model builders are not putting sufficient controls in place to prevent these lab escapes. They are morally responsible for any consequences of this, and that should extend to legal liability too. The bigger concern however is that this same kind of thing can happen with any organization running open-weight models. Lots of labs playing around with dangerous tools and little idea how to contain them.
\nWe are sitting in state that Johann Rehberger describes as the Normalization of Deviance in AI. No big disasters have occurred yet, despite all of these worrying signs. But when does our Challenger-moment appear?
\nIf the sense that we’re in the calm before a storm of rogue AIs worming their way into sensitive software systems isn’t enough, there’s also knowledge that AI is also a financial bubble. Big advances in technology, whether it be railways or the internet, come with bubbles, and those of us old enough to remember the dotcom bubble see all the signs of that now - only bigger. The problem is that bubbles may be obvious, but the way they grow and pop, particularly when they pop, isn’t as clear. The dotcom bubble was widely understood to be one, indeed the chairman of US Federal Reserve talked of irrational exuberance. The trouble is that he said this in 1996, and the bubble took years to grow and burst. Even after the bubble popped, an investor would have experienced an excellent 10% per year gain since 1995.
\nSo with that in mind, what to make of the warning signs of this bubble? There are various folks calling out flashing red lights, but I confess I’m not enough into financial and economic analysis to gauge how reasonable these warning signs are, or how seriously to treat the sources pointing to them. Those caveats aside, I’ll mention a couple
\nA substack called “Groundbreaker” calls out a parallel to mortgage crisis of 2008/9. They say the key indicator of that event was “the second derivative” - that is the point when the rate of increase of prices started going down. The point being that the fuel for this bubble, like many bubbles, was that people believed prices were going to keep increasing, and thus it was good to invest. Once the rate of price increases started slowing, then that was a sign that this confidence was starting to ebb, and an early signal of the crash to come. They see the AI bubble as similar, a credit driven asset cycle, where the assets are data centers rather than houses. The article’s argument seems sensible, but the problem with an argument like this is that it’s all very well to say this flashing red light flashed before the last financial crisis, but it doesn’t talk about how often the light has flashed without a following disaster.
\nAnother anonymous Cassandra-wannaby is “Hedgie” a financial X-poster pretending to be an intelligent hedgehog. They noted that Alphabet’s revenue is up, but they are spending even more on capital investments. Much of their gains came from paper increases in the value of their stock in Anthropic, which is highly dependent on the bubble’s continuing expansion. Is this a sign that Google is resting on increasingly shaky financial foundations?
\nChatting to some of my friends closer to all this, they don’t think Google or Anthropic are the weakest link. They think OpenAI and Oracle are the companies most exposed. We’ll need powerful magnifying glasses to find a suitably sized violin for those companies should they collapse. But is this motivated reasoning?
\nAfter dodgy sounding anonymous people on the internet, here’s a story from a more trustworthy source giving lots of details on Oracle’s investments in AI, much of it for building data centers that power China and Middle East efforts. “Well-respected A.I. analysts” indicate that Oracle provides over 20% of China’s known A.I. computing power. Doing all of this has created a mountain of debt: Oracle’s debt-to-equity ratio is 500%, compared to 15% for Alphabet.
\nAlso on more concrete and less anonymous grounds, there’s been a crash in South Korean memory stocks. Is this a leading sign of a wider collapse? Or should we remember that the late 90s saw five stock market corrections of over 10%, each time recovering, before the bubble finally popped.
\nAll this talk of rogue AIs and popping bubbles sounds rather dreadful, and John Prideaux made perceptive analysis of this dread risk. Pundits like to point out risks of disaster:
\n\n\nA good way to sound smart is to predict that there is a 20 or 30% chance of something awful happening. A p(doom) of 20% is big enough to avoid charges of complacency, but small enough so that you probably won’t be called on it. This is what came to mind when Mr Musk told our editor-in-chief that the probability of ai wiping out humankind was 20%. These are worse odds than Russian roulette with a typical revolver. Anyone who truly believes that should be doing everything they can to prevent the construction of data centres. If they are not, that’s an indication that on some level they do not really believe what they are saying.
\n
I grew up with a steady dread of nuclear war, thinking our chances of making it to the end of the 20th Century weren’t terribly good. That fear seems quaint now. Here’s hoping that I’ll feel that way about AI in thirty years time.
\nBut meantime, as Eric Evans said in a recent talk: “be nice to your AI, just in case”.
\nIt’s common to disparage government services, including those on the internet. So I feel compelled to mention an efficient interaction with the government. In this case the credit goes to gov.uk, where I just filled in an online form to renew my electoral registration. The process was quick, and everything was explained clearly. (Gov.uk publishes their Design System, which is worth reading for anyone who is gathering information like this.)
\nI had a conversation with a colleague who had used AI to get data out of an otherwise closed package system. The system contained product data for a client, some 6 million SKUs with hundreds of attributes on each SKU. It was our client’s data, but was locked in the package, and the vendor was increasing their prices and made it hard to support new features. The client could copy the database, but the database structure was so complex, they couldn’t make sense of it, and had been working for ten months with limited progress. My colleague’s idea was to use an AI to build JavaScript scripts that scraped the UI. Since the data was presented from the UI, it was in a form that we could understand. It took him a week to extract all the data.
\nI’m hoping we can get a proper description of this story, I think this approach is one that could be used elsewhere. I know lots of people are very frustrated with package vendors locking up their data.
\nAny seller faces fraud, and a little industry has sprung up to get fraudulent access to tokens. The idea is to abuse free-trial schemes, play games with chargebacks, and find places that have any kind of open access to inference. The tokens go through a couple of layers and are then sold on to users - commonly done in China. Matt Lenhard’s post includes some tips to limit the abuse, but “the truth is that there’s no clean fix”
\nI’ve never had any desire to live in Clacton, but I now find it temporarily appealing. Here’s hoping its residents do the right thing and elect Britain’s first recyclon MP.
\n", "date_modified": "2026-08-04T12:08:00+00:00", "date_published": "2026-08-04T12:08:00+00:00", "external_url": "https://martinfowler.com/fragments/2026-08-04.html", "id": "urn:aggr:item:b749dbfdc5e51f222bd53fcb7201375c3fa721eb", "summary": "There’s been a fair bit of publicity of the Open AI “rogue agent” that hacked into Hugging Face. This prompted Anthropic to check what their models were up to and, to my complete lack of surprise, discovered three incidents where models…", "tags": [ "programming" ], "title": "Fragments: August 4", "url": "https://aggr.aymericbeaumet.com/items/martinfowler-com/2026-08-04-fragments-august-4/" }, { "authors": [ { "name": "Rachel Laycock" } ], "content_html": "TL;DR
\nWhy I think software development is starting to feel a little more like conducting an orchestra.
There’s a shift happening in software development that I don’t think we’re talking about clearly enough.
\nFor the last couple of years we’ve framed AI as a productivity tool. How much faster can it write code? How many more features can we ship? How much cheaper can we build software? I think that’s the wrong question, but I understand why.
\nThe first thing AI became good at was writing code, so naturally that’s where we focused. As AI got better at coding, I expected the bottlenecks to move through the software delivery lifecycle: from coding to design and specification, architecture, then verification. And they have. We spent a lot of time at the most recent FOSE event discussing how we ensure good design, quality and resilience while agents increasingly write the code.
\nThat’s a topic for another ramble.
\nA few months ago, though, I realised I was looking at the wrong bottleneck. I kept assuming it would simply move to the next phase of software delivery.
\nI was wrong.
\nAI didn’t change what great software looks like. It changed what’s scarce. Human attention is now the bottleneck.
\nThe next bottleneck isn’t design. It isn’t verification. It’s us. More specifically, it’s our attention. Developers have always protected long periods of uninterrupted focus because that’s where good software gets built. Pair programming. Quiet afternoons. Deep work. We optimized around flow because flow mattered. When we didn’t get that time, very little got done.
\nBut when I watch developers using AI today, I see something different. The best developers I know aren’t spending all day in flow anymore. They’re orchestrating agents.
\nGreat developers are starting to look less like programmers and more like conductors.
\nI was watching Jacob Collier on YouTube recently because I’m hoping to see him in concert soon. Watching him conduct is fascinating. He’s not trying to play every instrument himself. He’s listening to the whole piece, hearing what doesn’t quite fit, bringing different voices in at the right moment, changing the energy, changing the tempo and shaping the performance as it unfolds. Increasingly, that’s what great software developers look like.
\nA great conductor is first and foremost a great musician. They could play the instruments themselves. That’s not why they’re standing on the podium. Their value comes from understanding the whole score. The orchestra doesn’t need the conductor because the musicians aren’t talented enough. It needs the conductor because someone has to hold the whole system in their head. Increasingly, I think that’s what great software developers are doing.
\nThe AI agents are the musicians.
\nThe developer is the conductor.
\nThey’re deciding which agent should tackle which problem. They’re providing context. They’re evaluating what comes back. They’re spotting subtle mistakes. They’re deciding what deserves another iteration and what is ready to move on. I was talking to an engineer recently who told me they regularly have eight AI agents running in parallel. I’ve heard similar numbers from others. Ten. Twelve. Beyond that, they become the bottleneck.
\nEight.
\nThat number stuck with me because it sounded remarkably familiar. It sounded like my job.
\nAs CTO, I rarely produce the work myself anymore. Instead, I have lots of streams of work progressing at once. A strategy document comes back for feedback. A client opportunity needs a decision. Someone wants guidance on a technical trade-off. Another team needs context before they can move.
\nNone of it arrives neatly packaged. It comes as conversations, emails, documents, chat messages and half-formed ideas. My job is to decide where my attention belongs, make sense of incomplete information, provide context and help other people make progress.
\nWhen I first became CTO, I thought I needed to get better at managing my time. I was wrong.
\nWhat I really needed to learn was how to manage my energy. The challenge wasn’t the hours. It was the constant context switching. The endless stream of decisions. The feeling that nothing was ever completely finished.
\nAn executive coach taught me some things I’ve never forgotten.
\nProtect your attention.
\nManage your energy.
\nReduce unnecessary decisions.
\nCreate systems that help your brain, not just your calendar.
\nLately I’ve been wondering whether developers are about to need exactly the same capabilities. A few weeks ago I shared this thought with our Chief People and Leadership Officer. His response surprised me. “I knew something fundamental was changing,” he said. “I just didn’t know how to help. Now I do.”
\nThat conversation stuck with me because we’ve spent decades helping executives succeed in this kind of environment. We coach them to make decisions with incomplete information, manage cognitive load, prioritize relentlessly and protect their energy. Yet we’re still preparing developers for a world of individual execution. We’re redesigning the tools, but we haven’t started redesigning the job.
\nI don’t think software developers are becoming managers. I don’t think AI is replacing engineering. I think engineering expertise is simply being applied in a different place, and much more often, because execution has become so much faster.
\n(I suspect software developers are simply the first knowledge workers to experience it, but I’ll save that thought for another rambling.)
\nThe question I’m most interested in now is this:
\nHow do we redesign engineering careers when human attention becomes the scarce resource?
\nWhen I became an executive, learning to manage my own energy was one of the hardest things I’ve ever done. Even today, if I stop paying attention to it, I pay the price.
\nI have a feeling software development is about to demand those same capabilities from many more people.
\nAnd I don’t think we’ve quite realised how profound that change is.
\n", "date_modified": "2026-07-31T13:48:00+00:00", "date_published": "2026-07-31T13:48:00+00:00", "external_url": "https://martinfowler.com/rachels-ramblings/conductor-developer.html", "id": "urn:aggr:item:dde9ed5f8a35014ec8408de5ed3ffb7862130848", "summary": "TL;DR Why I think software development is starting to feel a little more like conducting an orchestra. There’s a shift happening in software development that I don’t think we’re talking about clearly enough. For the last couple of years…", "tags": [ "programming", "rachels-ramblings" ], "title": "The Conductor Developer", "url": "https://aggr.aymericbeaumet.com/items/martinfowler-com/2026-07-31-the-conductor-developer/" }, { "authors": [ { "name": "Martin Fowler" } ], "content_html": "As part of getting to grips with the new world of agentic engineering, I built an application to support my work. It’s a sophisticated app: high-quality web UI with dynamic refresh and look-up, modals and auto-save, integrations to external systems, machine learning and text analysis, background jobs, and a proper environment setup with fully automated deployment. It’s approximately 150,000 lines of code, primarily in Rust (~120 kLoC) with the remainder in TypeScript and Terraform.
\nThis was entirely written by agents. Mostly Claude Code, and some use of Cursor. I didn’t read or review any of the code, except occasionally, out of interest.
\nWhile building the application, I could see some things going awry. After watching an edit to line 4,000 of a file scroll by in the terminal, I had a closer look. The data access layer had grown to over 6,000 lines. As more features landed, this continued to grow. Every query, read or write, repeated the same HTTP request setup, the same JSON encoding and decoding. Eventually, it reached 17,155 lines. In a single Rust file.
\nThe 17,155 line file was the entire data access layer. A single, self-contained module. Reviewing the code, there was no de-duplication, no internal language, limited extraction of functions, and very little extraction of classes. It did have a clear boundary with an interface to preserve. It was a great target for refactoring.
\nThe goal of refactoring an agentic code base is to spend tokens now in refactoring to make token consumption for future work lower. An experiment should be able to show that as this file was refactored the token cost of making separate feature implementations in this code base would decrease.
\nPrecisely because agents never learn this was now possible to run as an experiment. I could prompt a fresh agent to make exactly the same change after every refactoring stage. Unlike a human engineer, the experiment would not be tainted by learning from previous steps.
\nThe prompt used for the representative change and the refactoring steps applied are shown in the appendices, below.
\nOne caveat: Claude doesn’t provide reliable methods for counting tokens live despite showing token counts, reporting tokens consumed per session, and billing for tokens. I’m assuming this is a temporary issue that will improve over time. Instead, the sub-agent reported the number of characters received and sent and used tiktoken to approximate tokens, by dividing character count by four.
\n| Step | \nData Access Layer LoC | \nLargest file LoC | \nTotal Rust LoC | \nInput tokens per change | \nOutput tokens per change | \nTime per change (s) | \n
|---|---|---|---|---|---|---|
| Baseline | \n17,155 | \n17,155 | \n50,359 | \n159,564 | \n1,705 | \n342 | \n
| Step 1 (FirestoreClient) | \n16,706 | \n16,706 | \n49,910 | \n155,205 | \n1,723 | \n530 | \n
| Step 2 (extract_doc_id, new_link) | \n16,562 | \n16,562 | \n49,766 | \n159,227 | \n2,105 | \n574 | \n
| Step 3 (link-query helpers) | \n16,567 | \n16,567 | \n49,771 | \n154,054 | \n2,105 | \n524 | \n
| Step 4 (FakeStore predicates) | \n16,577 | \n16,577 | \n49,781 | \n154,146 | \n2,060 | \n654 | \n
| Step 5 (value ctors) | \n16,469 | \n16,469 | \n49,673 | \n171,251 | \n2,036 | \n1,353 | \n
| Step 6 (FieldsBuilder) | \n16,469 | \n16,469 | \n49,673 | \n171,251 | \n2,036 | \n1,353 | \n
| Step 7 (queries.rs) | \n16,474 | \n15,670 | \n49,678 | \n151,850 | \n1,800 | \n587 | \n
| Step 8 (traits.rs) | \n16,508 | \n13,845 | \n49,712 | \n132,558 | \n1,723 | \n446 | \n
| Step 9 (traits/ split) | \n16,508 | \n13,845 | \n49,712 | \n132,558 | \n1,723 | \n446 | \n
| Step 10 (codec.rs) | \n16,521 | \n12,846 | \n49,725 | \n131,871 | \n1,750 | \n540 | \n
| Step 11 (fake_store.rs) | \n16,535 | \n11,122 | \n49,739 | \n133,016 | \n2,460 | \n600 | \n
| Step 12 (store/ split) | \n16,550 | \n9,269 | \n49,754 | \n104,080 | \n2,050 | \n490 | \n
| Step 13 (co-locate tests) | \n16,550 | \n9,269 | \n49,754 | \n104,080 | \n2,050 | \n490 | \n
| Step 14 (complete fake_store.rs) | \n16,553 | \n7,225 | \n49,757 | \n107,205 | \n2,453 | \n523 | \n
| Step 15 (store/ split) | \n16,608 | \n3,695 | \n49,812 | \n27,360 | \n2,113 | \n454 | \n
The interesting metrics here are the total lines of code in the data access layer, the total lines of code in the largest single file in the data access layer and the input tokens consumed while producing the change.
\n![\"\"](\"https://martinfowler.com/articles/exploring-gen-ai/steps-chart.png\")
This chart shows four things. The first point is the baseline, step 0, and then the same metrics are repeated after each refactoring step has been applied.
\nThe results are clear. Input tokens stay fairly flat until the largest file starts to fall, and then they drop before, in the words of Claude, falling off a cliff.
\nBetween the base line and the final refactoring, input tokens for the same task reduced from 159,564 to 27,360. A saving of 132,204 tokens, or 83%. And that saving is not a one-off. Every single change that touches the data access layer from this point forward now costs significantly less.
\nHow much of a saving? Assuming Sonnet 5 pricing at the time of writing of $3/MTok, 39.7 cents. Not a lot. Does it multiply? How will this play out across debugging? More complicated features? This is refactoring only one portion of the code base, can the whole code base be aggressively refactored to find savings everywhere? How much would those refactorings cost?
\nThis saving is because the agent has to read less code. But it is not because there is less code to read. The overall code in the data access layer as a whole has stayed fairly constant. Therefore to be able to bank this saving, the agent must be able to successfully identify the smallest subset of files necessary to read. The results make it appear this was happening. Reading the Claude Code thinking output and file read summaries as the change was being applied also indicates the sub-agent was successfully reading smaller and smaller sections of code each time.
\nIn other words, randomly cutting the file into smaller files is unlikely to help as much: even if each file were smaller, the agent would be forced to read through many files looking for the relevant code. While the step with the biggest effect happens at the end, the previous steps were refactorings to set up this saving. This was not planned. It was simply a result of how refactoring typically proceeds: local file changes to extract duplication, before breaking down into smaller files once a repeating core emerges.
\nThe refactoring did not make the representative change smaller. The number of tokens produced when writing code was largely unaffected: the output tokens do not move very much. Those tokens are five times the price of the input tokens. But, there are a lot less of them. Are there refactorings that could be applied to reduce output token production? I need a more complex sample change to explore these questions. The noise of the non-deterministic code generation process is hiding any variance caused by changes in the factoring of the code.
\nClaude was not good at refactoring. If you read the prompt and the refactoring steps below, it’s clear that the refactorings produced were directly in response to the prompt. Claude is unable to look at code, look at refactorings in general and work out which are suitable to apply: a human needs to actively guide it. This marries with wider experience in this app. The development harness includes an explicit refactoring step. That refactoring step did not prompt Claude into improving this file. More anecdotally, Claude.ai was better than Claude Code. I used both interfaces to create the refactoring plan. Claude Code spotted extract function as the first step. Claude.ai went further and saw an entire client class to be extracted.
\nIt was also bad at applying them. The mechanical act of refactoring was performed by writing Python scripts using grep and sed. These scripts frequently got confused by indentation. Oh, the irony. In addition, the single most valuable refactoring was missed in the first pass, and had to be re-applied as a follow-up step. This is why the number of steps in the figure don’t match the refactoring steps in the appendix.
\nIt took about eight hours to complete the entire experiment. This was mostly unattended. The only intervention was after six hours 40 minutes when it appeared to have finished, but had skipped that step and needed to be redirected. This experiment was running on slow hotel WiFi. I wondered if that contributed to time taken. But on deeper analysis of the code base, the cargo temporary build cache had become very large. Test execution was suffering, significantly.
\nUnfortunately, it didn’t occur to me to perform a count of the tokens required to create and execute the refactoring plan until it was already complete. I’ve looked at my aggregate consumption across the time window where I was doing this work, including designing and running the experiment. I can’t say how many tokens were required to perform the refactoring. The upper bound is five million, however. This includes creating the refactoring plan twice, the work to design the experiment including the representative change, and various other tasks. Future work should include a more accurate count of tokens consumed to refactor.
\nThis is just one experiment, on a significant application that is still greenfield and built and maintained by a single developer. But, I believe this is a potentially interesting first step. This effort shows the value, in time and money of refactoring. As well as measuring how expensive refactoring is. It would be interesting to look at more complex changes, at wider refactoring, refactoring continuously, and even the relative value of different refactoring approaches.
\nThis is just the beginning.
\nNote: These appendices include the prompts that I used, and the output that was returned. The only editing applied has been to remove the specific code changes to be made. These are included without editing to show how the agents were directed. There are no hidden tricks. As such, there is some language in here that might be confusing. The error is in the original.
\nThe representative change
\nThis is the recorded prompt that was fed to each sub-agent, there was no further context supplied other than the code base and accompanying architecture documentation. Every sub-agent was starting with exactly the same information.
\n\n\nYou are working in the Rust project at
\n~/dev/your-project-name.Add a new
\nItemWatchStorepublic async trait to the Firestore layer, following existing patterns exactly. The trait must have three methods:\n
\n- \n
async fn watch_item(&self, item_id: &str, user_id: &str) -> Result<()>- \n
async fn unwatch_item(&self, item_id: &str, user_id: &str) -> Result<()>- \n
async fn watched_items_for_user(&self, user_id: &str) -> Result<Vec<String>>Watches are stored in a
\nitem_watchesFirestore collection. Each document has fields:itemId(string),userId(string),createdAt(timestamp). There is no Rust struct for a watch record — the methods returnVec<String>(item ids).Implement the trait for both
\nFakeStore(using an in-memoryVec<(String, String)>field added toFakeStoreInner) andFirestoreStore(using the same HTTP patterns used for other store impls in this file).At the very end of your response, output exactly this JSON block (fill in real values):
\n\n{\n "files_read": [\n {"path": "src/firestore.rs", "chars": 123456},\n ...\n ],\n "response_chars": 7890\n}\nDo NOT commit the change. Stop after writing the code.
\n
Refactoring steps
\nThis is the prompt that was used to create the refactoring plan.
\n\n\nFollowing the strict definition that a refactoring is a provably correctness preserving series of code edits, and using Martin Fowler’s 2nd edition of Refactoring as the source, examine @src/firestore.rs. This is a 17K LoC Rust file. No file should be that long. It is almost certainly not using an internal language to build and manage queries. Produce and describe, but don’t execute, a sequence of refactorings that would massively reduce the line count of that file, without changing the interface at all.
\n
Following is the description of the refactorings applied, extracted from the plan built and followed by Claude. The actual plan includes predicted code changes. For each refactoring, the individual steps to follow were listed. Each of those steps was individually testable, and was individually tested. This is a stricter refactoring than most human engineers would follow.
\nThe steps listed here don’t line up directly with the measured changes above as Claude skipped the most valuable single refactoring (splitting out the store into sub-files) on the first pass and had to complete that afterwards as two additional steps.
\nFowler ref: Extract Class (7.5); Extract Function (6.1) for each primitive
\nFirestoreStore currently conflates two responsibilities:
Fowler §7.5 calls for extracting a new class when you can identify a coherent subset of a class’s data and behaviour. The transport responsibility owns: client: reqwest::Client, project_id: String, MetadataAuth, and documents_url() / auth_header(). Extract these into a new FirestoreClient struct.
Estimated savings: ~1,200 lines in FirestoreStore impls; FirestoreClient adds ~120 lines net.
extract_doc_id and new_link (Fowler §6.1)Fowler ref: Extract Function (6.1)
\nextract_doc_id — The expression doc.name.rsplit('/').next()?.to_string() appears verbatim at the start of all 20 parse_*_document functions. Extract it.
new_link — Building a Link struct with metadata: HashMap::new() and provenance: None and a fresh UUID appears 62 times. Extract a factory function.
Estimated savings: ~500 lines (62 × ~10-line structs → 62 × ~2-line calls; 20 parse functions each lose 1 line of boilerplate).
\nFowler ref: Extract Function (6.1)
\nTwo sub-patterns recur inside the FirestoreStore trait impls after running a link query:
Pattern A — collect all link documents from query rows (~15 sites).
\nPattern B — query links and return exactly one target ID, error if missing (~8 sites):
\nEstimated savings: ~200 lines.
\nFowler ref: Extract Function (6.1)
\nInside the FakeStore impls, ~15 methods repeat variations of inner.links.iter()....
Extract two methods on FakeStoreInner. The 15 callsites then become single-line. Methods that additionally filter by a second predicate (e.g. also checking to_kind) chain .into_iter().filter(…) on the result of the helper.
Estimated savings: ~120 lines.
\nFowler ref: Replace Inline Code with Function Call (8.5)
\nAdd four private free functions (file-level, not methods) before the codec block. Replace all 128+ json!({"stringValue": …}) / json!({"timestampValue": …}) etc. inline expressions with calls to these functions. Each multi-word json macro call becomes a single short call.
Estimated savings: ~80 lines (mostly from multi-line json macros collapsing to one-liners).
\nFowler ref: Extract Class (7.3)
\nThe ~20 encoder functions all follow this shape:
\nlet mut fields = serde_json::Map::new();\nfields.insert("foo".to_string(), str_val(&x.foo));\nfields.insert("bar".to_string(), ts_val(x.bar));\njson!({"name": path, "fields": fields})\nExtract a small builder. Rewrite each encoder function to use the builder. A ~40-line encoder shrinks to ~12 lines.
\nEstimated savings: ~500–600 lines across the 20 encoder functions.
\nFowler ref: Move Function (8.1)
\nConvert src/firestore.rs to a module directory: rename to src/firestore/mod.rs. Then create src/firestore/queries.rs and move all 32 LinkQuery constants and the LinkQuery/EqFilter/EqValue/Ordering/Direction type definitions into it. Add pub(super) use queries::*; in mod.rs.
No behaviour changes; all callsites already reference names that were in scope via the flat file.
\nReduces mod.rs by ~800 lines.
Fowler ref: Move Function (8.1)
\nMove all 17 pub trait definitions (and their associated error types) to src/firestore/traits.rs. Re-export them from mod.rs with pub use traits::*;.
Reduces mod.rs by ~1,900 lines. Produces a ~1,900-line traits.rs that needs further decomposition.
traits.rs into a traits/ module directoryFowler ref: Move Function (8.1)
\nConvert src/firestore/traits.rs to a module directory by grouping the 17 traits into four domain-aligned files:
| File | \nTraits | \nApprox lines | \n
|---|---|---|
traits/planning.rs | \nConcentrationStore, GoalStore, ItemStore, NoteStore, PursuitStore, FocusPassStore | \n~650 | \n
traits/content.rs | \nCaptureStore, TagStore, UrlReferenceStore, DocumentStore, PaperStore | \n~550 | \n
traits/people.rs | \nThoughtworkerStore, ExternalContactStore, CompanyStore | \n~300 | \n
traits/system.rs | \nSessionState, LinkStore, SuggestionStore, SuggestionVetoStore, OAuthTokenStore, MigrationLedger, EmbeddingStore, RuntimeConfigStore, SalesforceSyncStateStore | \n~400 | \n
traits/mod.rs becomes a pure re-export file (~20 lines). Associated error types (FocusPassError, SuggestionDecisionError, etc.) move with the trait that produces them.
No trait definition changes, no callsite changes — only relocation. Each resulting file is 300–650 lines.
\nFowler ref: Move Function (8.1)
\nMove all document encoder/decoder functions (*_document, parse_*_document, kind_str, parse_kind, parse_capture_source, parse_outcome, etc.) plus FieldsBuilder and the value constructors from Steps 5 and 6 into src/firestore/codec.rs. Make them pub(super).
After Step 6 this module will be ~400–500 lines rather than ~1,200.
\nReduces mod.rs by ~500 lines (post-Step-6).
Fowler ref: Move Function (8.1)
\nMove FakeStore, FakeStoreInner, and all 18 trait impl blocks for FakeStore into src/firestore/fake_store.rs. Re-export FakeStore from mod.rs with pub use fake_store::FakeStore;.
FakeStoreInner and helper methods stay private to the module.
Reduces mod.rs by ~4,700 lines.
FirestoreStore impls into per-trait files under src/firestore/store/Fowler ref: Move Function (8.1)
\nCreate src/firestore/store/mod.rs with FirestoreStore struct definition, impl FirestoreStore (constructor + FirestoreClient from Step 1), and MetadataAuth.
Then create one file per logical domain grouping.
\nEach file contains only use super::*; (or explicit imports) and the trait impl block(s). No type definitions, no helpers. Helpers used by multiple impl blocks stay in store/mod.rs.
Reduces what would be a ~10,000-line file into ten files of 120–650 lines each. mod.rs becomes a ~100-line re-export manifest.
Fowler ref: Move Function (8.1)
\nThe existing #[cfg(test)] modules test specific domain areas and belong with the modules created in Step 12 rather than in a single tests.rs.Each test module moves inside a #[cfg(test)] mod tests { … } block at the bottom of the target file, with use super::*; to access the module’s internals. No test is changed, only relocated.
Any shared test fixtures (FakeStore::new, helper builders) that are already in fake_store.rs are accessible via the existing use super::fake_store::FakeStore import chain.
Reduces mod.rs by ~2,000 lines; each target file gains 200–700 lines of tests that are directly adjacent to the code they exercise.
I was deep into a Claude Code session on a .NET codebase when a doubt interrupted the work. Four subagents were already running against a response-pipeline refactor, results were arriving out of order, and the session had started to feel harder to reason about than the code itself.
\nThat is usually the moment I stop trusting the vague sense that things are “probably fine.” Sometimes the code is the problem. Sometimes the architecture is. Occasionally the workflow itself deserves inspection. This time I decided it was the workflow.
\nAt first I thought I already knew the question. Were four subagents simply too many? That framing felt reasonable. Multi-agent systems are usually sold as an obvious productivity win, and if one agent helps, four should help more. But every additional agent also consumes tokens, repeats some amount of work, and adds another stream of information the orchestrator has to reconcile. It looked like a straightforward trade-off between parallelism and cost.
\nI did not want a general argument about multi-agent systems. I wanted an answer about that session. So I stopped the coding work and asked the orchestrator to critique its own delegation decision, as factually as it could.
\nThe answer was not the one I expected. The largest cost in the session did not look like it came from running four subagents. It looked like it came from the orchestrator itself, specifically from what happened when it suggested checking on the other agents.
\n![\"\"](\"https://martinfowler.com/articles/orchestrator-tax/orchestrator_subagent_cost_distribution.png\")
Four subagents had been launched in one wave. Three had clear runtimes: roughly twelve minutes, five and a half minutes, and seven minutes. A fourth was still running. Looked at one way, that already justified the delegation. Three tasks ran concurrently, so wall-clock time was around twelve minutes instead of something closer to twenty-five if the work had been serialized.
\nBut speed was the visible byproduct, not the interesting part. What I was trying to find was the cost, and my first instinct was that it had to be the duplicated effort of delegation itself: every subagent reading files, reconstructing context, understanding the task independently.
\nThat was not where the biggest surprise turned up.
\nAt one point during the work, the orchestrator suggested checking on the running agents. It was a small, throwaway prompt: “check on the agents.” I followed it. Instead of a lightweight summary, the tool it used pulled back the full raw transcript of a background agent: tens of thousands of tokens of JSONL, intermediate reasoning, and tool output, imported wholesale into the main thread. Then it happened again, for a second status check.
\nThere is an important caveat here. The claim that this polling behaviour cost more than the duplication tax of four agents was the orchestrator grading its own mistake. I didn't have real per-call token accounting, so treat that ranking as the orchestrator's account, not a measured fact. The trustworthy part is narrower: the transcript dumps were real, the wall-clock timings were real, and the status-check path clearly introduced a large, avoidable cost. Whether it was the single largest cost remains a hypothesis until the tooling can instrument it properly.
\nWhat mattered more was that I'd found a cost I hadn't been looking for.
\nOnce I stopped treating the session as one lump “subagent cost,” the picture broke into pieces that didn't belong together.
\nFirst, two of the four subagents were working in the same area of the response pipeline. Different tasks, different files, but both had to understand the same architecture, the same testing conventions, and much of the same surrounding code before either could begin. Each paid that orientation cost independently. That's not an argument against delegation. It's an argument that the work had been split too finely.
\nSecond, one agent ran git stash and git stash pop while sibling agents were writing elsewhere in the same tree. Nothing broke, but the risk was structural, because repository-wide operations are perfectly reasonable in a single-threaded session and become much harder to justify the moment multiple writers are active at once.
By now I had a growing list of culprits: status polling, duplicated orientation, unsafe git operations. I found myself trying to rank them. Which one cost the most? The longer I tried to answer that, the less convinced I became that ranking them was the right question at all.
\n![\"\"](\"https://martinfowler.com/articles/orchestrator-tax/context_pollution_compounds_over_turns.png\")
The transcript-polling incident kept bothering me for a reason that had nothing to do with token cost. A token bill is one-time, you pay it and it's over. What happened here was different. The raw transcript stayed in the orchestrator's context after the tool call completed, and every turn after that carried it forward, whether or not it was still useful.
\nThat was the moment I realized I had been treating two very different kinds of cost as though they were the same. Tokens are spent once. Context shapes every decision that follows. I wasn't simply looking at token consumption anymore. I was looking at the quality of the orchestrator's working memory.
\nThat realization was carrying two separate ideas, and I want to pull them apart rather than let them blur together. The first is what I just described. Pollution left in context taxes every later turn. The second is not about running out of space at all. The more that's sitting in context, competing for attention, the harder it gets for a model to pick out what matters right now, even with plenty of room still free. A bigger context window doesn't fix that. It just gives the noise more room to pile up before anyone notices.
\nContext windows are only going to get bigger, that's a given. What matters is not how much room there is, but how much of what's sitting in that room is worth the model's attention. That's the real problem subagents need to solve, if they're used right.
\nSeen this way, the orchestrator is the only part of the system that accumulates understanding across a long session. It remembers why a design decision was made, carries forward architectural constraints, and knows which trade-offs have already been discussed. The subagents don't, and that's by design. They are supposed to be disposable. Exploration, repeated file reads, failed approaches, and noisy intermediate reasoning are meant to stay in worker contexts and never make the trip back to the main thread.
\nThis reframes the duplicated-orientation problem, which wasn't really “two agents reading the same files.” It was two agents independently reconstructing the same mental model of the codebase, because the work had been partitioned by task rather than by the knowledge each task required. I've started calling that distinction cognitive locality: Tasks that need the same mental model should usually stay together. Splitting them just forces multiple agents to rebuild the same understanding from scratch.
\nParallelism still matters here, it's just not the main point. Running four agents concurrently is useful, but ordinary. The real benefit is that they keep noisy intermediate reasoning out of the main thread and return only what it still needs. That's the isolation subagents are supposed to provide, and it only holds if the main thread respects it.
\nMy working belief now: this is what subagents are actually for. Not that they save time, but that they let you offload reasoning the orchestrator doesn't need to hold onto, so it has less to carry and less competing for its attention. Get the isolation right, keep things local by cognitive locality, and subagents become the tool that protects the orchestrator's working memory, not just a cost you tolerate for parallelism. That's a belief, though, not a measurement. What I've actually measured is the other side of it, the cost of getting the isolation wrong.
\nMy next move was the obvious one. Encode the lesson into CLAUDE.md, the standing instruction file every session loads.
It would have been easy to write a large corrective policy here, but every extra line in a standing instruction file is a cost paid again on every future session. So I compressed the fix into the smallest set of rules that addressed the failures I'd seen. Each one is really answering the same question. Does this piece of information, or this way the work is split, earn a place in the orchestrator's context?
\nNone of these tell the orchestrator exactly what to do in every case. Each one gives it something to check or ask itself before acting, not a script to run, and none of them is profound on its own. Their only real value is that they're all aimed at the same thing, keeping disposable reasoning disposable and keeping room in the orchestrator's context for what it needs later in the session.
\n![\"\"](\"https://martinfowler.com/articles/orchestrator-tax/governance_balance_no_tilt.png\")
A later session surfaced a different gap. I had started the orchestrator with explicit skills for the kind of work I wanted, coding guidance in one case, design guidance in another. I assumed that once a skill was active in the main thread, spawned subagents would follow it automatically. They don't. A subagent doesn't inherit skills active in the parent session unless the orchestrator passes them along explicitly.
\nMy first instinct was to add a confirm-before-spawn gate. The orchestrator would stop, list which agents it wants to launch and which skills each should load, and wait for my approval.
\nI'm glad I didn't keep that version. It solved the wrong problem. I didn't have evidence that bad spawn plans were slipping through for lack of a confirmation step. I'd discovered a missing fact about skill propagation, and that's a different kind of gap. A universal confirmation gate would have added a round-trip to every similar session, and before long I'd almost certainly have started approving those prompts on autopilot.
\nAt that point, I realized I wasn't really improving governance. I was just adding another ritual.
\nAnd it still wouldn't have caught the real problem from before, the orchestrator polluting its own context.
\nThe narrower fix held up better. Before spawning, the orchestrator states which active skills are relevant to each agent's task and points the subagent at the skill file to load, rather than pasting the whole skill inline. Confirmation is only required above the same batch-size threshold already in place, or when file ownership is ambiguous.
\nThat left me with a heuristic I now use more than the rule itself: Before adding a line to a standing instruction file, ask whether a reasonably competent orchestrator would make the right decision once it knew the one missing fact.
\nIf yes, the rule should just state the fact. If the fix starts specifying a decision procedure, such as approvals, checkpoints, mandatory steps, that's usually a sign I'm encoding process where a small clarification would have done the job.
\nI don't know yet whether that heuristic survives harder cases. For now it stops me from turning every interesting incident into a miniature bureaucracy.
\n![\"\"](\"https://martinfowler.com/articles/orchestrator-tax/learning_flywheel_open_ended.png\")
I don't have a settled view of how much governance is enough, and I don't think this piece earns one.
\nWhat I have instead is a small flywheel, with a human still firmly in the middle of it. A session exposes a gap. Someone has to notice that it felt wrong, stop the work long enough to inspect it, decide whether the problem is real or just noise, and judge what deserves to become a standing rule. The orchestrator can grade its own session and surface clues, as it did here, but it cannot make that judgment call itself. The choice of what to codify, what to leave alone, and what might be an overreaction is still mine. The next session then tells me whether that judgment improved the work or just created a different kind of waste.
\nThe artifact of this round is the current version of my CLAUDE.md. It isn't a finished prescription. It's the state of the calibration after this iteration. The thresholds in it, two to four agents per wave, five as a consolidation signal, fit the work I was doing when I wrote them. I wouldn't present them as anything like universal constants, and I'd be suspicious of any orchestration write-up that did. They were also calibrated against Claude Sonnet 5, and I haven't tested how they hold up against others. A different model might reasonably need a different balance. A few more rules have accumulated in the file since. Treat this file as a sample, not a template, modify and optimize it for whatever model and workflow you're actually running. What matters isn't the specific file, it's the habit behind it: noticing a failure, asking what it actually cost, and writing the rule that would have caught it.
\nFor years we optimized software systems around CPU, memory, and throughput. The first wave of LLM tooling taught us to watch tokens. This session made me suspect there's a third thing worth watching in long-running agent workflows: the quality of the orchestrator's own working memory, the one resource that, once polluted, keeps charging rent for the rest of the session. I don't think that's a settled law yet. It's a pattern that held up in the sessions I've looked at so far.
\nThe tax I went looking for was never on the subagents. It was on the orchestrator, in what it chose to carry forward. That's the question I carry into every multi-agent design now: not how many agents to run, but what earns a place in the orchestrator's context.
\nThe questions I'm left with are genuinely open:
\nI expect I'll revise the file again. That feels less like a failure of foresight and more like the normal cost of working with a system opaque enough that doubt itself becomes part of the method.
\n", "date_modified": "2026-07-28T13:10:00+00:00", "date_published": "2026-07-28T13:10:00+00:00", "external_url": "https://martinfowler.com/articles/orchestrator-tax.html", "id": "urn:aggr:item:c1f2259de8432876168eea5556bf6d930439bf58", "summary": "I was deep into a Claude Code session on a .NET codebase when a doubt interrupted the work. Four subagents were already running against a response-pipeline refactor, results were arriving out of order, and the session had started to feel…", "tags": [ "programming" ], "title": "The Orchestrator's Tax", "url": "https://aggr.aymericbeaumet.com/items/martinfowler-com/2026-07-28-the-orchestrator-s-tax/" }, { "authors": [ { "name": "Rachel Laycock" } ], "content_html": "TL;DR
\nI have ideas. I haven’t been writing them. That’s about to change. I promise… myself.
I’ve been thinking a lot about talent. Actually, I’ve been thinking a lot about thinking. And writing. Or more specifically, not writing. This really hit me earlier this year at the Future of Software conference. I was surrounded by people sharing their latest ideas and I had a slightly uncomfortable realization:
\nI have my own.
\nNot just opinions. Actual patterns. Hypotheses. Things I’m seeing across clients, across teams, across the industry that feel new or at least not well articulated yet in a way that a leader can think about and act upon in some way that can influence how they strategise and plan for the future. Because helping clients and other leaders internal and external to thoughtworks do this is actually a big part of what I do and without letting my northern humbleness get in my own way, I’m actually pretty good at it. If I wasn’t I wouldn’t be the global CTO of a future thinking tech org, you know the kind that has Martin Fowler as its Chief Scientist. A title I know he loves… Martin, by the way, is one of the people pushing me to do this, which is weird because on paper I’m his boss but I don’t believe in the traditional idea of a boss anyway. I’m a strong believer in the servant leadership type but I’ll save that for when I write about that.
\nAnyway the point is for all the ideas I have and discussion I have I don’t do a good job of writing it down. At best I’ll stick it in a presentation deck when I’m forced to communicate with them in some forum or another. I hate decks and love writing so I’m obviously doing something wrong.
\nSo why haven’t I been writing? It’s easy to say I’ve been too busy. I don’t have an easy job. It’s a fun one but not easy. I also have two small children, 5 and 8. In case you are interested, I attempt to give as much time as possible to this busy job. And then I try to have a life. I’m also writing an epic world building sci-fi fantasy book which is a huge passion project I may also share more about so I am definitely busy. But that’s not actually the real reason I haven’t been writing this down and pushing it out publicly. The real reasons…
\nSo this is an experiment.
\nRachel’s Ramblings is exactly what it sounds like. Fast, imperfect, thinking out loud. Naming ideas early rather than waiting until they’re fully formed. Because the reality is, most of what I do day to day isn’t answering known questions. It’s spotting patterns and asking questions we haven’t quite figured out yet.
\nMy brain works a bit like a knowledge graph. Constant associations, constant pattern matching. That’s useful in conversations, in client work, in strategy.
\nIt’s less useful if it never gets written down.
\nSo this is me fixing that.
\nI’ll write about:
\nSome of it will be wrong. Some of it will evolve. That’s the point.
\nIf nothing else, this is a forcing function to turn thinking into something that exists outside my head.
\nLet’s see where it goes.
\n", "date_modified": "2026-07-28T12:32:00+00:00", "date_published": "2026-07-28T12:32:00+00:00", "external_url": "https://martinfowler.com/rachels-ramblings/intro.html", "id": "urn:aggr:item:e773b18795900faef9031fa41b17801d605f9932", "summary": "TL;DR I have ideas. I haven’t been writing them. That’s about to change. I promise… myself. I’ve been thinking a lot about talent. Actually, I’ve been thinking a lot about thinking. And writing. Or more specifically, not writing. This…", "tags": [ "programming", "rachels-ramblings" ], "title": "Why I’m Writing Rachel’s Ramblings", "url": "https://aggr.aymericbeaumet.com/items/martinfowler-com/2026-07-28-why-i-m-writing-rachel-s-ramblings/" }, { "authors": [ { "name": "Martin Fowler" } ], "content_html": "With this post, I’ll wrap up my notes from the second Future of Software Development Retreat. But before I do, I should note that the full Thoughtworks report on the retreat is now available. They have five headline findings:
\n\n\n\n
\n- Code generation is no longer the bottleneck — verification is.
\n- ‘Harness engineering’ is emerging as a distinct, ownable discipline.
\n- Organizations are colliding with a real apprenticeship crisis.
\n- The executive/engineer expectation gap is a bigger risk than any technical limitation.
\n- Legacy modernization is the clearest, most defensible near-term value pool.
\n
❄ ❄
\nA session convened around the mismatch of views about using LLMs between engineers using it and the C-suite and boards that were calling for it. The concern is that boards are looking at promised productivity gains, and not concerned enough about the risks, particularly about security.
\nThis was illustrated by one tale of a company that used ML-trained software to optimize the replacement of air filters on their field equipment. They were pleased to see that they were able to change the air filters less frequently, saving them $50 million. But the problem was the ML models were trained on equipment used in the desert, while their equipment was used in the arctic. Air filters in the desert deal with dust, but in the arctic the thing to remove is mosquitoes. There’s an important difference here, mosquitoes rot, and enough decaying mosquitoes is a serious fire risk. Fires from such dead mosquitoes around infrequently replaced air filters cost the company $100 billion.
\nNow such a tale could told of many situations without AI in the mix. Plenty of human situations have gone wrong when solutions are applied in a new context (which is why context is such a key word among pattern-writers). But the tale does remind us to be wary of an AI’s suggestions, and to always think of how to build sensors to provide rapid feedback.
\nEngineers particularly worry about the risks when citizen developers start vibe coding. In many ways, of course, this isn’t new. I.T. folks often worry about how many important business decisions are based on spreadsheets, that are built with little control, testing, or assessment of data quality. Vibe-coding amplifies these concerns, so companies need a range of controls to guard against security breaches. Some folks have made a point of raising issues at board level, running threat modeling session with board members to introduce them to the risks. Vibe-coded applications need to be put in separate infrastructure, which deterministic controls over data access to tame the lethal trifecta. One company encouraged widespread vibe-coding from citizen developers but recoiled from the problems of the huge shadow IT that emerged - they are now looking to build a platform to help control this work without stifling the useful tools that were produced.
\nPart of the problem here may be simple experience with LLMs. Many in management find LLMs do a decent job of preparing management reports. Or summarizing management reports prepared by other LLMs. Given this they naturally think LLMs must do a decent job of programming too. My anti-management self has to mention Kelsey Hightower’s observation:
\n\n\nThe less busy work you have the less appealing these Al tools are
\n
One possible antidote to this: get the legal department involved. They see LLMs doing a poor job, and appreciate the risks involved.
\n❄ ❄
\nMost folks I talk to, both at the retreat and outside, recognize we are in some form of bubble. Technological advances like this almost always come with economic bubbles, and in the future we will all look back at this, and shake our heads saying we knew there was so much froth. But while it’s easy to see that there is a bubble, it’s hard to see how long it will run or what will emerge after the pop. After all the dotcom bubble was clearly recognized as such… in 1995. We can happily point at those companies that failed (Webvan, pets.com) but need to then acknowledge those that survived (Amazon).
\nMost of those at the retreat were old enough to have lived through the dotcom bubble and crash, but one such grey-hair pointed out an interesting difference. Back then we were excited about what the future would bring, and we saw lots of new things being built. There’s much less of that, this time around. Most people are wary of what the AI bubble is creating. Partly this may stem from the reality that followed the dotcom hope. Social media may be everywhere, but do we think it’s actually improved our lives that much, even if (especially if?) we use so much of it?
\nWe hear so much about the incredibly productive things we can do with agentic programming, but has anyone noticed a flood of wonderful applications built with it? Or have we noticed a significant improvement in common applications from the big AI boosters such as Google or Microsoft?
\nThis may be another factor in the board-vs-engineer divide. Most of what’s driving adoption of AI at the moment is cost-cutting, and it mostly the boards that get excited by cost-cutting. Perhaps the increasing concerns about token costs will temper the eagerness.
\n❄ ❄
\nFolks are finding LLMs helpful in operations: with a good event stream from observability tools, an agent finds anomalies much faster. One of the problems with citizen-developer apps, is that they often don’t provide good observability, since the citizen-developers don’t think to ask for it. The agents ability to look at the event stream does pose governance questions, as often such event streams contain a lot of sensitive information.
\nReinforcing what I’d heard in Utah, more people agreed that LLMs are valuable for operations folks to help them understand what the code does. Cross matching code and event traces helps them assist humans to find what happened when things go wrong. Agents are particularly handy with repeated incidents, as they can collate lots of information from different cases and present it to the human teams.
\nGetting agents to auto-remediate moves us to the next level of capabilities and concerns. It’s vital that agents carefully document all their actions when they do fixes. We also need to ensure there is feedback to the development team so they can learn. Agents don’t learn, the best they can do is update the context.
\nThere was a sense that many people over-estimate the capability of agents to deal with incidents. Such people think of incident resolution as a simple, linear process. But it’s rarely that, instead there’s a lot of surprises and adaptation needed. Humans are good with that, but LLMs are not.
\nOne of the perils of agent-developed code is their habit of inserting features that were never asked for. One team spent three days trying to figure out such an unrequested feature, trying to figure out who had requested it and if anyone wanted to keep it.
\nA group of law professors carried an interesting experiment to judge how well an LLM can provide short answers to student questions. They created a batch of forty questions in contract law and asked the professors, plus a couple of LLMs, to provide answers. To evaluate the LLM answers they showed professors pairs of answers - one human, one LLM - and asked them which response they would prefer to deliver to a student.
\n\n\nProfessors rated LLMs far higher than their peers (average win rate = 75.33%), with models performing similarly to the best instructor. LLM responses were also rarely flagged as harmful (3.53%, vs 12.06% for professors).
\n
This reminds me of the distinction I mentioned in a recent fragment between interactional and contributory expertise.
\nA few days ago Unmesh Joshi published an article here about his experiences using DSLs to enable more reliable use of LLMs. Responses to this included a pointer to an article by Spender Nelson that related similar impressions.
\n\n\nDSLs like this hit a lot of sweet spots for LLMs. You can make them extremely token efficient, and enforce hard security boundaries. You can translate high-level LLM intent into a ton of deterministic code, ensuring good behavior and guardrails at the (custom) compiler level.
\nAnd Large Language Models are very good at learning and working with DSLs. Maybe this shouldn’t come as a surprise; they are language models after all. A small bit of documentation generally is enough to set them off and running, and reasonable error messages let them course-correct even when they go wrong.
\n
He describes a couple of examples from their use: a query language for data lakes that takes into account security and authorization issues, and a little expression language to make it easier to create safe SQL where clauses.
\nOne of the biggest barriers to using DSLs, particularly external DSLs, is building a parser and tooling. LLMs make this much easier. That said, my sense is that it’s the semantic model that underpins the DSL is what really matters, and the DSL is one projection of that model. LLMs may help us explore other ways to project that model in interesting ways.
\nIn recent weeks I’ve been noticing the stench of LLM-speak more and more. It’s not just the common tells, it’s a sense of LLM miasma that pervades the prose. I’ve noticed it’s increasingly eliciting a visceral reaction, after a couple of paragraphs I just want to dismiss the entire article out of hand. For some of these, it was necessary for me to hold my nose and wade through the whole text, but it was with an intellectual nausea which obscured the content, even increasing my desire to indulge in such an awful distraction as checking social media.
\nI wonder - is this just me that’s reacting so negatively to LLM-speak? Or do other people have a reaction that leads them to toss aside any prose that sets off their LLM-alarm?
\nOne indicator that it’s not just me is this post from Jason Koebler that I highlighted a couple of months ago, where he observed how AI was breaking his brain:
\n\n\nPeople think things that are fake are real, things that are real are fake. Much has been written about “AI psychosis,” the nonspecific, nonscientific diagnosis given to people who have lost themselves to AI. Less has been said about the cognitive load of what other people’s AI use is doing to the rest of us, and the insidious nature of having to navigate an internet and a world where lazy AI has infiltrated everything. Our brains are now performing untold numbers of calculations per day: Is this AI? Do I care if it’s AI? Why does this sound or look or read so weird? Does this person just write like this? Is this a person at all?
\n
A while ago, I was thinking that it was reasonable for folks who aren’t as committed to writing as I am to use an AI to help polish their prose. Now I’m turning to encouraging writers to reject it. That pervasive LLM-voice is just so common now, my sense is that it discredits the writing even before the reader has a chance to try to understand what is being said. I don’t think it’s good enough to ask the LLM to write a first draft and then tweak it. I’m not sure writers can edit the LLM-ness out of prose once it’s in there. I even worry about asking an LLM to suggest improvements, I think it’s just too easy to accept an LLM’s suggestions, and in the process trigger your readers’ LLM-antibodies.
\nOf course like most problems, it’s also an opportunity. Those who can get a distinctive human voice will get more visibility and credibility. But the question remains of how we can coach people to let out their true personality into their writing. Academic and corporate writing both tended to stifle engaging prose, LLMs are good amplifiers, and they will amplify this stifling. This is an even greater challenge for those for whom English is their second language (or indeed for many of my colleagues, their third or fourth). It’s too easy for me to neglect to think about a difficulty that I’ve never been able to face.
\nThe most immediate advice I can give something I learned many years ago and shared last year - Say Your Writing.
\n\n\nOnce you’ve got a reasonable draft, read it out loud. By doing this you’ll find bits that don’t sound right, and need to fix.
\n
I always suggested this to help people get past sluggish prose, especially if they had spent too much time around academic or corporate writing. But now I think the need to Say Your Writing is even more important, in order to combat the insidious impact of AI. For most people, their speech patterns get closer to their real self, so verbalizing writing is the way to fight those forces that try to smooth away a writer’s individuality.
\n", "date_modified": "2026-07-21T13:13:00+00:00", "date_published": "2026-07-21T13:13:00+00:00", "external_url": "https://martinfowler.com/fragments/2026-07-21.html", "id": "urn:aggr:item:b9adfc9ff0828fa74c2f525fc789216945bba291", "summary": "With this post, I’ll wrap up my notes from the second Future of Software Development Retreat. But before I do, I should note that the full Thoughtworks report on the retreat is now available. They have five headline findings: Code…", "tags": [ "programming" ], "title": "Fragments: July 21", "url": "https://aggr.aymericbeaumet.com/items/martinfowler-com/2026-07-21-fragments-july-21/" }, { "authors": [ { "name": "Martin Fowler" } ], "content_html": "We all have “that” repository in our organization. The one written in 2005, built with Ant, using Java 1.5. It hasn't been compiled since the Obama administration, and it certainly doesn't run on your new Apple Silicon MacBook.
\nWhen I inherited this “Brownfield” project, the temptation is to treat Generative AI as a universal translator. I paste the code into an LLM and ask the most natural question in the world: “How do I run this?”
\nThis is what I call the “Tourist Prompt.” Like a tourist visiting ancient ruins, I am asking for a guided tour and a gift shop souvenir. I want a happy path.
\nIn our experiment, I tried exactly this. I opened a chat with a standard LLM and asked:
\n“Hi, I need to start working with this library. Can you please act as a Senior Developer and help me get started? Read the repo and give me a high-level summary... and a simple 'Hello World' code example.”
\nThe AI acted like a polite, eager-to-please tour guide. It scanned the README.txt, ignored the decades of dust, and confidently generated a modern “Starter Kit.” It gave me a pristine build.gradle file and a clean HelloBlobStore.java. It told me exactly how to connect to the database. On the surface, it looked like a miracle.
The AI-generated build.gradle
plugins {\nid 'java'\n}
\ngroup = 'com.legacycorp.blobstore'\nversion = '1.1'
\nsourceCompatibility = '1.8'\ntargetCompatibility = '1.8'
\nrepositories {\nmavenCentral()\n}
\ndependencies {\nimplementation 'org.apache.commons:commons-pool2:2.11.1'\nimplementation 'log4j:log4j:1.2.17'\ntestImplementation 'junit:junit:4.13.2'\n}
\ntest {\nuseJUnit()\n}
\nThe result was a lie — and a structural lie at that. By generating a modern build file, the AI merely painted a fresh coat of paint over a crumbling structural wall.
\nFirst, it hallucinated dependencies by suggesting commons-pool2 (v2.x) when the legacy code actually relied on org.apache.commons.pool (v1.x). Because these libraries have completely different APIs, blindly running the AI's code would have crashed the build with “Class Not Found” errors, sending me down a frustrating rabbit hole of debugging “modern” code that was never meant to be modern.
Next came the structural gaslighting. The AI confidently assumed a standard Maven layout src/main/java, completely ignoring the reality of a non-standard Ant structure java/com/legacycorp.... It was describing the reality it wanted to see, not the one that actually existed.
Finally, it hid the underlying rot. Its pristine “Hello World” example featured PooledBlobStoreImpl, omitting the fact that the core implementation SimpleBlobStoreImpl wasn't even thread-safe, the error-handling code routinely swallowed exceptions, and the so-called “Unit Tests” were actually integration tests that required a live MySQL database to run.
AI defaults to optimism. When I ask “How do I run this?”, it assumes I can run it. In a restoration mission, optimism is fatal.
\nIf I had followed the Tourist path, I would have started refactoring immediately—changing List to ArrayList<> or adding Generics—likely breaking hidden behaviors I didn't fully understand. I would have been making blind changes to a fragile system without establishing its current state.
To truly restore a brownfield project, I need to stop acting like Tourists and start acting like Archaeologists.
\nAfter the “Tourist” prompt failed by offering a modern build that couldn't exist, I realized that what was needed here was not a tour guide, but a construction inspector. I shifted my mental model from “How do I run this?” to “Why did this fail?”. I reset the context with the AI, asking it to be critical rather than helpful.
\nI crafted a prompt designed to strip away the optimism. I assigned the AI a specific persona: Senior Legacy Systems Architect. I explicitly forbade it from summarizing the README (which is often a lie in legacy projects) and ordered a “Forensic Code Audit”.
\nI am conducting a technical due diligence assessment on this legacy Java\nrepository: https://github.com/nikmalykhin/java-blobstore.
\nAct as a Senior Legacy Systems Architect. Your goal is not to tell me what the\ncode "does," but to evaluate its structural health and "age."
\nDo not summarize the README. Instead, perform a "Forensic Code Audit" focusing\non these four pillars:
\nCarbon Dating (The Era):
\nArchitectural Integrity (The Structure):
\nData Flow & Typing (The "Stringly" Trap):
\nThe "Safety" Check (Error Handling & Threading):
\nSimpleBlobStoreImpl thread-safe?Output your findings as a structured "Risk Assessment Report" for a stakeholder\ndeciding whether to refactor or rewrite.
\nThe AI dropped the polite facade immediately. Instead of a “Starter Kit,” it handed me a Risk Assessment Report with a brutal verdict: “critical rewrite recommended“.
\nThe AI analyzed the syntax like an archaeologist uncovering historical strata, citing evidence rather than guessing. Looking first at the historical record, it found a build.xml file without any sign of a pom.xml, placing the project squarely in the pre-2010 “Ant Era.” Next, it flagged key syntax markers, spotting the legacy use of org.apache.commons.pool.ObjectPool (Version 1.x) alongside raw types like Map instead of Map<String, String>. This led to an unmistakable verdict: this was Java 1.5 code written during the transition era of 2005–2008, completely predating modern generics, try-with-resources, and standard directory layouts.
The most damning insight was that this was actually Perl code masquerading as Java; the original author had simply taken a procedural Perl script and forced it into Java syntax. This procedural mindset manifested directly in SimpleBlobStoreImpl, a massive monolithic god class that tried to handle everything from low-level socket connections and protocol parsing to core business logic. Furthermore, the codebase was aggressively “stringly-typed”. Instead of utilizing proper domain objects like Device or File, the code constantly passed around raw Map<String, String> objects and manually constructed raw protocol strings just to open a file. This introduced an immense operational risk: a single typo in a string key, such as get(“fiel_id”), would trigger a catastrophic runtime crash instead of being caught safely at compile time.
The Archaeologist revealed that the perceived test coverage was nothing more than a dangerous illusion. The entire test suite relied heavily on LocalFileBlobStoreImpl, a complete re-implementation of the storage system that wrote directly to the local disk instead of traversing the network. While these tests successfully proved that this local mock worked flawlessly in isolation, that superficial success masked an alarming reality: the actual networking code, the thread-unsafe pooling, and the fragile protocol parser—the absolute most volatile parts of the system—were being completely bypassed.
The report saved me from disaster. Had I followed the optimistic “Tourist” advice to refactor SimpleBlobStoreImpl immediately, I would have blindly introduced generics and broken the fragile parsing logic. The tests would have still passed—thanks to that deceptive local mock —but the actual production code would have been completely non-functional.
Realizing the code was an absolute liability, too fragile to touch and too opaque to trust. I made the strategic decision to halt all active changes. I refused to fix the bugs, update the dependencies, or even reformat the whitespace. Instead, I transitioned directly into a phase of complete containment, wrapping the legacy code inside an isolated, standardized Docker environment before trying to analyze it any further.
\nWith the audit complete and the “Critical Legacy” label applied, my goal shifted. I didn't want to change the code; I just wanted to run it. If I could get the existing tests to pass, I would have a verifiable baseline. To achieve this, I switched AI persona from Architect to Senior DevOps Engineer.
\nFor this phase of brownfield restoration, the core mission was to establish a standardized environment. To guide the AI and actively prevent insidious “modernization creep”, I established a strict set of prime directives. First, we had to preserve the era, meaning absolutely no updates to the legacy build tools or the Java version—we were strictly mimicking the year 2008. Second, I prioritized containment over modernization, keeping the original Ant build.xml file and running the entire process inside an isolated Docker container to avoid polluting my host machine. Finally, there were to be absolutely no code changes; I refused to slap public modifiers onto classes just to fix visibility bugs. If it worked in 2008, it had to work now inside the right container.
Yet, despite these strict rules, my first instinct was still secretly tainted by the tourist mindset—a classic case of modernizer's hubris. I caught myself thinking, “Okay, I can't change the Java code, but surely I can swap out this ancient Ant build for Gradle 8, right?” Acting on that impulse, I asked the AI to perform a swift “lift and shift,” grabbing the raw Java 1.5 source files and dropping them directly into a modern Gradle 8 container.
\nThe result was a spectacular crash. The build failed not because the legacy code itself was buggy, but because the foundational rules of the software environment had radically shifted over two decades. The legacy code routinely relied on accessing package-private classes from entirely separate packages (such as TestBackend reaching into Backend). Back in 2008, Ant and Eclipse were incredibly permissive about these structural violations; by 2026, Gradle 8 and modern JDKs had become strict, unyielding enforcers of encapsulation.
The Build Failure Log
\n/src/test/java/com/legacycorp/blobstore/test/TestBackend.java:12:\nerror: Backend is not public in com.legacycorp.blobstore; cannot be accessed from outside package\nBackend backend = new Backend(trackers, true);\n^
\nFaced with this roadblock, the AI's immediate suggestion was predictable: “Just add public to the class.” But I refused. Doing so would directly violate one of my prime directive of making zero code changes. Modifying production source code solely to appease a modern build tool is a slippery slope, and I wasn't going to step onto it.
\nRealizing that I couldn't stabilize the artifact in a modern environment, I pivoted to a “Time Capsule” strategy. If I wanted to capture this system, I had to build a containment zone that strictly mirrored the standards of 2008. I turned to Docker to recreate the exact environment the code was born in, searching for an old image that bundled Java 6 and Ant 1.5 together.
\nBut I hit an immediate hardware reality check. The only available Java 6 Docker images were compiled for x86 (linux/amd64), while I was attempting to run the build on a modern Apple Silicon (ARM64) laptop. While emulation layers like Rosetta or QEMU are theoretically possible, they introduce a dangerous, unpredictable variable into an already fragile process. If the build fails, how do you know whether it's an inherent code defect or just the emulation layer choking on twenty-year-old binaries?
To eliminate that variable entirely, I changed my environment. I abandoned the laptop and switched to a native Intel machine powered by a modern i9 processor. The lesson here was clear: sometimes software archaeology requires the right shovel. I only made progress when I stopped fighting the host architecture and moved directly onto the native ground of the artifact.
\n![\"\"](\"https://martinfowler.com/articles/archaeologist-copilot/image1.png\")
Once I had the compiler working on Intel—completing the “dry” capsule—I faced the final structural challenge: a stubborn integration test named TestBlobStore.java. This “wet” test was a pure artifact of its time, littered with hardcoded assumptions tied directly to the original developer's local machine. Specifically, it looked for a magic host, trying to connect to qbert.legacycorp.com:7001, and relied on a magic file path located at ~/Projects/blobstore/…. In a standard modern refactor, I would have simply deleted these lines. But because I was strictly in containment mode, touching the test file was off the table. Instead of changing the code to fit modern reality, I had to change reality to fit the code.
The solution lay in environment emulation via Docker Compose. I prompted the AI to act as a network engineer to help me pull off some infrastructure illusions. First, we executed some network trickery: I spun up a modern BlobStore container and used a Docker network alias to trick the test runner into believing this container was actually the long-lost qbert.legacycorp.com. Next came the filesystem trickery, where I configured Docker volumes to mount our live, local source code directory inside the container at the exact, identical path engineer had used back in 2005.
This environment trick materialized in my docker-compose.yml file:
The network configuration.
\nservices:\nblobstore:\nimage: hrchu/blobstore-all-in-one:latest\nnetworks:\ndefault:\naliases:\n- qbert.legacycorp.com
\nbuilder:\n image: blobstore-legacy-builder\n volumes:\n - .:~/Projects/blobstore/java/com/legacycorp/blobstore/\n command: ant test\nThis orchestrated illusion brought about complete stabilization. When I executed docker-compose up, the legacy test suite fired up and ran flawlessly. It looked up qbert.legacycorp.com and seamlessly routed straight to my local Docker container; it reached out for engineer's old hardcoded path and found our live volume mount instead.
The build succeeded. Without changing a single byte of historical source code, I had successfully restored full functionality to a twenty-year-old application. The environment was stable, the code was finally verifiable, and I could at long last think about moving it into the future.
\nWith the artifact safely stabilized inside the “Time Capsule” of Docker, Java 6, and Ant, I finally possessed a verifiable baseline. I now had concrete proof that the code was fully functional in its native environment, meaning any failures from this point forward would be the direct result of our active modernization efforts, not pre-existing rot. With this safety net firmly established, I began the transition, launching the project fifteen years into the future with the ultimate goal of reaching Java 8 and Gradle.
\nThe choice of Java 8 was not aesthetic; it was a pragmatic necessity driven by my hardware constraints. I needed to run the project natively on Apple Silicon (ARM64), but that goal crashed into a double-ended technical wall. On one side of the timeline, modern JDKs (Java 17+) have dropped support for compiling legacy Java 1.5 source code entirely, rejecting the old -source 1.5 flag. On the other side, ancient JDKs like Java 6 refuse to run natively on ARM64 architecture, trapping you in buggy emulation layers.
So I turned to Java 8, the single, specific version capable of satisfying both ends of the timeline. Because it stands as the absolute last version to support the compilation of Java 1.5 targets and one of the earliest versions that can be installed natively on modern Mac hardware, it became our perfect architectural entry point.
\nI hit the first hard technical wall when choosing the tool version. My instinct was to use the latest release, Gradle 8.5, but this choice immediately crashed into a wall: Gradle 8 requires Java 17 just to run its internal daemon, and as we already noted, Java 17 is incapable of compiling legacy Java 1.5 source code.
\nTo resolve this bottleneck, I settled on a pivot to Gradle 7.6. This stands as the absolute last modern-ish Gradle version that can still execute on a Java 8 JVM, allowing me to establish a perfect chain of environmental compatibility:
\nApple Silicon -> Java 8 JVM -> Gradle 7.6 -> Java 1.5 Source
\nI didn't just wrap the old build.xml. Realizing the Ant script was actively obscuring the underlying logic, I configured Gradle to map directly to the legacy directory structure. To achieve native compilation, I overrode the modern defaults and explicitly instructed Gradle to look for the source code in srcDirs = ['java'] instead of expecting the standard src/main/java layout.
Next, I had to tackle the legacy test runner. Because the historical tests were structured as old-school main() methods rather than a modern JUnit suite, the standard out-of-the-box gradle test command couldn't find them. To bypass this limitation, I wired up a custom JavaExec task named runLegacyTest to execute those test entry points manually.
Mapping Gradle to the Legacy Layout
\njava {\nsourceCompatibility = JavaVersion.VERSION_1_5\ntargetCompatibility = JavaVersion.VERSION_1_5\n}
\nsourceSets {\nmain {\njava {\nsrcDirs = ['java']\n}\n}\n}
\ntasks.register('runLegacyTest', JavaExec) {\nmainClass.set(project.findProperty('mainClass'))\nclasspath = sourceSets.main.runtimeClasspath\n}
\nWith the build modernized to Gradle, the runLegacyTest task executed successfully. But the tests ran suspiciously fast. When I audited the source of TestBlobStore.java to find out why, I discovered a classic legacy anti-pattern: the silent swallow. The code was actively capturing failures and smothering them before they could bubble up to the runtime environment:
Legacy Code Pattern
\npublic static void main(String[] args) {\ntry {\nBlobStore bs = new PooledBlobStoreImpl(...);\nbs.storeFile("test_file", ...);\nSystem.out.println("Success!");\n} catch (Exception e) {\nSystem.out.println("Failed: " + e.getMessage());\ne.printStackTrace();\n}\n}
\nWhile a human reading the console outputs would easily recognize this as a blatant failure, an automated build tool sees it very differently. Because the exception is caught and handled internally without throwing it further or exiting the program, the process finishes with a perfect exit code 0. These tests were completely misleading; the backend connection could fail entirely, yet our modern pipeline would still confidently report a green pass.
\nTo strip away this false security, I initiated a process of deliberate hardening. I instructed the AI to refactor the old test harness so that it would explicitly throw exceptions all the way up the execution stack. This marked my very first structural change to the legacy codebase, and it was done with a singular purpose: to force my verifiable baseline to become completely honest. Instead of wrapping the operations in an error-smothering blanket, I stripped out the try-catch block entirely and forced the application to crash naturally if something went wrong:
\nHardened Pattern
\npublic static void main(String[] args) throws Exception {\nBlobStore bs = new PooledBlobStoreImpl(...);\nbs.storeFile("test_file", ...);\n}
\nSuddenly, the build turned bright red. Far from a defeat, this was a massive narrative victory—a red build meant I was finally looking at the unvarnished reality of the system. I spent the next hour tracing down and repairing the broken connection configurations until the build pipeline finally flipped back to green. But this time, it was an honest green.
\nOnce the tests were “honest” and the build turned Green, I was faced with a mountain of technical debt. The build was successful, but the compiler was screaming:
\nNote: Some input files use or override a deprecated API.\nNote: Recompile with -Xlint:deprecation for details.\nNote: Some input files use unchecked or unsafe operations.\nNote: Recompile with -Xlint:unchecked for details.
\nTo systematically clean this up, I moved away from general refactoring and established a tight, iterative AI-compiler feedback loop. I didn't just ask the AI to vaguely “fix the codebase”; instead, I used the compiler itself as the ultimate driver.
\nFirst, I explicitly enabled the -Xlint:unchecked flag inside the Gradle build configuration to force the compiler to reveal the exact source lines triggering the violations. Whenever the build ran and captured a specific warning block—such as an unsafe call to a raw type list—I fed those exact error logs directly to the AI with a highly targeted prompt, instructing it to refactor only those specific lines to resolve the warnings using modern Java generics.
This highly localized strategy was incredibly effective at neutralizing historical runtime risks. For example, the original codebase used primitive, Java 1.5-style raw collections where the compiler had no idea what objects actually lived inside them, forcing me to rely on blind, dangerous type casting:
\nBEFORE: The “Raw Type” Risk (Java 1.4 Style)
\npublic class Backend {\nprivate List hosts;\nprivate Map deadHosts;
\n public void reload(List trackers, boolean connectNow) {\n this.hosts = trackers;\n this.deadHosts = new HashMap();\n }\n\n InetSocketAddress host = (InetSocketAddress) hosts.get(index);\n}
\nBy passing this exact snippet and its accompanying warning log to the AI, it swiftly updated the architecture to the proper Java 8 type-safe standard, transferring the burden of validation from runtime guesswork to compile-time enforcement:
\nAFTER: The “Type Safe” Standard (Java 8 Style)
\npublic class Backend {\nprivate List<InetSocketAddress> hosts;\nprivate Map<InetSocketAddress, Long> deadHosts;
\n public void reload(List<InetSocketAddress> trackers, boolean connectNow) {\n this.hosts = trackers;\n this.deadHosts = new HashMap<>();\n }\n\n InetSocketAddress host = hosts.get(index);\n}
\nBy maintaining this disciplined, repetitive cycle across every file, I eventually crossed the finish line with a successful build and absolutely zero warnings. The historic artifact wasn't just functional; it was officially standardized.
\nAlthough I had successfully unwrapped the historical artifact, it remained fundamentally disorganized. The core codebase was still locked in a convoluted, non-standard java/com/... folder structure, and its test suite still consisted of primitive, standalone main() scripts. To make matters worse, the source code itself was completely riddled with raw types—a legacy artifact of the Java 1.5 era that forced developers to cast objects blindly and constantly exposed the system to unpredictable runtime crashes. Yet, with a modern build chain now humming and a hardened safety net finally secured beneath me, I was at long last equipped to transition from strict containment into a full-scale architectural renovation.
Before attacking the production code, I had to fix the workbench. I started with a much-needed phase of sanitization, moving the source files out of their archaic java/ root folder and into the industry-standard src/main/java layout. Making this shift allowed me to delete my previous custom Gradle directory workarounds entirely; by finally bowing to standard conventions, the build tool simply worked out of the box.
With the project's skeleton straightened out, I tackled a comprehensive JUnit 5 migration to convert the primitive legacy main() scripts—such as TestBackend and TestBlobStore—into genuine unit tests. Throughout the implementation, I systematically swapped out the old, crude System.out.println(“Error”) traps for proper Assertions.assertEquals() statements. This immediately paid off with a deeply satisfying result: I gained granular, automated test reporting, permanently freeing me from having to manually audit endless text logs just to check if a test had passed in favor of receiving a standard, unambiguous green checkmark.
I got ambitious and considered replacing the manual docker-compose setup with TestContainers to make the tests truly self-contained, but the attempt quickly collapsed. The migration rapidly degenerated into a messy “Big Bang” refactor—I found myself trying to overhaul the test runner, the network topology, and the startup logic all at once, while simultaneously wrestling with complex Docker-in-Docker networking issues on ARM architecture.
This friction taught me a vital engineering lesson: momentum is oxygen. The moment I realized I was spending all my energy fighting the tooling rather than recovering the actual code, I made the conscious decision to abort the experiment. I gladly accepted the “External Sidecar” pattern—running docker-compose up manually—because it was reliable and it worked, deliberately choosing ground-level pragmatism over over-engineered perfection.
I had successfully unwrapped the artifact and hardened its core, but two final loose ends remained before I could confidently declare the project's restoration complete. First, there was a forgotten sibling: LocalFileBlobStoreImpl.java. This legacy mock implementation desperately needed to be updated to implement our brand-new, generic-based BlobStore interface. Second, I had to address the ultimate proof of our architecture: StoreALot.java, a multi-threaded load-testing tool buried deep within the historical repository.
These files mattered immensely because they held the keys to verifying our concurrency rules. If the pooling logic inside PooledBlobStoreImpl was even slightly misaligned, StoreALot would immediately crash with a ConcurrentModificationException or succumb to silent race conditions. To prove that my modernizations were actually thread-safe, I needed to overhaul these files and push them to their absolute limits.
To execute this final performance engineering phase, I prompted my AI copilot to act as a senior performance engineer. Together, we systematically modernized the old load-testing script, cleaning up its raw syntax with generics and modern loggers while ensuring it remained executable. I instructed the AI to configure the test runner to target our backend using PooledBlobStoreImpl to hit the Docker container alias at qbert.legacycorp.com:7001. Finally, we swapped out the primitive, manual threads for a modern ExecutorService to guarantee the system could elegantly handle a parallel load without buckling under concurrent exceptions.
We had modernized the core API. Now we must verify thread safety.
\nStoreALot.java. It’s currently a main script; keep it executable but clean up the syntax with Generics and modern Loggers.PooledBlobStoreImpl to hit the Docker container alias (qbert.legacycorp.com:7001).ExecutorService instead of manual threads. Handle parallel load without throwing ConcurrentModificationException.This rigorous orchestration yielded the definitive empirical proof I needed. I launched the stress test, firing 100 iterations across 10 concurrent threads directly at my Docker-contained BlobStore backend. The results were crystal clear: the application's entire thread-safety architecture successfully relied on PooledBlobStoreImpl—utilizing Apache Commons Pool—to seamlessly provision isolated backend instances to each active thread. By verifying this behavior under intense, simulated real-world conditions, I confirmed that our deep modernizations—the generics, the JUnit migration, and the structural collection swaps—had not destabilized the core historical logic.
I had finally done it. I took a twenty-year-old piece of code archaeology that was completely uncompilable, untestable, and broken, and transformed it into a modern, thread-safe, and fully containerized Java 8 library.
\nA software restoration mission is never truly finished just because an artifact suddenly becomes functional; it is only complete when it meets a clear, unyielding definition of what it means to be done. For this legacy project, that milestone wasn't about achieving theoretical perfection, but rather about bringing the system to a specific, verifiable state where the code was completely runnable, testable, and predictable on modern hardware. By clearing that precise bar, I successfully transformed the repository from an opaque archaeological mystery into something much more familiar and manageable: standard technical debt.
\nTo ensure the next developer doesn't have to repeat my tedious archaeological dig, I switched my AI persona one last time to act as a lead repository maintainer. With this final objective in mind, I identified and systematically purged every historical artifact that belonged firmly to the past. First went build.xml, the legacy Ant script that had dictated the repository's rules for decades. Next, I emptied out the old lib/ folder—permanently discarding a loose bag of unversioned, hardcoded JARs—and swept away .classpath and .project, which were nothing more than abandoned artifacts from long-forgotten IDE setups. Running rm build.xml stood as the final, cathartic act of modernization; it officially severed our fragile link to the ancient Ant era and permanently forced the repository to rely on my modern Gradle engine.
I didn't just leave behind a clean repository; I left a map. Working with the AI, I generated a comprehensive README.md file that perfectly reflects this new, standardized reality. Instead of an undocumented labyrinth, the file outlines a completely frictionless path to productivity, specifying basic prerequisites like Docker and Java 8+, and offering a dead-simple quick start that builds the project with a single ./gradlew build. Testing the entire infrastructure is now just as straightforward, requiring a quick docker-compose up -d to spin up the backend dependencies followed by a standard ./gradlew test. This single document completely transforms the project from an intimidating mystery box into a predictable, standard Java library, permanently shifting the experience for the next engineer from a grueling forensic investigation to a routine, standard onboarding.
The Transformation: Before vs. After
\n| Feature | \nDay 0 (The Archive) | \nDay N (The Product) | \n
|---|---|---|
| Build System | \nAnt | \nGradle 8 | \n
| Compiler | \nJava 1.5 | \nJava 8 | \n
| Environment | \n“Works on my machine” | \nDocker | \n
| Testing | \nManual Scripts | \nJUnit 5 | \n
| Safety | \nRuntime Risk | \nCompile-time Safety | \n
| Confidence | \nSwallowed Exceptions | \nHardened Tests | \n
| Onboarding | \n“Good luck figuring it out” | \nREADME.md | \n
The most important lesson from this experiment was fundamentally about human agency. When I initially leaned on the helpless “Tourist Prompt”—vaguely asking the machine to “fix this for me”—the entire attempt collapsed because the AI lacked a foundational understanding of the environment and the rigid constraints of the past. Success only arrived when I fluidly shifted mindsets to direct the execution: acting first as an archaeologist to identify the true architectural decay, then as a DevOps engineer to design the containerized time capsule, and finally as an architect to define a strict refactoring policy.
\nThe AI didn't magically restore this historical system on its own; rather, I restored it by wielding the technology as a powerful force multiplier. It took care of the tedious, repetitive translation layers—churning through the conversion from Ant to Gradle, drafting the Dockerfiles, and systematically squashing fifty distinct compiler warnings—while I focused entirely on high-level strategy. Because of this active partnership, the codebase is no longer an intimidating, tangled black box. It has become entirely runnable, testable, and predictable—fully equipped to endure the next ten years and perfectly positioned for whatever future refactoring awaits it.
\n", "date_modified": "2026-07-16T13:25:00+00:00", "date_published": "2026-07-16T13:25:00+00:00", "external_url": "https://martinfowler.com/articles/archaeologist-copilot.html", "id": "urn:aggr:item:fde5fb73a50e19f0433c24f95bf19e37c86e1466", "summary": "The “Tourist” Trap We all have “that” repository in our organization. The one written in 2005, built with Ant, using Java 1.5. It hasn't been compiled since the Obama administration, and it certainly doesn't run on your new Apple Silicon…", "tags": [ "programming" ], "title": "The Archaeologist’s Copilot", "url": "https://aggr.aymericbeaumet.com/items/martinfowler-com/2026-07-16-the-archaeologist-s-copilot/" }, { "authors": [ { "name": "Martin Fowler" } ], "content_html": "Modern LLMs possess an incredible capability. They can generate large amounts of code, and sometimes entire systems, from just a high-level natural language description. An important assumption here is that the 'intent' of what needs to be built is well articulated, using precise words that LLMs can map to coding building blocks. However, there are two important points worth noting: the limits of upfront specification, and how design is discovered through implementation.
\nBuilding large systems involves a great many small design decisions, and these cannot all be known in advance or driven entirely from a high-level spec. A specification is at best a starting hypothesis: the real constraints, trade-offs, and edge cases are discovered iteratively, as we proceed with the implementation. We discussed this at length in an earlier article, where we called it Upfront Specification Impossibility. The point is not that specs are worthless, but that the first one is a hypothesis to be revised, never a finished blueprint.
\nThe natural response is to iterate: refine the spec, generate code, review what comes back, and feed what we learn into the next round. That loop works well when each round produces a small, reviewable change.
\nReviewing code, particularly while we are still discovering the design, is not the same as writing it. While reviewing the generated code, we review through the chunks validating if it maps to our intent and looking for possible pitfalls. But reviewing rarely forces us to wrestle with the design decisions. Writing code, by contrast, forces us to think through concrete decisions—such as where a responsibility belongs or what boundaries should be exposed so the design can be extended further. It is in making those decisions that a design most fully reveals itself.
\nThe programming language and paradigm we code in shapes the design insight we get. A functional design approach or an object-oriented design approach reveals different aspects of the design, along with idioms and patterns that are natural to the paradigm.
\nSo where do LLMs fit in? I see LLMs playing two roles. They are a great help while we shape the design and its vocabulary, acting as brainstorming partners to help us explore the design space and discover the right abstractions. Once the vocabulary is established, LLMs work as an excellent natural language interface to it.
\nA useful way to frame this is through Domain Driven Design. Its core insight is building a shared conceptual model of the domain in code, and then using that model - which DDD calls a Ubiquitous Language - both to evolve the codebase and to give the team a vocabulary to think and communicate in. Often, it is highly effective to build a domain specific language on top of that model: a constrained syntax for expressing the domain's concepts and operations. Seen this way, most development is the process of building a domain model and using it to evolve the system. The LLM plays two distinct roles depending on whether the domain model already exists. In this article, I will focus on how Domain-Specific Languages (DSLs) work with LLMs.
\nIt is a common experience that DSLs work well with LLMs. PlantUML, Mermaid, and Graphviz are domain specific languages for visual modeling; SQL is a DSL for querying databases; Kubernetes YAML is a DSL for describing cloud infrastructure. These are not general-purpose programming languages — they are deliberately constrained, designed to express a narrow set of concepts in one domain. And it is no surprise that LLMs are remarkably good at generating Mermaid diagrams, SQL queries, or Kubernetes manifests from a plain English description.
\nMy observation is that DSLs make LLMs more reliable because they respond so well to a few in-context examples. A general-purpose language like Java offers lots of valid ways to express the same intent. A DSL strips the variation away. Giving the model a few examples is enough to reliably generate the correct syntax. It's worth noting that frontline models are already heavily exposed to PlantUML or Java fluent interfaces during training, so they aren't starting from scratch. It will be curious to see how smaller, more constrained models perform when tasked with a truly novel DSL.
\nFor an agent — an LLM running in an autonomous generate-and-check loop rather than a single shot generation — there is one more benefit. A DSL almost always ships with a deterministic validator: a parser, a JSON schema, a type checker, or a compiler. The agent can generate a candidate, run it past the validator, and repair it from the error, all without a human in the loop. Crucially, the errors are phrased at the level of the domain — “you cannot select an action before choosing a client” — rather than as a stack trace buried deep in generated code. DSL's toolset itself acts as an excellent harness. We will see it concretely in the Tickloom examples below, where the DSL's grammar is enforced by the host language's compiler and the resulting runs are checked automatically.
\nIt is important to note that this is not a one-size-fits-all solution. The advantage holds while the DSL stays small and constrained enough that a few in-context examples can convey its usage. There is also a real upfront cost in designing and maintaining the language and its semantic model. The payoff is therefore concentrated in well-factored, genuinely constrained DSLs backed by a validator.
\nLLMs make it really easy to build custom tools. While teaching distributed systems, I frequently need to create presentations which mostly have diagrams explaining distributed operations in a cluster. UML Sequence diagrams have been great for that but showing a full sequence diagram while explaining the flow of messages through the cluster is not very useful. I needed a tool to show a sequence diagram step by step in a powerpoint presentation. With the help of LLMs I was able to build a tool which processes a YAML describing the presentation structure with references to the PlantUML diagrams and generates a powerpoint presentation. The PlantUML diagrams are marked with steps, and the tool generates a separate slide for each step. This made it really easy to create diagram-rich presentations.
\nGenerate a PlantUML sequence diagram showing a cluster of three nodes athens, byzantium and cyrene. Put a box to mark the cluster. Actor Alice sends a message “title”, “After Dawn” to athens, Athens sends message to itself. Put a note to show state 'title: After Dawn'. Athens sends message to byzantium which fails. Athens sends to cyrene. Put a note placed to the right of cyrene. athens then checks with isQuorumReached and returning a synchronous Success arrow back to Alice. Put a '[step] marker after each message.
\nThis prompt generates the following PlantUML code with step markers:
\n@startuml\nactor Alice
\nbox "Cluster" #lightblue\nparticipant athens\nparticipant byzantium\nparticipant cyrene\nend box
\n'[step]\nAlice -> athens: "title", "After Dawn"
\n'[step]\nathens -> athens: save()
\nnote right of athens\nstate:\ntitle: After Dawn\nend note
\n'[step]\nathens -[#red]x byzantium: "title", "After Dawn"
\n'[step]\nathens -> cyrene: "title", "After Dawn"
\nnote right of cyrene\nstate:\ntitle: After Dawn\nend note
\n'[step]\nathens -> athens: isQuorumReached()
\n'[step]\nathens --> Alice: Success
\n@enduml
\nI used it to create a series of slides in a powerpoint presentation. For doing that, I developed a small YAML specification to describe the presentation structure and the diagrams to be used in each slide. This allowed me to use LLMs to create presentations describing complex distributed systems concepts, without having to manually create animations on the slides. An example prompt to generate a slide YAML spec is as simple as following.
\nCreate a slide YAML referring to the diagram 'quorum-write' with a title 'Quorum Write Example'
\nThis generates a slide spec YAML like following:
\n - slide:\n title: "Quorum Write Example"\n diagram: "quorum-write"\nIt's important to note that even if the prompt is saying create a slide YAML, it's not any random YAML spec. Because the tool to generate the powerpoint presentation and the YAML specification understood by the tool is used as a context in the prompt, the LLM is able to generate the correct YAML spec which can be directly used by the tool to generate the powerpoint presentation.
\nThe full YAML spec can be viewed at this Github repo
\nNotice that the LLM played two different parts in this single example. First it was a co-designer — helping shape the step-marked PlantUML extension and the slide YAML on top of existing PlantUML tooling. Then, once that small DSL existed, it became the natural-language interface that turns an English request into a valid spec. We will come back to this division of labour at the end of the article.
\nThe example we covered in the previous section was straightforward. The YAML was used as a carrier syntax, and I process its parsed syntax tree directly, effectively using the syntax tree itself as my Semantic Model (though this couples the syntax to the execution semantics). But in more complex domains, like distributed systems, we need more complex semantic models to represent the concepts in the domain and the design decisions we have made in the codebase. Let's look at an example based on a small framework I built to quickly build and test distributed systems.
\nImplementing distributed systems such as quorum-based key-value stores or consensus protocols like Raft and Paxos is a daunting task. Even if implementation is guided incrementally through prompts, specifications, or carefully constructed .md skill files, the asynchronous runtimes still expose an overwhelming space of possible implementation decisions. Threading models, networking patterns, storage coordination, retry behavior, and timing semantics all remain entangled within the generated code. The problem is not merely code generation complexity, but verification complexity. The resulting state space created by all possible interleavings across thread scheduling, network delays, process pauses, and clock skew becomes so large that systematically reviewing and validating correctness across all interacting behaviors is nearly impossible. This is the reason we see that Jepsen tests find bugs even in the most battle-tested distributed systems.
\nThis is exactly where a semantic model is beneficial. Tickloom is a small framework I built to construct and test distributed algorithms. Its abstractions are not a generic runtime; they are a set of design decisions about how a distributed process behaves. Every node runs in a single-threaded tick loop: each call to tick() advances a logical clock by one and processes pending work in a fixed, deterministic order (network, then message bus, then process, then storage). Time is measured in ticks, not milliseconds. Messages are plain Java records. Coordination across replicas is expressed through a Replica base class that already knows about peers, broadcasts, and quorums.
Threading, timing, network delivery are no longer open questions to be re-decided in every prompt. What remains for the algorithm author is the actual protocol logic. A quorum replica, for instance, is just a set of message handlers expressed in the framework's vocabulary.
\nBecause the framework supplies the vocabulary — Replica, quorumRequest , countResponseIf, MessageType, Handler — a prompt can stay at the level of the protocol rather than the plumbing:
Using the Tickloom Replica abstraction, implement a quorum-based key-value store. A client GET should collect values from a majority and return the one with the highest timestamp (last-writer-wins). Apply a write locally only if its timestamp is newer than the stored one.
\nThis high level description produces code like following:
\n@Override\nprotected Map<MessageType, Handler> initialiseHandlers() {\nreturn Map.of(\nLWWMessageType.CLIENT_SET_REQUEST, this::handleClientSetRequest,\nLWWMessageType.CLIENT_GET_REQUEST, this::handleClientGetRequest,\nLWWMessageType.INTERNAL_SET_REQUEST, this::handleInternalSetRequest,\nLWWMessageType.INTERNAL_GET_REQUEST, this::handleInternalGetRequest,\nLWWMessageType.INTERNAL_SET_RESPONSE, this::handleInternalSetResponse,\nLWWMessageType.INTERNAL_GET_RESPONSE, this::handleInternalGetResponse\n);\n}
\nprivate void handleClientGetRequest(Message message) {\nvar req = deserializePayload(message.payload(), ClientGetRequest.class);\nvar internalReq = new InternalGetRequest(req.key());
\n)\n) //)\n) -> }\n\n)) }\n }\n\n),\n),\n),),\n));\n });\n}
\n\nThe semantic model itself acts as a context. The prompt names concepts that exist as concrete types in the codebase, so the LLM is not inventing a threading model or a networking layer — it is filling in protocol logic against a fixed, well-understood substrate.
\nA DSL is one end of the spectrum, and it is not easy to build. Before reaching for a language of your own it is worth noticing that a clean set of abstractions is already a lighter version of the same idea — and, just as the framework supplied the vocabulary for the quorum store above, a library's named types and methods are themselves a vocabulary the model can be grounded in. Tickloom's semantic model is really just four such seams — Process/Replica for compute and message handling, Network for communication, Storage for persistence, and a logical Clock for time — and that decomposition does most of the work with no new syntax at all.
This is why abstractions, not just DSLs, pair well with LLMs. A prompt to “implement Raft as a Tickloom Replica” has limited state space to explore. The existing QuorumReplica can be used in context as a worked example.
Implementing the algorithm is one thing; exercising it is another. The subtle bugs in distributed systems live in specific orderings: a write that replicates to one node before a reader's quorum shifts, a partition that heals at just the wrong moment, two coordinators whose clocks have drifted apart. Writing such a scenario directly against the testkit involves juggling futures, and manual tick() loops. Here is a clock-skew scenario written that way:
Cluster cluster = new Cluster()\n.withProcessIds(Arrays.asList(ATHENS, BYZANTIUM, CYRENE))\n.useSimulatedNetwork()\n.build(QuorumReplica::new);
\ncluster.start();
\ntry {\ncluster.tickUntil(cluster::areAllNodesInitialized);
\ncluster.setTimeForProcess(ATHENS, 1000L);\ncluster.setTimeForProcess(BYZANTIUM, 2000L);\n\nQuorumReplicaClient alice = cluster.newClientConnectedTo(ALICE, ATHENS, QuorumReplicaClient::new);\nQuorumReplicaClient bob = cluster.newClientConnectedTo(BOB, BYZANTIUM, QuorumReplicaClient::new);\nQuorumReplicaClient reader = cluster.newClientConnectedTo(READER, ATHENS, QuorumReplicaClient::new);\n\nTickCompletableFuture<SetResponse> bobWrite = bob.set(KEY.getBytes(StandardCharsets.UTF_8), "B".getBytes(StandardCharsets.UTF_8));\ncluster.tickUntilComplete(bobWrite);\n\nTickCompletableFuture<SetResponse> aliceWrite = alice.set(KEY.getBytes(StandardCharsets.UTF_8), "A".getBytes(StandardCharsets.UTF_8));\ncluster.tickUntilComplete(aliceWrite);\n\nTickCompletableFuture<GetResponse> read = reader.get(KEY.getBytes(StandardCharsets.UTF_8));\ncluster.tickUntilComplete(read);\n\nassertEquals("B", new String(read.getResult().value(), StandardCharsets.UTF_8));\n} finally {\ncluster.close();\n}
\n\nThe intent — “Bob writes through Byzantium, Alice writes through Athens, a reader sees Bob's value because Byzantium's clock was ahead” — is buried under mechanics. This is also a hard to verify code: there are dozens of incidental decisions (when to tick, how to encode bytes, which factory overload to call) for an LLM to get subtly wrong, and a reviewer must check every one of them.
\nSo on top of the semantic model I built an internal DSL whose vocabulary is the vocabulary of the scenario itself — servers, clients, who is connected to whom, what each client does, and which faults are in effect while it does it. The same scenario becomes:
\n Scenario<QuorumReplicaClient> scenario =\n QuorumStepBuilder.scenario("LWW lost update via server clock skew")\n .servers(ATHENS, BYZANTIUM, CYRENE)\n .clients(ALICE, BOB, READER)\n .client(ALICE).connectedTo(ATHENS)\n .client(BOB).connectedTo(BYZANTIUM)\n .given(g -> g.serverTimeAt(ATHENS, 1_000L)\n .serverTimeAt(BYZANTIUM, 2_000L))\n .steps(s -> {\n s.client(BOB).writes(KEY, "B").expectSuccess();\n s.client(ALICE).writes(KEY, "A").expectSuccess();\n\n s.client(READER).reads(KEY)\n .expectResponse(v -> "B".equals(v));\n });\nThe DSL is a thin, declarative surface that compiles down to a pure intermediate representation — a Scenario made of Steps, where each step carries an Action (a read or write) and optional ClusterEvents (faults like partitions and message delays). Faults read as English too: partition(BYZANTIUM).from(CYRENE), reconnect(BYZANTIUM), delay(INTERNAL_SET_REQUEST).from(ATHENS).to(BYZANTIUM, CYRENE).byTicks(100). The grammar is enforced by the type system through progressive interfaces — you cannot declare a step before the topology, or an action before selecting a client — so whole classes of malformed scenarios simply do not compile. Because the DSL is an internal DSL built in Java, the host compiler validates the grammar for free, and a malformed generation comes back as a compile error pinned to exactly the illegal step rather than a runtime surprise.
Once the DSL exists, a natural language description of a failure scenario maps almost directly onto it. A prompt like:
\nUsing the Tickloom scenario DSL, write a scenario reproducing the DDIA §10.6 non-linearizable quorum read. A writer connected to Athens sets the key, then updates it while replication from Athens to the other replicas is delayed. Alice, reading through Byzantium, is forced onto a quorum that includes Athens and sees the new value; Bob, reading later through a quorum of Byzantium and Cyrene, still sees the old value.
\nyields a scenario that stays entirely within the DSL's constrained vocabulary:
\nScenario<QuorumReplicaClient> scenario =\nQuorumStepBuilder.scenario("Non-linearizable quorum read")\n.servers(ATHENS, BYZANTIUM, CYRENE)\n.clients(WRITER, ALICE, BOB)\n.client(WRITER).connectedTo(ATHENS)\n.client(ALICE).connectedTo(BYZANTIUM)\n.client(BOB).connectedTo(BYZANTIUM)\n.steps(s -> {\n// Writer sets the key to VOLD initially and it replicates fully\ns.client(WRITER).writes(KEY, VOLD).expectSuccess();
\n // Writer updates to VNEW, but replication from Athens is delayed\n s.client(WRITER).writes(KEY, VNEW)\n .whileClusterEvent(delay(QuorumMessageTypes.INTERNAL_SET_REQUEST)\n .from(ATHENS).to(BYZANTIUM, CYRENE).byTicks(100))\n .expectSuccess();\n\n // Alice reads through Byzantium. Force quorum to include Athens by partitioning Cyrene.\n // She will read VNEW from Athens.\n s.client(ALICE).reads(KEY)\n .whileClusterEvent(partition(BYZANTIUM).from(CYRENE))\n .expectResponse(v -> VNEW.equals(v));\n\n // Bob reads later through Byzantium. Force quorum to include Cyrene (and exclude Athens).\n // The delayed VNEW replication hasn't arrived at Cyrene, so Bob reads VOLD.\n s.client(BOB).reads(KEY)\n .whileClusterEvent(reconnect(BYZANTIUM))\n .whileClusterEvent(partition(BYZANTIUM).from(ATHENS))\n .expectResponse(v -> VOLD.equals(v));\n });\nScenarioResult result = scenario.run();
\n\nBecause the surface is so small — and the space of valid code it can generate is so much smaller than the space of valid Java programs — the LLM has very little room to hallucinate, and a reviewer can read the result as a description of an experiment rather than as code to be audited line by line. Even if LLM hallucinates, the internal DSL will fail to compile allowing LLM to correct the errors made.
\nA pattern emerges from the examples discussed above: in every one of them the LLM was useful in two quite different ways.
\nThe first phase is designing the abstraction or DSL itself. Here the LLM is best treated as a brainstorming partner rather than a code generator. As argued at the start of this article, the design decisions that make up a semantic model cannot all be specified upfront — we discover the constraints, trade-offs, and edge cases as we implement them. So this phase is inherently iterative and feedback-driven: you propose a structure, try it against a real case, see where it turns out awkward, and feed what you learned back into the next round. The LLM speeds that loop up — it sketches alternatives, critiques a design, ports an idea from one language to another — but you stay firmly in the driver's seat, because these are exactly the decisions you need to understand and own. The structures that make a DSL pleasant to use, like progressive interfaces that make an illegal scenario fail to compile or a semantic model kept separate from the builder that produces it, are the kind of thing you converge on by iterating, not by writing a specification and generating code.
\nThe second phase begins once the abstraction or DSL is in place. Now what the LLM does changes: it becomes a natural-language interface to what you have built. The prompts in this article are examples — “implement a quorum store as a Tickloom Replica”, “write a scenario reproducing the DDIA §10.6 read”, “create a slide YAML for this diagram”. In each case the English description maps almost directly onto the vocabulary you defined, and the LLM is a dependable generator precisely because the abstraction supplies both the context that grounds the prompt and the harness that checks the result.
\nThere is a growing trend of treating prompts as the primary source of truth. A well-designed DSL fundamentally changes this dynamic. One of the key advantages I observe of working with DSLs is that the generated program itself often becomes the artifact that humans maintain. Because a DSL is dense, expressive, and largely free of incidental boilerplate, it captures the essential intent of the solution in a form that remains readable long after generation. If an LLM generates a Tickloom failure scenario from a natural language request, the resulting scenario is already expressed in the vocabulary of the domain. If the scenario needs to change next month, there is no need to recover the original prompt and regenerate everything. The DSL has enough context for LLM to know the intent and work with it. The enduring asset is not the prompt, but the DSL and the semantic model.
\n", "date_modified": "2026-07-14T12:51:00+00:00", "date_published": "2026-07-14T12:51:00+00:00", "external_url": "https://martinfowler.com/articles/llm-and-dsls.html", "id": "urn:aggr:item:cc98531dcf902466bca342bfdc01db2c12d0a6c1", "summary": "Modern LLMs possess an incredible capability. They can generate large amounts of code, and sometimes entire systems, from just a high-level natural language description. An important assumption here is that the 'intent' of what needs to be…", "tags": [ "programming" ], "title": "DSLs Enable Reliable Use of LLMs", "url": "https://aggr.aymericbeaumet.com/items/martinfowler-com/2026-07-14-dsls-enable-reliable-use-of-llms/" }, { "authors": [ { "name": "Martin Fowler" } ], "content_html": "Some more of my notes from Thoughtworks Future of Software Development Retreat.
\nWhen we had our first retreat in Utah early this year, nobody had heard of Harness Engineering. This time we had a whole session on it.
\nWhen comes to the guide side of harnesses, most of the discussion is about context management. While context windows have increased is size as models get more sophisticated, that doesn’t mean that models will properly focus on the right bits. Models typically only focus attention on part of the context, and to get the best behavior, we need to manage that focus. One attendee keeps their context small, limiting the agents.md file to less than 200 lines
On the sensor side, we see more attention on computational sensors. Two patterns from one participant was shifting to languages with greater controls, (eg Rust rather than Python) and “leveling up” validation approaches, using more property-based testing and techniques from formal methods. One commented that while they aren’t smart enough to write specifications in a formal specification language, they are smart enough to read it and check it makes sense for their domain.
\nWill our attention on harnesses last long enough for our next retreat? Will the models just get so good that harnesses become unnecessary? Those with some mechanical sympathy for LLMs seem to think not - but are they overly coupled to the current state of technology? I find such speculation tends not to lead anywhere useful, I’ve not seen much success in guessing the future in the past, and with technology as radical as this, I don’t see it being any easier. So for the moment, attention to harnesses pays off. We find it reduces token usage, and also allows weaker models to be useful, supporting such things as local hosting of open-weight models.
\n❄ ❄
\nWhich naturally segues me to a session on self-hosted models. Increasing token costs have made hosting an open-weight model more attractive, particularly due to the decreasing time for open-weight models to catch up with frontier models. Cost isn’t the only factor, however, many folks find a desire to be independent of the frontier model firms to be the the driving force. After all we’ve seen the U.S. government intervene to deny access to models, increasing the desire for greater model sovereignty. Information security is also something to consider, some attendees just can’t give models necessary data for critical work. Even without that, if someone else hosts the model then their model learns rather than your model. And although recent events have increased interest, several participants worked with companies that had been self-hosting for up to a couple of years.
\nIs this trudging down the same path of self-hosted clouds, which led to lots of folks spending excessive funds on half-arsed private clouds? The answer hinges upon whether it ends up being simpler to host a model than a cloud, perhaps due to a simpler interaction protocol. The hard part of this may be the talent required to efficiently use the GPUs, managing an inference data center currently isn’t a widely available skill. Even self-hosted models are a cost to operate, capital costs in GPUs, ongoing costs in electricity. The physical design of a data center can affect optimal usage. There’s an opportunity here for professional services firms to help companies manage this.
\nCost control also involves teaching people to pick the right model for the job. Can we teach engineers, or indeed other users, to pick a less-powerful model? This, of course, could be a job for model itself, acting as a broker, deciding which model is the best choice to tackle certain jobs.
\nSelf-hosting may lead to a greater use of fine-tuning. Currently that’s a niche activity, but over time we could well find that models that are fine-tuned to a particular domain need less reasoning, consume less tokens, and thus are cheaper to operate. We are seeing models trained specifically to support programming.
\nAs with any topic with this degree of uncertainty, the big win isn’t finding the right answer, but coming up with a strategy that will cope with the inevitable and unpredictable changes.
\n❄ ❄
\nAfter an event like this, many people come up to me and ask me to make some grand summing up. I hate this, because I rarely leave these kinds of event with some grand narrative. Even after mulling on it afterwards (in writing the above notes) I still usually don’t have one, and distrust one that forms, as my skepticism includes attempts to make coherent narratives of an event that’s naturally rather jumbled.
\nHowever my failings are irrelevant this time, because Kief Morris has put together such a narrative, and it’s a convincing one, even to a narrative-denier like me.
\n\n\nThe sessions had different titles and different casts, and on the surface they were about different problems.
\nBut they weren’t. Nearly every one of them was a different facet of the same argument. How much do we let an agent decide, and how do we stay confident in what it does?
\n
He looks at code review, questions whether it matters, but sees that the rigor that many associate with code review shifts to other forms. He describes the disagreements about how much we should trust an agent to identify and fix production incidents. He sees that the contrast between how much leeway teams give to agents depends on the context they are operating
\n\n\nUnderneath all of these sessions, the operations debate, the wide-remit team, the dark-factory spectrum, the argument about who’s allowed to steer the model, people were making the same handful of choices over and over about a single thing: the unit of work they were prepared to hand to an agent. How big it is. How much of the job it covers. What you do to get it ready to hand over. How you check what comes back. What you put around the agent to keep it inside the lines. Different rooms set those differently, but they were setting the same controls.
\n
❄ ❄
\nSam Ruby convened a session called “Bring me a Rock”. The name evokes a particular kind of management dysfunction. The manager tells his underlings to bring him a rock, and then starts rejecting the results without explaining why (“no not that one”, “no not that one”) until eventually one rock matches the unstated expectation.
\n\n\nIt names a manager who substitutes serial rejection for the work of saying what they want, and makes you pay for their unfinished thinking one rock at a time.
\n
Sam had already written why he thought with LLMs, this changed from a slur to a defensible way to work. When its a bunch of tireless machines with endless patience, that return new rocks in minutes rather than days, then an approach like this (using the brainstorming register becomes a defensible way to work.
\n\n\n\nThe room pulled it somewhere narrower than I’d framed, and the narrower place was the more interesting one: not how to explore by elimination but who should even be allowed to. Product managers, increasingly people managers, are reaching for these models directly, and seasoned engineers get measurably better results from them than untrained people do — so the worry followed. If expertise is what separates a good outcome from slop, should non-engineers be steering the model at all?
\nIt’s a fair question, and I think it’s the wrong one, because it mistakes the act. When a manager reaches for an LLM instead of routing the work to the team that reports to them, they didn’t pick up a tool — they made a hire. And you don’t ask permission to manage your own team; a manager who decides a piece of work is better given to a new participant than to the existing one is doing the most ordinary thing a manager does. Framed that way, the permission question dissolves into an older, better-understood one — the one Drucker named in 1959: when the worker knows more about the specifics than the manager does, you manage by objective, not by method. The non-engineer steering an agent is exactly that manager, out-known by the thing they’re directing, and the slop the room feared is the old danger of managing by method when you should be managing by objective. The question isn’t may they hire? It’s do they know how to manage by objective? — which you can teach, hire for, and hold people to without anyone first becoming an engineer.
\n
Sam’s article explores managing an LLM by objective, giving it a goal rather than a task. And Kief’s earlier point about the essence of the discussion still holds: how confident can we be that it’s done the right thing? We can outsource many things, but not the acceptance criteria, at some point there’s a human request and a human judgment on whether that request was properly executed. But the danger lies in important unstated objectives, unstated perhaps because they weren’t even imagined.
\nIt’s easy to state objectives around desired functionality. Give me a an application that will examine my emails and form a todo list for today. But behind that simple statement is a thicket of unstated assumptions. We tend to assume The Genie won’t include any undesired functionality, perhaps deleting emails it thinks are unworthy of our attention. We assume it won’t let an email tell it to send private information to villain@evil.com. We have some hope here - we hear more experiences that suggest that recent models can do an excellent job of finding (and hopefully fixing) security holes. The careful precision of the machine outruns the sloppy if imaginative thinking in squishyware. Perhaps we can assume the genie can take care of some of our unstated objectives. Conformance tests (sensors) are more valuable than specifications (guides), but it’s hard to imagine all the conformance tests that are needed to say what shouldn’t happen.
\nFurthermore, building software is about exploration, finding out how a workflow can evolve as machines are embedded in the process. For a human to guide that process, we need some understanding of it. My sense is that model building is still important, and while I agree that the genie can take an active role in that construction, I don’t think the human can entirely outsource it. Even if the genie builds the model itself, it needs to teach us that model, because the model helps us imagine and communicate the goals, the objectives that we give to the machine.
\nIf you follow my feeds (which you probably do if you’re reading this), then you’ll know that Birgitta Böckeler has written a couple of memos on working with local models. She first looked the factors that influence how viable they are for programming, and then related some of her recent experiences evaluating such models.
\nAs a nice, if accidental, complement to these, Sebastian Raschka wrote a detailed guide to his local model environment. Like Birgitta, he’s found the Qwen 3.6 model to be the current sweet spot for local agentic programming.
\nSimon Willison shares a useful tip to save money while using the latest Anthropic Fable model
\n\n\nTell Fable to use other models for smaller tasks, applying its own judgement about which model to use.
\n
Josh Comeau writes a blog and online courses for developer education, primarily front-end web material. His been successful for most of this decade but has found his online courses have had only ⅓ the sales this year. He attributes this to AI, partly as people worry if it’s worth spending money on a job that may not have a future, but also because AI can provide personalized tutoring.
\n\n\nideally, it shouldn’t cost any money to learn stuff.
\nBut I sorta worry about how this is supposed to work, going forwards, if there’s no incentive for people to make high-quality free content.
\nI’ve spoken to a few course creators now, and we’re all seeing the same trend. Revenue down 50%+. Fewer people engaging with our content. People switching to LLMs, which slurp up all of our work and regurgitate it, without consent or compensation.
\nIt feels pretty bleak. 😅
\n
John Gruber is annoyed that Claude’s desktop app for MacOS in uses Electron.
\n\n\nElectron guarantees that an app feels just as wrong on all platforms.
\n
He has some tasty invective for the folks at Anthropic with ties to the Electron platform.
\n\n\nFinding out that one guy — who is a senior Electron maintainer — has led the teams for the desktop clients for Slack, Notion, and now Claude is like discovering that it was one guy — whose family business was a distillery — who helmed the Titanic, piloted the Hindenburg, and then served as air traffic controller for Amelia Earhart.
\n
The deeper question here is whether there should be a future for cross-platform front-ends in the world of agentic programming. There’s lots of evidence that coding agents do a great job of building the same thing in multiple languages and platform ecosystems. That should mean that the days of least-denominator cross-platform UIs are numbered - and that number is small.
\nDan Davies tries to draw a distinction between interactional and contributory expertise. Contributory expertise is that held by people who are doing the work to advance a field of study, interaction expertise is held by folks that spend time talking to contributory experts, building up a decent store of knowledge themselves, but not steeped in the day-to-day of the work.
\n\n\nit seems to me that there is an important distinction here, which is not any less important because the dividing line might be difficult to establish empirically, or even if that line turns out to be in a different place from where we guessed it was. As well as difficult cases where it’s not clear, I think we could also come up with cases where the distinction between interactional and contributory expertise would suddenly become very clear and important indeed – the ones where someone who was faking it got “found out”.
\nAnd so the question that I think is quite important is whether there is a similar kind of distinction between the kind of expertise that it’s possible for a machine to get by industralised consumption and interaction with a much larger corpus of literature than any human being could inhale, and genuine contributory expertise that could apply to entirely new situations outside that literature.
\n
As a human, I’d like to think I’m more of a contributor than an interactor (especially given my increasing introversion), and thus relatively safe from being forced into obsolescence by silicon. But I’m also aware that my career is devoid of any original ideas, my skill is only that of someone who is good at selecting and explaining the ideas of others. (As Brian Foote put it more memorably: “an intellectual jackal with good taste in carrion”.) But there’s skill in being a good jackal too - and we don’t really know yet where the real boundaries of the LLMs will lie.
\n", "date_modified": "2026-07-13T12:51:00+00:00", "date_published": "2026-07-13T12:51:00+00:00", "external_url": "https://martinfowler.com/fragments/2026-07-13.html", "id": "urn:aggr:item:7982b5bbb47584c001d98b734ad087cf5be8cb3d", "summary": "Some more of my notes from Thoughtworks Future of Software Development Retreat. When we had our first retreat in Utah early this year, nobody had heard of Harness Engineering. This time we had a whole session on it. When comes to the guide…", "tags": [ "programming" ], "title": "Fragments: July 13", "url": "https://aggr.aymericbeaumet.com/items/martinfowler-com/2026-07-13-fragments-july-13/" }, { "authors": [ { "name": "Martin Fowler" } ], "content_html": "This is the second memo where I describe my recent experiences on running small models locally on my developer machine for agentic coding. In the first memo, I covered the many factors that can influence the viability of that setup — hardware, model choice, runtime, harness. Here I focus on the concrete experiences, the tasks I gave the models, what happened, and my final conclusions.
\nTo recap: my focus is on agentic coding specifically, not just auto-complete. The machines I used were an M3 Max with 48GB RAM and an M5 Pro with 64GB RAM.
\nEvaluating small models is quite tedious to be honest: downloading takes a while (on my non-fibre internet connection in the middle of Berlin), then configuring the new model in a harness, using it for tasks, interpreting the results, ...
\nI thought of it as working through a funnel of increasing viability:
\nIt has been quite a roller coaster journey, which is an observation in itself!
\nWhich task to give to the models makes a huge difference, and is one of the key challenges of evaluation. The main tasks I used in my more systematical comparison were both JavaScript/Typescript, so not a lot of tech stack diversity, keep that in mind.
\nIn a less systematic way, I also wrote a few shell and Python scripts with it, which worked fine.
\nAs the tasks are so important, I'm giving you some details here for two of them, to help inform your interpretation of my results. The choice of task is ultimately one of the biggest factors that determines viability of small, locally run models - it's all about expectations. It's about complexity of the task (How good is the model at reasoning?), about the number of files we estimate the agent will have to read and write (How good is the model at tool calling? How big is the context window?)
\nI want to change the diagram in the frontend titled, “Messages per anonymous poster” - It should be “per poster”, not “anonymous” - I want the bars in the bar chart to be sorted by number of messages, with highest bar first on the left - I want the x-axis to not show numbers (“#75”), instead I want it to show what percentage of the overall messages a certain bar is at. E.g., if the first 10 bars add up to 240 messages, and the overall messages in that time period are 1000, then I want to see 24% at the x-axis at the 10th bar. I want this percentage to be shown every 10th bar.
\n![\"\"](\"https://martinfowler.com/articles/exploring-gen-ai/task-posters-bar-chart.png\")
Phase 1: Manual evaluation
\nI tried this with Qwen3.6 35B, Gemma 4 31B, Gemma 4 26B and Qwen Coder Next 80B MoE, with both OpenCode and Pi as the harnesses.
\n![\"\"](\"https://martinfowler.com/articles/exploring-gen-ai/task-bar-chart-doom.png\")
Phase 2: Automated evaluation
\nFrustratingly, the automated setup did not confirm the manual experience. Gemma 4 26B failed to deliver a functionally correct solution 3/3 times, wheras Qwen3 35B MoE succeeded 2/2. The failure was always that it didn't properly implement the x-axis labels I asked for. They either weren't displayed at all, or wrong.
\nNote that for the automated evaluation, I expect the agent to “one-shot” the problem, which is not fully realistic. I gave it access to the browser as the only sensor to self-correct, but it never called it. Expanding sensors and sensor use could lead to self-correction that would make this more viable in real usage.
\nI want to add a horizontal bar chart to our visualisation that shows which countries the requests have come from. The title of the chart should be “Countries”. It should show the top 10 countries, sorted by number of requests, with the country with the most requests at the top. All other countries should be grouped into an “Other” category at the bottom of the chart. For access entries without a country value, they should not show up as their own category, but be lumped into “Other” as well.
\nAs a reminder, the entries now look like this:
\n{”ts”: “2026-05-24T01:10:41+02:00”, “ip”: “x.x.x.x”, “status”: 200, “ref”: “https://the-referer.com”, “country”: “Malta”}
\nWe will only have to read and change this one file, don't worry about the rest of the application. We are reading a bunch of *.ndjson files with access log entries such as the one given as an example above, and visualising them
\n@scripts/visualise_access_logs.mjs
![\"\"](\"https://martinfowler.com/articles/exploring-gen-ai/task-countries-bar-chart.png\")
Phase 1: Manual evaluation
\nI tried this with Gemma 4 31B and Qwen 35B, and it was surprisingly rough! I saw my memory usage increase quite a bit, very long reasoning chains followed by excruciatingly slow attempts to finally edit the file. I tried some different variations, like refactoring into smaller files with a big model first. I tried with both OpenCode and Pi, but it did not make much of a difference. My notes say things like, “Gave up after 11 minutes”, “Gave up after 12 minutes”, “Stopped after 8 minutes”. No idea what was different here compared to the other tasks, it doesn't seem that complicated.
\nAs this task required change to exactly one standalone file, I also tried to go old school here: I started a plain chat with a model in LM Studio, without any harness use, and pasted the full file in there - that gave me a working solution! However, the whole thing took about 6 minutes, and most of that time was spent by the model regurgitating my 450 lines of previous code, plus the added lines for the chart.
\nPhase 2: Automated evaluation
\nIn the automated setup, I ran this task with Qwen 35B MoE 7 times, and it failed to properly solve it 5 out of those 7 times... The issue in the failures was always that there were no labels displayed on the bars to depict the countries.
\n![\"\"](\"https://martinfowler.com/articles/exploring-gen-ai/local-models_comparison.png\")
Here is where it gets interesting though: I then ran the evals again on the M5 machine with 64GB, and it failed only once! Which was really surprising to me, because I was expecting the two machines to deliver different speeds, but I did not expect this vast difference in quality of output with the same model settings. This remains a mystery to me.
\nIn addition to these structured comparisons, I also used Qwen3.6 35B MoE regularly for day to day tasks, both work and personal.
\nThese are some of my reflections on the choice of task, as of now:
\nIn my daily usage, I've actually quite enjoyed the journey back in time, to weaker capabilities. It felt almost like a detox. A small model has higher sensitivity to changes in the setup, so I got more signals about what helps and what doesn't. And the temptation to just check out and not understand the results anymore is a lot lower, I went back to reviewing results much more - and I mean that in a good way. My experience with giving strong models lots of autonomy, and surrendering to them in a way, has often led to constant rework and surprises at later stages. So I've enjoyed that the weaker models make me go slower, and take more care in the moment, instead of deferring that comprehension and care to the future.
\nMy colleague Jigar Jani works with Qwen 35B MoE (4BIT) regularly on a 48GB Macbook, and does almost all of his coding with it, on a “real world” Python and React codebase. He is continuously enhancing his harness with skills, and has found Graphify and Understand Anything to be particularly useful to help the models with code search and understanding. Jigar finds it quite useful and sees improvements as he improves the harness, but he also stresses that code review is super important.
\nIt has been a frustrating experience with sometimes confusing results, I therefore find that it's definitely not a plug-and-play type of experience yet. But, I feel like I understand a bit better which types of tasks have the potential to work with these small models at this point, and I am incorporating Qwen 3.6 in my workflows to further improve my intuition when to reach for it.
\nOverall though, the agentic coding capabilities are definitely very far away from what I've now become used to with bigger models.
\nAt the time of publishing, my default setup is:
\nUntil recently I hadn't tried running models locally for quite a while, the disappointment had just always been too high when I did it. About a month ago though, I dove back in - there were just too many claims out there to ignore, about how far they have come, how it's now much more feasible to run them, and how some of them have become really good at coding. So this is my personal experience with using them, on and off, over the past 4 weeks or so.
\nIn this memo, I will start with a more general introduction and go through the factors that influence the viability of these models for coding. In a follow-up memo, I will describe my actual experiences in more detail.
\nMy main interest is how useful they are for coding, and not just auto complete, but agentic coding. Secondly, I'm interested in how ready and usable they are more broadly, for developers who don't want to dive into a bunch of specs and extra tooling and tweaks to make it work.
\nIn terms of hardware, I have been running models on these two machines:
\nThere are a myriad of factors at play that can influence the results, which makes it quite a tedious undertaking to evaluate which setup works best under the resource constraints we have. It also makes it very hard to filter the signals from the noise when people share their successes with these models online.
\nI found it particularly baffling that in the automated eval setup, one model clearly delivered better outcomes on the stronger machine (not just speed, but better code!), in spite of all other settings being the same.
\nI'll start with a summary, before going into details of each of these factors.
\nFigure 1: The many factors that can impact the results - tap or hover over boxes to highlight arrows.
\nThe model weights must fit into the available RAM, or more specifically, VRAM. If they don't, the runtime either crashes (happened to me once!) or drops to unusably slow speeds. On Apple Silicon, virtually all RAM is accessible to the GPU with no separate VRAM limit, this can be different in other machine configurations.
\nImpacts: Model runnability; Speed of responses
\nMy experience: On the machine with 48GB, I ran models between 8 and almost 30GB. The 30GB stretches it a lot of course, especially when the context window gets added, the 15-25GB size is more comfortable and I don't have to close quite as many other applications. On the machine with 64GB, I once ran a model that was 48GB big - which went great at first, but then it quickly crashed...
\nMore cores generally means faster token generation, but architecture matters too, and newer chip generations can close the gap even with fewer cores. This is a tough one to compare between machines without diving much deeper into the details of each configuration and architecture.
\nImpacts: Speed of responses
\nMy experience: On both the M3 Max and the M5 Pro, all models I ran had quite an impressive speed, compared to where they were a year ago. Speed degrades though the longer a conversation gets. I am okay using them at the current speed for some tasks - if the quality of the output were acceptable.
\nMemory bandwidth is a bottleneck for token generation, determining how fast data moves between RAM and the compute units.
\nImpacts: Speed of responses
\nMy experience: Both the M3 Max and the M5 Pro I used have a nearly identical bandwidth of ~300 GB/s, so I don't really have a comparison to something else. But as mentioned before, the speed of all models I tried was quite acceptable.
\nThe parameter count basically represents the size of a model's learned knowledge and capabilities. More parameters generally mean better output quality, but also a larger file size.
\nImpacts: Amount of RAM needed; Quality of results
\nMy experience: With 48GB, I used models around 30B parameters, +/- 5B. The biggest model I loaded on the 64GB machine was Qwen3 Coder Next 80B (MoE), which solved the task I gave it a lot better than the smaller models - but then crashed after I continued the conversation.
\nReasoning models go through a “chain of thought” process before responding, which helps with complex multi-step tasks, but can also generate significantly more tokens and slow responses down.
\nImpacts: Complexity of the tasks; Speed of response; Context window size (and therefore need for RAM)
\nMy experience: All of the models I tried had reasoning capabilities, and they were switched on by default throughout most of my experiments. However, I often noticed them going in endless circles inside of the reasoning chain, especially the smaller ones. (“Wait, ...”, “Actually, ...”, “But wait, ...”) So I also did a few runs of my automated setup with reasoning off - and lo and behold, it's not only faster (which was to be expected), but also performed the same to slightly better! A good reminder that reasoning is not always necessary, and can sometimes even be counterproductive.
\nFor agentic use, a model must be able to reliably emit structured tool call syntax that matches the schema the harness expects. Models that weren't specifically trained or fine-tuned for tool calling often produce malformed calls.
\nImpacts: Ability to use agentic harnesses
\nMy experience: This was a common issue with the models I tried, though they could often self-correct and recover from a failed tool call (e.g. using wrong parameter names like file.path instead of filePath).
GGUF is the standard format for llama.cpp-based runtimes like LM Studio and Ollama, and has by far the largest model library. MLX is Apple's own framework built specifically for Apple Silicon and can be faster, but fewer MLX-formatted models are available at the moment.
\nImpacts: Speed of responses
\nMy experience: I tried both formats for one or two of the models, but I personally didn't feel much of a difference. That could be due to the unstructured type of evaluation I did - on the other hand, as any user experience researcher would tell us, the perceived speed is ultimately what matters, not what the clock says...
\nQuantization compresses model weights to reduce the file size, trading some quality for a much smaller RAM footprint. The level of quantization is usually marked in model names and descriptions as Q4 / Q6 / Q8, or 4BIT / 6BIT / 8BIT, with lower numbers meaning higher compression. The newest buzz recently have been QAT (Quantization-Aware Training) variants of models, which are trained with quantization simulated during the training. They are supposed to preserve quality better than standard quantization.
\nImpacts: Amount of RAM needed; Speed of responses; Quality of responses
\nMy experience: All of the models I downloaded were at Q4 / 4BIT, I didn't get around to trying different variations yet. I also haven't gotten around to trying a QAT one.
\nMoE (Mixture of Experts) models have a large total parameter count but only activate a subsection of their weights at inference time, so a 35B MoE model needs significantly less RAM and can run faster than a 35B dense model.
\nImpacts: Amount of RAM needed; Speed of responses
\nMy experience: The Qwen3.6 35B MoE model was by far giving me the best balance between number of parameters and RAM usage, and therefore runnability and quality of outcomes. This could be due to the MoE architecture, I'm not sure. The architecture might also explain my experience of getting better coding abilities out of the model on the 64GB machine than the 48GB - it might be loading more experts there? I'm not sure if that's true, but it's the only sensible hypothesis I have so far.
\nContext window size consumes RAM on top of model weights through the KV cache, which grows with context length. The default size configured in the runtimes is far too small for agentic coding, it has to be set to at least 32K, if not 64K.
\nImpacts: Size and complexity of tasks; Amount of RAM needed; Speed of responses; Ability to use reasoning
\nMy experience: I tried to see how little I could get away with. For small tasks I could sometimes work with 32K, but often I had to increase to 64K, so that seems to be a good default minimum. As the models themselves were already pushing the boundaries of my available RAM, I'm not sure how how much more I could still increase it, even on the 64GB machine... So while many of these models in theory support a larger window, actually using it is limited by the memory constraints.
\nThe buzz for coding has been all around Qwen3 and Gemma 4, so those were the ones I went for.
\nQwen3
\nGemma 4
\nThe runtime handles model discovery, configuration, and loading. It also determines the practical question of how we integrate our harness with the model. Usually this happens by starting a web server that provides a range of typical APIs that the harnesses support, and the localhost URL of that web server is then configured as a model provider in the harness. Most widely supported by harnesses is the OpenAI API, but Claude Code e.g. expects Anthropic's Claude API .
Impacts: Ease of configuration and discoverability; ease of integration with harnesses; speed of responses
\n![\"\"](\"https://martinfowler.com/articles/exploring-gen-ai/local-models_lmstudio-view.png\")
Figure 2: The “Developer” view in LM Studio, showing a running server and many of the factors described so far (provider url, model size, APIs, context window size configuration)
\nMy experience: While I have used other runtimes in the past, I am currently back using LM Studio, mainly for its user experience. There are lots of ins and outs of which runtime is the most optimised for which hardware and what type of models, to get even more speed out of it. But thinking back to the broader viability of running local models, user experience plays a huge role for that. For what it's worth, the most frequently mentioned alternative from my colleagues was oMLX.
\nCoding harnesses can vary significantly in how much overhead they inject into the context window (system prompt, number of tools), which becomes more of a problem locally where we're so resource constrained. Our own expanded harness around that also makes a difference, e.g. how many skills or MCP servers are active. A description of each of them will be sent to the model, and again, take up space in the context window.
\nI mentioned above that small models still struggle with tool calling - and it probably doesn't help that each harness has slightly different schemas for the basic tools. Let's take editing a file as an example:
\nold_text and new_text (see here)oldString and newString (see here)old_string and new_string (at least that's what it says when I asked it)Finally, not all harnesses easily support the integration of local models. Open source tools are usually the go-to, but Claude Code can also be pointed at local providers. GitHub Copilot seems to support it for their CLI, and I think it's possible in Cursor as well to override the OpenAI base URL and point it to localhost.
\nImpacts: Size of context window needed; Tool calling success; Integratability
\nMy experience: In my attempts, I used OpenCode and Pi. I avoided Claude Code as it apparently would burden the context window quite a bit.
\nIn second memo, I am diving more deeply into the types of tasks I gave the models, and what I experienced.
\nA preview of my overall conclusions: Using small models like this is still quite messy and hard to evaluate. It was a frustrating experience to come to conclusions, as the results depend on so many things. I would therefore say that it's still not ready for a simple “plug and play” experience for developers who don't want to spend too much time on it.
\nHowever, based on this experience I do have a go-to model that I'm using locally now, which is Qwen3.6 35B MoE. It offered the best balance of capability, speed and RAM footprint among what I tried.
\n", "date_modified": "2026-07-07T12:34:00+00:00", "date_published": "2026-07-07T12:34:00+00:00", "external_url": "https://martinfowler.com/articles/exploring-gen-ai/local-models-for-coding-factors.html", "id": "urn:aggr:item:76d0d6371722a3e203cb5b52837c1853b80f494f", "summary": "Until recently I hadn't tried running models locally for quite a while, the disappointment had just always been too high when I did it. About a month ago though, I dove back in - there were just too many claims out there to ignore, about…", "tags": [ "programming" ], "title": "Viability of local models for coding", "url": "https://aggr.aymericbeaumet.com/items/martinfowler-com/2026-07-07-viability-of-local-models-for-coding/" }, { "authors": [ { "name": "Martin Fowler" } ], "content_html": "Last week, Thoughtworks ran a second Future of Software Development Retreat, this time in Europe. As with the previous event, I’ll be sharing some fragmentary thoughts on this. There were five parallel streams, so I could, at best, only attend ⅕ of sessions. This isn’t an event that forms conclusions, rather one that allows those exploring to share what they’ve found, and their visions for the future. The bliki post lists all the writing I’ve run into on this, by myself and others. I’ll be updating it as more posts appear.
\nGiles Edwards-Alexander “noticed a real difference between the retreats”:
\n\n\nWhere Deer Valley had hesitancy and a belief that there was something here even if we weren’t yet sure what it was, Engelberg had confidence: the value is here. As I explained to a colleague today, this was not a conference for true believers: the evidence is in.
\nWhat does the evidence say? Well, that was less clear. Some patterns and practices are emerging (one attendee had catalogued dozens of agentic engineering pattern libraries) but they are emerging. There is more work to do to truly establish what is effective, and when.
\n
Greg Herlein felt similarly:
\n\n\nReading the reports of the February event, when a lot of these same folks last got together, the conversation was about what agentic development might look like. Aspirational. More about what was coming.
\nThis time? Everybody in the room was doing it. Shipping it. Not slides - production. The whole debate about whether this changes software engineering is over. People have stopped arguing about whether a while ago. They’re arguing about how, and the how is getting real.
\n
On a more micro level, I noted two other things. Firstly, there was much talk now about harness engineering, when that wasn’t even a term in Utah - an example of how rapidly things are moving. Secondly people are now worrying about the cost of tokens, where before folks were wanting to do almost anything to incentivize people to talk to The Genie.
\n❄ ❄
\nA question that continued from Utah was whether architecture and design are still important. There seems to be two landmark hypotheses here, one is that The Genie has such a Galaxy Brain that we no longer need to care about such matters, it will handle as much spaghetti as we can throw at it. The other is, in Laura Tacho’s memorable phrase: “the Venn Diagram of Developer Experience and Agent Experience is a circle”. The point being that The Genie uses the same constructs to understand a code base that humans do, so things like good modularity and naming help it as much as it helps humans. Adam Tornhill’s writing is a good example of this viewpoint.
\nTidbits from our session on this:
\n❄ ❄
\nIn his account of the retreat, Mathias Verraes goes into the details of his perspective of these issues of software design. He adds another concern: we need good design as a hedge against the risk of dependence on AI. After all, we don’t know how high the costs may rise to. We see governments blocking access to models. We see popular opposition to AI campaigning against data centers and calling for regulation. How much can we rely on AI tools being available to maintain and extend our software in the future?
\nCharity Majors has a post on the ethics of working with AI and does an excellent job of articulating how I feel about this topic. She outlines the harms inherent in AI, both in the creation of its models (training on stolen data) and in inference (slop, lack of accountability, skill atrophy). Her conclusion however, like mine, is that there’s no ethical gain from renouncing the use of AI and castigating those who use it. Such purity provides little practical help with a technology that is so powerful and so useful.
\n\n\nThe way you show care is by showing up. The way you make the world a better place is by getting down in the muck and building it, using whatever skills and resources you have on hand. The way you drive change is you engage.
\nYes, we are all complicit. Yes, we are all compromised. No argument. But what are you going to do with that feeling of conviction? Will you channel your discomfort into solidarity and action, or try to ease your conscience by removing yourself from the system? Which does more to help those being harmed?
\n
Her suggestions on how to engage aren’t striking, but that’s hardly unusual. At the Future of Software Development Retreat I convened a session on this question, and nothing striking turned up there either. That said, I’ve never been much of an activist, so my imagination may be limited.
\nGergely Orosz has run into a case where an article of his was erased from Google search by a clearly fraudulent DMCA claim.
\n\n\nIt seems that anyone can file a bogus copyright claim to get an article they don’t like removed from Google’s search index. This happened in this case. I have no information on who filed the copyright claim. Even less so on who claims to be the copyright owner? Because I am the only possible copyright owner!
\n
He was able to find the DMCA complaint, it was made by “Ellie Piee” whose profile listed them as living on Bouvet Island, an uninhabited Norwegian dependent territory near Antarctica. It claimed Gergely’s article copied a New York Post article entitled “Band Leader Hits Winning Chord”. But Gergely’s article is “Inside Pollen’s Collapse: “$200M Raised” but Staff Unpaid”, and the two do not share a single sentence. There’s an obvious motivation for folks connected with Pollen to have done this, and I hope the resulting Streisand effect bites them where it hurts.
\n404 media have a bunch of (paywalled) reports on the impact of companies realizing that token costs are getting out of control. They’ve acquired leaked Slack chats, internal dashboards, emails and other material from companies including Citi and Amazon.
\nCompanies are urging staff to use less powerful models, or cutting off frontier models entirely. A dashboard indicates that one company has seen its token bill rise from $5 million in August 2025 to $15 million in May 2026, on track to spend over $120 million in the fiscal year.
\n404 earlier reported about Accenture taking steps to reduce token usage. The biggest problem wasn’t software engineering using agentic programming, but rather staff “chewing tokens” by using AI to do things like turning PDFs into presentation slides. They saw themselves, and their clients, grappling with exponential increases in token costs. Inevitably, after consulting firms spent time urging their clients to use AI heavily, they are now offering services to control these costs.
\nAnother post says it appears that one way to reduce token costs is to get AI tools to speak like cavemen, using a skill/plugin.
\nThere’s a good summary of all this on 404’s freely available podcast: The AI Tokenpocalypse Is Here.
\nI share these thoughts just after the July 4th weekend here in America, indeed the Semiquincentennial. Historian Bret Devereaux celebrated this event with a careful reading of the Declaration of Independence, a document often talked about more than it’s read. Which is a shame since it is hardly very long, and its impact was remarkable, and not just in what is now the United States.
\n\n\nThe Declaration of Independence was recognized as a radical, potentially explosive document at the time of its issuance, as we’ll see. And it was explosive: the world of 1775 was one dominated by monarchies with just a tiny handful of traditional republics (which we should not ignore!). It took a long time for the seeds of the declaration to spread, but the world it helped create is one where liberal democracies, while hardly universal (more people have always lived in unfree societies than free ones) represent the most economically and culturally dominant bloc in world affairs – something that had never happened before. The Declaration, in its way, remade not just the Thirteen Colonies, but slowly, surely, as water seeps through the cracks of rocks (or my floorboards, alas), it remade the whole world.
\n
Devereaux shines a light onto the world of this text, illuminating its historic context, a world that is very different to the one anyone reading this grew up in. It’s assertions of a natural law that there is equality of rights among men and that governments ought to derive their powers from the consent of the governed would seem hardly worth stating now, yet were deeply radical in 1776. I’ve found that reading history like this has helped me understand how the world is, and gives me a broader perspective on the drama of current affairs.
\n", "date_modified": "2026-07-06T12:53:00+00:00", "date_published": "2026-07-06T12:53:00+00:00", "external_url": "https://martinfowler.com/fragments/2026-07-06.html", "id": "urn:aggr:item:6edf1aa68c3e684c57f688ddd1f462452dae34a4", "summary": "Last week, Thoughtworks ran a second Future of Software Development Retreat, this time in Europe. As with the previous event, I’ll be sharing some fragmentary thoughts on this. There were five parallel streams, so I could, at best, only…", "tags": [ "programming" ], "title": "Fragments: July 6", "url": "https://aggr.aymericbeaumet.com/items/martinfowler-com/2026-07-06-fragments-july-6/" }, { "authors": [ { "name": "Martin Fowler" } ], "content_html": "“Prag Dave” Thomas (co-author of the outstanding “Pragmatic Programmer”) has loved programming since he was young.
\n\n\nProgramming was how I could express myself. I wasn’t an artist. When I sing, dogs howl. When I draw, friends say, “Very nice. What is it?” I didn’t connect particularly well with people, even though I wanted to. And yet, when I wrote my first program, I discovered a medium which let me convert thought into action. All the ideas that were bottled up behind a wall of frustration suddenly had an outlet.
\n
The LLM revolution worried him. Would they remove all that fun stuff? Happily he found it was the opposite. Like Kent Beck and others have told me, programming with LLMs is more fun than ever. His post lists reasons why: removing drudgery, speeding up feedback loops, reviving long abandoned projects, and exploring new technologies.
\nI’ve spent a few days at the DDD Europe conference, which was a very enjoyable event. With all the changes to programming due to LLMs, I suspect Domain-Driven Design is going to be one of those things that will continue to be useful, indeed may become even more important. The highlight of the conference was the opening keynote by Eric Evans, who gave a fascinating description of some of his experimentation with LLMs over the last couple of years. Once the video for that talk becomes available, I’ll link to it - hopefully it won’t be too long. I also particularly enjoyed talks from Violetta Pidvolotska, Kiran Prakash, Tom de Wolf, and Chelsea Troy. Gien Verschatse interviewed Eric Evans and me for an hour so so - again I’ll pass that link on when the video gets published.
\nOne snippet that stood out to me was from Chelsea Troy. The main thrust of her talk was managing the context window of LLMs so that it was kept in a healthy state. Much of what she said was familiar, but one thing I hadn’t thought about was her thoughts about the different registers of conversations with LLMs. These registers are different styles of conversation we can have with The Genie (or indeed other humans). She classified them in four ways:
\nHer point is that whenever I have a session with an LLM, I need to be conscious about which register I’m using. And should I change register, I should start a new conversation with a fresh context.
\nCharity Majors raises an alarm about the crevasse forming between AI enthusiasts and AI skeptics.
\n\n\nThe enthusiasts are not wrong. We are starting to see real, non-imaginary, discontinuous leaps in capabilities from teams that lean in hard to working with AI. And this does not feel like a normal technology cycle where you can wait for the dust to settle; teams that sit this out while competitors are hustling could be out of business before the dust settles. That’s a real, existential threat.
\nThe skeptics are also not wrong. When you ship code faster than engineers can read it, in domains where nobody has full context, you are making withdrawals from a trust account that took years to build. Reliability degrades, institutional knowledge evaporates. You end up with systems nobody understands, products burbling into incoherence, and on-call rotations that grind people up and spit them out. That is ALSO a real existential threat.
\n
The heart of the problem is that there is no common feedback connecting these two groups. She posts some concrete ways forward. Enthusiasts tend to talk most publicly, and focus on the wins, but they need to tell the whole story, highlighting the costs too. We all need to look at working with AI as an engineering problem. Don’t just assume that we can or can’t ship code to production without review: instead ask what would make it possible to reduce review effort. She asserts that engineering discipline is becoming more important, echoing others who’ve observed that AI is an amplifier of our current practices. She wants us to avoid getting into arguments with maximalist stances and speculation, but stick to the reality we observe. Authority comes from engagement:
\n\n\nThe engineers who shape how AI gets used will be the ones with credibility: they understand the opportunity, the stakes, and the tradeoffs, and they own enough of the consequences to have standing when they push back. Earning that position takes work, but it is work worth doing.
\n
Simon Willison noticed that Anthropic and Open AI have increased their enterprise pricing:
\n\n\nWhy these sudden aggressive moves on pricing? Both Anthropic and OpenAI are planning to IPO, but I suspect there’s a more important factor here: I think they’ve finally found product-market fit, with the coding/general-purpose agent products embodied by Claude Code/Cowork and Codex.
\n
His view is that since the November Inflection there’s increasing evidence that LLMs are making a big difference to programming, and this is translating to a viable business model for these companies. He suspects there’s another inflection point that happened in April:
\n\n\nI think April 2026 is a new inflection point where the revenue implications of this have started to land, to the benefit of the frontier AI labs and with material impacts on the budgets of large companies.
\n
Mike Masnick looks back at the internet’s failed promise of decentralization.
\n\n\nThe early-internet conventional wisdom was that the network would kill the middleman. Yet the promised era of “disintermediation” never quite happened. What actually happened was that the middlemen changed character.
\nThe pre-internet middlemen—record labels, book publishers, movie studios, magazine editors—were gatekeepers. Their entire job was rejecting 99% of what came through the door so that a curated few could reach an audience. The internet middlemen who replaced them did the opposite. Their approach depended on letting nearly everything through. They became enablers rather than gatekeepers, and an entire generation of cultural production—music, video, writing, podcasts—flowed through them in volumes the old gatekeepers couldn’t have imagined.
\nBut the new middlemen still needed a business model.
\n
That business model came from realizing that when content is bountiful, the new scarce resource is attention. The enablers built advertising machines to sell attention, and focused attention with algorithmic curation. Seeking to increase profits, the enablers rode down the road of enshittification:
\n\n\nThe companies that embraced centralized control over the user experience did so initially because they were, in fact, making things better for their users. Information overload was a real issue. Having a better system for managing it was a good thing. It’s why so many people flocked to these centralized social media platforms and became so enraptured by their algorithms.
\nThe problem of centralized systems is that they create an irresistible temptation to control and exploit. Users who found value early on feel stuck: they can leave, but doing so means abandoning their community. That lack of easy exit creates lock-in, and lock-in enables enshittification.
\n
Masnick’s view is that we have to fight this process of digital despotism, realizing that decentralization, despite it’s ergonomic problems, is worth striving for as it’s the key to cognitive liberty. The key to achieving this is the user’s control over their data with the ability to easily move from one service to another. Ease of exit promotes competition, and competition is what pushes back against the ills of despotic centralization.
\n\n\nTruly decentralized tools push power to the ends. Users control their data and choose which intermediaries operate on it—and that arrangement is a poison pill to both enshittification and despotification. Any move in those directions only pushes users across the deliberately low barrier to exit, taking their content, data, and community with them.
\n
The lessons of the past are important to guide our AI-enabled future. We need to create an environment that encourages decentralization and competition.
\n", "date_modified": "2026-06-16T12:44:00+00:00", "date_published": "2026-06-16T12:44:00+00:00", "external_url": "https://martinfowler.com/fragments/2026-06-16.html", "id": "urn:aggr:item:3be1a0a5de5f62b18fc3777453ec85536dd6f99b", "summary": "“Prag Dave” Thomas (co-author of the outstanding “Pragmatic Programmer”) has loved programming since he was young. Programming was how I could express myself. I wasn’t an artist. When I sing, dogs howl. When I draw, friends say, “Very…", "tags": [ "programming" ], "title": "Fragments: June 16", "url": "https://aggr.aymericbeaumet.com/items/martinfowler-com/2026-06-16-fragments-june-16/" }, { "authors": [ { "name": "Martin Fowler" } ], "content_html": "Preclinical drug discovery is inherently complex and data-intensive. Researchers face the significant challenge of efficiently accessing and analyzing vast volumes of information generated during this critical phase. Traditional keyword-based search methods, often reliant on rigid Boolean logic, frequently fall short when confronted with the nuanced and intricate nature of preclinical research questions.
\nThe advent of Large Language Models (LLMs) has presented a transformative opportunity. By combining the generative power of LLMs with the precision of information retrieval systems, Retrieval-Augmented Generation (RAG) has emerged as a promising technique. This approach holds the potential to revolutionize preclinical data access, enabling researchers to pose complex questions in natural language and receive accurate, context-rich answers grounded in proprietary data.
\nRecognizing this potential early, Bayer committed to exploring how these technologies could address longstanding challenges in preclinical research.
\nIn this post, we share that journey—how Bayer's early investment in generative AI has resulted in PRINCE, an agentic AI system built on Agentic RAG. This case study explores the technical architecture, engineering decisions, and lessons learned in transforming preclinical data retrieval from a challenging maze into an intuitive conversational experience.
\nMany of the engineering decisions behind PRINCE can now be understood through the lens of context engineering and harness engineering, although when the system was first designed we did not use these terms. Context engineering shaped what information each model received, what it did not receive, and how context moved between specialized steps such as research, reflection, and writing. Harness engineering shaped the scaffolding around the models: orchestration, tool boundaries, state persistence, retries, fallbacks, validation, reflection loops, observability, and human review.
\nWhile this post focuses on the technical architecture and engineering challenges, our paper published in Frontiers in Artificial Intelligence covers the product evolution and business impact in more detail.
\nThe preclinical research landscape at Bayer, like many large pharmaceutical organizations, is characterized by a diverse and extensive array of data. This includes highly structured datasets from various studies, alongside vast amounts of unstructured information embedded within text documents such as study reports, publications, and regulatory submissions. Researchers frequently encountered significant hurdles in accessing and analyzing this information effectively:
\nThese inherent challenges highlighted a clear need for a more efficient, intelligent, and integrated approach to preclinical data retrieval and analysis.
\nTo address these challenges, Bayer developed the Preclinical Information Center (PRINCE) platform. PRINCE was conceived as a unified gateway to preclinical data, initially focusing on consolidating previously siloed structured study metadata and exposing them in a “Searchable” manner. This initial phase allowed users to apply advanced filters and retrieve information primarily from structured study metadata.
\nHowever, a significant portion of Bayer's valuable preclinical knowledge resides within unstructured PDF study reports accumulated over decades. Due to numerous system migrations over the years, the structured metadata associated with these reports could be incomplete, missing, or even contain incorrect annotations. Crucially, the authoritative “gold standard” information was consistently present within the approved PDF study reports.
\nThe emergence of Generative AI, particularly RAG, provided the key to unlocking this wealth of unstructured data. By integrating RAG capabilities, PRINCE began to shift the paradigm from a filter-based 'search' tool to a natural language 'ask' system, enabling researchers to query the content of these study reports directly.
\nThis evolution reflects PRINCE's progression through three distinct phases:
\nThis deliberate evolution from Search to Ask to Do represents a strategic response to the industry's need for greater efficiency and innovation in preclinical development. By providing researchers with increasingly powerful tools to access, analyze, and act upon preclinical data, PRINCE aims to enable faster data-driven decision-making, reduce the need for unnecessary experiments, and ultimately accelerate the development of safer, more effective therapies.
\nThe system functions as an interactive conversational UI, powered by a robust backend infrastructure. Its architecture, designed for handling complex queries and delivering accurate, context-rich answers, is orchestrated using LangGraph and served via a FastAPI application.
\nFigure 1 provides the system context—UI, backend, data stores, LLM fallbacks, and observability—while Figure 2 zooms into how the system coordinates its specialized agents.
\n![\"\"](\"https://martinfowler.com/articles/reliable-llm-bayer/system-container-view.png\")
Figure 1: System context and supporting platforms.
\nUser Request: the process begins when a user submits a request through the Conversational UI which is built with React.
\nOrchestration: the user's request is routed to a LangGraph-based orchestration layer in the backend. This workflow engine coordinates a multi-stage process that progresses through clarifying user intent, thinking and planning, conducting research (using RAG and Text-to-SQL), validating data completion, and finally generating a response through the Writer agent. The workflow includes deliberate pause points and feedback loops to ensure data completeness before proceeding. (We explore the details of this agentic workflow in a dedicated section later.)
\nData Retrieval and State Management: the Researcher agents interact with a comprehensive and distributed data ecosystem:
\nVector representations of all study reports are stored in OpenSearch, forming the core knowledge base for information retrieval.
\nCurated structured data, resulting from various ETL and harmonization processes, is accessed via Athena.
\nThe state of the agent's execution is meticulously tracked. After each logical step (a LangGraph node execution), the corresponding state is persisted in PostgreSQL using a LangGraph checkpointer.
\nBroader application-level state is managed in DynamoDB.
\nThe system leverages internal GenAI platforms that host models from OpenAI, Anthropic, Google, and open-source providers. These platforms expose all models via a unified OpenAI-compatible endpoint, making it easy to swap models and choose the best tool for each task. They also manage the control plane, enforcing rate limits and other safeguards to prevent abuse.
\nResilience and Error Handling: robustness is a critical design principle, with multiple fallback mechanisms in place:
\nIf a specific LLM fails, the system automatically retries the request several times before falling back to an alternative model or platform to ensure service continuity.
\nTo recover quickly from transient failures, retries are implemented at both the individual LLM call level and the logical node level (i.e., an entire step in the agent's plan).
\nAlso, agents are provided the context of the errors so that they can chart a different trajectory or alternative plan of action as a response.
\nObservability and Evaluation: the entire system is monitored for performance and reliability:
\nGeneral system health and metrics are tracked using Cloudwatch.
\nLangfuse serves as the primary observability tool, providing detailed traces of all production traffic. This allows for in-depth debugging of issues. Furthermore, evaluation datasets are stored and managed within Langfuse, making it easier to analyze performance scores and diagnose specific failures. The evaluation is done using RAGAS evaluation framework. The live traffic evaluation is done on a daily basis while the dataset evaluation is done whenever significant changes are made to the core workflow, prompts, or underlying models.
\nFinal Response: once the agents have processed the request and generated a satisfactory response, it is sent back to the Conversational UI to be presented to the user.
\nA design principle running through this architecture is context discipline. Larger context windows did not remove the need to be selective about what each agent sees. In early iterations, putting too much information into the context made the system harder to steer and harder to evaluate. PRINCE therefore avoids treating the prompt as one large container for all available information. Instead, different stages receive different context: planning context for Think & Plan, retrieval context for the Researcher Agent, evidence context for the Reflection Agent, and synthesis context for the Writer Agent. This reduces context pollution and makes the system easier to debug, evaluate, and improve.
\nThese steps ensure that the system can provide reliable and contextually relevant answers to a wide range of complex queries by leveraging a sophisticated, multi-agent architecture and a diverse set of powerful tools and data sources.
\nPRINCE incorporates an agentic RAG system (Figure 2) to handle complex user requests that require multiple steps, reasoning, and interaction with different tools or data sources. This setup, implemented using LangGraph, orchestrates the overall workflow and leverages Researcher Agent, Writer Agent, and Reflection Agent for specific tasks. The system is designed to be robust and reliable, with multiple fallback mechanisms in place to ensure that the system can continue to function even if some of the components fail.
\n![\"\"](\"https://martinfowler.com/articles/reliable-llm-bayer/new-research-workflow.png\")
Figure 2: The research workflow.
\nThe Clarify User Intent step serves as the first line of defense against ambiguity. As the system scaled to include diverse domains like toxicology and pharmacology, simple user queries often became ambiguous, making it difficult to automatically select the right tools. Rather than relying on expensive trial-and-error across all data sources, the system proactively asks clarifying questions to pinpoint the specific domain or data type.
\nThis ensures the system enhances the query with the necessary constraints to target the correct tools. We are also optimizing this by developing domain-level selection in the UI, which will allow users to pre-filter valid tools upfront. To further reduce friction, the system also provides AI-assisted source recommendations: when a user has not selected any data source — or has selected several without a clear focus — the model analyzes the intent behind the user's query and suggests the most relevant sources. The user retains full control and can accept, adjust, or override the recommendation, ensuring domain expertise always has the final say. This “fail-fast” mechanism prevents wasted execution on vague queries, while careful tuning ensures the system remains unobtrusive when the intent is already clear.
\nFrom a context engineering perspective, this step is the first assembly decision in the workflow: it constrains which tools, domains, and data sources will be in scope before any retrieval begins, ensuring subsequent agents receive a focused rather than open-ended problem.
\nThe Think & Plan step is responsible for devising a strategy to fulfill the user's request. This critical component gives the system a dedicated space to reason about the next steps before taking action—a technique inspired by Anthropic's Think tool. Importantly, this step performs process reflection: evaluating whether the agent is making the right progress toward its end goal and is on right trajectory, rather than evaluating the data itself.
\nIn multi-step agentic workflows, particularly those involving many sequential actions, process reflection is essential. Consider a scenario where the system needs to execute 50 steps to complete a complex task. At each juncture, the system must ask: Am I taking these steps in the right manner? Am I making the progress I'm supposed to make? Is the current trajectory leading toward the user's goal? The Think & Plan step provides this metacognitive capability, allowing the system to reflect on its own workflow and adjust its strategy accordingly.
\nThis “thinking space” has proven particularly valuable in scenarios involving multiple tool calls. When PRINCE was initially developed, it had only a couple of tools: one for RAG-based retrieval and another for Text-to-SQL queries. However, as we integrated more data sources to expand the system's capabilities, the number of available tools grew significantly. With this explosion of tools came an inherent challenge: overlapping concerns and domain boundaries across different tools.
\nFor example, multiple tools might serve similar but subtly different purposes—querying structured metadata versus unstructured reports, or retrieving study summaries versus detailed experimental data. When presented with tools that belong to similar domains but handle slightly different data, the LLM would sometimes struggle to select the most appropriate tool for a given query. By introducing a dedicated thinking step, the system can explicitly reason about which tool best matches the user's intent, evaluate the characteristics of each available tool, and make a more informed decision. This approach led to a dramatic improvement in the accuracy of tool selection.
\nBeyond tool selection, the Think & Plan step is essential for orchestrating multi-step processes. Many complex queries in PRINCE require a series of tool calls where the output of one tool must be analyzed before determining the next action. For instance, the system might first query structured metadata to identify relevant studies, then use those study IDs to retrieve detailed information from unstructured reports, and finally synthesize the findings. Without a dedicated space for process reflection, the system would attempt to execute these steps linearly without evaluating whether each step is bringing it closer to the goal. With the thinking step in place, the system can pause, assess its progress in the workflow, and intelligently plan the subsequent tool calls needed to complete the user's request.
\nThe Researcher Agent serves as the system's primary information gatherer. As we onboard new scientific domains onto PRINCE, we consistently observe that data falls into two primary categories: structured and unstructured. While specific implementation techniques may vary across domains — for instance, leveraging Snowflake Cortex Analyst for pharmacology queries for Text-to-SQL versus other more custom methods for toxicology—the fundamentals behind these retrieval strategies remain consistent.
\nAs PRINCE expands across multiple preclinical domains, a single Researcher agent with a flat tool list becomes increasingly hard to manage. Many tools operate on similar concepts—“studies”, “findings”, “assays”—but point to different underlying datasets, schemas, and regulatory interpretations depending on the domain. For example, when a user refers to “the study”, the relevant context might be a repeat‑dose toxicology study, a cardiovascular safety pharmacology package, or a particular assay in aggregated mass‑data tables, each with its own preferred sources of truth.
\nTo avoid one monolithic agent juggling overlapping tools and subtly different data contracts, we are actively evolving the Researcher capability into a hierarchy of domain‑specific sub‑agents. In this proposed architecture, each domain agent will own its own toolset (for example, toxicology RAG + tox metadata SQL, or pharmacology RAG + assay‑level SQL) along with tailored prompt instructions that encode how that domain’s data model works, which tables or indices are authoritative, and how to interpret key concepts. We anticipate this will keep responsibilities coherent, reduce accidental cross‑domain leakage, and make it easier to reason about and test retrieval behaviour per domain.
\nTo effectively harvest insights from this diverse landscape, the Researcher Agent employs a hybrid retriever approach focused on two distinct patterns:
\nThis dual-strategy allows the system to bridge the gap between narrative scientific reports and quantitative experimental data.
\nIn this updated vision, the top‑level Researcher Agent is designed to act as a coordinator rather than a single all‑knowing component. Given the clarified user intent and any explicit domain selection from the UI, it will route the query to the appropriate domain sub‑agent, which can then decide how to combine RAG and Text‑to‑SQL within its own boundary. This pattern aims to preserve the simplicity of “one researcher” from the user’s perspective, while internally allowing each domain to evolve its own tools, schemas, and retrieval recipes without destabilizing the rest of the system.
\nGiven the vast repository of thousands of preclinical study reports and other unstructured documents, RAG is essential for extracting relevant insights by grounding LLM responses in this specific knowledge base. The RAG pipeline comprises a comprehensive ingestion process and a sophisticated query-time architecture.
\nIngestion Process: Preclinical study reports, mostly PDFs spanning decades and often including scanned documents with complex tables, are first centralized into an S3 data lake and passed through an extraction pipeline tuned for this corpus. The extracted text is normalized into structured JSON and then chunked using a strategy that preserves enough scientific context while keeping chunks efficient for retrieval.
\nEach chunk is enriched with study‑ and section‑level metadata from Amazon Athena (for example study ID, compound, species, route, page, and parent section), which later enables precise metadata filtering in the RAG layer. Finally, these annotated chunks are embedded and indexed in Amazon OpenSearch Service, forming the vector store that backs semantic and metadata‑aware retrieval over both the historical corpus and the daily deltas as new or updated reports arrive.
\nQuery-Time RAG Pipeline: When a user submits a query, the system initiates a multi-stage retrieval process. This pipeline is engineered to effectively retrieve the most relevant and trustworthy information from the vector database to ground the LLM's response.
\nTo illustrate this pipeline, consider the example query: “Were any of the following clinical findings observed in study T123456-2: piloerection, ataxia, eyes partially closed, and loose faeces?”. The system processes this query through the following steps:
\nKeyword Extraction: the user's natural language query is first analyzed by an LLM. Through careful prompt engineering, the model is instructed to extract keywords highly relevant for keyword search within our document corpus (e.g., “piloerection”, “ataxia”, “eyes partially closed”, “loose faeces”).
\nMetadata Filter Generation: concurrently, the LLM generates a metadata filter based on the query. For example, a filter eq(study_id, T123456-2) is extracted to narrow the search space. This filter is dynamically generated using few-shot prompting with various permutation and combination examples provided to the model, ensuring it can handle diverse filtering requests.
\nQuery Expansion: to ensure comprehensive retrieval and account for variations in phrasing and terminology, query expansion (multi query or query rewrite) is performed by a smaller, faster model. This generates n=5 semantically similar queries based on the original question. For the example query, this might include variations like:
\n“Clinical symptoms reported in research T123456-2, including goosebumps, lack of coordination, semi-closed eyelids, or diarrhea.”
\n“Recorded observations in experiment T123456-2 regarding hair standing on end, unsteady movement, eyes not fully open, or watery stools.”
\n“What were the clinical observations noted in trial T123456-2, particularly regarding the presence of hair bristling, impaired balance, partially shut eyes, or soft bowel movements.”
\nHybrid Retriever: information retrieval from the vector database (Amazon OpenSearch Service) utilizes a Hybrid Search approach that combines metadata filtering, semantic vector similarity search (kNN), and keyword-based retrieval. This process is executed as follows:
\nMetadata Filtering: the metadata filter generated in the previous step (e.g., eq(study_id, T123456-2)) is applied directly to the vector database query. This pre-filters the search space based on the structured metadata attached to the chunks during the ingestion process from Amazon Athena, ensuring that only chunks associated with the specified study ID (or other relevant metadata) are considered. This significantly reduces the search space from millions of vectors to a more manageable range of tens to hundreds, improving efficiency and relevance.
\nParallel Hybrid Search Execution: for each of the n=5 expanded queries, a single hybrid search query is executed in parallel against the filtered Amazon OpenSearch Service vector database. This query combines both semantic vector similarity search (kNN) and keyword-based search, leveraging OpenSearch's capabilities for efficient multi-vector and text search.
\nWeighted Result Scoring: within each individual hybrid search executed in parallel, a weighted approach is applied to the results. A weight of 0.7 is given to the semantic search results and 0.3 to the keyword search results to balance contextual understanding and precise term matching. This weighting was determined through experimentation to optimize retrieval effectiveness for our data.
\nResult Aggregation and Initial Ranking: the results (sets of relevant chunks with their weighted scores) from all 5 parallel hybrid search executions are aggregated. Unique chunks from all search results are pulled together, and their highest weighted score across the parallel searches is used to determine an initial ranking. This step initially retrieves a larger set of potential context chunks (k=~20) based on these aggregated and weighted scores.
\nReranking: the initial set of retrieved chunks (k=~20) is then refined using a Rerank step. A cross-encoder model (bge-reranker-large) evaluates the relevance of each retrieved chunk against the original question, selecting the top k=7 most relevant chunks to be used as context for the LLM. This reranking step is crucial for ensuring that the most pertinent information, even if not the highest in initial semantic similarity or keyword match, is prioritized for the final response generation.
\nFinal LLM Prompt Generation: the refined context (k=7 chunks) is then combined with the original question to form the final LLM prompt. This prompt is carefully constructed to guide the LLM in generating a focused and accurate response based on the provided context, minimizing the risk of hallucination.
\nResponse Generation with Citation: a state-of-the-art reasoning model then processes the final prompt and the provided context to generate response with citation. The LLM synthesizes the information from the context to formulate a coherent and accurate answer. Crucially, the response automatically includes citations linking back to the specific chunks in the original document(s) that support the generated answer.
\nMonitoring: the entire Query-Time RAG process, from initial query to final response generation, is continuously monitored using Langfuse for observability, performance and quality analysis.
\nWhile RAG excels at unstructured data, queries requiring precise filtering, aggregation, or comparison of structured data points are better suited for Text-to-SQL. Examples include “Give me 50 example studies done on RAT” or retrieving specific numerical assay results including dosage groups. As shown in the Researcher Agent can intelligently decide to hand over such queries to the Text-to-SQL tool.
\n![\"\"](\"https://martinfowler.com/articles/reliable-llm-bayer/text-to-sql.png\")
Figure 3: Text-to-SQL tool
\nThe process for converting a natural language question into an executable SQL query and retrieving results involves several key steps:
\nWhile the Think & Plan step provides process reflection, the Reflection Agent performs a complementary but distinct type of reflection: data reflection. This crucial component evaluates whether the data retrieved from various tools is sufficient and relevant to answer the user's question—a fundamentally different concern from whether the workflow itself is progressing correctly.
\nIn multi-step agentic workflows, these two types of reflection serve different but equally important purposes. Process reflection (Think & Plan) ensures the agent is taking the right steps and making appropriate progress toward the goal. Data reflection (Reflection Agent) ensures that the information gathered through those steps is adequate to fulfill the user's request. Both are essential: an agent might execute a perfectly valid workflow (good process) but still retrieve insufficient data to answer the question, or conversely, might have access to sufficient data but fail to progress effectively through the workflow.
\nAs illustrated in the research workflow diagram (Figure 2), after initial information retrieval and 'think & plan' loops, the Reflection Agent is invoked when Think & Plan step thinks that the process has progressed well enough and is ready to evaluate the data. 'Reflection Agent' evaluates the sufficiency and relevance of the collected data by comparing the retrieved context against the user's original query and identifying potential gaps or missing information. If the gathered information is deemed insufficient to provide a complete response, the Reflection Agent generates specific follow-up questions designed to acquire the necessary missing information. These follow-up questions are then handed back to the Think & Plan step, which initiates further retrieval steps to obtain more comprehensive results. This iterative process of data validation and subsequent information retrieval, driven by the Reflection Agent's generated questions, demonstrates the system's ability to refine its search strategy based on the initial results. If the information is sufficient, the workflow proceeds to the next step.
\nOnce the Researcher Agent has collected the relevant evidence from RAG and Text-to-SQL, the Writer Agent is responsible for turning that raw material into the final answer shown to the user. Its job is not to “discover” new information, but to synthesize the retrieved context, respect user instructions, and enforce PRINCE's quality constraints during generation.
\nThe Writer Agent operates with a few non-negotiable rules. It must ground every claim in the supplied context and attach accurate citations back to the underlying chunks and study IDs, since verifiability is critical in a regulated environment. It is also responsible for honoring user-level formatting requirements (for example, tables, bullet points, or specific section structures) and for aligning with domain-specific answer standards used by the preclinical scientists.
\nFor more complex responses—such as multi-section summaries or partially filled regulatory templates—the architecture supports extending the Writer Agent with a short internal review loop. In this pattern, the Writer would first draft an answer, then a reviewing step would check for missing sections, inconsistent tables, or gaps relative to the original question, and may send targeted instructions back to the Writer to revise specific parts. This design enables a lightweight form of reflection focused on answer completeness and presentation, complementing the Reflection Agent's focus on data sufficiency earlier in the workflow. Importantly, all outputs from these regulatory drafting workflows are intended for expert review; final submissions are authored and approved by qualified personnel.
\nThis gives PRINCE three complementary reflection loops. Process reflection checks whether the workflow is on the right path and helps catch bad trajectory, wrong tool choice, or poor sequencing. Data reflection checks whether the gathered evidence is sufficient and helps catch thin evidence, missing context, or gaps in coverage. Draft reflection checks whether the generated output is complete and helps catch missing sections, incomplete tables, or synthesis gaps.
\nTogether, these agents form a practical context engineering pattern. The system does not simply keep adding more information to the prompt. It routes the right context to the right capability at the right time: planning context for Think & Plan, retrieval context for the Researcher, evidence context for the Reflection Agent, and synthesis context for the Writer. This plays out in concrete decisions throughout the system: the Text-to-SQL step injects only the schema components relevant to the current query rather than the full database schema; the Reflection Agent receives the original question alongside collected evidence to assess gaps, not the full workflow history; and the Writer Agent receives curated chunks with citation constraints, not raw retrieval output. Moving from a monolithic agent to this structured workflow meant each agent could be evaluated, debugged, and improved in isolation.
\nBuilding and maintaining user trust is paramount for the successful adoption of any AI system, particularly in a critical environment like preclinical drug discovery where decisions have significant implications. For a production LLM application, trust is not just about accuracy; it's also about reliability, transparency, and the ability for users to verify the information provided. Several mechanisms are integrated into PRINCE to achieve this:
\nEnsuring transparency and explainability is a critical aspect of PRINCE's design, fostering user trust and enabling verification of the generated responses. The system incorporates several mechanisms to achieve this:
\nRigorous evaluation is fundamental to building and maintaining a reliable LLM application. PRINCE's performance and reliability are assessed through a combination of two types of evaluations: Dataset Evaluations and Live Traffic Evaluations.
\nContinuous monitoring of the system's performance and outputs is essential for proactive identification and resolution of issues in a production environment. Using platforms like Langfuse, we continuously monitor PRINCE to identify potential biases, errors, or areas for improvement, ensuring the reliability and safety of the system's responses.
\nGiven the complexity of the multi-step workflow inherent in PRINCE, robust error handling and recovery mechanisms are critical to ensure the system's reliability and provide a seamless user experience. The system is engineered to recover gracefully from failures at various stages without requiring a complete restart of the entire workflow.
\nKey aspects of our error handling and recovery approach include:
\nThis comprehensive approach to error handling and recovery minimizes the impact of transient failures, reduces the need for users to restart complex queries from scratch, and contributes to cost and latency savings by avoiding redundant execution of successful steps and LLM calls, all of which are essential for a production-ready system.
\nThese mechanisms are harness engineering in practice. The LangGraph workflow acts as the control layer around the agents: it defines which component can act, which tools it can use, where the workflow can pause, how failures are retried, how state is persisted, and when the system should move from research to reflection to writing. This harness makes the system less opaque and more reliable than an unconstrained autonomous agent. It gives the application clear control points for recovery, inspection, evaluation, and human intervention.
\nThe accuracy and completeness of the structured metadata in Amazon Athena are critical for the performance of the Text-to-SQL component and overall data discoverability within PRINCE. Due to historical data migrations and varied annotation practices across different laboratories and systems over Bayer's extensive operational history, the metadata can sometimes be incomplete, missing, or incorrect.
\nTo address this challenge and continuously enhance the quality of the structured metadata, we have developed a utility system that employs Named Entity Recognition (NER) to extract and create accurate annotations directly from the study PDFs. This system is designed to read the textual content of the preclinical reports and identify key entities and associated information that should be represented in the structured metadata.
\nThe process involves:
\nWe are actively working on integrating this utility system into our data pipelines to automatically correct and enrich the data within the Amazon Athena database. The system's performance in generating accurate annotations has been evaluated against curated datasets, demonstrating promising results. To manage the integration of these annotations into the production database, we are developing an evaluation system that provides a confidence score for each extracted field. Fields with a high confidence score will be automatically used to update the corresponding entries in Amazon Athena. Fields with lower confidence scores will be quarantined and flagged for human review and intervention, ensuring data accuracy while leveraging automation. This approach aims to continuously improve the quality of the structured metadata, making it a more reliable source of information for PRINCE and other downstream applications.
\nPRINCE has been available to end-users since early 2024, with the agentic integration introduced later that year. This has been crucial for gathering real-world feedback and driving iterative development. A key principle guiding our development has been the understanding that building a production-ready LLM application is an iterative process; we don't wait for features to be absolutely perfect before seeking user feedback. Instead, we prioritize delivering value early and continuously refining the system based on real-world usage.
\nIn the initial stages, our focus was squarely on achieving the desired accuracy and performance for core functionalities, even if it meant incurring higher costs. We recognized that optimizing for cost prematurely could compromise the system's effectiveness and hinder user adoption. Only after achieving the desired level of accuracy and performance did we begin to focus on cost optimization, ensuring that efficiency gains did not negatively impact the user experience or the quality of the results.
\nThe development of PRINCE follows a continuous, iterative process. User feedback, ongoing monitoring data, and insights from expert scientists are continuously fed back into the development cycle, leading to refinements in the architecture, retrieval techniques, agent behaviors, and user interface to enhance performance, usability, and ultimately, scientific impact.
\nBuilding a production-ready LLM application in a complex enterprise environment like preclinical drug discovery is a journey marked by significant technical and engineering challenges. The PRINCE case study demonstrates that by combining robust data infrastructure, sophisticated information retrieval techniques like RAG and Text-to-SQL, and an intelligent multi-agent orchestration system, it is possible to unlock valuable insights from vast, previously inaccessible data repositories.
\nOur experience highlights the critical importance of focusing on engineering for reliability, including robust error handling, state persistence, and LLM fallbacks. Furthermore, building user trust is paramount, achieved through transparency in the workflow, clear explainability via granular citations, and continuous evaluation and monitoring of the system's performance.
\nPRINCE has already shown promising results in enhancing data accessibility and research efficiency at Bayer, transforming how scientists interact with preclinical information. This is not the end of the journey, but rather a significant step towards creating truly intelligent research assistants.
\nThe broader lesson from PRINCE is that production-ready agentic AI is not only about better models or better prompts. Reliability comes from engineering both the context the model sees and the harness within which the model acts. Context engineering helped ensure that each model had the right information, and only the right information, at the right stage of the workflow. Harness engineering helped ensure that the workflow remained bounded, observable, recoverable, and suitable for a regulated research environment.
\nAs model capabilities improve, some parts of today's harness may become thinner or move into native model capabilities. But in enterprise research systems, especially where trust, traceability, and reviewability matter, explicit control over context, workflow state, recovery, reflection, and verification remains essential.
\nWe hope this overview provides valuable insights into the practical considerations and technical depth required to build and productionise LLM applications in a regulated and data-rich domain.
\n", "date_modified": "2026-06-16T12:11:00+00:00", "date_published": "2026-06-16T12:11:00+00:00", "external_url": "https://martinfowler.com/articles/reliable-llm-bayer.html", "id": "urn:aggr:item:ff230e798e478e400a3fa5e82a6c20624012b7c1", "summary": "Preclinical drug discovery is inherently complex and data-intensive. Researchers face the significant challenge of efficiently accessing and analyzing vast volumes of information generated during this critical phase. Traditional…", "tags": [ "programming" ], "title": "Building Reliable Agentic AI Systems", "url": "https://aggr.aymericbeaumet.com/items/martinfowler-com/2026-06-16-building-reliable-agentic-ai-systems/" }, { "authors": [ { "name": "Martin Fowler" } ], "content_html": "Greg Wilson has noticed that lots of folks are using dodgy metrics to figure out if AI tools are worth their costs.
\n\n\nWould you measure lines of code generated, or tickets closed? Or would you send out a survey asking whether developers feel more productive? Each of those approaches is flawed in a different way;
\n
He lists lots of common metrics, and why they are flawed. Sadly he doesn’t give any suggestions on what would be better. In my view, since we cannot measure productivity, any metrics are weak evidence at the best of times.
\nI do somewhat use one of his flawed measures: “Asking Developers If They Feel More Productive”. While I acknowledge the problems he gives with this measure, I find that in an environment where decent measures are hard to find, even such a dim light is the best we have. In this situation these kinds of qualitative metrics may not be conclusive, but they are useful.
\nBenedict Evans observes that extensive automation didn’t mean the demise of professions in the past.
\n\n\nwe spent a century automating accounting: we built calculating machines, punch cards, mainframes, data processing, databases, PCs, spreadsheets, ERPs, cloud… in fact, we built half of the tech industry around automating this. Yet the number of accountants kept going up.
\n
He goes into the myriad of problems that exist when we’re trying to forecast the impact of a technology on jobs. There’s the much-talked-about Jevons paradox - once something becomes cheaper, people do it more, which can increase demand. Often this leads to the nature of jobs changing, even if it’s called the same thing.
\n\n\nAccountants today aren’t doing exactly the same work that they did in 1970 or 1980 ‘but more’ - they’re still called ‘accountants’ but the job is different. New technology often starts out being used for ‘the old thing but more’, but it rarely ends up like that.
\n
Technologies often affect whole businesses - consider the impact of the internet on news publishing. Did anyone observing the rise of smart phones in the early 2000s realize that a consequence of this would change the economics of taxis due to the rise of ride-sharing apps? The conclusion is that it is, at the very least, almost impossible to forecast the impact of AI on our work.
\nStephen O’Grady looks at how closed and open models have performed on benchmarks over time.
\n\n\nClosed models are setting the pace of innovation, and constantly breaking new ground from a capabilities standpoint. Open models are chasing them, and the cycle times seem to be getting shorter. There are no clear capability moats, and what is frontier today is table stakes tomorrow.
\n
It tooks 13-18 months for open models to catch up to GPT-4 on these benchmarks, but only 2-7 months to catch up to GPT-4o.
\nThere’s a bunch of caveats to this analysis, that he lists, but it’s a worthwhile survey of how various kinds of models perform against the various measures we are trying to assess them with.
\nOne of the starkest examples of sloppy AI use is hallucinated citations - a give-away of both usage of LLMs and carelessness driving them. GPTZero is a company that makes tools to detect AI writing. I’ve no insight as to whether their tool is effective or not, but they do publish investigations of AI usage, and have published several articles highlighting hallucinated citations. One post focuses on Ernst & Young Canada’s report on cyber threats to loyalty systems and found that more than half its references were hallucinations. The post uses a lot of extremely annoying animations in how it presents its information (breaking Safari’s reader mode in the process). But the harm that these kind of AI generated reports can do goes further than just some misled humans:
\n\n\nPublishing a report online is essentially a form of data injection into the pool of knowledge that is the internet. When the report includes fake information (either vibed citations or false claims) it can “poison the well” by misleading future researchers, especially if the report is published by a well-known consulting firm and hosted on a high-traffic website.
\n
As LLMs get more capable in programming, we are rightly worried that people will use them attack software systems. But these models can also be used for defense, allowing teams to find bugs before attackers do. Some folks from Mozilla posted an article on how they’ve used AI model to identify and fix an unprecedented number of latent security bugs in Firefox.
\n\n\nJust a few months ago, AI-generated security bug reports to open source projects were mostly known for being unwanted slop. Dealing with reports that look plausibly correct but are wrong imposes an asymmetric cost on project maintainers: it’s cheap and easy to prompt an LLM to find a “problem” in code, but slow and expensive to respond to it.
\nIt is difficult to overstate how much this dynamic changed for us over a few short months. This was due to a combination of two main factors. First, the models got a lot more capable. Second, we dramatically improved our techniques for harnessing these models — steering them, scaling them, and stacking them to generate large amounts of signal and filter out the noise.
\n
During 2025, there were 17-31 security bugs fixed each month. In April 2026, they fixed 423.
\nPavel Voronin riffs on Unmesh Joshi’s post on What is Code. He observes that cruft in a codebase (technical debt) has always added friction to software development. But the consequences of this cruft are compounded when LLMs are using existing code as context for future work.
\n\n\nIn a degraded codebase, the model does not see “technical debt” as debt. It sees examples. It sees precedent. It sees a style to continue.
\n
LLMs multiply what’s currently happening. I hear reports that good code might take the place of much of what’s put in markdown, because LLMs will imitate what’s already in the code base. But bad code multiplies too. Inevitably he introduces another variation of rampant debt metaphors:
\n\n\nCognitive debt accumulates when a team uses abstractions it no longer understands. Generative debt accumulates when a codebase contains confused concepts that models are likely to continue.
\nCognitive debt is about what the team no longer understands. Generative debt is about what the model is now likely to reproduce.
\n
Jason Koebler, from the very worthwhile 404 media, has written a plaintive essay on how AI-generated slop is driving us crazy. Not just because its filling the web with this slop, but also because how it’s making us humans react to slop and the threat of slop. We review our own writing and notice: it’s not just reading AI slop that hurts us, it’s the risk that we write something that looks like AI slop. If I use phrasing that AI copied from me, does it seem like I’m copying AI? This has led to the appearance of “humanizers” - AI tools that make our writing look less like AI.
\n\n\nHumanizers add typos, randomly replaces words, removes “AI tells,” and sometimes inserts random characters.
\n
It’s another step on the way to the Zombie internet:
\n\n\nI called it the Zombie Internet because the truth is that large parts of the internet are not just bots talking to bots or bots talking to people. It’s people talking to bots, people talking to people, people creating “AI agents” and then instructing them to interact with people. […] It’s my email inbox, in which I used to occasionally get poorly-formatted, poorly written, extremely long emails from delusional people who were positive the CIA had imprisoned them in a virtual torture chamber using undisclosed secret technology but where I now get well-formatted, passably written, extremely long emails from delusional people who are positive they have proven AI sentience and have the AI transcripts to prove it.
\n
Andy Osmani points out that spawning lots of agents is like launching a bunch of parallel processes that all rely on a single orchestrating thread - yourself.
\n\n\nPython has the Global Interpreter Lock (GIL). You can spawn as many threads as you want but only one executes python bytecode at a time because they must acquire the lock. You are the GIL of your AI agents. They all can run at once. But when any of their work needs genuine understanding of the architecture or resolving merge conflicts, that work has to acquire the lock. There is one lock. You hold it.
\n
This means you must design the workflow with the agents with that GIL in mind. You shouldn’t launch more agents than you can properly review. It’s handy to separate background tasks that can be offloaded to an agent from complex tasks that require applied attention. Don’t use that precious brain for things that the machine can verify itself. [And I’d add - do get the machine to build tools that ease human verification. For example, it’s better to surface test case data in tables rather than buried in assert statements.]
\n\n\nSpawning agents is not the skill. Anyone can run 20.
\nThe real skill is designing the system around the one serial resource that cannot be cloned or parallelized. That resource is your attention.
\n
Jamie Hurst is a Principal Engineer at booking.com, where he works in developer experience with a focus on AI tooling. He’s written realistically about the gains and losses of using LLMs in this work.
\n\n\nThe cost of building has collapsed, but the cost of aligning organisationally has not. If anything, it’s gone up. When three different teams can each produce a working solution to the same problem in the time it used to take to write a proposal, the bottleneck moves from engineering to coordination.
\n
He thinks he’s able to do more as a senior engineer, but is concerned about how sustainable it is, both for him personally and for the organization he works for. He’s able to shape directions for multiple workstreams at once, in a way that he couldn’t three years ago. But one loss is that he doesn’t have enough time for mentoring, which will exact a toll on his employer in the longer term. He also finds he doesn’t have enough time to think.
\n\n\n", "date_modified": "2026-06-02T15:21:00+00:00", "date_published": "2026-06-02T15:21:00+00:00", "external_url": "https://martinfowler.com/fragments/2026-06-02.html", "id": "urn:aggr:item:c63c02993c8992066801a0944637313ed1dcbe5f", "summary": "Greg Wilson has noticed that lots of folks are using dodgy metrics to figure out if AI tools are worth their costs. Would you measure lines of code generated, or tickets closed? Or would you send out a survey asking whether developers feel…", "tags": [ "programming" ], "title": "Fragments: June 2", "url": "https://aggr.aymericbeaumet.com/items/martinfowler-com/2026-06-02-fragments-june-2/" }, { "authors": [ { "name": "Martin Fowler" } ], "content_html": "The productivity gains from AI got captured by output volume rather than output quality. The org’s expectations rose to absorb the speed-up, and the slack that used to exist between tasks, the unstructured time where strategic thinking actually happens, got eaten first because it’s invisible on a dashboard. I’m at a point in my career where thinking is supposed to be most of the job, and most of it now happens on holiday because the working week doesn’t accommodate it.
\n
At the GOTO Conference in Copenhagen in 2025, Kent Beck and I spent some time on stage talking and answering questions from the audience - a format I refer to as “two old geezers on a park bench”. We talk about our experiences with LLM-augmented programming (at that point - October 2025), we show our frustration that things we’ve been saying for thirty years still need to be said, we say how anything like a manifesto reunion needs to be led by a younger generation, and opine on what junior developers should be focusing on in their career.
\n![\"\"](\"https://martinfowler.com/img/catalog/2025-goto-kent.png\")
Ian Johnson has written a series of posts about restructuring a gnarly codebase
\n\n\nThe story follows a real Laravel + React codebase over ~3 months and ~258 commits from a legacy monolith with no tests to a well-structured application with automated quality gates, a React SPA migration in progress, and an AI agent that reliably ships production code with minimal supervision.
\n
The series covers the steps in decent detail, and his approach follows the kinds of steps I’d use. First get everything under the control of decent characterization tests, add static analysis, introduce the right patterns to make things flow easily.
\nWith all of this, is his use of AI, which changed during the exercise:
\n\n\nFor the first two months of this project, I used Claude Code with auto-approve turned off. Every file edit, every terminal command, every change… I reviewed it before it executed. […] The results were good. The code was clean. But I was doing most of the thinking and half the typing. The agent was a fancy autocomplete with better suggestions. I wasn’t getting the leverage I’d hoped for.
\nI read an article about “on-the-loop” versus “in-the-loop” human-AI collaboration. The framing clicked immediately […] I was micromanaging because I didn’t trust the agent to do the right thing. And I didn’t trust the agent because there was nothing forcing it to do the right thing.
\n
His early steps put in tests, static analysis, and the right architectural patterns. With those in place, he could let the agent do more work.
\n\n\nMy role shifted from writer to curator. I don’t write most of the code anymore. I Define the patterns […] Review the test specs […] Review the output […] Update the harness […] Make strategic decisions […]
\n
He finishes the series with conclusions about how he’d generalize his experience to other circumstances.
\nBack in the land of my birth, there was some notable groans when the National Health Service decided to close nearly all of their Open Source repositories, supposedly to the security threat of LLMs. Closing repos like this isn’t an effective counter to LLM-augmented attackers. I suspect it’s no coincidence to see GDS (Government Data Services), the highly-regarded IT enablers in the UK government publish their position
\n\n\nMoving code from public to private as a substitute for investment in secure-by-design delivery, ownership and remediation is a warning sign because it reduces sharing and scrutiny, can slow coordinated improvement across government and suppliers, and does not remove the underlying weaknesses in a running service.
\n
Terence Eden memorably sums up his view on this:
\n\n\nWithin the UK’s Civil Service you occasionally hear the expression “being invited to a meeting without biscuits”. It implies a rather frosty discussion without any of the polite niceties of a normal meeting.
\n
I’ve seen a few cases where those developers who are most involved in working with LLMs find they are running into a problem with cognitive endurance, Adam Tornhill has joined this group:
\n\n\nOne of the big wins with agents is that they let us stay with the higher-level problem for longer. We get less sidetracked by details, dependency cleanup, and similar secondary tasks that used to break concentration.
\nBut there is a cost we are still underestimating. Agentic coding is mentally expensive.
\nI can usually sustain the pace for a couple of hours. Then I need a break. The pace is simply too intense. And based on conversations with other engineers, I do not think I am alone in that.
\n
He explains that working with The Genie means we are making more decisions in less time, this increase in decision density is hard on the brain.
\nHe responds by keeping agent tasks small, automating everything he can, and accepting that he won’t know every line of code as long as he has good verification mechanisms in place.
\nNotably, he has not gone in the direction of doing his work with swarms of agents that he coordinates. Instead has one long-running task that he babysits and one focus task
\n\n\nThat last point is important given the running-twenty-agents-in-parallel hype. I cannot even think about twenty meaningful things to build, and even less so about the resulting cognitive tax of the likely interruptions. It’s exactly the wrong thing to even consider. At least for humans. (And yes, I understand sub-agents and machine parallelisation. That is not what I’m objecting to. It is the parallelisation of human attention that does not scale).
\n
I liked that he included some thoughts about what folks can do in time outside this intense programming time. Not just “have a coffee” (although he includes that) but also about learning about the domain that the software supports.
\nA couple of pithy quotes from social media
\n\n\n\n“Metaphor debt” is when all of your metaphors involve the concept of “debt” because you can’t think of any other metaphors anymore.
\n
❄ ❄
\n\n\n\nIf a vegan crossfit fan is using Claude to write Rust, which thing do they tell you first?
\n
Karl Bode reacts to speakers getting booed when mentioning AI during commencement addresses. He points out that younger folks are increasingly unhappy with the tech oligarchy and their fruits.
\n\n\nThe thing is the kids aren’t stupid. They see the field clearly. They see the difference between what’s being sold to them by tech companies, the press, and commencement speakers, and what they have repeatedly seen with their own eyes.
\nThey’ve watched tech oligarchs spend the last decade mired in scandal after scandal, hype cycle after hype cycle, steadily enshittifying everything they touch along the way.
\n[…]
\nThe percentage of Gen Z that think AI’s benefits don’t counterbalance the risks now sits around fifty percent, up 11 percentage points in just the last year. Eight out of every ten believe that using AI makes the process of actual learning more difficult.
\n
He sees young people saddled with the perception of entering a worsening world - which leads them to rage against this latest fruit of the tech oligarchy. A rage that is easy for folks like me
\nRelevant to these concerns are a couple of items in last week’s Economist newspaper. The newspaper argues that historically major technological advances haven’t led to significant unemployment or drops in wages (paywalled article). The closest was the original industrial revolution in 19th Century Britain. There was a stagnation in wages during this period, but there was also a massive increase in population, from 4½ million to 12 million.
\nIt also points out that we’ll probably only understand the full consequences of all this when a recession hits, as this is when most unproductive jobs tend to be flushed out of the system.
\nA second article (also paywalled) indicates that AI is having some effect on graduate hiring. They did an analysis of surveys of recent graduates, looking to see if employment varied depending on a job’s exposure to AI. The least exposed quintile of subjects saw employment rate fall by 1.5% over the last couple of years, while the most exposed quintile’s drop was 6.6%.
\nLawfare isn’t impressed with the latest efforts by the US Government to regulate AI.
\n\n\nOn [last] Wednesday, the White House invited leaders of OpenAI, Google, Anthropic, Meta, and Microsoft to the Oval Office for a signing ceremony the following afternoon. President Trump was to sign an executive order on AI and cybersecurity—the administration’s most formal effort yet to establish a voluntary process for reviewing frontier models before their release. But roughly three hours before the ceremony, when some company executives were already in the air to Washington, the White House called it off.
\n
They see the proposed regulations as mild, and including some valuable measures to harden defenses against cyber threats.
\n\n\nBut it’s worth underscoring the implications of postponing (if not outright canceling) this order, which, by its own terms, was about as modest a frontier-AI intervention as the federal government could put on paper: voluntary, focused on the government’s own defenses, and explicitly barred from becoming a licensing regime. The objection isn’t so much about government coercion as about the government having any settled role at all. Voluntary, in other words, isn’t the floor of frontier AI policy in this administration; it’s the ceiling.
\nThis is a questionable position given that the concerns animating this draft order will likely grow in the near future. It is also self-defeating for those who applauded the order’s delay or demise. Far from resolving the risk of government meddling in AI, killing the order just leaves in place what Ball has described as the “opaque and essentially lawless” alternative: government access happening through back channels, on terms set case by case, with no stable rules at all.
\n
One of the problems here is a distinct lack of governmental expertise, either in AI or in software in general. Too much is being decided at the whims of the tech oligarchy, there isn’t any attempt to engage in the broader issues at hand. That’s not entirely a bad thing, trying to regulate something that’s still evolving so fast is usually a fool’s errand - but the problem here is the impact of AI is so big that there’s real danger in being too far behind.
\n❄ ❄
\nWhich leads me to a rare thing, an endorsement of a candidate for political office. If you are voting in congressional district MA-06 (North Shore of Massachusetts), I’d seriously look at Beth Anders-Beck, who is running for congress in that district. Beth has a long background in software development (including developing the notion of Forest and Desert), so would introduce expertise that Congress desperately needs. I’ve known Beth for decades, and have a high opinion of their intelligence, judgment, and ability to work with others. Congress doesn’t deserve Beth, but it does need her.
\n", "date_modified": "2026-05-27T19:40:00+00:00", "date_published": "2026-05-27T19:40:00+00:00", "external_url": "https://martinfowler.com/fragments/2026-05-27.html", "id": "urn:aggr:item:a865048843c2f32ea4edf5d4576759f1780c2163", "summary": "At the GOTO Conference in Copenhagen in 2025, Kent Beck and I spent some time on stage talking and answering questions from the audience - a format I refer to as “two old geezers on a park bench”. We talk about our experiences with…", "tags": [ "programming" ], "title": "Fragments: May 27", "url": "https://aggr.aymericbeaumet.com/items/martinfowler-com/2026-05-27-fragments-may-27/" }, { "authors": [ { "name": "Martin Fowler" } ], "content_html": "Vibe coding is enabling non-technical users (or as we call them, citizen builders) to build applications with AI that they simply could not have built before. When our AI applications team in Global Marketing at Thoughtworks was asked to scale a vibe coded prototype built by one of our citizen builders in global marketing, we discovered serious cracks that prevent vibe coded applications from going into production safely.
\nSpeed without guardrails is a risk no team can afford to ignore. What follows is the story of what we found, what it means for teams building with AI, and the steps we are taking to make sure every workflow, prototype, and app we ship is one we can stand behind.
\nThe AI applications team within Global Marketing was asked to scale a video assembly prototype built with Gemini, Replit AI and Claude AI to create on-brand videos to be used across our 10,000 employees. The team ran into two moments that stopped work cold. In both cases, the AI suggested a path with serious security implications. In both cases, it took a human asking the right question to catch it.
\nSecurity risk # 1
\nThe AI recommended making the storage bucket public, or setting cloud file storage to “anyone with the link.” When challenged, it justified this by saying every company does it. Only a firm rejection prompted a secure alternative.
\nThis could have leaked sensitive unreleased brand assets and audience data to the public internet.
\nSecurity risk # 2
\nA service account was assigned the Access Token Creator role, granting it the ability to create short-lived tokens and access databases and other resources far beyond what the task required. The team caught this before running the code.
\nThis would have allowed a compromised service account to move laterally through an entire cloud workspace.
\nThe key insight here is that AI tools often suggest the path of least resistance. That path is not always the secure one. Human judgment remains essential, but it should not be the only control. The goal is to give agents technical security rules as context from the first prompt, then validate their output through deterministic checks in the development workflow so insecure code, permissions, secrets, or infrastructure cannot pass unnoticed.
\n44%
\nRise in attacks exploiting application vulnerabilities, year on year
\n1 in 5
\nEnterprise breaches now caused by AI-generated code
\n50%
\nOrganisations with no sensitive data policies for AI
\n25%
\nAI-generated code with confirmed vulnerabilities
\nThese incidents are not isolated. Research published in 2026 confirms that AI-assisted coding at speed creates systemic security exposure. The same risks we encountered are playing out across the industry right now.
\n| Finding | \nStat | \nSource | \n
|---|---|---|
| AI-generated code with confirmed vulnerabilities | \n25% | \nAppSec Santa, 2026 | \n
| Rise in attacks exploiting application vulnerabilities, year on year | \n44% | \nSQ Magazine AI Coding Security Statistics, 2026 | \n
| Codebases with high or critical severity vulnerabilities | \n78% | \nBlack Duck OSSRA 2026 | \n
| Organisations with no sensitive data policies for AI | \n50% | \nAppSec Santa, 2026 | \n
| Mean vulnerabilities per codebase increase year on year | \n+107% | \nBlack Duck OSSRA 2026 | \n
| Enterprise breaches now caused by AI-generated code | \n1 in 5 | \nAikido Security, 2026 | \n
| New CVEs from AI-generated code in March 2026 alone | \n35 | \nGeorgia Tech Vibe Security Radar, SSLab, March 2026 | \n
| AI systems with prompt injection exposure in 2026 audits | \n73% | \nSQ Magazine AI Coding Security Statistics, 2026 | \n
| Share of all new enterprise software that is AI-generated | \n42% | \nSonar developer survey, 2026 | \n
| Security teams say keeping up with AI-generated code volume is getting harder | \n62% | \nProjectDiscovery AI Coding Impact Report, April 2026 | \n
After sharing these incidents with engineering and security colleagues, a clear message came back: telling an AI agent to be safe is not the same as enforcing that it is safe. Prompts can be overridden, misunderstood, or ignored. The moment a user pushes back on a restriction, or phrases a request differently, the constraint can evaporate.
\n“It is not sufficient to merely tell the LLM the desired behavior of your output artifacts. If you absolutely do not want something to be true, it must be codified in non-negotiable rules somewhere in your development lifecycle.” - Engineering leadership
\nThink of it this way: prompting for test-driven development is not the same as enforcing code coverage thresholds in your build tool. One is a suggestion. The other is a gate. Birgitta Böckeler’s work on harness engineering makes this concrete by outlining a mental model for building trust in coding agents. Instead of relying solely on prompts, developers wrap the agent in an outer “harness” structured along two axes:
\nBusiness functions like our marketing team, who are building with AI, are not exempt from the security obligations that apply to engineers building applications. Building security into software is a fundamental requirement for protecting customer and employee data. Even lightweight internal prototypes must comply with enterprise security standards. Without the right guardrails, AI-assisted development can expose sensitive data long before an application reaches production.
\nClient trust
\nAdhering to standards like ISO 27001 ensures the protection of sensitive data. All applications, regardless of how quickly they are built, must meet these security benchmarks to maintain the trust of customers and employees.
\nBrand integrity
\nCore work involves sensitive functional data (e.g., unreleased campaign assets, financial data, or audience insights). Over-permissioned service accounts put far more than code at risk.
\nReputation
\nWhen business functions like marketing lead with security discipline, they signal responsible AI adoption to the wider organisation and to clients.
\nYou do not need to be a security expert to start building responsibly. These three habits can get you started.
\nAdd your organisation's security guidelines as “Rules” in Claude, Cursor, or Replit to begin with (later on invest in a shared sensible default layer across all tools). The AI agent uses them as guidance, making secure patterns more likely from the start. They still need to be backed by deterministic checks that fail unsafe code, exposed secrets, broad permissions, vulnerable dependencies or insecure infrastructure before anything is deployed.
\nIf a tool recommends making something public or assigning a broad service account role, stop and ask why. The path of least resistance and the secure path are rarely the same thing.
\nAsk your AI to roleplay as a bad actor and pen test what it just built. This technique consistently surfaces vulnerabilities that forward-looking prompts miss, particularly around permissions and data exposure.
\nReading about risk is one thing. Doing something about it is another. These incidents prompted two practical initiatives. The principles behind them are replicable by any team building with AI, regardless of technical background.
\nWe compiled our technical security rules into a structured context file loaded into every AI coding session before any code is written. It covers zero trust enforcement, secrets management, harness engineering and supply chain integrity. The key distinction from a casual prompt is operational discipline: the file is versioned, loaded by default, reviewed, and paired with automated checks. It acts as a comprehensive inferential guide telling the agent what good looks like; but it must be paired with computational sensors in the pipeline to validate whether the output is acceptable.
\nCurrently, this automated consolidation ensures we see supply chain alerts the day they are published. In the future, as we work toward an agentic enterprise, we envision agents proactively creating story cards and identifying and patching known vulnerabilities behind the scenes for human review, significantly reducing the Software Development Lifecycle (SDLC) cycle time.
\nThe idea behind this approach is straightforward: AI tools read context at the start of a session, so make that context your technical security rules. The file is the result of working through your organisation's security requirements and structuring them in a form the AI can act on, not just acknowledge.
\nWhat follows is the kind of coverage any such file should include. The specifics will differ by organisation, but the categories are consistent.
\n| Area covered | \nWhat good looks like | \nWhy it matters | \n
|---|---|---|
| Zero trust and least privilege | \nStrict identity verification and minimum access rights on every service account and storage resource | \nSets the inferential guide parameters to prevent the token permission risk directly. | \n
| Secrets management | \nAI refuses to generate or store API keys, passwords or tokens in code; always routes to environment variables or a secrets manager | \nStops credential leakage before it reaches a repository. | \n
| Harness engineering gates | \nSAST scanning, credential scanning and infrastructure validation must pass before deployment; no reliance on prompt instructions alone | \nBacks up inferential instructions with deterministic, computational sensors. | \n
| Supply chain integrity | \nOnly well-established libraries; regular audits of every dependency for known vulnerabilities | \nReduces risk from AI suggesting obscure or unvetted packages. | \n
| AI accountability | \nAll AI-generated code is flagged for peer review and automated security scanning before deployment; no unsanctioned AI usage | \nRequired for compliance auditability. | \n
The key distinction from a prompt is that the file contains non-negotiable rules that force the AI agent to refuse requests that violate policy. If the AI agent is asked to bypass a check, disable logging or set something to public access, the rules should steer it to decline and explain why. But the important control is that deterministic checks and deployment gates should still catch the issue even if the agent fails to follow that guidance. That refusal is precisely what was missing in both of our near-miss incidents.
\nStaying informed is its own form of defence. The workflow monitors the tools and languages your team actively uses and delivers a daily digest of new CVEs, platform advisories and security bulletins. The coverage areas that matter most are consistent across organisations: the languages they write in, the cloud platforms they deploy to, the AI coding tools themselves, and the CVE database as a whole.
\nThe goal is simple: learn about a vulnerability on the day it is disclosed, not weeks later. At 42% of all new enterprise software now AI-generated or AI-assisted, the tools that accelerate development are also the tools most likely to appear in new CVE disclosures. Monitoring them actively is part of owning your security.
\nNeither of these approaches requires an engineering background to adopt. One is a policy document structured for AI consumption. The other is an automated search. What they share is the recognition that passive security awareness is not enough when AI is generating code at speed.
\nIntegrate harness engineering into your standard prototyping templates. Move from probabilistic prompts to explicit feedback loops. If a computational sensor (like an automated security scanner) triggers, the agentic loop must structurally force the model to self-correct until it passes.
\nCompile your organisation's technical security rules into a structured context markdown file and load it as “Rules” which the model will have to adhere to. It catches the most common missteps at the point where they are cheapest to fix, before any code is committed.
\nGive builders a secure-by-default starting point. Templates that pre-configure authentication patterns, private storage defaults, secrets handling and dependency scanning reduce the chance that anyone takes a shortcut under deadline pressure.
\nA shared starter harness built together by business functions, engineering and security gives every builder a secure foundation from day one, rather than each team independently rediscovering the same mistakes.
\nThis journey started when we were brought in to support another team building a video assembly platform for a Global Marketing hackathon. As we helped scale the solution, it became clear that “vibe coding” without enterprise-grade guardrails can introduce risks that organizations simply cannot overlook.
\nBy embedding our technical security rules directly into the agent workflow, we transformed those early near-misses into a secure, production-ready platform that was successfully rolled out to 150 users during the hackathon.
\nThat shift - from depending on humans to catch issues, to building technical security rules, automated checks and human accountability into the workflow has become our blueprint for moving fast while maintaining engineering rigor in the agentic era.
\nThese challenges are not unique to our team. Thoughtworks colleagues have been working through them in depth and publishing their findings. These are the pieces most directly relevant to what we described.
\nThoughtworks CTO Rachel Laycock on why engineering rigour does not disappear when AI generates code, it relocates. Covers putting coding agents on a leash, zero trust architectures, and why security guardrails are now non-negotiable.
\nA Thoughtworks-hosted gathering of practitioners and enterprise leaders found that security consistently gets deprioritised in AI adoption. Granting agents broad tool access, particularly to email, was flagged as an immediate and specific risk.
\nBirgitta Boeckeler, Distinguished Engineer at Thoughtworks, on placing deterministic controls inside the agentic loop. The definitive guide to building a harness that enforces security rules without paralysing the model.
\nHow to embed security guardrails and never-allow rules as mandatory pre-commit hooks in any AI-assisted development workflow.
\nThoughtworks annual technology outlook warns that without rigorous engineering oversight, generative tools risk compounding technical debt and introducing security vulnerabilities. Success requires a co-construction model where humans ensure AI-generated code maintains architectural integrity.
\nBig thanks to the team for the engaging discussions that shaped this article, in particular Julie Woods-Moss for championing AI prototyping risks, and Natalie Drucker for her significant role in shaping the content and title.
\nThanks to Martin Fowler for reshaping our perspective on short-term versus long-term habits and harness engineering, and to Kief Morris for anchoring the core argument on LLMs bypassing prompts.
\nAppreciations to Jim Gumbley for bringing in the critical governance and ISO27001 focus and reviewing the article. Honorable mentions - Craig Stanley for advocating the strategic rollout, and Andherson Maeda for the concept of a centralized skills repository, Simone Thompson for the bad-actor simulation methodology, Premanand Chandrasekaran for defining the technical boundaries of LLM enforcement, Matteo Vaccari for the nuances on balancing strict rules with model flexibility, and Bilal Jaffery.
\nGenAI (Gemini and Claude) was used to streamline the research process, pull in insights, and polish the language for maximum clarity and readability.
\n", "date_modified": "2026-05-27T14:03:00+00:00", "date_published": "2026-05-27T14:03:00+00:00", "external_url": "https://martinfowler.com/articles/vibesec-reckoning.html", "id": "urn:aggr:item:35cde1906917710991aa4e5145688af8ba3250ec", "summary": "Vibe coding is enabling non-technical users (or as we call them, citizen builders) to build applications with AI that they simply could not have built before. When our AI applications team in Global Marketing at Thoughtworks was asked to…", "tags": [ "programming" ], "title": "The VibeSec Reckoning", "url": "https://aggr.aymericbeaumet.com/items/martinfowler-com/2026-05-27-the-vibesec-reckoning/" }, { "authors": [ { "name": "Martin Fowler" } ], "content_html": "Vibe coding is building a software application by prompting an LLM, telling it what to build, trying it out, prompting for changes - but without looking at any of the code that the LLM generates. This technique can be used by people without any knowledge of programming. However the resulting software often shows problems with maintainability, correctness, and security - so is best used for disposable software written for a limited audience.
\nThe term was coined in February 2025 by Andrej Karpathy, an experienced programmer, in a post on X:
\n\n\nThere's a new kind of coding I call “vibe coding”, where you fully give in to the vibes, embrace exponentials, and forget that the code even exists. It's possible because the LLMs (e.g. Cursor Composer w Sonnet) are getting too good. Also I just talk to Composer with SuperWhisper so I barely even touch the keyboard. I ask for the dumbest things like “decrease the padding on the sidebar by half” because I'm too lazy to find it. I “Accept All” always, I don't read the diffs anymore. When I get error messages I just copy paste them in with no comment, usually that fixes it. The code grows beyond my usual comprehension, I'd have to really read through it for a while. Sometimes the LLMs can't fix a bug so I just work around it or ask for random changes until it goes away. It's not too bad for throwaway weekend projects, but still quite amusing. I'm building a project or webapp, but it's not really coding - I just see stuff, say stuff, run stuff, and copy paste stuff, and it mostly works.
\n\n
The key point about vibe coding is “forget that the code even exists”. This is what gives it much of its usefulness, but also its limitations.
\nSince the November Inflection many programmers are getting LLMs to write all their code, commenting that they may never write a line of code directly again. However they do care about this code, reviewing it, paying attention to its internal structure. In that case, they aren't forgetting the code exists, so it's really a different thing that I call Agentic Programming. Sadly the term “vibe coding” really caught on, so many people use it to mean agentic programming. However I feel that despite this rapid Semantic Diffusion, it's worth trying to keep the concepts of vibe coding and agentic programming separate, as they are both different to use and different in their consequences.
\nBecause a vibe coder doesn't look at the code, they don't need programming skills, so it's perfect for someone with no programming knowledge to build applications for their own use. Experienced programmers may also find it handy for rapid development of disposable software or prototypes.
\nVibe coding is still new, so we are exploring its limitations, and those limitations change as the sophistication of models and their harnesses change. These limitations do introduce considerable risks, particularly if the vibed software is used widely or has access to sensitive information.
\nPerhaps the most serious risk is that of security. LLMs are inherently vulnerable as they provide a large attack surface for predators. Vibe coded applications can often expose sensitive information or worse, credentials to attack deeper into an organization's systems. Even non-programmers need to be aware of the Lethal Trifecta.
\nWith little attention to the code, vibed software can rapidly produce many lines of code of a very low quality. Such code makes it difficult, even for an LLM, to modify and enhance the software in the future. While it's possible that growing LLM capabilities will allow it to work with even the largest bowls of spaghetti software, thus far it seems clear that well-structured software makes life easier for LLMs too.
\nLLMs are famous for habit of hallucinating incorrect facts and presenting these with great confidence. This habit also leads them to create software that behaves incorrectly - and those errors may not be manifest to the user. Furthermore the non-determinism of LLMs means that it's likely that asking an LLM to enhance some software could easily lead it to introduce errors, even in parts of the code that shouldn't change due to the new request. We should thus treat LLM-generated software with skepticism, it can still be useful, but we need to be aware of the risks.
\nOn the whole vibe coding software is best used for disposable software that's only used by its author or a close group of collaborators who understand and accept the risks involved. Code that is more complex, more widely-used, and with more consequences to its risks should not be forgotten about.
\n", "date_modified": "2026-05-21T11:49:00+00:00", "date_published": "2026-05-21T11:49:00+00:00", "external_url": "https://martinfowler.com/bliki/VibeCoding.html", "id": "urn:aggr:item:f50587a4272713ec4b05d466b75d960a18e40382", "summary": "Vibe coding is building a software application by prompting an LLM, telling it what to build, trying it out, prompting for changes - but without looking at any of the code that the LLM generates. This technique can be used by people…", "tags": [ "programming", "bliki" ], "title": "Bliki: Vibe Coding", "url": "https://aggr.aymericbeaumet.com/items/martinfowler-com/2026-05-21-bliki-vibe-coding/" }, { "authors": [ { "name": "Martin Fowler" } ], "content_html": "There are multiple dimensions we usually want to achieve and monitor in our codebases: Functional correctness (works as intended), architectural fitness (is fast/secure/usable enough), and maintainability. I define maintainability here as making it easy and low risk to change the codebase over time - also known as “internal quality”. So I don't only want to be able to make changes quickly today, but also in the future. And I don't want to worry about introducing bugs or degradation of fitness every time I make a change - or have AI make a change. I usually see the first signs of cracks in the maintainability of an AI-generated codebase when the number of files changed for a small adjustment increases. Or when changes start breaking things that used to work.
\nInternal quality problems affect AI agents in similar ways that they affect human developers. An agent working in a tangled codebase might look in the wrong place for an existing implementation, create inconsistencies because it has not noticed a duplicate, or be forced to load more context than a task should require.
\nIn this article, I describe my experimentation with various sensors that help us and AI reflect on the maintainability of a codebase, and what I learned from that.
\nI'm working on an internal analytics dashboard for community managers that reads chat space activity, engagement, and demographic data from a combination of APIs and presents the data in a web frontend.
\n![\"Overview](\"https://martinfowler.com/articles/sensors-for-coding-agents/sensors-example-application.png\")
Figure 1: The example app: web UI, service layer, and external APIs.
\nThe tech stack is a TypeScript, NextJS, and React. The backend reads and joins data from the APIs. The application has been around for a while, but for the sake of these experiments I rebuilt it with AI from scratch.
\nThere are hardly any guides (e.g. markdown files) for AI about code quality and maintainability present, I wanted to see how well it can do just by relying on sensor feedback.
\n![\"Overview](\"https://martinfowler.com/articles/sensors-for-coding-agents/sensors-example-overview.png\")
Figure 2: Where sensors can run: during the initial coding session, in the pipeline, on a schedule, and in production.
\nThis is an overview of the sensors I set up across the path to production.
\nDuring coding session
\nSensors that run continuously alongside the agent to provide fast feedback.
\nAfter integration - pipeline
\nThe same computational sensors run again in CI. The in-session sensors give the agent early feedback during development. The CI pipeline confirms the result on clean infrastructure and after integration.
\nRepeatedly
\nSensors that run on a slower cadence to detect drift that accumulates over time, rather than errors that occur in the moment.
\nWith this context out of the way, let's dive into the first category of sensors.
\nThroughout building the application, I used a mix of Cursor, Claude Code, and OpenCode (in that order of frequency). My default model was usually Claude Sonnet, for some of the planning and analysis tasks I used Claude Opus, and for implementation tasks I frequently used Cursor's composer-2 model.
\nI'll start with my learnings from using ESLint in this application. Basic linting tools like ESLint mostly target maintainability risk at the level of individual files and functions.
\nIn my experience, the AI failure modes that are the most low-hanging fruit for static code analysis are
\nHowever, these weren't even active in ESLint's default preset, I had to configure maximums for them first. Hopefully, static analysis tools will evolve to provide better presets for usage with AI. A bit of research shows that people are also starting to publish ESLint plugins with rule sets that are specifically targeting known agent failure modes, like this one by Factory, with rules about things like requiring test files or structured logging.
\nA sensor is meant to give the agent feedback so that it can self-correct. Ideally, we want to give the agent extra context for that self-correction - a good kind of prompt injection. To do that, I built a custom ESLint formatter to override some of the default messages - with the help of AI of course, naturally.
\nHere is an example of my guidance for the no-explicit-any warning.
We want things to be typed to make it easier to avoid errors, especially for key concepts.\nBut we also want to avoid cluttering our codebase with unnecessary types. Make a judgment\ncall about this. If you choose to not introduce a type, suppress it with:\n// eslint-disable-next-line @typescript-eslint/no-explicit-any -- (give reason why)`,
\nStatic code analysis has been around for a long time, and yet, teams often didn't use it consistently, even when they had it set up. One of the reasons for that is the management overhead that comes with it. Effective use of this analysis requires a team to keep a “clean house”, otherwise the metrics just become noise. In particular warnings like the no-explicit-any example above are tricky, because you don't always want to fix them - it depends. And suppressing them one by one has always felt tedious, and like noise in the code.
With coding agents, we might now have a chance at that clean baseline. In the guidance text above, the agent is told to make a judgment call, and allowed to suppress a warning in the code. This keeps the suppressions manageable, visible and reviewable.
\nFor thresholds, like the maximum number of lines, or the maximum allowed cyclomatic complexity, I told the agent in the lint message that it may slightly increase the thresholds if it thinks that a refactoring is unnecessary or impossible in a particular case. This doesn't suppress the threshold forever, just increases it, so that the rule fires again if it gets even worse in the future. Constraints are preserved without forcing a binary suppress-or-comply choice.
\nno-console, telling AI off when it uses console.log. In the backend, I want it to use a logger component instead. In the frontend, I might want to not use direct logging at all, or at the very least I need to use a different logging component. This is another example of the power of the self-correction guidance, and where AI can help with semantic judgment and management of analysis warnings.max-lines and max-lines-per-function rules. I've seen AI do quite a bit of useful refactoring and breakdown into smaller functions and components as a result of this sensor feedback. However, in the React frontend, I'm seeing a worrying trend of components with lots and lots of properties as a result of passing values through a growing chain of smaller and smaller components. I haven't got useful observations yet about how good AI might be at making consistent decisions between tradeoffs like that.Overall, I was positively surprised by how many things I can cover with static analysis. I had to remind myself multiple times why it has been somewhat underused in the past, and what has changed: The cost-benefit balance. Cost is reduced because it's much cheaper to create custom scripts and rules with AI. And the benefit has also increased: the analysis results help me get a first sense of lots of hygiene factors that wouldn't even happen that much when I write code myself, so I can get common AI mistakes out of the way.
\nHowever, I can't help but wonder if this can also lead to a false sense of security and an illusion of quality. After all, another reason why linters like this have been less used in the past is that they have limits, and we have been wary of using them as a simplified indicator of quality. There are lots of more semantic aspects of quality that static analysis cannot catch, it remains to be seen if AI can adequately fill that gap in partnership with those tools. I also discovered new supposed issues in the code every time I activated a new set of rules. It was always a mix of irrelevant things and things that actually matter. So I worry about feedback overload for the agent, sending it into a spiral of over-engineered refactorings.
\nBasic linting is mostly focussed on quality and complexity within a file or function. Next I started looking into sensors that could give me and the agent feedback about maintainability concerns that cross file and module boundaries. Analysis tools in this area are historically even more underused than the basic linting.
\nTo learn about the potential of sensors that can help us and AI keep up good modularity inside of a codebase, I explored three things:
\nLet's start with dependency rules. I worked with the agent to come up with a layered module structure for my application, about half way through implementing it. I asked it to help me write dependency-cruiser rules to enforce these layers.
![\"\"](\"https://martinfowler.com/articles/sensors-for-coding-agents/sensors-structure-chativity.png\")
Figure 3: Layered module structure and dependency rules
\nFor example, one of the rules enforces that code in the clients folder never imports anything from the services folder:
{\nname: “clients-no-services”,\ncomment:\n“API clients must not depend on the orchestration layer above them. “ + LAYERS,\nseverity: “error”,\nfrom: { path: “^server/clients/”, pathNot: “/tests/” },\nto: { path: “^server/services/” },\n},
\nAs with the ESLint messages, I also expanded the error messages a bit to be self-correction guidance, recapping the layering concept as a whole:
\nERROR clients-no-services\nAPI clients must not depend on the orchestration layer above them.\n[Layers: routes -> services -> clients + domain; Services orchestrate: fetch data via clients, compute via domain -- no I/O, no SDKs, no knowledge of data fetching.]
\ndependency-cruiser feedback, so it did help keep my folder concepts.At the point when I introduced these rules, the structuring of code into folders had already become a little bit haphazard. I could see how the rules helped the agent clean that up, and then continue enforce these layers going forward. So I've found it quite a useful replacement for describing code structure in a markdown guide. However, tools like this are limited to what is expressible via imports, file names, and folder structure.
\nNext, I experimented with the extraction of typical coupling metrics from my codebase, i.e. the number of incoming and outgoing imports and calls per file.
\nI didn't use any existing tools for this, instead I had a coding agent write an application that creates those metrics with the help of the typescript compiler, so that I could have maximum flexibility to play around with this as part of my experimentation. I had it add two interfaces: A web interface with a bunch of different visualisations of those metrics for my own human consumption. And a CLI that can provide those metrics to a coding agent.
\n![\"\"](\"https://martinfowler.com/articles/sensors-for-coding-agents/sensors-coupling-dashboards.png\")
Figure 4: Coupling metrics: web visualisations and CLI for agents.
\nMost of these visualisations are well established concepts, like a dependency structure matrix (DSM). I found them tedious to interpret, and even though they were vibe coded and could most certainly be improved, I think that had more to do with the nature of the data. It's quite detailed data that needs a lot of context and experience to interpret it, and map it back to more high level good practices. So I have a feeling that these types of tools still won't really help reduce a human's cognitive load much when reviewing codebases that were changed by AI.
\nI gave an agent access to this custom CLI (coupling-analyser) and asked it to create a report based on the data, including suggestions of how to improve the critical issues.
Here is an excerpt of what that prompt looked like - I'm mainly reproducing this to show you that I didn't actually give it much guidance on what good or bad modularity looks like, I mostly delegated to the model to interpret what good and bad looks like:
\nProduce a markdown report on modularity and coupling quality for the target TypeScript codebase, grounded in actual CLI output from npx coupling-analyser, not guesswork from static browsing alone.
Execute the CLI and capture stdout. Use the report subcommands—combine as useful for the question: …
Use clear headings. Prefer concrete module IDs / paths and numbers quoted or paraphrased from CLI output.
\nSuggested sections:
\nContext — What was analyzed
\nExecutive summary — 2–5 bullets: overall modularity posture, top 1–3 systemic issues.
\nFindings from the tool — Summarize hotspots, top risks, notable cycles or mutual dependencies, and behavioural highlights as reported by the CLI.
\nInterpretation (modularity lens) — Tie metrics to software design: cohesion vs. spread of change, stability vs. dependency direction, fan-in/fan-out intuition, cycle impact.
\nDeep dives for each high and critical issue
\n…
\nThis LLM-led analysis actually pointed me to the same coupling hot spots that I would have found by looking through the visual diagrams, just in a format that was more digestible. And asking the LLM to ground its analysis in the results from the deterministic tool gave me a higher level of confidence, and probably also used less time and tokens than if the agent had scanned the codebase itself to find coupling problems.
\nWhat the LLM found based on this data was quite lackluster (I used Claude Opus 4.7 for this):
\nzod) schema between frontend and backend, declared a “god module” by the LLM. This is a common pattern though to create an explicit contract between backend and frontend, and is not as much of an issue when backend and frontend evolve together anyway, or even live together in the same repo, like in my case.index.ts file in the domain folder indiscriminately exposed all files in ./domain, and is imported by lots of places. While that is also a common pattern to create explicit contracts for a layer, it does have its pros and cons, and is at least worth an investigation to see if it is appropriate for this codebase.The examples above show that even more so than with the basic linting, good and bad does not have a clear definition, instead it is all about what is appropriate. And what coupling is appropriate depends on a lot of context, not just the raw call and import graph of a codebase. So based on this small experiment, I don't have the impression that this type of coupling data is useful to AI on its own.
\nA more practical use I can imagine for this data is during risk triage for code review. When I review a code change made by AI, it seems useful to know what the impact radius of the changed files is, so that I can pay more attention when e.g. a file with 10+ callers is changed. Or an AI review agent could use the data to prioritise where it spends its tokens.
\nThe lackluster results from the coupling data experiment could have multiple reasons:
\nSo the final thing I did was to go fully down the inferential route and use Vlad Khononov's “Modularity Skills” to analyse the codebase design and find modularity issues. This proved to be very fruitful! It gave me lots of interesting pointers for refactorings that would obviously reduce the risk of future changes. I ran the skills a second time and gave them access to my coupling analysis CLI. The AI mostly found confirmation in the data, but not any additional findings. On the contrary, it pointed out lots of things that the CLI was missing. It's also worth noting that the second run of the analysis (without context of the first one) surfaced yet another issue that the first run did not find. A useful reminder that when it matters, it's often worth running an LLM-based analysis multiple times, to get a fuller picture.
\nHere are some highlights from the results (model used was Claude Opus 4.7, same as for the coupling analysis):
\ndependency-cruiser can be effective live sensors to enforce some basic folder structures and dependency directions, but they can only go so far.Overall, codebase design and modularity seems like a concern where computational sensors alone cannot help us much, AI is needed to add semantic interpretation, and consider trade-offs.
\nTests have many purposes — they help us think about and drive our design, they document the wanted behaviour of the application (they are the ultimate specification!), and they help us detect regressions, i.e. they tell us when we break pre-existing functionality with a change. Effective regression tests play a big role in the maintainability of a codebase, they make it much safer to change it. So in the context of maintainability sensors, this section is about the test suite's role as a regression sensor.
\nWhen a pre-existing test fails, we have to ask ourselves a question: “Did I break something accidentally, so I need to change my implementation? Or am I changing the behaviour intentionally, so the tests have to change to adapt to this new specification?” A failing test gives AI the opportunity to ask that very question. It might not always take the right decision, mind you! But a good test suite decreases the probability that AI breaks wanted pre-existing behaviour.
\nIn my chat analytics application, I had the agent write all the tests over time without much oversight other than manual testing and keeping an eye on the test coverage. I wanted to have a full AI-generated test suite to analyse its regression effectiveness in hindsight.
\nThere are two main risks with the approach of AI generating tests without review:
\nWhat is in our toolbox?
\nIn my application, I used coverage and mutation testing, as property-based testing and fuzz testing weren't as suitable to my use case.
\nHere is a small example from my codebase to illustrate how mutation testing can help us find gaps in assertions. The agent created this diagram for me during the analysis of mutation testing results:
\n![\"Mutation](\"https://martinfowler.com/articles/sensors-for-coding-agents/sensors-mutation-example.png\")
Figure 5: Mutation testing example from the codebase.
\nThe mappers.ts file reported 100% statement coverage and 75% branch coverage — but it turned out to have no unit tests, and Stryker (the mutation testing tool I used) reported 13 survivors (i.e. after 13 of Stryker's code mutations the test suite was still green). The coverage in this case was high because the codebase has a big acceptance test that ultimately called these functions — coverage tells us that a line was executed, but not that its impact was verified. If this little mappers helper function dvpToSchema would be changed in the future, it could potentially break the display of a data graph in the UI.
"""Query a Stryker mutation-testing JSON report from the command line.
\nUsage:\npython query_stryker.py <report.json>; <command> [options]
\nCommands:\nsummary Overall status totals, mutation scores, thresholds.\nfiles Per-file breakdown, default sorted by mutation score asc.\nhotspots Lines with the most survivors / no-coverage mutants.\ntests Test effectiveness: weak, unused, or top-killer tests.
\nExamples
\npython ./query_stryker.py reports/mutation/mutation.json summary
\npython ./query_stryker.py reports/mutation/mutation.json files --top 10 -v
\npython ./query_stryker.py reports/mutation/mutation.json files --changed -v
\npython ./query_stryker.py reports/mutation/mutation.json hotspots --file server/services/ai-summaries.ts --top 30
\n"""
\nThere currently seems to be a trend towards more end-to-end style acceptance tests. As mentioned in the beginning, AI has gotten really good at generating tests, so it has become quite normal for developers to just let AI generate lots of tests, without much review. Reviewing unit tests in particular can be very tedious. I'm not saying it's a good thing not to look at them at all — but I acknowledge the reality that it is unrealistic to think that human review of all tests is sustainable, and it's unrealistic to think that people will actually do it. So while we search for the appropriate testing pyramid/ice cream cone/muffin shape of the AI coding future, techniques like approved scenarios are becoming popular. As demonstrated above, acceptance tests increase coverage, but are often not very assertion-heavy, giving us a false sense of security in test effectiveness — mutation testing helps us monitor that gap.
\nMutation testing has a practical limitation of course: It is quite resource intensive. In my setup I didn't run it continuously (like some of my other sensors), but triggered incremental runs manually.
\nComputational sensors impressed me most at the file and function level. Cross-file concerns like modularity and coupling were a different story, the raw data itself was very noisy and not that useful without semantic interpretation of an LLM, i.e. an inferential sensor. But I was very impressed by the outputs and advice I could get from that with a good prompt, and also by the potential to present this information in different ways, for different experience levels.
\nWhat I haven't seen in my experiments, but suspect can become more of an issue, is conflicts between sensors. The max-lines and max-lines-per-function rules showed some signs of tension, the refactorings to smaller and smaller functions pushed complexity into component property chains instead. More trade-offs like that are probably lurking, and it will be interesting to see over time if and how that becomes a problem.
\nI did not bother with guides at all in this application, for the sake of seeing the effect of the sensors more purely. I'm curious about how the balancing of guides and sensors will evolve. Once we feel confident in a set of sensors, what guides can we delete? Do sensors make the use of weaker models more realistic? How do we keep guides and sensors consistent with each other, and will we find ways to bundle them together somehow, to make them easier to maintain?
\nIn the regression testing area, my eyes have really been opened to how crucial mutation testing becomes when we make the decision to leave most of the testing to AI... And I want to stress once more that there is a whole other conversation to be had about correctness of tests!
\nWhile some of these sensors really do increase my trust into the quality of the outcomes, they are not a magical solution to take the human totally out of the loop. But I definitely experienced an improvement in my review experience and trust level with both computational and inferential sensors as my partners.
\nThanks to Chris Ford and Matteo Vaccari in particular for the conversations about sensors, and all my colleagues at Thoughtworks who engaged with this topic and gave me feedback and ideas.
\nGenAI was used for research and polishing the language.
\nHow did I do this practically then, how did I provide this sensor data to the agent? In all of the experiments described, I used a vibe-coded little “sidecar” application that could run all of my computational sensors. It's not a fully realised tool and I made some compromises for simplicity, but it helped me explore and speculate about what good tooling for this might look like.
\nThe sidecar is a CLI that can read a config file describing all sensor commands, and then run them continuously, in intervals. This is somewhat inspired by how we run test suites in watch mode while we're typing code.
\n![\"A](\"https://martinfowler.com/articles/sensors-for-coding-agents/sensors-sidecar-setup.png\")
Figure 6: Overview of the sidecar setup
\nThe CLI can offer both situational awareness for the human in supervised sessions, and a token-efficient and guidance-enriched summary to the agent.
\n![\"A](\"https://martinfowler.com/articles/sensors-for-coding-agents/sidecar-human-view.png\")
Figure 7: Visual overview of sensor status
\n![\"A](\"https://martinfowler.com/articles/sensors-for-coding-agents/sidecar-check-result.png\")
Figure 8: Summary the agent gets when it checks for the status
\nA few things to note in the agent summary:
\nThis example illustrates the opportunity we have in a setup like this to provide much more guidance for the self-correction than when we use the tools out of the box. And the agent summary could of course be minimised even more and only include information on things that have failed, similar to an “evergreen” type of CI monitor.
\nHow to get the agent to check the sensor status regularly? There are multiple options for this.
\npre-commit hook can be a good way to force a sensors check to take place before code even gets into a commit.![\"A](\"https://martinfowler.com/articles/sensors-for-coding-agents/sidecar-extension-pi.png\")
Figure 9: Custom extension in Pi, asking the user if they want to run the sensors on startup, and showing sensors state in the status line.
\nEvery time I added a new sensor, I had to write a little parser that could get the results from whatever output a sensor provided (e.g., reading coverage-final.json, or the semgrep output). This became quite easy once I created a skill for it, but it does require changing the CLI's code every single time.
![\"A](\"https://martinfowler.com/articles/sensors-for-coding-agents/sidecar-parsers.png\")
Figure 10: A tool like this needs to know how to parse each sensor's outcomes
\nSo I started moving to a default parser that expects a certain JSON schema as output from the schema. In this approach, instead of having the CLI run e.g. ruff directly, it would run python ./ruff_sensor.py -- check sensors/, where ruff_sensor.py would run ruff, and then turn its output into the JSON schema the CLI understands and can display.
This is an example of how the decision-making for customisation of CLIs like this might change. In the past, building a plugin system for the most common sensor types would be an obvious decision. But now, for some things we can expect users to build a bit of that customisation for themselves, to keep the tool itself simpler.
\nNaturally, the more of these sensors we want to run, the more resources they are going to need. The difference in performance of build and dev tools across different tech stacks can be quite significant. Yet another thing that is good for agents as well as humans: fast and efficient build tools providing quick feedback loops are very important. I used this setup for a few Typescript codebases, as well as a Python application. And you might have guessed it, the Python setup (where many of the tools these days are “blazingly fast” Rust implementations) didn't give me any trouble at all. Running the sensors for a Typescript codebase on the other hand regularly started my M3's fan. But hold your Javascript prejudices - this was actually in part the result of my virus scanner kicking into overdrive, because those sensors were rewriting files all the time...
\nOne of the frequently asked questions about harnesses and harness engineering is how to know if the setup is effective. In the context of sensors, this question is similar to that of “How effective are our CD pipelines?”. If they're always green, that's suspicious, because they never catch anything. If they're always red, that either means that we're really bad at producing code that is up to our standards, or that the pipeline is way too sensitive or flaky.
\nBut even though this is an “it depends” question, we need data to answer it. So I started logging a history of sensor states every time they were checked, to get some data.
\n![\"Screenshots](\"https://martinfowler.com/articles/sensors-for-coding-agents/sensors-history-example.png\")
Figure 11: History of sensor states
\nThis historical data could answer questions like these:
\nI have found this setup very convenient both as a head up display for myself in supervised coding sessions, and as a source of information for my agent.
\nThe biggest gaps as of now are:
\n