The Pragmatic Engineer

The revised rules of engineering management with Will Larson

Gergely Orosz — Wed, 21 Oct 2026 16:58:21 +0000

Before we start: I’m very excited to announce an The Pragmatic Engineer Unplugged live event: with Diogo Almedia (the creator of Jev) and Bryan Cantrill (cofounder and CTO, Oxide). 4 November (Wednesday), San Francisco, the SFJAZZ Center. It’s free to attend, but spaces are limited and registration is required.

If you’re in SF or in the area, perhaps see you there? RSVP here

Listen and watch now on YouTube, Apple, and Spotify. See the episode transcript at the top of this page, and timestamps for the episode at the bottom.

• turbopuffer – vector and full-text search built on object storage: fast, 10x cheaper, and extremely scalable. Under the hood, the team are rewriting the tpuf storage engine. Follow along until it hits production at turbopuffer.com/v3

• What if you only needed a single test to harden your system? Get a free introduction to property-based testing: through a live, online webinar hosted by Hillel Wayne (a recent podcast guest), author of Logic for Programmers and Developer Educator at Antithesis. RSVP here.

• Entire – Git hosting, rebuilt for the agentic era. Entire hosts your code in-region, and is up to 89x faster than any other competitor. Mirror from GitHub with a single click – I’ve already done so.

What’s changed in “the unwritten rules” of engineering management since the 2010s, what drove the shift to today, and what has it done to the engineer-to-CTO career path? Today’s guest is one of the best people I know who can tackle these questions.

Will Larson is CTO of Imprint, was an engineering executive for 10+ years, and is the author of several popular books on engineering leadership, including ‘The Engineering Executive’s Primer,’ ‘Staff Engineer, An Elegant Puzzle,’ and ‘Crafting Engineering Strategy.’ Previously, he was also CTO at Carta and Calm, and an engineering leader at Stripe and Uber.

In this episode, we explore how engineering leadership has evolved from management practices shaped in the long post-financial-crisis ZIRP era to the altered expectations of engineers and managers today.

Is middle management an increasingly precarious career path, and might managers need to reskill and return to hands-on technical work – or even consider retirement? Will makes the case, and we hear why the traditional path to engineering executive roles is changing, as well as why taking one step backward might sometimes be the best way forward.

We discuss Will’s early career, and get his first-hand account of Digg’s disastrous relaunch, his own path to CTO, and he tells us why career ladders still matter. We also explore how AI is changing engineering roles and bringing technical decision-making back to leadership.

www.youtube.com

Here are some standout talking points from our conversation, during which Will recommends middle managers to go back to line management and re-skill like he has done. He didn’t ask me to, but I want to mention that he’s hiring for several key engineering manager roles at Imprint to use agents to rethink how software teams work.

1. Will shares an inside account of how a 10 website sank into irrelevance after a v4 rewrite. Digg was the “king” of social media sites in the late 2000s; bigger than Reddit, and one of top-10 most visited websites, globally. That all crashed to a halt in 2010 when the company embarked on a massive rewrite called Digg v4 – and it was a disaster. It was the start of Digg’s downfall.

Will worked at Digg during this rewrite, and shares how Digg was already losing traction and traffic to Facebook, and that Google changed its algorithm which caused 60-70% traffic loss. From the outside, Digg looked like it was growing, but its business prospects were shrinking fast. The company had about 120 people and was burning through money fast.

Into this entered the v4 rewrite, which was carried out to a hard deadline set by Kevin Rose, the newly-returned CEO at the time. The v4 version alienated passionate users, and had major performance and reliability problems.

Will and a small engineering team fixed the issues. The root cause was Python memory leaks introduced by devs who didn’t know Python so well. Will has shared more detail on his blog.

Even so, Will doesn’t think the rewrite killed Digg: with or without that rewrite, Digg would’ve met the same fate of being overtaken and replaced by Facebook and Reddit.

2. At Yahoo, a “black market” for servers popped up after the cofounder insisted on personally approving new servers. A hilarious story of how incentives drive behavior: Yahoo cofounder, David Filo, wanted to drive accountability by teams justifying every server purchase. This created a “black” market and Will recalled that one engineer on his team had 30 servers left from a canceled project that he traded with devs who could not get approvals elsewhere, in exchange for stuff he needed. Also, servers didn’t have to be returned, so Yahoo ended up running a large number of outdated, power-hungry machines!

3. In the 2000s, having annual 1:1s with managers was normal. Will worked at Yahoo in 2008-2010 and had two 1:1 sessions with his manager: one in which they asked Will about someone else’s performance, and the next one when he handed in his notice after it finished. It was pretty common for 1:1s with managers to be rare things at the time.

4. Too many Staff+ engineers only want to work on fun problems. At Stripe, Will led an org of about 250 people, where lots of the most senior engineers regularly sought more “engaging” problems to work on. Will eventually found his optimal response:

“I only have suffering to offer, but this suffering is what’s going to make the business succeed.”

However, he felt that he was seen by some as guilty of violating an unwritten rule of management by not finding enough – or interesting enough – problems for them to engage with. Today, at Staff+ levels, engineers putting business problems first is a baseline expectation.

5. ZIRP shaped engineering management practices in the 2010s, Will believes. ZIRP stands for zero interest rate period – and we did a deepdive on the impact ZIRP had on engineering management. He recommends the article ‘ZIRP explains the world’ by Ranjan Roy. From Will:

“Back then, your ability to hire and close and retain became the differentiator between successful companies. The less successful ones simply couldn’t retain great people.

[This dynamic on hiring and retaining] pulled leverage back into the engineer’s hands and created a very different environment. So, companies were pitching themselves as a more relaxed environment, more focused on you, and offering more career development. And I think in ways that I think were often kind of self-defeating.”

6. “Hire more engineers” used to be the answer from many engineering leaders in the 2010s. Today, it’s not, he says:

“I tell my teams that I work with now that hiring is the least interesting way to solve every problem. We should almost never talk about hiring.

But in the 2010s, we basically told managers they couldn’t think about technological solutions. And then what do they have? They have process solutions, they have scheduling, or they have people to hire.”

7. Focus on engineering wellbeing could have led to less reliable software. During the 2010s, several companies chose to not put the highest-paid engineers oncall, and for it to be handled by others. But when engineers are oncall for the software they build, it leads to better, more reliable software. Back then, not having oncall was a way to hire and retain talent, but it also led to lower quality as engineering teams did not feel the pain of outages caused by their own software.

Side note from Gergely: I wonder if AI agents will make a difference here. One of the worst things about oncall is getting awoken by the pager. Could we design loops that automatically mitigate incidents that we expect and are well-prepared for?

8. Software engineering change is more extreme, and it’s driving away some engineers. Will knows someone who decided to retire from tech because AI has destroyed the ability to be “in the flow.” Engineers are being more productive, but enjoy it less. By focusing on the outcomes, Will says he’s having a great time.

9. The tech lead manager role is now very in demand. Will wrote previously about how the tech lead manager role is a career trap, and I have a similar opinion in the deepdive ‘Engineering leadership skillset overlaps:’

Things are changing inside companies; EMs are starting to manage 12-14 people, instead of 4-8, as before, while Directors are managing more TLMs. At Imprint, some of Will’s best hires have been managers who also want to write software. The TLM role is perfect for this. In his book, he describes a great TLM as someone who has the capability, but no desire, for bureaucratic work; by default, they’re builders.

10. Engineering managers used to expect not to make technical decisions, but that’s outdated now. Will recalls:

“One of the worst parts of being an engineering manager in the 2010s was when you joined the ranks of engineering management, one of the first lessons you’d get was: ‘you need to never make a technical decision again.’ That’s literally what I was told.”

It was the same advice I received when becoming an EM: let your engineers make the technical decisions, get out of their way, and support them in growth. The tide is very much changing these days, with more tech companies expecting managers to be hands-on and in the details.

11. Great engineering executives (CTOs, VPEs) need to be technologists and bureaucrats. Will believes an engineering executive has two jobs:

Be a bureaucrat: manage incentives, create and deal with bureaucracy, and know how to navigate it. The job often includes running functions you know nothing about
Be a technologist: know what to build, how it works, and whether a technology choice makes sense or not

Will reckons that until recently, many engineering executives from the middle-manager path were bureaucrats before they were technologists.

12. Blunt message for middle managers: retire or re-skill. Will left a team of 250 at Carta to head up 25 tech people at Imprint. He purposely took a “step back” in his leadership career because he believes things are changing so radically that the engineering leadership profession that we know will not exist in three years.

Amid all this, Will says that line management can be a lot more engaging for executives these days because the tools for line management and the tools for writing software are really good. However, the tools of bureaucracy are still pretty terrible, he believes. That’s why some talented engineering folks get jaded about the work:

“For a lot of folks, there is this sense the industry has betrayed them and that they’re owed something that they’ve worked for. And the thing is, they’re right!

They have worked incredibly hard, but the rules have changed. Since the 2010s, we’ve forgotten that our industry is actually very volatile. We really do just get paid to do things that are valuable to the companies. There’s no entertainment orientation to why these companies employ us.”

• The end of ZIRP and what it means for startups and the tech industry

• Getting an Engineering Executive Job

• What is a Staff+ Engineer?

• Inside Stripe’s engineering culture

• Engineering career paths at Big Tech and startups

• Staying technical as an engineering manager

• Performance calibrations at tech companies

• Scaling Uber with Thuan Pham (Uber’s first CTO)

• Hypergrowth startups: Uber and CloudKitchens

00:00 Intro

08:20 Stories from Yahoo!

12:47 Digg’s v4 rewrite: the insider story

26:02 Will’s path to eng leadership

30:03 Uber, Stripe, Carta, and Calm

35:11 What does a CTO do?

38:09 Will’s role at Imprint

42:48 How engineering management has evolved

51:08 ZIRP’s impact on tech

57:21 Career ladders

1:01:03 Eng 1 vs Eng 2

1:06:58 How software engineering roles are changing

1:16:25 Revised rules of engineering leadership

1:21:57 The risks of becoming a bureaucrat as an EM

1:25:57 How to become a CTO?

1:31:33 Middle management vs. line management

1:41:36 Will’s writing

1:52:13 Book recommendations

Where to find Will Larson:

• X: https://x.com/Lethain

• LinkedIn: https://www.linkedin.com/in/lethain

• Website: https://lethain.com

Mentions during the episode:

• Apple IIc: https://en.wikipedia.org/wiki/Apple_IIc

• Multi-user dungeon: https://en.wikipedia.org/wiki/Multi-user_dungeon

• mIRC: https://en.wikipedia.org/wiki/MIRC

• Jet Program: https://jetprogramusa.org

• Yahoo Search BOSS: https://en.wikipedia.org/wiki/Yahoo_Search_BOSS

• David Filo on LinkedIn: https://www.linkedin.com/in/david-filo-230184169

• Digg: https://en.wikipedia.org/wiki/Digg

• Kevin Rose’s website: https://www.kevinrose.com

• Mrbabyman on Instagram: https://www.instagram.com/mrbabyman

• Jeremy Grosser on LinkedIn: https://www.linkedin.com/in/synack

• The Phoenix Project: A Novel About IT, DevOps, and Helping Your Business Win: https://www.amazon.com/dp/0988262592

• Site Reliability Engineering: How Google Runs Production Systems: https://www.google.com/books/edition/_/iXnw0QEACAAJ?kptab=getbook

• Stripe: https://stripe.com

• Carta: https://carta.com

• Calm: https://www.calm.com

• David Singleton on LinkedIn: https://www.linkedin.com/in/davidpsingleton

• Imprint: https://imprint.co

• ZIRP explains the world:

• Linear: https://linear.app

• The Mythical Man-Month: Essays on Software Engineering: https://www.amazon.com/dp/B0722G5TKZ

• Revised rules of engineering leadership: https://lethain.com/revised-rules-of-engineering-leadership

• Parkinson’s Law: https://en.wikipedia.org/wiki/Parkinson%27s_Law

• Slack agents: https://slack.com/ai-agents

• Jira: https://www.atlassian.com/software/jira

• Muse: https://muse.ai

• Seeing like a State: How Certain Schemes to Improve the Human Condition Have Failed: https://www.amazon.com/dp/0300078153

• Salary Negotiation: Make More Money, Be More Valued: https://www.kalzumeus.com/2012/01/23/salary-negotiation

• How to be a tech influencer: https://lethain.com/tech-influencer

• No idea blog: https://www.noidea.dog

• Being glue: https://www.noidea.dog/glue

• An Elegant Puzzle: Systems of Engineering Management: https://www.amazon.com/dp/1732265186

• Staff Engineer: Leadership beyond the management track: https://www.amazon.com/Staff-Engineer-Leadership-beyond-management/dp/1736417916

• The Engineering Executive’s Primer: https://www.amazon.com/Engineering-Executives-Primer-Will-Larson-ebook/dp/B0CV4QGPXD

• Crafting Engineering Strategy: How Thoughtful Decisions Solve Complex Problems: https://www.amazon.com/Crafting-Engineering-Strategy-Thoughtful-Decisions

• Thinking in Systems: https://www.amazon.com/Thinking-Systems-Donella-H-Meadows/dp/1603580557

• Practical TLA+: Planning Driven Development: https://www.amazon.com/Practical-TLA-Planning-Driven-Development-ebook/dp/B07FMHX7M2

• Formal methods with Hillel Wayne: https://newsletter.pragmaticengineer.com/p/formal-methods-with-hillel-wayne

—

Production and marketing by Pen Name.

What’s the ROI of software engineers’ spending on AI?

Gergely Orosz — Tue, 20 Oct 2026 17:34:18 +0000

AI spending has greatly increased at most companies this year, but has it actually changed shipping velocity and PR output? We analyse data from 27 companies – with some interesting findings

A software engineer whose company is looking into its own spending on AI tools pinged me with a very timely question recently, asking what other companies spend on AI, per engineer. They were also keen to find out what the PR throughput was before and after the spending on AI at other places.

Spending on AI agents is ballooning at all the tech companies I talk with this year. With 2027 coming around fast, many companies are seeking to put a realistic number on how much they really spend on AI.

The change in PR throughput is a very simple metric, and one of those “readily-available” measurements at nearly all companies, It’s also one that probably has not been “gamed” by devs yet, and it’s possible to compare a “before AI” and “after AI” measurement with this metric.

To find out more and to try and answer the reader’s question, I asked around and got to talk with engineering leaders and senior+ engineers at 27 workplaces. I also gathered aggregate data from CodeRabbit and Gitar. This article summarizes what I found, covering:

Stats from CodeRabbit and Gitar. Weekly PRs by devs using the AI code review tool increased by around 2x compared to a year ago. Meanwhile, Gitar sees “AI-native” teams ship 60+ PRs/dev/week.
AI spending patterns of 27 companies. Spending caps are common, and there’s no connection between company size and AI spend.
High spenders ($1,000+ per month). A quant shop has seen AI costs increase 15x since January, while a two-person .NET shop spends $3,000/dev/month, plus details from six more companies.
AI spend at Big Tech. Meta and NVIDIA encourage devs to use AI heavily, while engineers at Apple are far more cautious.
Medium spenders. Details from five companies spending $300-500/month/dev.
Low spenders. Companies spending below $300/month/dev, many of which use subsidized subscriptions. Including details from a publicly traded company, a nonprofit, fintech Lendscape (UK), and startups Lonestone (France), Paulina24 (Germany) and 5 others.
What’s the ROI of AI? A question that few companies seem able to answer – at least for now.

The bottom of this article could be cut off in some email clients. Read the full article uninterrupted, online.

Read the full article online

The team at CodeRabbit polled data engineers using the platform, anonymized it, and has kindly shared it with us. They saw a 113% increase in weekly PRs:

On average, devs are creating more than twice as many PRs as they did a year ago. Source: CodeRabbit

There’s a 127% rise in monthly PRs:

Monthly PRs have a very similar trajectory as weekly ones. Source: CodeRabbit

When tracking the same group of 20,000 developers, the increase is nearly identical across the platform: a 128% increase:

Tracking 20,000 devs for a year. Source: CodeRabbit

Across customer groups, there seems to be “tiers” of PR frequency. Here’s data from Gitar (Sonar’s AI code verification agent) on PRs/week/dev across hundreds of customers:

Gitar’s categorization of teams, based on PR output, per dev, per week

I suspect that a lot has to do with company culture: whenever you join a new team, you pick up the pace pretty quickly. If the company is one where ~5 PRs per week is considered a high output, then new joiners get used to that pace. If it’s a place where most people ship 20-30 PRs to production per week, then new joiners pick up that pace!

I got details on AI spending and changes in the rate of PRs from 27 different companies. I anonymized most names by request. At a glance, here’s how spend looks like:

Spend split across the 27 companies

A few common themes from this data set:

Company size and spend on AI are unrelated. In the data set, a 2-person .NET startup spends $3,000/month/dev, and a publicly traded company with 1,500 engineers, limits devs to $150 of AI spend per month.
Spend peaked during the summer, and is falling. June and July saw peak monthly spend at nearly all companies, and costs have roughly halved for those on API plans. A mix of Opus 5.5 and moving to open models is the reason for the drop at most companies.
Spending caps are common at medium-and-above spenders. It’s normal to have a spending cap in place at medium-and-above spenders, where additional spend needs to be justified. Most people say the caps are easily avoided.
Subscriptions conceal concrete AI costs, but some subsidies are ending. Several companies mentioned that their subsidized AI subscriptions are coming to an end, so they need to prepare to either optimize costs, or have their bill increase 5-10x.
Claude and Opus 5.5 seem to be the favorite models. Within this rather limited data, Claude and Opus 5.5 dominated answers about where most of the spending goes. Cursor, Codex and GitHub Copilot were also mentioned.

Here are details of companies spending considerable amounts each month on AI coding tools.

This company has seen its AI spend balloon overall: total spend was $8M /month in September, a 16x increase on the $500K/month spend in January!

The average spend is $1,600 /dev/month (up from around $80 in January). Most are going to Claude, and the biggest spenders are not devs, but traders. A current engineer there told me:

“Lots of traders are using Claude. Engineering is learning the importance of rate limiting for traders who accidentally run expensive queries without realizing it.”

The company has a $25,000 limit per employee for AI, which is extremely high, more than at NVIDIA! The engineer explained this generous limit:

“The last few years have, generally, been extremely profitable for trading firms. For us, last year was our most profitable year ever – and we’re already on track to beat it this year. So, the spend on AI is well within what these companies can afford.”

AI outlay is expected to grow: this firm is building its own internal GPU and CPU farms to run models, and are building models expected to beat state-of-the-art (SOTA) models like Fable, according to their own data.

ROI is measured differently at trading firms. The ROI of any spend is the profit or loss made by a trading desk, and AI spend is now part of the costbase. So, provided the desk’s profit increases, no questions are expected.

Outside of Big Tech where the monthly AI spend can be $10K+ for some people, the largest per-developer bill comes from a two-person .NET shop in Europe. This agency builds assessment games for a niche, global HR platform. Everything is built in C# and Blazor, running on Azure. A single tech stack was a great decision, especially now, they told me.

$3,000 /month/dev (€2,500/dev/month) is what they spend on AI tools. Most of the spend goes on Claude Opus, and GitHub Copilot. One of the devs told me:

“We barely type code anymore. We do a lot of feature development, but also a ton of backlog cleanup. Most of our systems are 5-10 years old.

We do about 60 PRs/dev/week (so around 10-15/day). It’s amazing how fast we’re clearing out our ~10 years-old backlog.

If you have high agency and know what to build, these are fun times!”

PR frequency: up 5-10x. As this engineer told me:

“We used to do 1-2 PRs per day, and now it’s 10+. It feels like our two-person team is doing the work that would have taken about 5-7 people before AI.”

$1,500 /dev/month is the default spending cap, but devs can ask for an increase. A current software engineer told me they spent $2,200/month in September. with Opus 5. With the latest version, Opus 5.5, this dev expects their spending to be cut in half.

PR frequency has not much changed. This engineer told me:

“We aim to maintain the previous PR standard in terms of PR size and quality. We have, however, built an agent to review PRs and point out, e.g., large PRs, verbose code, and similar things, and that part is going well.”

$1,200 /month/dev is the budget limit, but a Staff Engineer at the company reports that fewer than 5% of devs hit this mark.

PR volume: it’s up (no data). However, the velocity of PRs merged has not seemed to change much! From the same engineer:

“We have a complicated codebase in my team and even frontier models like Opus 5.5 are still doing weird things when someone delegates thinking to them.

We still do human PR reviews, and they catch a lot of stuff that agents and agentic review misses, like changes that are not needed at all, or those that are 10 times bigger than they should be.”

The engineer identified areas where AI has helped a lot:

Triaging of customer issues. What previously took a whole day sometimes, is now finished in a few minutes with agent work and a maximum one hour of human verification.
Research. Devs can get the info on why something was done in code based on Slack/design docs/PRs history, within minutes. Previously, this would take a day or two of manually digging into stuff.
Proof-of-concept (POC) prototyping. Creating a POC with very complicated features can be done in a few hours for Product Managers to play with. Previously, this would take a week or two!

When the focus is on quality, velocity probably does not increase all that much. The engineer had this observation:

“Overall, I personally see accelerated slop produced all around me. However, when the team cares about the quality in production code and wants to keep it like before AI, then actual development speed had not increased much.”

$1,000 /dev/month spend is the average, according to an engineering lead. While not in place yet, the company is considering a $1,000/dev/month soft cap.

PR frequency: 2x In 2025 there were around 4 PRs/dev/week; Today, this is closer to 8 PRs/dev/week. The lead engineer observed a 2-3x increase in PRs by experienced engineers compared to a year ago.

$1,000 /dev/month spending cap: this is the default, companywide, and can be spent on any provider, any way that devs want. It’s a “soft” spending cap and it’s trivial to request a larger budget: just fill out a short form to justify it, and it’s approved fast.

A current team lead shared with me the numbers they’re seeing:

Most backend engineers use around $1,500/month
Most team leads / Staff+ engineers are using around $2,500/month.

PR frequency: flat. The team lead observed no real increase compared to a year ago.

PR size: 3x. Meanwhile, PRs have exploded in size and have 3x as many lines of code per change request as before. I wonder if this could be attributed to LLMs being pretty verbose, even when coding.

$2,000 /month/dev allowance for devs. This is through a Claude or Codex Max subscription, plus credits used.

This company has a public #ai-limits Slack channel, which publicly pings those nearing their limit, as well as top spenders and reminds them to rein in their AI usage, if possible; it sounds a bit like a “public embarrassment” strategy!

Getting an increase is more complicated than at most companies. As a current engineer explained to me:

“When needing a limit increase, you run an AI skill that looks for common reasons why you might be burning tokens. It looks through things like whether you are running too many loops, or using subagents, and more. It gives suggestions on how to fix those.

If the skill doesn’t find things to improve, then submit an approval request to the manager for a budget increase. Personally, I’ve rarely seen these increases granted, so everyone treats the $2K as a hard cap.”

Velocity: unchanged. The engineer told me they don’t see anything getting done faster than before:

“We are perhaps marginally faster in a few places. AI has not fundamentally changed any of our processes, and I wonder if this is one reason. Most people are still sleeping on AI to be honest, and are adopting it very slowly. There are the odd outliers like myself and a few others who’ve been able to push a ton more, particularly on greenfield projects or anything without blockers around OSS.”

$4,000/dev/month maximum capped spend. This is a hard cap. Engineers at this company use GitHub Copilot and can choose between the models. Right now, Opus 5.5 is the top choice for devs across the company, a current team lead told me.

$1,000 /dev/month is what the average spend feels like, though from this team lead looking around the org. People are not getting close to the $4K monthly cap.

PR frequency: 4x compared to the start of this year for a single team, led by a team lead, who told me:

“We had about 1.5 PRs/dev/week in January, and this is around 4 PR/dev/week today. However, these numbers are skewed because we have one engineer whose productivity has skyrocketed, while the others have not seen all that much change.”

Velocity: 3x more stories closed than last year. This team lead has been using Scrum and Kanban for years at the company, which helps compare “business value delivered”, not just PRs. They told me:

“For the last 5 years, we used Scrum + Kanban and always used a plan board. On average we close 3x more user stories per month since feb/march. It tells me we deliver more business value.

And, for the first time ever we close more user stories than we have incoming! Keep in mind, this is all while not the whole team is there yet with AI.”

Some of the most “AI-pilled” places are in Big Tech, which is also home to some of the least “AI-pilled” approaches of all. Here are details about Meta, NVIDIA, and Apple:

The Pulse: OpenAI software engineer accused of spying

Gergely Orosz — Thu, 15 Oct 2026 16:55:18 +0000

Also: insider details on how and why Anthropic moved off Git, a new trend of AI-assisted coding rounds inside of Big Tech, and more

Oct 15, 2026

∙ Paid

The Pulse is a series covering events, insights, and trends within Big Tech and startups.

Today, we cover:

OpenAI software engineer accused of spying. A software engineer working on OpenAI’s Agents SDK has been arrested after allegedly planting a secret audio recording device in a meeting room at the AI company. The FBI and US Department of Justice are in…

Uber’s AI infra, from the inside

Gergely Orosz — Wed, 14 Oct 2026 17:06:56 +0000

Listen and watch now on YouTube, Apple, and Spotify. See the episode transcript at the top of this page, and timestamps for the episode at the bottom.

• turbopuffer – The turbopuffer engineering team is doing something really cool: they are completely redesigning their storage architecture from first principles to make search faster, cheaper, and more reliable at scale. And they’re documenting all of it! Follow along their rewrite at turbopuffer.com/v3

• Linear – one of the big challenges for us devs is the sheer amount of code to review. Linear Diffs is a new feature to review PRs inside of Linear, with all the context in one place. It even supports stacked PRs. Check it out

• WorkOS – how do you deal with agents and permissions? WorkOS built Airlock, the authorization layer for AI agents. It evaluates every request against the agent’s intent and your rules and either allows it, denies it, or routes it to a human for approval. Works with Claude Code, Codex and MCP gateways. Give it a spin

What do developer tools look like inside Uber, today? They are very different from what you would see at most places. For example, can you guess what service a system called DevPods, Code Inbox, Slate or Wisdom would be providing to engineers?

DevPods, Code Inbox, Slate and Wisdom: four of the 20+ internal tools Uber engineers use,day-to-day

Today’s episode is a deepdive into the day-to-day tools of devs at Uber, our tour guide being on of Uber’s longest-tenured engineers: Uday Kiran Medisetty. Uday is one of the few Distinguished Engineers at the company, and leads much of the work on agentic AI, today.

With Uday, we talk through how Uber’s developer infrastructure prepared it for AI agents. We look at the systems Uber built before generative AI and the new tools that connect agents to its code, internal services, and engineering knowledge. Uday explains how engineers use background agents, why tools like Slate make development (and bugfixing!) more efficient, and how code review is changing. We also cover Uber’s approach to token costs and how it measures productivity as the engineer’s role changes.

www.youtube.com

Here are 11 interesting learnings from today’s episode:

1. Uber’s service count changed erratically: in 2015, when Uday joined, there were two services in total. By 2019, there were thousands. In 2026, there will be about the same number of services than in 2019, even though the business is about 10x larger than back then!

2. Uber built most services in-house because off-the-shelf software could not handle its scale – or simply did not exist! Examples of services Uber built because it had to:

uOwn: shows who owns a service or asset. Back in 2015, no such vendor existed.
M3DB: Uber was emitting about 500 million metrics per second, with no vendor being able to handle this load, so they built it themselves. Observability company Chronosphere was built on top of the open sourced M3DB, and we previously covered how Uber built its observability platform.
Cadence: a durable workflow engine for multi-day flows like driver onboarding. No such system existed at the time when Uber needed it.Temporal was built on top of the open sourced Cadence framework.
Morpheus: a system combining feature flags and experimentation, including switchback tests, with support for two- and three-sided marketplaces that Uber operates. Again, no such solution existed when Uber needed it.

3. At Uber, cities are a first-class concept that go all the way down to physical infrastructure. Another reason Uber had to build a lot of its custom tooling is how the concept of cities is a “basic primitive” that needs to be handled across systems like observability, feature flagging, deployments, and even service sharding. For example, in the early days, some services had their database sharded by cities. Even today, each city is pinned to a cloud region, and can fall over to another region at any time. We previously covered how Uber moved from bare metal to the cloud.

4. A single AI model gateway has paid lots of dividends. In 2023, Uber’s platform team made the decision that everything built with AI needs to be model- and harness-agnostic. They built Model Gateway to support switching between Anthropic, OpenAI, other vendor models, and now also supports open models they run. Having the Model Gateway in place made it easy to extend with these parts:

AI Guard: a middleware run by the security team. It checks all requests and responses, and measures latencies (and flags overly slow models or queries)
Spend tracking: spend is tracked per user
Visibility: audit logs and session traces

5. AIFX allows engineers to switch harnesses without losing settings. Uber currently supports Claude Code, Codex, and OpenCode as harnesses. AIFX (short for AI Foundations) is a wrapper on top of all of these, so when an engineer changes harnesses, all their settings, skills and plugins move with them.

6. Uber has a hybrid approach to AI spend: engineers get allocated a spend tier. When they hit the limit, their manager can sign off to spend more in that month. Basically, for the right type of work, there is no spend limit, but Uber doesn’t want engineers to carelessly burn tokens.

A plugin called CostCheck works across sessions and gives tips on how to improve an engineer’s cost profile. Uber has reduced token costs by about 50% the last few months, as we previously covered.

7. All PRs are still reviewed by engineers, but Uber built tools to help devs decide how much time they need to spend on each code change. Uday admits that code review has become a big bottleneck inside the company, and Uber built Code Review to make it easier and more efficient for devs to do these reviews.

By the time a dev reviews a change, AI code review has run, screenshots and videos of the feature have been added (when applicable), and risk scores are assigned to the change.

8. Uber had to declare an internal “code red” for its CI system earlier this year. This was because CI jobs more than doubled in a 12-month timeframe, pushing load on CI systems overly high. To keep build speeds at bay, the team moved simple CI jobs (eg linting, build and template rules) to the “inner loop:” so agents run these commands with the afx validate command before opening a PR.

9. Uber tags PRs by the type of work each of those PRs do. In the past, Uber measured “PR velocity:” how long it took for a PR (or, as Uber calls it: a “diff”) to get to production. With agents, doing so makes less sense, because it’s easy to create dozens or even hundreds of PRs, and many of those PRs might have no business impact on the product.

So Uber now tags all PRs by type: feature, bugfix, maintenance, testing, or docs.

They are seeing the percentage of “feature PRs” going up, and the percentage of “maintenance PRs” going down.

I find this categorization to be a clever approach!

10. The job of a software engineer inside a place like Uber is to build context. As Uday reflects on how the job has changed:

“The job of engineering is, in some sense: ‘how do you build the right context for the agents to produce reliable output?’

You need to get really good at context engineering.

Also, [being good at] multitasking has become a thing. Before, it would take so much time to create a PR, so you would be laser-focused on it. But now you’re constantly switching between so many things. You need to be able to verify your work and pick any code smells from the agent [to avoid] shipping regressions into production.

Context engineering, ability to multitask, and ability to review and verify the outputs: these skills have become much more important now because you’re able to parallelize a lot more than before.”

11. Uber’s AI tooling was seemingly built in a year: in reality, it was built on top of a decade worth of dev tooling investment. Minion – built in 2025 – is powered by DevPods, which Uber created in 2017. Uber’s MCP Gateway was fast to build, thanks to existing Thrift and Protobuf registries. Jaeger tracing made Slate routing and Context graph easier, and so on.

In Uday’s words:

“All the things that we built to make life easier for engineers: it makes it easier for our agents as well!”

• How Uber uses AI for development: inside look

• Scaling Uber with Thuan Pham (Uber’s first CTO)

• Inside Uber’s move to the Cloud

• The Platform and Program Split at Uber

• Hypergrowth startups: Uber and CloudKitchens with Charles-Axel Dein

• How Uber Built its Observability Platform

• How Uber is Measuring Engineering Productivity

• Developer Experience at Uber with Gautam Korlam

00:00 Intro

02:48 Uday’s early days at Uber

05:40 Uber’s move to microservices

07:55 Uber’s engineering team, then and now

09:05 Developer tooling before AI

16:03 DevPods

19:28 Building for AI agents

29:27 MCP gateway

34:37 Agent Skills repo

38:46 The context graph and Cortana

47:54 PRD and ERD tool

50:45 Minion

54:46 Testing and code review with Bazel, Slate, and Capri

1:12:26 Edge Gateway, Shepherd, and Wisdom

1:21:33 The changing role of engineer and measuring productivity

1:30:40 Uday’s favorite new products

Where to find Uday Kiran Medisetty:

• X: https://x.com/udaykiran

• LinkedIn: https://www.linkedin.com/in/udaykiran

Mentions during the episode:

• Scaling Uber with Thuan Pham (Uber’s first CTO): https://newsletter.pragmaticengineer.com/p/scaling-uber-with-thuan-pham-ubers

• How Uber uses AI for development: inside look: https://newsletter.pragmaticengineer.com/p/how-uber-uses-ai-for-development

• Bazel: https://opensource.google/projects/bazel

• Developer Experience at Uber with Gautam Korlam: https://newsletter.pragmaticengineer.com/p/developer-experience-at-uber

• Hypergrowth startups: Uber and CloudKitchens with Charles-Axel Dein: https://newsletter.pragmaticengineer.com/p/high-growth-startups-uber-and-cloudkitchens

• Instinct: https://instinct.com

• Muse: https://ai.meta.com/muse

• Grok Bot: https://x.ai/bot

—

Production and marketing by Pen Name.

Building resilient AI-powered software, and planning for when things go wrong

Gergely Orosz — Tue, 13 Oct 2026 16:03:33 +0000

Last week’s guest on the podcast was Sam Newman, who had released a book I’ve personally long awaited titled Building Resilient Distributed Systems. It’s an especially timely book for many of us software engineers, as we’re seeing more outages with nearly all services and systems – much of it caused by more code deploys thanks to AI agents writing more code; and more bugs causing more incidents in production.

Sam’s book being “fresh out of the oven,” it naturally also touches on what AI changes as we design resilient systems. In fact, Sam told me that he took a total of three years to finishe the book – instead of the two he originally planned – because he went back and reworked all parts of the book, updating parts where AI is relevant to talk about.

I asked Sam if he would consider sharing an excerpt from the book, with sections especially relevant for anyone building systems that ought to be reliable. Sam graciously agreed, and today’s issue covers a few important topics, spread across two chapters from his book. We get into:

Disaster recovery (DR). Recovering from catastrophic failures: planning for cloud region/data center/site-wide outages, the concept of RPO and RTO, DRO objectives and SLOs.
Disaster Recovery Patterns. Backup-and-restore, pilot light failover, warm failover and multisite active-active.
Infrastructure availability when planning for DR. Ongoing GPU and CPU shortages mean you cannot expect infrastructure availability at all times, so you need to understand constraints your vendors have, and to prioritize critical services.
Resiliency of AI-powered software. Model provider resilience, the challenge of models changing, and the need for observability.
How to handle uncertainty. Tactics like being multi-model and multi-vendor, making costs transparent, and frequently analyzing vendors.

Even though this article is pretty long, we cover less than a full chapter from the book's 16 chapters. If you build reliable systems for a living, I warmly recommend the book itself, and you can get it from the publisher.

My usual disclaimer: as with all my recommendations, I was not paid to recommend this book, and none of the links are affiliate ones. See my ethics statement for more details.

Excerpt from Chapter 10: Disaster Recovery.

We’ve already discussed how to handle a variety of failures that impact individual service instances. But what happens when a catastrophic failure causes widespread outage of large parts of your system? Fire, flood, electrical supply issues, and errors in the underlying cloud infrastructure are all outside of your control, and at one fell swoop, they can render your attempts to make your system more robust totally redundant.

Handling these situations falls under the auspices of disaster recovery (DR), a set of procedures that should kick in when something really bad happens. As with much of what we’ve looked at in this book, trade-offs abound. As you reduce your period of acceptable downtime or data loss, cost and complexity increase, so picking the right approach for your system can be a delicate balancing act. In this chapter, I’ll give you some concrete targets to work with to help drive your own DR approach.

The exact terminology you use to discuss your infrastructure will likely vary based on implementation details, and this in turn can make discussions around generally applicable patterns more difficult. When running on the public cloud, for example, you’ll probably be familiar with the concept of a cloud region, which is where your cloud resources live. When running on a private cloud or in a more traditional on-premises setup, you’ll instead think in terms of data centers.

As we discuss the various options around handling DR, the patterns that we’ll see later in this chapter can be used for either public or private cloud scenarios. Therefore, to keep things simple, I’ll use the term site as a general descriptor of where your computing infrastructure lives.

There are some specific issues to consider around mixing cloud vendors as part of a DR strategy, and we’ll look more at those in Chapter 11.

A spectrum of DR options are available to you, and they can range greatly in terms of cost, complexity, and recovery. To help you decide which DR approach is most appropriate, we should first try to understand how to determine the appropriate requirements for your context.

A simple way to define technical requirements around DR is to make use of two concrete measurable targets, recovery point objective and recovery time objective. In Figure 10-1, you can see how these objectives relate to an incident:

Figure 10-1. The key objectives to define as part of selecting an appropriate DR strategy

The recovery point objective (RPO) defines the maximum amount of time that can have passed since the data was last backed up or synchronized. Put another way, RPO defines the upper duration of potential data loss. If the data for your system backs up once an hour, then it’s possible to lose up to an hour of data if your system has an incident. Therefore, your RPO would be an hour.

The recovery time objective (RTO) defines how long it’s acceptable for your system to be unavailable in the wake of an incident. If all your computers became unavailable, perhaps because of a data center–wide outage, how long would it take you to have your services running in an alternative data center? Your RTO includes all the time needed to recover data from the last recovery point, make computing resources available for your workloads, and redirect traffic to your new site—so there are more variables to consider in terms of bringing your system back online than just time.

If asked what RPO and RTO you’d want in a vacuum, you’d probably give a very simple answer: you’d want an RPO of zero (i.e., a guarantee of no data loss), and an RTO of zero too. This is an understandable wish, but as you reduce your RPO and RTO, cost and complexity increase dramatically—and in technical terms, it’s not possible to guarantee a zero RPO or RTO in any case.

An additional complication is that different types of workloads might require different RPOs and RTOs. Consider a bank, for example. Having a low RPO to ensure that payment information isn’t lost would be key, and having a low RTO to ensure that people can still take money out of their account is vital. On the other hand, the part of the system that enables new customers to sign up for an account could have a higher RTO and RPO, as the impact to the business is less severe.

So when deciding on the appropriate RPO and RTO, there needs to be a discussion with stakeholders regarding end-user needs and regulatory and legislative requirements, which are balanced against the cost and complexity of implementing the required solution. For example, your product owner might initially say they need an RPO of 10 minutes, but if you could tell them it would take months of work to implement (during which other work has to be deprioritized), they might reduce their target, perhaps accepting an RPO that is easier for you to deliver. Your target RPO and RTO should also exceed anything specified in any service level agreements you have.

At first glance, RPO and RTO would appear to make for good service level objectives (SLOs). After all, they are both important measures that directly impact user experience. However, there are some challenges around managing RPO and RTO as SLOs.

When determining if we are meeting an SLO, we need to collect lots of service level indicators (SLI) to confirm whether or not we are on target. So if we had an SLO that defined 99th percentile latency over a one-month period of less than 500 milliseconds, we would analyze the latency-related SLIs for that monthlong period to make sure we were meeting the target. With RPO and RTO, though, what would our SLI be?

The reality is that we don’t experience disasters regularly enough to gather useful SLIs. We can of course define related SLOs around things related to our DR approach. For example, it would be possible to define an SLO around the speed of data replication, as this is something we are doing in production and can programmatically measure. But at best SLOs are only tangentially measuring RPO and RTO.

So RPOs and RTOs don’t really fit neatly into the SLO concept. Of course, that doesn’t mean that you can’t still test your system to make sure it is delivering on your DR objectives—something we’ll discuss more later in the chapter.

Another excerpt from Chapter 10: Disaster Recovery.

With the discussion of RPO and RTO in mind, let’s take a look at some common patterns used to implement DR in the wake of a major incident. In Figure 10-2, we can see an overview of each of the four patterns we will explore, arranged (broadly speaking) in order of increasing complexity and improving RTO and RPO.

Figure 10-2. The key objectives to define as part of selecting an appropriate DR strategy

If starting from scratch, many of these patterns can build on one another, allowing you to incrementally increase the complexity (and cost) of your DR solution over time. For example, a backup and restore approach requires rerouting, a pilot light setup will add the need for cross-site data replication and infrastructure automation, and a warm failover adds on the cost of having some infrastructure ready to go. So if you aren’t sure where to start, working your way through the patterns from left to right in Figure 10-2 makes a lot of sense.

If you have different RPO and RTOs for different parts of your system, you may also end up using a mix of these patterns. Note that they all require some form of rerouting of traffic, and most require some form of cross-site data replication. Those options are discussed later in this chapter.

Recovery by restoring data from a backup and loading it into a replacement system.

As outlined in Figure 10-3, with this pattern, data is periodically backed up to a separate site. In the event that the primary site fails, infrastructure needs to be provided at a new site, and the backed-up data restored.

Figure 10-3. The backup and restore DR pattern

This pattern is very straightforward to implement, consisting of a few steps:

1. Back up data regularly.

All data that you want to make available to the failover site needs to be regularly backed up, with the backup data stored somewhere other than the primary site.

2. Provision infrastructure on the failover site.

This would ideally be automated, as manual infrastructure provisioning would likely lead to longer RTO and an increased chance of errors.

3. Restore data from backup to the failover site.

Data is loaded from the backup location into the newly provisioned infrastructure.

4. Switch traffic from the primary to the failover site.

Once infrastructure is available and the data is restored, traffic can be switched to the secondary.

Ideally, your failover site runs the same core technology stack as the primary site, meaning that any investment in infrastructure automation on your primary can also be used on the failover (for example, being able to reuse Terraform scripts). Keeping the code to automate setup of the primary and the failover the same is very useful, as it means that you’ll have a higher degree of confidence that spinning up the infrastructure on the failover site will work. If your failover site uses a different technology stack—meaning it’s not possible to share automated infrastructure provisioning code between the primary and secondary sites—then it becomes even more vital that your DR failover process is regularly tested. A caveat to spinning up infrastructure on demand here comes in terms of guaranteeing that infrastructure is actually available when you need it, something we’ll explore later in this chapter.

This pattern is generally applicable in situations where your primary site is unavailable and you can tolerate a long RTO, and especially in those situations where you want to keep costs low. It’s important to note that regular tests of a backup and restore process are necessary to ensure that it actually works. One approach I’ve found useful is to perform a backup and restore to a secondary site as part of other regular activity (for example, a regular load test exercise).

The RPO of this pattern will depend on the frequency of backup. The RTO will be the longest of the patterns we’ll look at, and its time will depend primarily on how long it takes to restore data from the backup and provision replacement resources. With a fully automated process and smaller data volumes, this could be in the minutes to hours range. Manual recovery processes will likely extend these timelines greatly, as will the size of backups involved—terabytes of data take longer to send than gigabytes! Another major factor will be availability of the required computing resources. If you end up having to purchase replacement equipment, this pattern could easily result in a timeline of weeks or longer.

Data is constantly synced to a failover site, with infrastructure provisioned after failover.

With a pilot light setup, data is continually replicated from the primary to the failover site, as shown in Figure 10-4. Infrastructure is not provisioned until a failover is required, ensuring that costs can be kept down when working with public cloud infrastructure.

Figure 10-4. The pilot light failover DR pattern

This pattern builds on a lot of the work done with a backup and restore, the main change being that the data is continually synchronized to the failover site. As such, the steps involved look fairly similar to the backup and restore pattern:

1. Replicate data continually.

As data is committed to the primary, it is continually replicated to the failover site. This can be done in a variety of ways, such as by making use of native database technology or application-level replication.

2. Provision infrastructure on the failover site.

The assumption is that databases and datastores are already present on the failover site, so only remaining resources like compute would need to be provisioned.

3. Switch traffic from the primary to the failover site.

Once infrastructure is available and the data is restored, traffic can be switched to the failover.

From an implementation point of view, the new source of complexity comes from the replication of data from the primary to the failover. For systems designed with this process in mind, it might be somewhat trivial to implement. However, things become more complex when adapting existing systems if you’re making use of a data product that doesn’t support this replication strategy natively. This is worthy of a more in-depth discussion later in this chapter, especially as the next two patterns also rely on this capability.

When compared to a backup and restore, the pilot light failover allows for a much shorter RPO because of more frequent data replication, while also reducing RTO by eliminating the time taken for data restoration.

The cost-saving of a pilot light failover really makes sense only in the context of the public cloud. Costs are reduced primarily in terms of not paying for infrastructure unless it’s used. But if your failover site is your own private infrastructure that you have to pay for regardless of whether it is used, then some form of a warm failover makes a lot more sense. At that point you’re already paying for the hardware, so you might as well use it.

Data is synced to a failover site, with at least some infrastructure already provisioned to handle traffic.

With a warm failover, as shown in Figure 10-5, data is continually replicated from the primary to the failover site, with at least some infrastructure already provisioned in advance of the failover itself.

Figure 10-5. The warm failover DR pattern

Having some infrastructure provisioned even before the failover occurs results in a reduction in RTO for some functionality. How much infrastructure is provisioned on the failover can vary, but the idea is that some capacity already exists to allow for a much faster failover.

A common approach would be to prioritize the subset infrastructure that is provided on the secondary site for the more critical capabilities of the system, allowing for a shorter RTO for the most important functionality.

Cold, Warm, and Hot Failovers: A brief note is warranted here regarding the term warm. You’ll find various solutions and vendor products promising cold, warm, or hot failover, with the only real distinguishing factor between these terms often being that a hot failover would give you a lower RTO in comparison to a warm failover, and a warm failover would give you a lower RTO than a cold one. Given that what constitutes hot or warm from vendor to vendor and from tech stack to tech stack varies so greatly, I just decided to pick warm and stick with it.

Naming things is hard.

The “How it works” and “When to use it sections” are cut off from this excerpt, but are present inside the book.

The system is distributed across two or more sites, with writes being able to be sent to any site.

With a multisite active-active setup, data is replicated constantly between two or more sites, as you can see in Figure 10-6. Traffic for any customer can be served from any site at any time.

Figure 10-6. The multisite active-active DR pattern

This pattern sidesteps the need for a failover, as it is assumed that traffic for any user can switch between sites at any time. As such, it delivers the lowest RTO and RPO of any pattern we’ve looked at so far, though at much higher cost and complexity.

The “How it works” and “When to use it sections” are cut off from this excerpt, but are present inside the book.

Yet another excerpt from Chapter 10: Disaster Recovery.

At the time of writing, GenAI is being used by more and more industries. Whatever your views on the usefulness of this technology (or its broader societal impact), the nature of GenAI workloads for both training and inference purposes has driven ever-increasing demand for infrastructure, in turn leading to reduced capacity for many users. Even if you aren’t running what you consider to be specialized GenAI workloads (for example, those requiring specific GPUs), you may well see the impact of these shortages more broadly.

Aside from the ever-growing cost of GPUs, memory costs have spiked significantly in 2025 and 2026, impacting general server availability. I’ve already spoken to clients who are struggling to run “normal” (non-AI) workloads on some of the large public cloud providers, as they can’t secure enough capacity. Being able to shift regions in some cases is helping, but guarantees of future availability seem fairly weak. Even buying your own hardware doesn’t seem to be a quick fix, with lead times for servers seemingly extending further and further with each passing week.

This has to be taken into account when considering the patterns detailed previously. If your chosen approach assumes you can dynamically provision infrastructure when a failure occurs, you will have to look at what guarantees you have around the capacity you want. In a scenario where a major cloud vendor’s region has failed, you may find yourself fighting over scraps with everyone else.

There are three ways to handle this scenario: reserve capacity; prioritize critical services; or lastly, consider running a fully redundant set of infrastructure at all times.

Not every vendor offers the ability to reserve capacity, and even then it is limited to certain cloud services. On AWS, for example, you can pay a fee to guarantee the availability of EC2 instances (managed virtual machines) when you need them. This has a cost impact, as you pay a fee for the reservation whether or not you use the instances (the silver lining being that if you did end up using them, the cost would be cheaper than normal in-demand pricing). This doesn’t help you, though, when using cloud resources for which reservations aren’t possible—you can’t purchase guaranteed availability of Lambda invocations, for example.

Next, consider prioritizing critical services. If there is a cost to guaranteeing availability (or if only a limited amount of infrastructure is available), then it’s vital that this goes to workloads that have a higher criticality. This is where having separate RTOs and RPOs on a service-by-service basis comes into play.

Finally, if you could justify the complexity and expense of having a fully redundant set of infrastructure ready to run your system, then you’d have the confidence that your computing resources will be available, with the caveat that this option can be extremely costly. But that cost can be mitigated if you can make alternative use of the redundant resources.

Back in Chapter 4, I mentioned BankCo, where we ran calculations across a large pool of machines. A large portion of these computers were desktop PCs sitting idle in an office situated outside London. These PCs formed part of the bank’s DR strategy: in the event that traders had to evacuate the bank’s main location, they could relocate to the secondary site and continue working. It was a requirement that these standby PCs were constantly running to ensure that we knew that they worked, and so that updates could be pushed out to them. This was obviously back in the day when most of the critical work relied on thick-client applications, so they couldn’t be accessed via the web.

We were able to make use of these DR workstations to grow our pool of available workers. This meant that when the DR site wasn’t in operation (which was most of the time), we were able to carry out our calculations much faster, as we could parallelize the work over more machines. If the DR site’s machines were unavailable to us, we still had a set of our own servers on which we could run the calculations to suit our normal targets.

So if you are thinking about securing ring-fenced infrastructure as part of a DR approach, see if you can think of useful ways to put it to good use when it’s not needed for production workloads. You could use the spare infrastructure for a load test environment, for example; the load test could easily be aborted when DR is needed.

Excerpt from Chapter 7: Resiliency and AI

We’ve looked at how AI is changing how we build and operate our software. But we are also integrating AI into the products we build, using it to hopefully improve our software for our users. What new challenges does this create for ensuring our systems are resilient?

Foundation model providers OpenAI and Anthropic don’t have a great track record in terms of providing high levels of availability for their services, at least when compared to other managed service providers, like public cloud vendors. While I’m sure this will improve over time, having a potentially flaky third party in the mix of your critical functionality can be an issue.

A solution here is to consider ensuring that your software can still operate even if the model is unavailable. This could be achieved by limiting functionality in the case of the model being unavailable (or you getting rate limited or running out of tokens!)—this is an example of the graceful degradation pattern, which we’ll look at more in Chapter 11.

Another option is to have backup vendors for a given model to hedge against your normal model provider having availability issues. For example, you can use OpenAI models on Azure, a handy alternative if accessing the models via OpenAI’s own services proves to be too unstable, and in a similar way Anthropic’s models can also be accessed via Google’s Gemini Enterprise Agent Platform. When using backup vendors like this, there may be a lag in terms of how quickly the latest models may be available, but given that you may not always want to be on the bleeding edge of the models you use to embed in your software (something we’ll discuss in just a moment), this is probably fine.

Consistency of behavior plays a big role in helping us ensure that our system is behaving appropriately. While dealing with the unexpected is a key concept in resiliency (or “graceful extensibility,” as David Woods put it), that doesn’t mean we want to build software where surprise is constant. Quite aside from the fact that LLMs are nondeterministic, small changes in models can also lead to quite different behavior.

I’ve seen reports where a model has started behaving differently, without any major change in behavior being flagged by the vendor. Additionally, moving to a newer version of a model can fundamentally change how your prompts may be handled.

It is therefore vital to have a good set of tests to validate that the AI portions of your code are behaving appropriately—and this should be a mix of traditional functional tests, together with the use of AI evaluation tools (evals). You want to make sure that when the unexpected occurs, you are ones who get surprised first, rather than your users.

I can recommend the second edition of Gayathri Mohan’s Full Stack Testing,3 for those of you wanting to explore testing in general, including verification of AI-based systems. For a deeper dive specifically into evals, I’d suggest Evals for AI Engineers by Shreya Shankar and Hamel Husain.

The telemetry we receive from our own software and from third-party services alike is key to making sure our applications are working the way that we expect. As we discussed back in Chapter 3, we prefer a small variation of latency rather than a broad variation of latency. If the time it takes to successfully complete an operation varies wildly, it can be difficult to know where we should start triggering timeouts, for example. It also becomes more difficult to define latency-related SLOs.

Once you send a prompt to an AI model or instruct an agent to do something for you, the latency for the operation can vary greatly, and in ways that can be difficult to predict from the outset. Some of this can be put down to variability in the performance of the provider at a given point in time, but it can also be the result of rate limiting by the provider, or your prompt could just be taking longer to process.

In situations where the prompts sent to models come directly from your users, then you will likely have even less certainty about how long it should take for a model to come back to you. Has a call hung, or is it expecting a large output?

Some techniques to solve this can include dynamically setting timeouts based on the number of tokens being generated, or monitoring the streaming output from a model so you can track when the response starts coming back rather than waiting for the full response.

Excerpt from Chapter 7: Resiliency and AI

In the previous sections, I’ve outlined why the world of AI is so uncertain right now. However, I want this book to focus on practical, actionable advice. With that in mind, I want to outline some key steps you can take to make sure that you’re prepared for the uncertainty ahead.

With model behavior, speed, and quality changing from version to version, it’s vital that you build your software to support changing models. Not only do you want the ability to change models to make use of the latest and greatest, but if costs start to increase from one vendor, you want to make sure the barrier to finding another vendor isn’t too great.

It is worth highlighting that mixing models from different vendors in a single flow can exacerbate issues. For example, a flow that involved models from Anthropic, OpenAI, and Google’s Gemini would be impacted if any of those models had availability issues. All those models have to be available and functioning correctly at the time the flow is executed. Also, prompts that work against one model may not be entirely portable to another—a prompt tuned against one model will have to be reevaluated when moving to a different model, even if you stay within the same vendor ecosystem.

Taking a multimodel approach extends to your vendors too. If a vendor you are dependent on is in turn dependent on a single vendor’s model or API, you and your vendor become vulnerable to volatility in that model or to being impacted by a sudden price increase. Cursor was forced to pass on a significant cost increase to its customers in 2025, as it relied on the use of Anthropic’s services, and Anthropic changed its pricing structure, which led to Cursor having to pay much more. Coincidentally, Anthropic’s Claude Code is a direct competitor of Cursor. If Cursor had been able to use AI models from another firm, it might have been less vulnerable to this issue.

Keep Your Options Open: Make sure you aren’t wholly dependent on a single model or AI platform. If model quality degrades, or prices from OpenAI or Anthropic go up, you want to make sure that you have the flexibility to go elsewhere.

Given concerns about rising costs from vendors, and given that pay-as-you-go pricing models can lead to surprising and unsustainable bills, having the ability to see on a regular basis how much money you are spending, and where, can be really important. This doesn’t need to be about limiting spend necessarily; it can be about understanding how you spend money and understanding what value that brings.

James Stanier, CTO at digital healthcare firm Nordhealth, has advocated for being able to see spend on a per-person basis. Rather than this being about shaming people who spend too much money, he makes the case that this visibility drives better understanding of how the software is being used and helps drive the adoption of tools that are proving to be valuable. In his 2025 talk at LeadDev Berlin, “LLMs: An Operator’s View”, Stanier explains that if you see a person spending a lot of money, this shouldn’t be seen as a bad thing, as typically it showsomeone trying to get more done. He cites an example of one of his QA engineers blowing through €1,000 in a single week on Claude Code, which was a much higher spend than normal. After pairing with the engineer in question to understand where this money was going, Stanier was able to see that this spend allowed the engineer in question to more than double his productivity.

Having the information about spend available to you, and being able to talk to people about how they use these tools and the benefits (or problems) they cause, will help you all make much better and more informed decisions.

It can also help you spot egregious overspend. Uber recently revealed that it had burned through its entire annual AI budget in just the first four months of the year, and Microsoft announced it was pulling Claude licenses from its employees, due in part to cost, apparently.5 These stories might cause one or two raised eyebrows, but they pale in comparison to the recent (admittedly unsubstantiated) report from Axios that an unnamed enterprise company spent $500 million on Claude licenses after failing to put cost controls into place.

Tip: Make your AI spend visible! It can help you understand how people are using it and also help you spot when the bills start climbing precipitously.

Vendor management is a common activity that many firms carry out. It typically consists of a regular review of the main vendors that the organization makes use of to help it make better decisions in the future. For example, seeing that a large amount of money is being spent with a particular vendor despite the quality of service from said vendor degrading might cause an escalation to either get the price reduced or get the quality of service improved—or it might kick off a process to find a replacement.

In my last role at Thoughtworks, I helped lead our internal technology operations. Although my main job was to help provide support to internal delivery teams, I was also involved from time to time with our own vendor management activities. As part of these activities, we would regularly assess the general health of the vendors themselves. During the last review I was involved with, we were actively concerned about the health of one of our main technology suppliers; the supplier was in a very competitive market, and its market share was dropping. Although we were happy with the quality of service it provided (and with the cost), our fear was that wider market forces would drive it out of business. As a result, we decided to get to know the vendor much better but to also prepare a list of alternative solutions that we could select if the vendor in question didn’t survive.

Given the degree of uncertainty around the AI sector in general, I’d recommend regular vendor management activities. If you already have a normal vendor analysis cycle, consider increasing its frequency for AI vendors. If possible, find out if any vendors are in turn dependent on another provider, as in the case of Cursor being dependent on Anthropic’s cloud services. I’d be more concerned about a vendor that is in turn wholly dependent on a single company than I would be about a vendor that has more flexibility about whose services it uses. If you’ve never done vendor management before, consider instituting a lightweight process so you can build up a better understanding of how exposed you are to risk in this space.

With the concerns around the potential security vulnerabilities that come from the use of AI, I’d also recommend keeping your threat model up-to-date with regular threat assessments. As with vendor reviews, consider increasing the frequency of your normal threat assessments in areas with heavy use of AI, as this technology and how we use it is changing very quickly.

Review Your AI Vendors Regularly! Given the volatility in the space, make sure you review your AI vendors much more frequently than normal vendors.

I realize that this chapter may have ended on somewhat of a negative note. But I wanted to be clear that we have real, practical use cases for AI in helping us build more resilient distributed systems. Most of the use cases I outlined in this chapter are being used daily by people all over the world. The key thing is that the solutions I have shared here still put humans at the heart of decision making. It’s clear that, due to the challenges I have covered here as well, we are a long way from a truly autonomous AI SRE.

Despite the obvious benefits of using AI, we do need to be fully aware of the downsides at both the micro and macro level. At the micro level, LLM-powered AI requires us to be OK with nondeterminism, whereas there are many activities we perform in the context of building resilient systems that need to be fully repeatable and deterministic. At the macro level, concerns about the AI sector as a whole mean we are dealing with an ever-changing (and hopefully) improving set of products built on very shaky financial foundations. It doesn’t matter how good the tooling is if it’s too expensive to use (or the company making it goes bust).

As is the nature of things, this uncertainty around AI’s role in our industry and society as a whole will fade as we learn more and as the market settles down. But be warned: I think we are still very much in the middle of the storm, and the calm waters may be several years away yet.

A lot of this chapter was about uncertainty, and one thing that can make the resiliency of a system quite uncertain is what happens when machines fail. So in Chapter 8, I’ll explore how to make sure our systems still work even when machines let us down.

Gergely, again.

Thanks to Sam for sharing this excerpt of the book! While this was a long article in itself, we covered less than one of 16 chapters in the book. I found the book an excellent read, and recommend getting it if you’re building production systems:

Get the ebook or print book

My sense is that AI is making “resiliency basics” more important than before. Most of the book is not about AI – in fact, AI is barely mentioned in many of the chapters. But as AI agents generate more code and allow building software faster than before: we now have more software (or more complex software!) to operate.

And while there are AI tools and vendors that promise operating software to be easier with AI: I don’t really buy that it is, at least not at scale. Having the right monitoring and alerting in place is becoming more important, as is responding to outages, and being aware of tricky situations like thundering herds, retries, idempotency, timeouts, and many others.

Preparing for disaster recovery seems like a practical thing to do when your service is important enough. I wanted to include excerpts on disaster recovery, because I observe plenty of “disasters” (that is, widespread outages) happening across the industry where recovery time becomes surprisingly long. And some of this could be because the team had not prepared for, or exercised, disaster recovery beforehand.

A good example is how Coinbase’s global trading service went offline after a single AWS zone had problems. As I summarized the incident in The Pulse, back in June:

“Having no automated failovers is incredibly amateurish for an operation of Coinbase’s scale. Coinbase moves about 5.2 trillion dollars per year, and is valued at around $40B. The outage interrupted around $7 billion-worth of financial activity, based on my napkin math.”

It’s smart to learn from the mistakes of others, when we can: and the Coinbase incident felt like one of these.

I hope you found these topics relevant, and thanks again to Sam for writing the book, and sharing relevant parts with us.

No posts

The Pulse: new trend of building “internal vibe-coding apps” at tech companies

Gergely Orosz — Thu, 8 Oct 2026 16:55:51 +0000

The Pulse is a series covering events, insights, and trends within Big Tech and startups.

Today, we cover:

New trend: building internal vibe-coding apps at mid-sized companies. Ramp and Stripe built platforms for non-engineers to build internal websites and tools with, and both are taking off in those workplaces. I expect more companies to do the same.
Do us engineers really enjoy hard problems? Or do we actually like pattern-matching with backend problems? A provocative post by Cloudflare engineer, Sunil Pai, suggests there are other motives.
New open models launch in the EU and US. Kolibri, Mistral Large 4, and Beam by Reflection AI could challenge China’s dominance in open weight models.
Industry Pulse. Why Figma doesn’t let any agent use its MCP server; Google Cloud adds Swift support on the server side, Anthropic’s two-week sprint to speed up Claude Code, Coinbase dumps React Native shortly after Shopify announces doing so, Claude Opus 5.5 formats the C: drive, and more.

When I visited Ramp’s headquarters last month in New York, I talked with the four-person development team behind ‘Glass,’ which is Ramp’s “internal Lovable.” This name refers to Lovable.dev, the leading AI vibe-coding tool.

Ramp’s ‘Glass’ is basically a vibe-coding platform for non-engineers to create complex applications which run internally. They can share the applications they make with others to use and modify. This is what the landing page looks like:

The landing page for Glass. Source: Ramp

Users can connect various internal systems as “connectors” that can then be used as data sources for their apps:

Setting up connectors in Glass

The team showed me a few popular apps, including a custom CRM built by someone in marketing, which had 50+ fields and filters on-screen. It seemed overwhelming to me, but the creator reportedly finds it extremely powerful to use.

Since then, I’ve learned that many other mid-sized-and-above companies are building similar “internal Lovables” that don’t usually see the light of day in the outside world.

Building resilient systems with Sam Newman

Gergely Orosz — Wed, 7 Oct 2026 17:19:58 +0000

Listen and watch now on YouTube, Apple, and Spotify. See the episode transcript at the top of this page, and timestamps for the episode at the bottom.

• Get a free introduction to property-based testing: through a live, online webinar hosted by Hillel Wayne (a recent podcast guest!), author of Logic for Programmers and Developer Educator at Antithesis. What if you only needed a single test to harden your system? RSVP here.

Sam Newman wrote one of the most-read books about microservices (“Building Microservices”), but he calls them an architecture of “last resort.”

In this episode of the Pragmatic Engineer podcast, Sam explains his thinking on this, and he’s certainly well placed to do so; he was in the room when the term “microservices” was coined. We discuss what teams get wrong when adopting microservices, why independent deployment matters, and how microservices can help teams work more autonomously.

We also peek into his new book, ‘Building Resilient Distributed Systems’, and Sam explains his three rules for distributed systems, why observability is essential, and also why it’s vital to take business context into account when deciding whether to fail open or fail closed on errors.

We also explore how AI is changing software development, from specs versus code as a source of truth to cognitive debt and cognitive surrender. Sam shares how modular architecture can help teams experiment with AI while maintaining understanding of the systems being built.

www.youtube.com

1. In the room when “microservices” was coined. In the early years of the 2010s, James Lewis – Sam’s colleague at Thoughtworks – kept running into companies building service-oriented architectures where each service could be replaced quickly. At an architecture symposium in the Lake District, England, James pitched a name for this trend: “micro apps,” and in a room of about 10 people (including Sam), someone else suggested “microservices.” Lewis and Martin Fowler then wrote the first article on the topic in March 2014, ‘Microservices: a definition of this new architectural term’, and Sam published the book Building Microservices the following year.

2. Sam spent 18 months teaching Googlers how to write good automated tests. In 2007, Sam was embedded at Yahoo and Google as a ThoughtWorks consultant, alongside people from Pivotal Labs. Their job was to teach engineers how to write automated tests, and code that’s testable. He recalls that some Google devs did not believe it was possible to write automated functional tests for websites, until Sam and his colleagues showed them how Selenium works!

3. Sam spent 10+ years at a large company, three years at startups, and has been independent for more than a decade. After 13 years at Thoughtworks, he quit to avoid becoming “institutionalized.” He recalls thinking he could hang around at Thoughtworks and grow bitter. “So I’ll just go now.” He then worked at three different startups before going independent, which he’s now done for 15 years. The biggest challenge of being your own boss? He says: “The problem of working for yourself is that the boss is frequently an a*****e. So, current Sam frequently is cursing past Sam for the commitments he made!”

4. So, what’s a “microservice?” Sam gives two definitions: a clear one, and a softer one:

Independently deployable service. The “clear” definition: as long as, upon a change to a system, it can be independently deployed – without any other services needing to be deployed –, you can call it a microservice.
Clear boundaries around the business domain. The “softer” definition: services split by business function (not by technical layer) can also be called microservices.

5. Three rules of distributed systems: Sam sometimes struggled to recall the eight fallacies of distributed computing, and so he boiled them down to the three that cause the most problems:

1. Information takes time to travel. You cannot send information instantaneously from point A to point B. This is basically the “speed of light” constraint.

2. Sometimes, the thing you want to talk to isn’t there. That is, resources can and do go offline.

3. Resources are not infinite; you will eventually run out of things like CPU, memory, storage, and networking bandwidth.

In Sam’s experience, most distributed systems outages are caused by #3; resource pools running out.

6. Dealing with idempotency: keys or fingerprints? Idempotency is performing the same operation multiple times and it having the same effect as if done once only. For example, if you repeatedly hit the “down” button for an elevator, nothing more happens than what happened upon that first press: multiple button smashes don’t bring more elevators your way. Payments are a classic example where idempotency is needed: when you retry or refresh a payments page, you should not be charged twice.

There are two common ways to deal with idempotency:

Idempotency keys: a client sends a unique identifier (usually a UUID) that is the idempotency key. The server makes sure operations with the same idempotency key are executed only once. It’s clean to build but hard to retrofit, which involves changing both the client and server.
Fingerprints: the client sends nothing, but the server creates a “fingerprint” of the request by hashing relevant fields. It’s easy to retrofit later (only need to change the server), however, legitimate requests can be rejected as duplicates, and duplicate requests could still happen. Sam describes a Danish payment system he worked on that rejected purchases of the same value within a 5-minute period while it did the fingerprinting.

7. Four concepts of resiliency: Sam frequently references the four outcomes for military systems, explained by David Woods:

Robustness: systems continue to function as intended, for example, when Kubernetes replaces a failing pod. A downside of robustness is that it can add complexity, and more complex systems have more ways to fail.
Rebound: how quickly the system recovers when something goes wrong.
Extensibility: dealing with surprises and being able to stretch the system’s support to deal with new needs
Adaptability: the system changing to support new functions. Resilient systems are adapted, even between incidents.

8. What if code is a side effect of collaboration? When Sam was at Thoughtworks, some clients refused to do pair programming as a practice because they could only see one developer typing and the other doing nothing, which is a waste. Martin Fowler’s dictum that “programming is not typing” helped Sam to explain to clients that typing out code is an output, but not an outcome.

Sam conceives programming as building a system collectively, and code is one of the mechanisms delivering the system. Now that with LLMs we no longer spend most time typing out code, we’ll discover the next bottleneck in creating good software. Could it be delayed feedback cycles?

9. Hedging the vendor options is sensible when integrating AI into systems. It’s hard to tell which AI companies have long-term, sustainable business models and which don’t, so it’s the responsible thing when designing systems to hedge the providers. Aim to be multi-vendor, multi-model in your choices, so you can easily change the vendor or model layer. Also, consider if you can swap out LLM-powered functionality for deterministic code that runs faster and cheaper.

10. We must resist “cognitive surrender” to AI. Sam believes that more AI usage is leading to less critical thinking – or could we be using it wrong? He says: “The original pitch of AI was it was going to free us from drudgery. We’d have less toil, and we can focus more on critical thinking and smarts. However, much of our current use of AI is not freeing us up from critical thinking! It’s causing more context switching and leaving less time for thinking. People are working longer hours; we’ve got studies that show this.

But [longer hours and more context switching] is not necessarily AI’s fault. It’s just how we’re using it.”

Putting in the effort often leads to better results. For example, when writing Building Resilient Distributed Systems, Sam used NotebookLM for research, but he manually clicked through each link it surfaced, and checked whether it was information he wanted to reference.

11. Starting with modules encourages better software architecture. Also: production is truth. I asked Sam for advice on what to do to get better at designing systems:

Start by thinking carefully about module boundaries and the connections between them
When you use AI, allow it to roam freely, but only inside the module structure that you have designed!

12. Much of the tech world might still be naive about what an LLM is. As Sam said:

“I think the tech world in general is quite naive about AI and fundamentally misunderstands what an LLM is. For example: why did the LLM delete my database? Well, because it has no concept of causality. They have no concept that if I do A, B happens.

LLMs are not world models. World models have been around for a long time. They predate LLMs. So if you had the appropriate model and the appropriate inputs, world models could at least get to the point of ‘if I do X, then Y happens and I’ve been told not to do Y.’

I think we expect LLMs to do more than they actually can do, because they seem so smart. This leads us to overestimate their capabilities a bit too much. But the nature of LLMs is why all the guardrails around LLMs are really not going to be the right long-term solution.”

Sam’s comments feel very timely, given Opus 5.5 just formatted the hard drive of a dev who ran the model with the —dangerously-skip-permissions tag, and was surprised to see all of his data gone.

Opus 5.5 decided it would be a good idea to format the C: drive. Sam warned about this kind of outcome due to the nature of LLMs. Screenshot source: X

Finally, recommended reading to get better at software architecture. Sam’s recommendations:

Fundamentals of Software Architecture by Neal Ford and Mark Richards.
Balanced Coupling by Vlad Khononov
On the criteria to be used in decomposing systems into modules: a paper by David Parnas from the 1970s original papers from the early 1970s, such as
An architecturally-evident coding style: making your design visible: a paper by George Fairbanks, published in 2010. It’s an approach that allows the code to hint at the system’s architecture

• Scaling Uber with Thuan Pham (Uber’s first CTO)

• What is good software architecture at Netflix?

• The past and future of modern backend practices

• What is reliability engineering, and the history of SRE

• How to debug large, distributed systems: Antithesis

• Designing Data-intensive Applications with Martin Kleppmann

• Building Bluesky: a Distributed Social Network

00:00 Intro

03:16 Sam’s path into tech

09:32 Thoughtworks

20:55 The rise of microservices

33:37 Are specs becoming more important than code?

45:22 Building Resilient Distributed Systems (Sam’s new book)

52:35 Three rules of distributed systems

56:16 Observability

1:02:20 Resilience tradeoffs

1:07:54 Idempotency

1:15:58 Thundering herds

1:21:03 Business context and resilience decisions

1:25:51 Resilience engineering: four concepts

1:32:42 AI and resilience

1:36:26 AI’s limitations and where to use it

1:40:06 Cognitive debt and cognitive surrender

1:45:02 Modular architecture and AI software factories

1:51:53 Resources for learning software architecture

1:55:30 Where to find Sam

Where to find Sam Newman:

• X: https://x.com/samnewman

• LinkedIn: https://www.linkedin.com/in/samnewman

• Website: https://samnewman.io

Mentions during the episode:

Uptime Labs: incident response training, backed by science. In the episode, Sam said this about them:

“I’d like to give a shout out to Uptime Labs who are based here partly here in London. They do excellent instant training that you can do online in a web browser. It’s fantastic. They are really great.”

• Acorn Electron: https://en.wikipedia.org/wiki/Acorn_Electron

• Commodore 64: https://en.wikipedia.org/wiki/Commodore_64

• Alstom: https://www.alstom.com

• Fortran: https://en.wikipedia.org/wiki/Fortran

• Thoughtworks: https://www.thoughtworks.com

• TDD, AI agents and coding with Kent Beck: https://newsletter.pragmaticengineer.com/p/tdd-ai-agents-and-coding-with-kent

• How Kent Beck shapes the software engineering industry: https://newsletter.pragmaticengineer.com/p/how-kent-beck-shapes-the-software

• Extreme programming: https://martinfowler.com/bliki/ExtremeProgramming.html

• Growing Object-Oriented Software Guided by Tests: https://growing-object-oriented-software.com/

• Dave Farley’s website: https://www.davefarley.net

• Jez Humble on X: https://x.com/jezhumble

• Continuous Delivery: https://continuousdelivery.com

• Mike Bland’s publications: https://mike-bland.com/publications

• How AI will change software engineering – with Martin Fowler: https://newsletter.pragmaticengineer.com/p/martin-fowler

• James Lewis on LinkedIn: https://www.linkedin.com/in/james-lewis-microservices

• Building Microservices: Designing Fine-Grained Systems: https://www.amazon.com/dp/1492034029

• Microservices Guide: https://martinfowler.com/microservices

• Ben Christensen on LinkedIn: https://www.linkedin.com/in/benjchristensen

• Scaling Uber with Thuan Pham (Uber’s first CTO): https://newsletter.pragmaticengineer.com/p/scaling-uber-with-thuan-pham-ubers

• Stop being skeptical about AI for development with Charity Majors: https://newsletter.pragmaticengineer.com/p/stop-being-skeptical-about-ai-for

• Understanding Spec-Driven-Development: Kiro, spec-kit, and Tessl: https://martinfowler.com/articles/exploring-gen-ai/sdd-3-tools.html

• Chris Ford on LinkedIn: https://www.linkedin.com/in/ctford

• Beyond Vibe Coding with Addy Osmani: https://newsletter.pragmaticengineer.com/p/beyond-vibe-coding-with-addy-osmani

• Building Resilient Distributed Systems: https://samnewman.io/books/building-resilient-distributed-systems

• Honeycomb: https://www.honeycomb.io

• Monzo: https://monzo.com

• How Generative and Agentic AI Shift Concern from Technical Debt to Cognitive Debt: https://margaretstorey.com/blog/2026/02/09/cognitive-debt

• Fundamentals of Software Architecture: A Modern Engineering Approach: https://www.amazon.com/dp/1098175514

• Balancing Coupling in Software Design: Universal Design Principles for Architecting Modular Software Systems: https://www.amazon.com/Balancing-Coupling-Software-Design-Addison-wesley/dp/0137353480

• Learning Domain-Driven Design: Aligning Software Architecture and Business Strategy: https://www.amazon.com/Learning-Domain-Driven-Design-Vlad-Khononov-ebook/dp/B09J2CMJZY

• Software Fundamentals: Collected Papers by David L. Parnas: https://www.abebooks.com/9780201703696/Software-Fundamentals-Collected-Papers-David-0201703696/plp

• Just Enough Software Architecture: https://www.georgefairbanks.com/book

• Modern Software Engineering: https://www.youtube.com/c/ContinuousDelivery

—

Production and marketing by Pen Name.

The state of the tech industry in 2026

Gergely Orosz — Tue, 6 Oct 2026 16:25:03 +0000

I recently delivered the keynote at the LDX3 engineering leadership conference in New York, attended by 2,000+ engineering leaders, CTOs, Director+ and Staff+ engineering folks, in which I attempted to create a snapshot of where the tech industry is at this exact point.

In the middle of my keynote at LDX3

It came after I had the chance to visit the AI labs OpenAI and Anthropic, meet innovative startups like Ramp, Uber and others, get new, unpublished data from GitHub, Factory AI and Linear. The focus of the talk is trend inside of AI labs, VC-funded startups and Big Tech. Thanks to the kind access to these teams!

Before the conference, I spent weeks identifying and recording trends that didn’t exist a year ago – or were much more nascent back then – and also things that are currently broken, and others that remain unchanged. Today, we cover:

Speed of change. In the tech industry, change has never been this large-scale, this fast. Industry legend Martin Fowler confirms.
What’s changed: practically nobody writes code by hand anymore, working with 5-10 agents more common, the fading of the IDE, and 13 more changes from the last year.
What’s broken: assumptions about code output broke, code reviews became theatrical, quality and reliability is down, and more.
What’s still the same: teams are still important, as is planning; non-engineers are still not shipping code, and more.
What comes next? Cloud coding agents and harnesses, engineers to stop reading the code, companies building a new type of AI infra, and more are already starting, and should accelerate.

You can watch the full talk, which is 29 minutes long:

www.youtube.com

Watch the full talk

Paid subscribers can also access the slides for the presentation:

People in tech are more than used to change; seeing the internet go from zero to everywhere, the arrival of mobile communications and smartphones exploding in popularity, cloud computing rising from a niche concern, and the development of programming languages like Go and Rust, and frameworks like React (web) and Jetpack Compose (Android).

Despite this, the scale and pace of the impact AI is having at present is still without precedent. That’s how industry veteran Martin Fowler described it at The Pragmatic Summit:

“Nothing has hit with the magnitude of AI. This is a whole size difference from anything that we’ve faced before.

On a smaller scale, we were very much involved in the growth of object-oriented languages. [object-oriented languages] scared a lot of people, but it didn’t scare us so much because we were part of it.

The internet had a huge impact upon us all. And, of course, we were spreading the challenge of agile software development. [Agile] had a very big impact on a lot of organizations because you could tell by how hard they resisted it.

But [for all those other changes] we kept talking about how important they were and how valuable they were and trying to persuade people of the importance of them. It may sound surprising, but even for the internet, there were people who wouldn’t think that was important!

For AI there’s no argument about how important it is. You cannot put blinkers on to deny the importance of this thing.”

Martin Fowler (left) at The Pragmatic Summit. See the full video

My own take is similar; we are without doubt at the beginning of a massive technological transformation, and after it, software will be built differently from how we’ve been doing for decades, although some parts of development will remain the same. At the very least, the tooling and best practices will look different, and things are changing faster than ever.

As the calendar enters Q4 of this year, we see this has solidified into a mega trend since the end of 2025 year when models greatly improved at coding. Related to this, in the very first article of this year we asked the question: when AI writes almost all code, what happens to software engineering?

Today, there are plenty of signs that most engineers have stopped writing code by hand, and it was also a sign of the times recently when the Ruby on Rails creator sparked a “death of coding by hand” debate. Personally, I’m excited by the new opportunity that AI creates, and it’s exciting to experience this revolution firsthand. As mentioned in that deepdive, I’m certain that my way of coding will change drastically in 2026, and there’ll be plenty of knock-on effects.

For more on this topic, check out this recent edition of The Pulse.

I like talking to engineers at AI labs because their working practices are often a few months ahead of the rest of the industry. Claude Code creator, Boris Cherny, revealed the new way he gets things done on the podcast:

“I have 5 terminal tabs. Each one of them has a checkout of their repository. I’ll round robin and start Claude Code in each one. I also run 5-10 Claudes on Claude Web, in parallel with my local Claudes.”

A few months after these comments, on last week’s pod with Cockroach Labs cofounder, Peter Mattis, – who’s an extremely productive developer who wrote circa 100K lines of code/year, pre-AI – said he’s doing something similar:

“Oftentimes, I’m doing things in parallel. I find my cognitive overhead is about 5-10 agent sessions concurrently. But sometimes those sessions will have many subagents doing things.”

I also asked a former coworker at Uber, who’s also an extremely productive software engineer, how he works these days. Dima Zaytsev (now a software engineer at Linear), told me:

“It used to be a simple world: one mouse, one keyboard, one screen. You physically can’t work on more than one thing at once. Now, you no longer have that limitation.

I end up having 5-10 (literally) worktrees locally that I rotate between. Prompt one agent, and while it is working, move on to another to test output or review code.”

Nearly all of the most productive software engineers I’ve met – who were hand-writing code a year ago – no longer write the code by hand and also run several parallel agents. What a change!

Fresh data that GitHub shared with me:

Agent-authored PRs: ninefold increase in eight months’ time. Source: GitHub

I’ve given GitHub grief for its ongoing reliability issues, but seeing how rapidly agent-generated PRs are growing makes me a lot more empathetic for the load which the platform is dealing with.

Food for thought: there were more agent-authored PRs in August 2026 on GitHub than human-authored ones! It’s reasonable to speculate that the number of agent-generated PRs will be permanently exceeding human-generated ones on the platform, going forward.

To give a sense of the number of AI-generated PRs: as of today (October 2026), there is likely to be 3x as many fully AI-generated PRs, per month (probably 75M+), than human generated ones at the end of 2023 (25M in December 2023). And the pace of AI-generated PRs does not seem to be slowing down, at least not now.

Agents are not just generating PRs, they’re also opening tickets. Some more exclusive data, this time from my friends at Linear, which added first-class MCP support for agentic creation of issues:

More agent-created issues than human-created ones since July. Source: Linear

Both engineers and non-engineers use agent skills, and lots of companies are building their own agent skills repositories for all staff to create, share, and evaluate. An interesting data point on skills usage trending heavily up comes from autonomous software factory vendor Factory AI, which shared with me:

83% of Factory AI users use skills, up 2.5x from February. Source: Factory AI

I have great respect for software engineer Steve Yegge, who’s good at identifying emerging trends and learning about them. In a podcast episode, he said the IDE is effectively over:

Gergely: “Speaking about the job as developers, you’ve said something that can be triggering for a lot of people. You’ve said, on the AI Engineer Summit, that if you’re still using an IDE now, you’re a bad engineer.”

Steve: “Yeah, well, you got to be a little provocative. Let me put it this way. Okay, I’m not going to say you’re a bad engineer because I know some very, very good engineers, better than I am, who are still at level one or two in my chart [of AI usage]? But I feel profoundly sorry for them.

I feel pity for them like I’ve never felt in my life. For these grown people who are good engineers or used to be. And they’re like, ‘yeah, you know, I use Cursor and I ask it questions sometimes. And I’m really impressed with the answers. And then I review its code really carefully.’

And then I check it in and I’m like: ‘dude, you’re going to get fired. And you’re one of the best engineers I know!!’ “

Steve described his levels of AI usage like this:

No AI usage
AI agent in the IDE, with strict permissions
AI agent in the IDE, with “YOLO” mode (permissions off)
AI agent in the IDE, no longer looking at the code, but interacting with the agent
CLI-first: abandoned the IDE
Running several agents in parallel
Running 10+ agents
Built a custom agent orchestrator to run 30+ (or 100+) agents

Steve believes that “AI-pilled” engineers will abandon the IDE, and I’m seeing data from the market which backs that prediction:

Antigravity 1.0 was the last IDE released, based on a VS Code fork. It was released in November 2025. After that, no major IDE that is a VS Code Fork has launched. In May 2026, when Antigravity 2.0 launched, the product moved away from the IDE concept.
Codex launched as a non-IDE in Feb 2026, and they’re glad they did so, despite being torn about moving away from the IDE concept, at the end of 2025.
Cursor moved away from the IDE. In April 2026, Cursor relaunched and got rid of the IDE interface. It now looks pretty similar to the Codex and Claude desktop apps. When I talked to their team over the summer, they told me they maintain the “legacy” VS Code fork version for existing enterprises, but the future is agentic for them.
JetBrains seems in a hurry to pivot from IDEs. JetBrains has built some of the most beloved IDEs for developers, but the company is now also pivoting to JetBrains Air, and agentic development environments.

How every coding tool looks these days – gone is the IDE? Source: Cursor

Industry legend Kent Beck told me something interesting when we discussed why IDEs are slowly vanishing:

“It’s not that we don’t need more perspective and context for our human-based decisions; it’s that the context has changed.”

I wonder if what is really happening is the IDE evolving into something different to work better with coding agents. Steve Yegge also talked about how the IDE needs to evolve into a conversation and monitoring interface, but not an editor, given that we no longer write the code.

I’d also add that IDEs need to evolve into a validation and verification interface: how do we know that the agents’ output works as expected, how can we verify what tests it passes, how it looks, and whether you can try out whatever the agent builds. Tools like this are surely coming and will be widely adopted.

In August, I posed a question on social media about whether it’s possible to be a serious tech business without having built an AI harness:

I asked it because most mid-sized-and-above companies have built their own agent harnesses, at this point! A few examples:

Ramp building Inspect – a case study we covered in depth
Stripe (Minions), Uber (Minion), Block (Goose), Shopify (River)
Google (Agent Smith), Meta (Devmate), Amazon (Kiro + Crew), Dropbox (Nova), Spotify (Honk)
DoorDash (Flux), Grab (LLM-Kit), WorkOS (Horizon), Hubspot (Crucible)
Monzo (Agent Chip), Sierra (Pinecone), Harvey (Spectre), Browserbase (bb)
… and many, many more!

LDX3 keynote: much dev work starts in Slack for startups

What I’ve learned by visiting startups is that more and more development kicks off inside Slack. At OpenAI, it’s “@Codex, implement this”; within Anthropic, it’s “@Claude build this”, while inside Linear, it’s “@Linear work on this”. More startups have their own Slack integration of their coding agent – often a custom one.

Coding agents work really well when deeply integrated into a company’s stack, when the Slack agent kicks off a coding agent that runs in the cloud.

We have covered in-depth how OpenAI is building an agentic software factory, and a lot of feedback from readers about that article has been an “we are too!” type response. Many readers say that their company is building a similar “agentic software factory.”

The “agentic software factory” adds AI functionality to a bunch of existing systems such as the CI/CD system; for instance, in being able to interact with CI/CD via Slack, or creating brand new systems with agents in them like agentic code review tools, OpenAI’s agentic deploy system, or the Perf Factory.

We’re seeing many migrations that would have taken years to complete, now taking mere weeks or months:

Anthropic: migrating the package manager Bun from Zig to Rust in 11 days (versus an estimated 1.5 engineering years)
OpenAI: the API layer is being migrated from Python to Rust. The migration is taking around 5 months, versus several years that it would have taken, without AI
Airbnb: migrating 3,500 end-to-end test files from Enzyme to React Testing Library: took 6 weeks (instead of years)
Asana: migrating 4,000 end-to-end test files from Enzyme to React Testing Library: 2 weeks, instead of an estimated spread out over five years
Uber: migrating 600,000 JUnit 4 tests across 15 million lines of code (!!) to JUnit 5: took 4 months (instead of estimated several years)

In May, we covered the trend of companies wanting to cut back on AI spend within engineering departments, and last month, I reported on how tech companies are moving to open AI models to save 50% or more in token costs. Uber is a good example, where, despite token usage trending upwards, costs have stayed flat since May thanks to them running open models and doing smart model routing:

Uber’s per-token costs are down 50%+. Source: The Pulse

Three weeks after my article, Bloomberg confirmed the exact same trend. As I covered in my original deepdive, lower-cost models and smart routing account for the majority of cost savings at most companies. If you’re only doing two things, consider those techniques.

Most efficient way to cut back on token costs. Image source: Databricks

Katelyn Lesse, Head of Engineering for Claude Platform, told me how her team works inside Anthropic. From our deepdive:

“On an individual project, you often cannot have more than two people working on it. This is because each engineer is already running several agents. And so as an engineer, you’re already fighting against your agents, which are stepping on each other’s toes on implementation. And in this setup, you just cannot have that many humans, who also come with all their agents!”

These days, I see the same thing happening at companies both small and large.

13. Engineering specializations are disappearing. Specifically, there seems to be less demand for iOS, Android, and frontend specializations, and more for “generic” software engineers. We covered this trend with data in the State of the software engineering job market in 2026, observing how mobile and frontend demand is dropping (while AI & FDE demand surges)
14. Teams are getting smaller. Projects are being done by fewer engineers. Inside startups, team sizes seem to be shrinking.
15. Reduced junior recruitment. We cover how it’s harder for graduates & interns to get hired in the State of the software engineering job market in 2026 deepdive.
16. Agentic infra is becoming its own discipline. Most mid-sized-and-above companies are building out agentic platforms, and infra teams are increasingly morphing into agentic infra teams.

It’s been a commonly held belief that code quantity grows roughly linearly. But with agents, both the lines of code and number of commits are growing exponentially, as shown in data from GitHub:

A software engineer at a mid-sized startup told me something that’s pretty much an open secret across the industry, including at places with code reviews in place.

“Everyone is playing the theater of doing reviews, but with the volume of changes that get thrown your way, I observed that people just find a path of least resistance: give up on reviews and just stamp everything with LGTM.

We are gradually phasing out this process ourselves with some shortcuts that you are allowed to take, and that has been a major productivity boost.”

At the LDX3 conference, there was a strong reaction among the audience when they heard the bolded sentence out above. Today at most companies, there is indeed a “theater of code reviews”. The fact is that no engineer can keep up with 5-10 times more code to review, so most don’t thoroughly review the code.

Fresh data from Linear shows that PRs which are reviewed solely by AI agents are a rising trend, and I expect it’ll continue:

AI-only PR reviews are up. Source: Linear

We see this in many of the digital products we use. As Mario Zechner, the creator of Pi, said on the podcast:

“Everything is broken. It sure feels like software has become a brittle mess, with 98% uptime becoming the norm instead of the exception, including for big services. And user interfaces have the weirdest bugs that you’d think a QA team would catch. I give you that that’s been the case for longer than agents exist. But we seem to be accelerating.”

We covered the phenomenon of quality being worse with more AI usage in March, in ‘Are AI agents actually slowing us down?’

It’s common knowledge that there’s a GPU shortage and memory shortage in the market. In addition, there’s also the growing trend of CPU shortages. As I covered last month:

“Apparently, it’s now nearly impossible to get CPUs on spot instances without long-running connections with cloud providers. Also, reserving specific CPUs now needs to be done months in advance, and cloud providers will even turn down certain reservations because they don’t have enough CPUs or the right type of CPUs.”

If you’re at a company with a non-trivial amount of CPU usage, securing capacity is something worth doing now, as I wrote recently:

“The best time to secure more CPU capacity is most certainly right now. I’m hearing rumors that certain cloud regions no longer accept new tenants because all CPU capacity is leased, or negotiations elsewhere are difficult. I’m also hearing that customers are already paying today to reserve capacity that will only come online in data centers from December. This seems predatory by providers, but demand is so high that this is how they likely prioritize new capacity allocation – while earning much higher profits than usual.

If your company has dynamic workloads, and you’ve used spot instances in the past, now could be a good time to allocate fixed capacity – even if it’s more expensive. If you expect meaningful growth, doing so now might mean having options at some cloud providers or in some regions.”

Software engineer Dima Zaytsev (currently at Linear, formerly my colleague at Uber) told me something that felt very relatable on AI and productivity:

“There’s lots of context switching: there’s always another agent that waits on your response.

Also, it’s just more work. When AI was ramping up, I felt more productive than my peers: I did in an hour what others needed a day for. Now, the expectation has become that everyone works on parallel things. And so, I end up feeling I have more work compared to pre-AI.”

An interesting, unexpected trend that has emerged is CTOs, Heads of Engineering, and VPs of Engineering resigning from their jobs, often with nothing specific lined up. From a current CTO who asked to remain anonymous:

“This change [with AI] makes builders want to build. We want to build again, so we can build more, and build faster with better tools, so we can increase the “talent density” of the team by doing more with fewer great people, and so on.

It’s an amazing time to build things. And it’s only getting better. And people just want to be close to it. They want to build and see how far they can go with these new tools.

I have a bunch of friends in my social circle who are the “CTO drop outs” you speak of. More still who are moving from management to IC roles in large companies. And their stories are largely what I describe above. They simply see this as a truly exciting time to build.

In the end, building is why they got into this field in the first place.”

We cover this in the deepdive ‘Headed for the exit: the great engineering leader career break’.

In the LDX3 keynote, I also covered areas that are mostly unchanged from before AI:

Teams are still important. As Katelyn Lesse, Head of Engineering for Claude Platform told me in July:

“One thing I’ve heard from some people is “we have two humans and a bunch of agents.” I reply that this isn’t where we’re at. I still have teams whose job it is to own a piece of software, iterate on it, own oncall, and so on. While each of these humans are supercharged by AI, the size and shape of the team is still similar. We still have two-pizza teams.”

Katelyn works at one of the most “AI-pilled” companies globally, so if teams are still as important there as they were before AI, then it’s not a stretch to say that the team structure is still relevant at companies across the industry.

Planning still happens – at least for complex work. Complex projects still have a lengthy planning phase. Also from Katelyn, on the planning phase of Claude Managed Agents:

“Our planning process looked more like a typical pre-AI planning process. You know how every team has the project, where everyone comes up with some version of the same idea and people keep floating and circling it around until you finally do it? Managed Agents was this for our team. When we started the project, we had documents dating back up to two years about ideas and suggestions.”

Tests and validation are still very important. From Jarred Sumner, creator of Bun, who’s currently at Anthropic:

“The AI writes pretty much all the code, but we have the AI write pretty much all the tests as well. Before AI, for production-ready software, we spent about the same time writing tests as we did on writing code. This is still the case with AI. You need to have a way to trust your code, and tests are probably the best way to.”

At each company I talked to in advance of the LDX3 keynote, I observed that a lot of focus is going into validating the output of AI agents. This makes sense: code review no longer works like it did before, we have more code, and we need to make sure that it won’t break production!

Non-engineers are still not shipping prod code. In the last few weeks, there’s been stuff on social media about AI enabling product managers/designers/non-technical people to ship production code. So, I asked the AI labs, startups, and other companies.

I can report that I did not find any single company where PMs/designers/non-technical people ship to production!

What I did learn about are cases where these folks create bugfixes – often unknowingly! – or new features with their agents that end up with devs to review. There’s also a massive amount of prototyping. But it’s still engineers who are responsible for deciding what can and what cannot be released to production.

We’re rediscovering old patterns that work great with AI. This came up in our podcast with Matt Pocock. Matt noticed that agents try to build software layer by layer, which causes bugs between the layers. Reading The Pragmatic Programmer, he discovered the concept of the “tracer bullet.” When he instructed the agent to use “tracer bullets” to build an app (aka implement a “golden path”), the agent started to produce better code.

As mentioned in the podcast, Matt is now reading classic software engineering books to find other “leading words” that guide agents efficiently.

In general, I’m noticing that “old” best practices are helpful when building better software when working with AI. This includes writing unit tests, building a “golden path” first (aka a “tracer bullet”), architecting an application upfront, and even using design patterns. The amusing thing is that we’re talking about decades-old best practices here.

It’s possible to look ahead at where the tech industry is headed by identifying trends that are underway, and which look certain to conitnue:

Cloud coding agents + harnesses will dominate. Most devs at companies will run AI agents in the cloud, instead of locally. Ramp offers a blueprint for this in our deepdive about their cloud agent, called Inspect. It seems that inside innovative tech companies, the “build your own cloud agent harness” principle is trending.

I also expect vendors like Anthropic, OpenAI, SpaceX, and others to start pushing their cloud coding agent offerings, and for more startups to build their own cloud harnesses.

As engineers, we’ll stop reading the code. It’s too early to tell when this will happen; this year, next year, or further in the future, but it’s likely that few of us will read the code that AI agents produce properly, going forward.

As a rule, I pay a lot of attention to Honeycomb cofounder and CTO, Charity Majors, due to her being a “default sceptic” about new technology, as well as a standout engineer and someone who speaks her mind. On our podcast in August, she asked:

“What would it take for you to be comfortable shipping code without you reading it and understanding it? Because that is engineering.”

Charity made the point that Ops and QA have already had decades to figure out how to ship code to production that they didn’t write and don’t understand – all while making sure it works! It seems unavoidable that most software engineers will also join this group. This is ironic, since software engineering, for the longest time, was about engineers writing and understanding the code!

Companies will build brand new types of internal infra. We’ll see a “reinvention” of these systems to work well with agents:

CI/CD, with agentic evals part of the pipelines
Agentic software factories, and figuring out what is and isn’t practical
Agents becoming part of deployments, observability, incident management
Agentic observability will surely become its own problem space and specialization: how can we make sure the customer-facing agents deployed work as expected, and how to ensure that systems which agents build and modify actually work as expected?

A “golden age” of refactoring/migrations/full-on rewrites. Migrations and rewrites that had been delayed for years because they’d take similar amounts of time to complete, should take no longer than weeks. That means there’s little excuse to delay any longer!

Engineers understand more about how capable AI agents are with rewrites/refactors/migrations today, and we’ll also clean up tech debt much faster, while having no reason to have a system in a state that we’re unhappy with!

AI “fluency” and “positivity” matter in recruitment at startups. A trend I have not written much about – but which is already happening – is startups selecting engineers for jobs who demonstrate a positive attitude about AI. As the Director of Engineering at a Series D startup told me:

“AI positivity was something we started to screen for in our hiring process. We want people to join who want to help us push the limits with what we can build with AI.”

The reality is that AI coding agents are already everywhere, and agentic systems look set to proliferate very soon. This will bring great demand for software engineers who are willing and able to build next-generation systems, and it’ll be a baseline criterion to be excited about the problem space.

This is not too dissimilar to the way that startups hire engineers who buy into the idea of growing fast, or when companies favor engineers who pick up the tech stack already in use, instead of insisting upon the one that they personally use.

Engineers with deep domain knowledge will become more in demand. In New York, I had a conversation with Titus Winters, lead author of Software Engineering at Google. He told me this observation:

“What you need to succeed anywhere is intelligence, wisdom, and charisma. Intelligence means: how to do it. Wisdom means: what to do. Charisma means: convince others to do it.

When intelligence becomes commonplace, wisdom and charisma become much more important.”

In the context of software engineering, “wisdom” is most easily earned by becoming a domain expert. So, if you work at a fintech company, learn about the finance industry and its customers in order to become a more valuable, in-demand engineer! That also applies if you’re at an agriculture tech company, or work in any other domain.

Finally: remind yourself why you got into the tech industry. At the end of my podcast with Peter Mattis, the cofounder and CTO of Cockroach Labs, we talked about how it feels pretty exhausting right now to keep up with the pace of change. He said:

“Remind yourself why you got into this in the first place. I got into software engineering because I like building stuff. Now I can build faster, and without some of the compromises that I had before!”

No posts

The Pulse: Firebase’s global outage & poor response

Gergely Orosz — Thu, 1 Oct 2026 16:43:19 +0000

The Pulse is a series covering events, insights, and trends within Big Tech and startups.

Today, we cover:

Firebase: global outage & poor handling by Google. The Firebase iOS SDK crashed after a backend change, crashing all apps which used Firebase analytics for 2-6 hours. Google did not update the status page or offer any postmortem, which is a head-scratcher from a company known for standout incident management practices.
OpenAI’s platform play from AWS playbook? OpenAI is becoming a platform where it’s possible to allocate ChatGPT spend on open models and AI offerings from among 16 partners, not just OpenAI models. It’s fair to ask if Anthropic will consider a similar platform play.
More data on companies moving to open models. Vercel’s AI gateway shows 60% of model spend goes to open weight models, and OpenRouter also shows open models are being more used than closed ones.
Why CTOs and VPEs are quitting en masse: another take. What if it’s not “founder mode”, but about people who love building software, feeling like they can do it solo (or with a small team) with AI tools?

Distributed databases with Peter Mattis

Gergely Orosz — Wed, 30 Sep 2026 16:30:05 +0000

Listen and watch now on YouTube, Apple, and Spotify. See the episode transcript at the top of this page, and timestamps for the episode at the bottom.

• Linear – most of us work with agents in a “single-player” setting. Linear’s take is agent work should be teamwork: your teammates can follow the session of the agent, check out the PR it produces, and join the review. Works with Codex, Cursor, Linear’s own agent, or custom agents. Check it out

How is it that a software veteran who regularly shipped ~100K of database-grade code to production each year, pre-AI, feels like he’s even more productive today, with no drop in quality? Peter Mattis is co-founder and CTO of Cockroach Labs, and an original creator of GIMP. He also worked on Gmail and distributed storage at Google.

In this episode, Peter reflects on his journey from open source to Google to founding a database company, and we explore how to keep systems fast, reliable, and correct at scale, from Gmail’s early storage challenges to the tradeoffs in building distributed databases.

Peter tells us how AI has brought him back to writing code after his work shifted toward management, and why he believes AI can improve quality and multiply the impact of domain experts. We also consider the future of code review, and Peter has some advice about how to level up our engineering skills.

www.youtube.com

1. Peter initially turned down Sergey Brin’s offer to join Google because of the commute. The very first version of the famous Google logo was made in GIMP, the free, open-source raster graphics and image editing software created by Peter and his college roommate Spencer Kimball.

The first version of the Google logo, created by GIMP

In 2001, Sergey reached out to Peter, invited him to an interview and then made an offer. But Peter said “no” because he lived in San Francisco and did not fancy the hour-long commute to Mountain View. Instead, he joined another startup, but that didn’t go anywhere. When Google reached out again, he took the chance.

2. Peter almost did not ship GIMP after learning about an even more ambitious photo editor. Peter and Spencer worked hard on GIMP, but a few weeks before launching it, they saw an announcement about another program that promised to do everything GIMP did – and then some! Peter and Spencer felt discouraged but shipped anyway – and the rest is history. They never heard about the other ambitious project again. “There’s always going to be someone else working on your idea,” says Peter. “You can’t get dissuaded if they pre-announce it. To founders: assume that dozens of people have the same idea you have, but most won’t ship it! Know that it’s a competition; you’ve got to enjoy that aspect of it – don’t be afraid of it.”

3. B-trees were used in the first version of Gmail. Gmail launched on 1 April 2004, offering 1GB of email storage for free, an offer seen as so ridiculously generous at the time that most people assumed it was an April Fool’s joke! B-trees played a role in Gmail’s storage layer, thanks to storing email threads. Under the hood, each incoming message was matched to a thread using the search index. Threads and their unread counts were tracked by B-trees.

4. Colossus reduced Google’s file storage overhead by 33%, while increasing redundancy. Before Colossus, Google File System (GFS) stored three full copies of data. For Colossus, which became GFS’s successor, Peter and the team pioneered Reed–Solomon erasure coding in a distributed file system. Data was stored twice, but redundancy increased!

5. The fastest way to send a packet around the globe is through space! Peter keeps “speed of light numbers” in his head – similar to what turbopuffer founder Simon Eskildsen does with ”napkin math” calculations. For example, a network round trip within a zone went from milliseconds when Colossus was built to about 100 microseconds today (10x faster!). But sometimes the bottleneck is the speed of light, and as light speed is faster in a vacuum than anywhere else, that means the fastest global route is straight up to Starlink, across by laser, and then back down. Find out more in ‘ Delay is not an option: low latency routing in space ’.

6. Peter twice “beat” standard library data structures in performance terms. At Google, one of his colleagues noticed that std::map showed up in memory profiles. Peter looked closer and figured out that the std::map implementation is a red-black tree, meaning every node has two pointers. Peter built a B-tree with nearly the same semantics which was both faster, due to spatial locality, and smaller because it used fewer pointers. They used this data structure inside Google. Peter also did something similar with Go’s map: it was performant, but he built a Swiss Table implementation that was faster. That implementation later made it into the Go library, with the Go team helping to finish it!

7. If you squint hard, everything in distributed databases and storage systems starts looking like a B-tree. These B-trees are a recurring theme in this podcast episode: the backend of Gmail, the std::map replacement, CockroachDB’s range index, etc. There’s even a paper on this phenomenon, The Ubiquitous B-Tree.

A B-tree: a very useful data structure. Source: Wikipedia

8. For consensus in a distributed database, at least three replicas are needed. With only one, recovery after it crashes isn’t possible. With primary and secondary replicas, neither one will know if the other received the last write following a crash. So, CockroachDB uses three replicas by default – and up to five for some system tables – and customers can add more, although this will result in higher latency.

9. CockroachDB’s design was inspired by Google’s Colossus data storage system and Spanner. Colossus was append-only, meaning you could only add to files and not mutate them. Google’s distributed database, Spanner, was built on top of Colossus, and so Spanner’s design decisions came from Colossus’ append-only nature. But if you cannot update files and only append to them, it’s not possible to use B-trees as the database’s design structure since B-trees need the files to be updated. An ideal data structure for append-only filesystems is the Log-structured merge tree (LSM tree), where writes are appended to a log file for durability:

A log-structured merge tree. Source: Wikipedia

Google popularized LSMs with LevelDB, and RocksDB later forked LevelDB. RocksDB was the underlying storage engine that CockroachDB used in its early years. In 2019, Peter wrote and open sourced Pebble, which is now CockroachDB’s storage engine.

10. Peter, the former C++ readability reviewer at Google, predicts we will stop reviewing code: At Google, every code change needs to be signed off for “readability” by a language readability reviewer, and Peter was a C++ readability reviewer for years. But these days, he’s reviewing less code and foresees this continuing. As he puts it: “What I’m finding is the agents are getting better; you’re having to give less and less scrutiny. I don’t know if it’s going to be this year or next, [but] we’re materially going to stop looking at the code in the same way we don’t look at assembly anymore.”

11. Non-engineers at Cockroach Labs built ~1,000 internal apps (!!) in a couple of months. Earlier this year, Cockroach Labs launched an internal platform for non-devs to build apps (think of it as an “internal Lovable”). Folks jumped at it; HR professionals built new tools they’d only dreamt of, while the CFO created the dashboards they’d always longed for, and many others too. Peter and the engineering team were surprised at this uptake, but it chimes with how OpenAI saw nearly all its non-engineers move almost their entire token spend from ChatGPT to Codex within just 4 months.

12. Does “flow” still exist with AI? I asked Peter this, and he believes it does, but it’s different from the “coding flow” state: “It definitely feels a bit different. It’s maybe a bit less intense, but you’re managing more things cognitively. I’m thinking of ideas that normally would’ve taken me a week to experiment with. And I think of multiple of these experiments and then fire them off all simultaneously. [It feels as] if I was a college professor with a whole swarm of research assistants, and they’re all off doing things and it’s coming back really rapidly.”

• Inside Google’s Engineering Culture

• Resiliency in distributed systems

• How to debug large, distributed systems: Antithesis

• Pushing software engineering limits with “napkin math”

• Designing Data-intensive Applications with Martin Kleppmann

• Formal methods with Hillel Wayne

00:00 Intro

02:42 Peter’s path into tech

04:00 Building GIMP

09:30 Working on Gmail at Google

14:51 Google’s infra: google3, build files, Bazel, and Colossus

21:30 Distributed storage bottlenecks

23:59 Latency, throughput, and availability

30:04 Contributing to libraries

41:52 Google Spanner

46:10 CockroachDB

52:00 Manual vs. automatic sharding

55:28 Consistency models and strong consistency

1:00:03 Raft consensus

1:06:15 How AI brought Peter back to coding

1:19:12 Peter’s tools and agentic workflows

1:23:08 How AI can improve quality

1:26:39 Code reviews: are they done?

1:29:17 100x engineers

1:35:33 Peter’s advice for leveling up your engineering skills

Where to find Peter Mattis:

• X: https://x.com/petermattis

• LinkedIn: https://www.linkedin.com/in/peter-mattis-46549144

Mentions during the episode:

• Cockroach Labs: https://www.cockroachlabs.com

• GIMP: https://www.gimp.org

• Usenet: https://en.wikipedia.org/wiki/Usenet

• Larry Page: https://en.wikipedia.org/wiki/Larry_Page

• Sergey Brin: https://en.wikipedia.org/wiki/Sergey_Brin

• Paul Buchheit on X: https://x.com/paultoo

• Y Combinator: https://www.ycombinator.com

• Delay is Not an Option: Low Latency Routing in Space: https://discovery.ucl.ac.uk/id/eprint/10062262/7/Handley_hotnets.pdf

• Bazel: https://opensource.google/projects/bazel

• Colossus under the hood: a peek into Google’s scalable storage system: https://cloud.google.com/blog/products/storage-data-transfer/a-peek-behind-colossus-googles-file-system

• Pushing software engineering limits with “napkin math”: https://newsletter.pragmaticengineer.com/p/pushing-software-engineering-limits

• turbopuffer: https://turbopuffer.com

• Trimodal Nature of Tech Compensation in the US, UK and India: https://newsletter.pragmaticengineer.com/p/trimodal

• Quicksort: https://en.wikipedia.org/wiki/Quicksort

• B-tree: https://en.wikipedia.org/wiki/B-tree

• Go: https://go.dev

• Google Goggles: https://en.wikipedia.org/wiki/Google_Goggles

• Spencer Kimball on LinkedIn: linkedin.com/in/spencerwkimball

• Ben Darnell: https://en.wikipedia.org/wiki/Ben_Darnell

• The Ubiquitous B-Tree: https://dl.acm.org/doi/pdf/10.1145/356770.356776

• Paxos: https://martinfowler.com/articles/patterns-of-distributed-systems/paxos.html

• Raft is so fetch: The Raft Consensus Algorithm explained through “Mean Girls”: https://www.cockroachlabs.com/blog/raft-is-so-fetch/

• DoorDash: https://www.doordash.com

• RocksDB: https://rocksdb.org

• Rubber duck debugging: https://rubberduckdebugging.com

• Claude Code: https://claude.com/product/claude-code

• Codex: https://chatgpt.com/codex

• Building Claude Code with Boris Cherny: https://newsletter.pragmaticengineer.com/p/building-claude-code-with-boris-cherny

• Programming TypeScript: https://www.oreilly.com/library/view/programming-typescript/9781492037644

• Building Codex with Tibo Sottiaux: https://newsletter.pragmaticengineer.com/p/building-codex-with-tibo-sottiaux

• Terence Tao Digests the Jacobian Conjecture Counterexample: How Claude Fable 5 Broke an 87-Year-Old Math Problem: https://www.developersdigest.tech/blog/jacobian-conjecture-counterexample-fable

• Jeff Dean on LinkedIn: https://www.linkedin.com/in/jeff-dean-8b212555

• Sanjay Ghemawat on LinkedIn: https://www.linkedin.com/in/sanjay-ghemawat-763485428

—

Production and marketing by Pen Name.

Why has Shopify dropped React Native?

Gergely Orosz — Tue, 29 Sep 2026 15:53:19 +0000

Before we start: given this article is about native mobile development, I want to offer my 2021 ebook, ‘Building Mobile Apps at Scale: 39 engineering challenges’, for free to all readers. (normally costs $20). The book remains relevant on the challenges to solve for large-scale mobile applications, and lists the technologies covered below in this article, Kotlin Multiplatform included.

Claim your free copy here

This offer is valid until Friday, 2 October. On checkout, simply select “The ebook: PDF & EPUB versions of the book.” No credit card required! Feel free to share the link with colleagues who build mobile apps, or work on them.

Recently, Shopify dropped the bombshell announcement that Native is now the future of mobile development at the e-commerce platform. This triggered a lot of reaction in the mobile development community, on the scale of the stir Shopify generated when it originally moved over to React Native six years ago.

But times have changed, and we all know the agent of change this time: AI agents! This latest generation of models has improved in coding capability to the extent that Shopify no longer is satisfied with React Native (RN), as it was only last year.

This article looks into what led here, why Shopify made the change, and also the broader mobile ecosystem. We cover:

Why Shopify chose React Native in 2020. Android apps took too long to build, first and foremost, and it was nice to have consistent iOS and Android apps.
Still happy five years later. Shopify moved all six apps over to RN and most things were going well, and the 2020 move was seen as a good move.
Why ditch React Native now? AI is now very good at writing mobile code as well as backend code, while React Native adds abstractions that native does not. Shopify rewrote the Shop app to native in just 12 weeks(!!). Additional details from the Shopify mobile team.
Haven’t we seen this back-and-forth before? Airbnb did something similar by adopting React Native in 2016 before moving back to native just two years later. The underlying reason was the same: performance.
Going native since 2019. Moving to native used to take a long time, and Notion has been at it for seven years. When Notion finishes the process, a fair question will be whether a native editor actually slows down their web iteration speed, with or without AI.
Kotlin Multiplatform (KMP) would like a word. KMP allows the writing of business logic in Kotlin and sharing it across iOS and Android while each platform stays native. It’s a technology that feels relevant, and could become more popular with AI.
What’s next? It’s easier to write native iOS and Android apps than before, but that’s also true for React, Flutter, and KMP apps.

Let’s rewind to 2020, when Shopify decided to go all-in on React Native (RN). Back then, mobile was a very important channel for the platform, with seven out of 10 customers using it on mobile devices. As Head of Engineering, Farhan Thawar wrote at the time:

“Each quarter, the majority of buyers purchase on mobile (with 71% of our buyers purchasing on mobile in Q3 of last year [in 2019]). Black Friday and Cyber Monday (together, BFCM) are the busiest time of year for our merchants, and buying activity during those days is a bellwether. During this year’s BFCM, Shopify merchants saw another 3% increase in purchases on mobile, an average of 69% of sales.”

With mobile so important, it’s no surprise the company built native mobile applications for iOS and Android; after all, native mobile gives full control of a platform’s capabilities, integrations, and performance, so why bother with cross-platform technology?

It turns out one reason was the delay in launching native apps on Android, which Shopify struggled with, along with other companies.

During the pandemic at the start of the decade, the live conversations app Clubhouse was a massive hit after launching in March 2020. The app was iOS-only for the first 14 months as the team was unable to release an Android app. And at the end of the year, usage absolutely exploded:

December 2020: 600,000 global installs (downloads) ( source)
January 2021: 3.5 million global installs ( source)
February 2022: 8 million global installs ( source)

Clubhouse app: voice chat blew up with Covid-19. Source: TechCrunch

Despite all that popularity, Clubhouse was unavailable on Android at the time!

It took 14 months to launch a beta version of Clubhouse on Android, and by the time it did in May 2021, the platform had passed its peak a few months earlier and usage numbers were by then trending down. It took until July 2021 to roll out the Android Clubhouse app globally – 16 months after the iOS version – by which time it was too late to catch the growth wave, and that spelt doom for the promising business.

How did this happen? There were a few main reasons:

Android is more complex to build well for than iOS: more device types to support, and much older OS versions to support with different APIs.
Getting good performance on low-end Android phones was hard.
I also wonder if Clubhouse’s team was engineering for a scale that never materialized. Perhaps they tried to build the “perfect” app to launch globally, but missed the growth wave, meaning the app ended up “over-engineered” compared to a different version that could’ve been released sooner, in a more timely fashion that benefitted the business.

Building a native Android app from scratch – even with an existing iOS app as a blueprint – was a challenge during the 2010s and early 2020s. Clubhouse is an extreme case of a business which went wrong, partly due to a sluggish Android launch.

But customer habits could have played a part: at the time, most US startups prioritized iOS apps and features first because in the US, the most valuable users were increasingly found on iOS. Prioritizing iOS made business sense for companies with mainly US customers, but not for ones with a global footprint, where regions like Asia featured mainly Android smartphone users.

Shopify’s first positive experience with React Native was in 2018 with the Shop app (then called ‘Arrive’). The company summarized (emphasis mine):

“At the end of 2018, we decided to rewrite one of our most popular consumer apps, Arrive (which is now Shop app) in React Native. Arrive is no slouch; it’s a highly rated, high performing app that has millions of downloads on iOS. It was a good candidate because we didn’t have an Android version. Our efforts would help us reach all of the Android users who were clamoring for Arrive. It’s now React Native on both iOS and Android and shares 95% of the same code. We’ll do a deep dive into Arrive in a future blog post.

So far, this rewrite resulted in:

fewer crashes on iOS than our native iOS app
an Android version launched
a team composed of mobile + non-mobile developers.”

Moving over to React Native was one of the easiest ways to speed up the rollout of Android versions of new apps. In 2020, when Shopify announced the move, that was the stated intention:

“Our popular Shopify Ping app (now called Shopify Inbox) which has enabled hundreds of thousands of customer conversations is currently only iOS. In 2020, we’ll be building the Android version using React Native out of our San Francisco office and we’re hiring.”

Last year, Shopify declared it was satisfied with the move to RN in a recap article, ‘ Five years of React Native ’. The summary was quite the victory lap:

“To recap, we decided to switch to RN for three main reasons:

Write it once - Stop building the same features twice, once on iOS and once on Android

Talent portability - Enable devs to work fluently across iOS, Android, and Web

Ship more value - Spend more time delivering value to users instead of chasing feature parity

We’re happy to share that our transition has been quite successful:

Not having to build the same features twice has given us a step change in productivity

Engineers are able to work across web and mobile allowing teams to do more with the same number of people and unlocked new growth opportunities

Maintaining feature parity between iOS and Android has become a non-issue, freeing up capacity to ship a lot more value

Our apps are blazing fast (<500ms screen loads) and stable (>99.9% crash-free sessions)

We continue to leverage native wherever it is the best tool for the job, giving us the best of both worlds

Over the past 5 years, we have migrated all our apps to React Native. Instead of using a one-size-fits-all approach to do so, each team chose when and how to migrate their app. This allowed them to continue shipping features while also aligning with our strategy of leveraging RN.”

They weren’t kidding: in five years, six mobile apps were migrated to React Native:

App migrations to React Native

Shopify’s engineering team said they loved a lot about React Native. Mustafa explained:

“Speed. RN apps are fast. We’ve achieved sub-500ms (P75) screen loads in the Shopify app. We’ve achieved similar performance in all our apps. Just like native, you have to apply good patterns and techniques to eliminate performance bottlenecks.
Hot reloading. This was one of our biggest pain points with native. Given the size of our code bases, it took several minutes for even the most trivial changes to be compiled and run on an emulator/physical device. This wastes time and breaks developer flow. React Native’s hot reloading completely eliminates this problem.
TypeScript is awesome as the “shared” language for devs. TypeScript has become ubiquitous, and we’ve seen great success with developers transitioning between React web and React Native.”

There were also some new challenges:

Worse debugging: “Debugging in React Native is flaky and configuring it correctly in VS Code takes some work. iOS and Android on the other hand have powerful debugging capabilities that just work”
Native code and devs still important. You want native code for performance-intensive code, certain animations, long-running background jobs and specialist APIs (like home screen widgets).
More third-party libraries. “The React Native framework is not as comprehensive compared to Native, so you end up having to use more third-party libraries.”

But how quickly things can change in tech…

There was widespread surprise this month when Shopify announced that Native is now the future of mobile at Shopify, meaning that RN is suddenly history.

From Head of Mobile, Mustafa Ali (emphasis mine):

“In January 2025, I wrote that the future of React Native was bright and that Shopify planned to keep investing in it. That was true based on what we knew then. React Native was working well for us, and it remains an excellent framework. But since then, coding models have gotten dramatically better, and for our apps and our team, building the same feature in Swift and Kotlin no longer carries the cost it used to.

Native still means building and maintaining software on two platforms, that cost has not disappeared. What changed is that agents can now do enough of the implementation, translation, testing, and review work that it’s no longer the deciding factor it was in 2020.

React Native apps can be fast. Ours are. We are making this change because agents have reduced the advantages of sharing implementation, while the advantages of building for each platform remain. Native keeps us closer to platform capabilities and first-party tooling, with fewer framework and dependency layers between our code and the platform.”

Of course, there have always been pros and cons in choosing native over cross-platform. Pre-AI, this is how it looked:

Native vs React Native development, observed across these four dimensions

What’s changed is that AI coding agents are now much more capable at generating iOS and Android code.

There’s no doubt that fully native apps are well-built and always more efficient than those built on a cross-platform interpreter like React Native. Nonetheless, RN has come a long way to be much more performant – with one additional abstraction layer between RN code and native code – especially since the New Architecture of 2024, Let’s visualize how it works:

How React Native renders views on iOS and Android

React Native’s core philosophy is to minimize updates done on the UI using expensive rendering operations. So, RN maintains a React Host Tree of visual elements on a screen (think a “logical tree”) which transform into a Host View Tree, the representation on iOS and Android. React itself cleverly minimizes re-rendering operations, which saves on compute resources.

As mentioned above, a well-written native app naturally outperforms a React Native version. The easiest way to see why this is, is to sketch what happens with native apps rendering, compared to with React Native:

React Native vs native rendering

React Native can theoretically be more performant, in the unlikely event the Native version has a poorly written structure, or if UIKit / Jetpack Compose or any other native library is inefficient enough for this to happen! But native has many fewer abstraction layers.

One common question since Shopify moved to native is how come the Shop app had not moved over to the New Architecture to improve things like startup performance. I reached out to Head of Mobile, Mustafa Ali, about that. He told me:

“We actually moved other big Shopify apps to the New Architecture. Shopify and Point of Sale were both migrated, and I believe we were one of the first large teams outside Meta to do it.

We got some performance gains with this migration, but they were modest. App startup got about 10% faster on Android and 3% on iOS, and some complex screens actually got slower until we tuned them.”

The team has written in detail about the move to New Architecture. Nonetheless, being on native code provides a lot more opportunities to improve performance; after all, you can now control the complete rendering stack! Heck, you could also throw away the Apple-built system libraries like UIKit or ComposeKit and build more performant ones. Shopify doesn’t plan to go this far, but Head of Engineering, Farhan Thawar, told me they expect performance wins:

“I think we’re just at the beginning of how much better these native apps will be. I expect they will get faster – like with better startup time and more responsive interactivity – as we work through the codebase post migration.”

One benefit of React Native was that it wasn’t necessary to have both an iOS engineer and an Android engineer. But with AI, one engineer can build software on both platforms, Mustafa Ali told me:

“One cultural benefit of switching to RN in the past is that we don’t have a separate iOS and Android team, unlike most native shops.

We have one team, and the same developer builds a feature on both iOS and Android. This keeps the context in one place and there’s no communication overhead.

With LLMs making it easy to contribute code outside of your main expertise, I expect this to become more common in the coming months.”

Farhan put it even more simply:

“Basically, React, as the shared language, is now replaced by the English language, thanks to LLMs.”

Another downside of going native is having two codebases to maintain, which tend to diverge from each other in behavior, features, and bugs. But with AI, Shopify doesn’t see this as a big problem any more. Mustafa:

“Maintaining two separate codebases was a big deal before LLMs, but not anymore.

Agents are quite good at porting a feature from Android to iOS or the other way around. They are also good at comparing the two implementations over time. They can spot gaps in the code and in side-by-side testing, and fix them.”

Shopify is also building a shared verification test suite to validate the two apps as identical. Again, from Mustafa:

“We now have one shared test suite that validates business logic implemented in both Swift and Kotlin. This means that a feature can’t ship until it passes the exact same tests on both platforms. That logic runs headless on desktop, which gives agents a very fast feedback loop to iterate on.”

Smart! It also confirms my sense that verifying code is becoming very important for working with agents, as is building dedicated verification layers.

Let’s return to the comparison table and update it for how native + AI changes the equation for Shopify:

Native vs React Native development, with more capable AI agents

My personal sense is that the downsides of native development disappear with AI tools getting so good, and that Shopify now has full control over the performance and architecture of its native apps.

With the decision to go all-in on native with AI, the e-commerce business is committing to migrate all of its apps. With AI, it should be faster than the migration to React Native was. Mustafa, again:

“We are going to migrate all our mobile apps to Swift and Kotlin using AI throughout the process.

Shop has already shipped as a fully native app in just 12 weeks, the Shopify app is underway, and the rest will follow soon. We’re moving quickly, but not by lowering the bar.

Every rebuild must meet or exceed the performance, stability, accessibility, and product quality people expect today. This isn’t just the same apps rewritten in different languages. We’re rebuilding them so both humans and agents can understand, test, and change them quickly.”

Best of luck to the team with this migration! I’m looking forward to hearing about what they learn with this move.

The Pulse: RoR creator sparks new “death of coding by hand” debate

Gergely Orosz — Thu, 24 Sep 2026 16:44:27 +0000

37signals – the creator of Ruby on Rails (RoR) – moves over to agents generating nearly all its code. Also: Amazon and Meta struggle to hire engineers, code reviews will probably also go away and more

Sep 24, 2026

∙ Paid

The Pulse is a series covering events, insights, and trends within Big Tech and startups.

Today, we cover:

Writing code by hand: is it over? In his Rails World keynote, David Heinemeier Hansson (DHH) declared the end for writing code by hand for professional work – at 37signals at least. Is this change now unstoppable?
Amazon and Meta struggle to hire and …

Design Engineering with Maggie Appleton

Gergely Orosz — Wed, 23 Sep 2026 17:07:22 +0000

Listen and watch now on YouTube , Apple , and Spotify . See the episode transcript at the top of this page, and timestamps for the episode at the bottom.

• turbopuffer – not only a vector and full-text search engine built on object storage, but I also think that they have one of the most refreshing brands in tech. They have charts showing p50, p90 and p99 performance on their landing page , hand-crafted ASCII diagrams , and a team that goes exceptional lengths to deliver for their customers.

• O’Reilly Early Release: Scaling AI Adoption in Engineering – As an engineering leader or CTO, why is it so hard to get business value from AI? CTO Peter Bell brings strategies and tactics that work. The book is available for free, compliments of season sponsor Antithesis. Download your copy here (The book currently has 4 chapters ready, additional chapters will be sent out as the author finishes them.)

• Entire – Git hosting, rebuilt for the agentic era. Entire hosts your code in-region, and is up to 89x faster than any other competitor. Mirror from GitHub with a single click – I’ve already done so.

What can everyone else learn from designers and design engineers? As it turns out, there’s plenty, as I discovered when one of the best design engineers in the industry, Maggie Appleton , came onto the Pragmatic Engineer Podcast. She’s a staff research engineer at GitHub Next , where she builds prototypes to explore how software engineers might collaborate with AI in new ways. Maggie is at the intersection of design, anthropology, and web development, and was the first designer hired by AI startup Elicit, and Lead Design engineer at AI startup, Normally.

Today’s episode is more visual than usual because Maggie brought her notebook along, so there are peeks inside its pages of prototypes and more:

Where the design process starts: pages in Maggie’s notebook

We got into designers’ work and how their design processes are adapting to and changing with AI. We explore why Maggie starts projects with pens and notebooks, what distinguishes design engineers from other designers, and why understanding engineering constraints leads to better collaboration with engineers.

We also discuss how Maggie uses jigs to gain more control over AI agents, why human judgment and style still matter when models can generate designs, and how inconsistent AI capabilities can mislead us.

www.youtube.com

1. Post-graduation, one potential career path led to a job inventing torture techniques for the US army. Maggie said ‘no thanks’ and resolved to work in tech instead. Maggie studied cultural anthropology and her background has helped her through her tech career to date. Software is built by people and relationships matter.

2. Maggie got a frontend engineering education from illustrating React tutorials. She spent four years as an illustrator at the developer education company Egghead , rising to art director. To illustrate the lessons, it was necessary to understand what she was drawing: React components, useEffect, and JavaScript functions. Note from Gergely: I followed Maggie’s work after her excellent illustrations work on Dan Abramov’s Just JavaScript course . Here’s an animated explainer by her for that course:

3. Some folks believe the best UI interface already exists. In 2021, the AI startup where Maggie worked was trying to launch a new interface to speed up scientific research using LLMs – a year before ChatGPT was released. Months of intense work went into a “new UI for AI”, but it turned out that scientific researchers didn’t want innovations like infinite canvases with cards, composable Notion-like documents, and more. They wanted the same, simple tables they were deeply familiar and comfortable with! Maggie says the experience taught her that starting with a familiar primitive is sensible – even when innovating.

4. The nomenclature matters! Also, problem solving is at the heart of design – just like in engineering. Maggie sees design and software engineering as related by being about problem solving. The difference lies in the materials. Coming up with the names and verbs to describe new things which will then be adopted and used by people can be hard work. Easy when building an online sneakers store, harder when building a new product for AWS.

5. Notebooks are an important part of the designer’s toolkit. Maggie often starts her projects by sketching out ideas. She finds it faster to sketch out an idea by hand than to describe it to a tool like Claude Code. Plus, when you sketch out an idea physically, it will still be there in the notebook the next day. In contrast, if it gets put into a tool instead, it’s a lot harder to go back to it, dozens of prompts later!

6. Maggie regularly builds her personal Figma called “Jigs,” which is also the name of a woodworking device that helps with a specific job. She regularly asks a coding agent to build a prototype that has sliders and color pickers so she can tweak it in realtime, like having a personal Figma!

A jig: interactive prototype where colors, sizes, and animation speed can be tweaked

7. Maggie has stopped looking at the code at work. When a PR is generated, she doesn’t look at the code, and this approach fits when building prototypes. Once she knows what to build, she composes a detailed spec, listing out how the agent will verify its work. Previously, she did keep an eye on the code, but that’s not needed with the new generation of models.

8. Planning with AI agents breaks when there’s too much text. “I have this theory that planning is a really bad experience at the moment,” Maggie says. “An agent grills you with a set of choice A, B, or C questions a hundred times over. By question 20, you’re quite tired and your brain starts shutting down [because] you can’t make this many decisions in this short of time. Also, it told you A is recommended. Then you just start being like, ‘Yep, enter A, I agree with you.’”

9. “Capability gaslighting” is when frontier models convince users they’re an expert but fail the same task the next day. Maggie coined the term “capability gaslighting” for how models impress users before failing badly soon afterward. Too often, we keep believing in models because we’re convinced they’re capable. The same is true for agents, so we should be vigilant when working with LLMs.

10. We need new types of artifacts for humans and agents to work better together, Maggie believes : “There’s this world that agents live in: there’s weights and models and skills and MCPs,” she says. “Then you have your human side: it is physicality and texture and light and materials and all these things agents don’t understand. Trying to find artifacts that allow us to meet in the middle and create stuff together is a really hard challenge because you’ve got two totally different types. I just find myself frustrated that agents cannot look over my shoulder, looking at my notebook and understanding what I’m drawing, and how they cannot help me move my ideas along.”

11. Engineers should try treating the AI agent as a patient tutor when learning about design: As engineers, we can ask AI agents to teach us about design: they’re good at explaining things like when to change up line height, what a good sidebar looks like, or how many characters to squeeze into a line, etc. In the past, acquiring product design skills was hard, but AI agents make it a bit easier

I hope you enjoyed this episode, and many thanks to Maggie for educating all of us engineers!

• What is “loop engineering?”

• Design-first software engineering: Craft, with Balint Orosz

• Are AI agents actually slowing us down?

• Vibe Coding as a software engineer

• How Codex is built

• How Claude Code is built

• From Chrome DevTools to AI Engineering, with Addy Osmani

00:00 Intro

03:24 From anthropology to tech

10:18 What does a designer do?

18:23 How Maggie works

24:55 The case for planning with physical tools

31:53 Why Maggie is learning woodworking

33:13 Design engineers and engineering constraints

38:49 How Maggie uses Figma

40:30 Design at GitHub Next

45:12 How has AI changed design

50:37 When models design and why humans are still needed

53:30 UX and UI

58:29 Capability gaslighting

1:00:33 One Developer, Two Dozen Agents, Zero Alignment

1:07:21 Craft and AI tells

1:14:17 Visual gardens, home-cooked software, and barefoot developers

1:21:02 Advice for engineers and lessons from anthropology

1:25:34 Book recommendation

Where to find Maggie Appleton:

• X: https://x.com/Mappletons

• LinkedIn: https://www.linkedin.com/in/maggieappleton

• Website: https://maggieappleton.com

Mentions during the episode:

• MySpace: https://myspace.com

• Egghead: https://egghead.io

• Elicit: https://elicit.com

• How Kent Beck shapes the software engineering industry: https://newsletter.pragmaticengineer.com/p/how-kent-beck-shapes-the-software

• TDD, AI agents and coding with Kent Beck: https://newsletter.pragmaticengineer.com/p/tdd-ai-agents-and-coding-with-kent

• Design Patterns: https://refactoring.guru/design-patterns

• Sketch: https://www.sketch.com

• Figma: https://www.figma.com

• GitHub Next: https://githubnext.com

• Codex: https://chatgpt.com/codex

• AI Skills with Matt Pocock: https://newsletter.pragmaticengineer.com/p/ai-skills-with-matt-pocock

• Bret Victor’s website: https://worrydream.com

• Stop Drawing Dead Fish:

player.vimeo.com

• The Shape of AI: Jaggedness, Bottlenecks and Salients:

• One Developer, Two Dozen Agents, Zero Alignment: https://maggieappleton.com/zero-alignment

• Buzz: https://buzz.xyz

• Why Ramp built its own in-house coding agent, Inspect: https://newsletter.pragmaticengineer.com/p/why-ramp-built-inspect

• Oh my craft: https://x.com/jorgemanru/article/2091307201117688066

• Pinterest: https://www.pinterest.com

• Robin Sloan’s website: https://www.robinsloan.com

• Barefoot doctor: https://en.wikipedia.org/wiki/Barefoot_doctor

• Addiction by Design: Machine Gambling in Las Vegas: https://www.amazon.com/Addiction-Design-Gambling-Princeton-Classics/dp/0691278288

• Bret Victor - Stop • Drawing Dead Fish:

www.youtube.com

• Buzz from Jack Dorsey and Ace https://github.com/block/buzz

• Home-Cooked Software and Barefoot Developers: https://maggieappleton.com/home-cooked-software

• Addiction by Design: Machine Gambling in Las Vegas - https://www.amazon.com/Addiction-Design-Machine-Gambling-Vegas/dp/0691160880

• Dialkit / Dial Kit by Josh Puckett - https://github.com/joshpuckett/dialkit

• Matt Pocock’s “Grill Me” skill: https://www.aihero.dev/skills-grill-me

• “O My Craft” article by Jorge Monrubia - https://www.linkedin.com/pulse/oh-my-craft-jorge-manrubia-etm8e

—

Production and marketing by Pen Name .

How will AI change operating systems? Part 2: Windows

Gergely Orosz — Tue, 22 Sep 2026 17:17:31 +0000

AI is changing how us software engineers build software, and developers’ tool preferences are rapidly evolving with it – like how AI coding harnesses have gotten very popular. Likewise, future versions of the world’s leading operating systems look certain to feature more support for agentic tools.

To find out how things are changing, we talked with the folks at tech giant Microsoft who shared in detail their plan for Windows, and how AI will play a part. It comes after the company’s previous AI efforts led to it being dubbed “Microslop” by some users online.

The Windows team’s vision is opinionated and includes building new agentic primitives, alongside reversing some decisions that have annoyed engineers. Microsoft wants developers who have shunned the system to return. The big question is: will it succeed?

For more, check out a previous article on how the leading Linux distribution, Ubuntu, is changing, thanks to AI , with a focus on hardware support for GPUs, NPUs and DPUs, a bet on local-first LLMs, and a focus on AI developer tools.

Today, we cover:

How many devs use Windows anyway? It’s hard to get exact numbers, but macOS appears far more popular at startups than Windows, while Linux may also be on track to overtake Microsoft’s OS in developer popularity.
Agent identity & discovery. Windows ships with native support for agent identification, plus a centralized registry of locally available MCP servers to be included with the OS.
Isolate agentic tools. Microsoft is building an OS-agnostic isolation mechanism that should make it easy for developers to build agents that run tools safely.
Running models locally : Windows is betting big on local models. WindowsML is a new hardware abstraction layer for building and running local AI models across GPU, NPU, and CPU. The OS aims to ship Small Language Models (SLMs) without requiring an NPU in the future. We’ve also seen an impressive demo of a local model running on a Surface laptop using NVIDIA chips.
Building agents on Windows: one goal of the OS is to allow building of agents with opinionated frameworks and libraries that devs can just assume are available.
More dev-friendly : Windows was plagued by questionable product decisions that resulted in a cluttered Start menu and search, to the chagrin of many developers who lost interest in the OS. Microsoft wants them back and is addressing criticisms – finally!
Linux on Windows (WSL): Windows ships with an embedded Linux called Windows Subsystem for Linux (WSL). It’s proving quite popular with devs. Windows embracing Linux might just be the strategy to get devs to switch from both native Linux and macOS, as counter-intuitive as this strategy sounds.
Windows & hardware: since its early days, Windows has gone out of its way to support a wide variety of hardware. The OS was x86-based from the beginning, but Windows on ARM is starting to finally look like a good alternative. A look into why ARM support took so long, and the upcoming NVIDIA collaboration.

We’ve talked with people on the Windows team to learn about the thinking behind Microsoft’s strategy for the operating system: Pavan Davuluri (EVP, Windows and Devices), Scott Hanselman (VP, Member of Technical Staff, Microsoft CoreAI and GitHub), and Logan Iyer, (CVP, Windows Platform and Developer). Many thanks for your time!

While Windows remains the most popular desktop operating system for mainstream computer users at around 63% market share as per Statcounter – the strong sense is that its popularity is down among developers.

A year ago, the 2025 Stack Overflow survey polled professional users’ choices of OS. It found that a minority of devs overall use Windows – although it remained the single most popular OS. Despite its top ranking in the survey, the number of devs using Windows was actually way down on Microsoft’s XP-era peak. Meanwhile, MacOS and various Linux distributions also showed considerable market share:

Elsewhere, a JetBrains survey asked devs about their OS usage for developments, also last year. The results:

Interestingly, half of devs use more than one OS for development. This JetBrains survey suggests that several devs jump between operating systems for their work. The survey also found macOS close to overtaking Windows as the most-used standalone OS for development work.

Only last week, myself and Ivan ran two social media surveys on X and on LinkedIn , most likely shown overwhelmingly to people who are also The Pragmatic Engineer readers. We offered a single choice – with no option to select multiple OS usage, Linux on Windows, or via WSL. We were surprised to find Windows in third place behind Linux, based on nearly 10,000 combined responses:

What OS are you using to build software on? Based on 9,937 responses, surveyed in Sep 2026

Based on current research, it’s likely that Windows’ share of the developer market is on the wane, but it’s hard to tell by how much. Inside VC-funded startups and Big Tech, it’s an open secret that Macs have been the most popular developer machine for some years now, partly thanks to the superior hardware performance of M-series CPUs, and also because these companies are not looking to save money when purchasing developer machines. It’s very likely that our own social media survey over-indexes on this group!

Either way, Microsoft has both market share and developer goodwill that it needs to win back. Based on what we’ve heard from them, Microsoft is attempting this.

But where is OS usage at with readers of The Pragmatic Engineer? To figure this out, please cast a vote on the primary operating system you use when developing software. After the vote, you can see the results:

What OS is your primary OS that you develop software on?

MacOS
Windows
Linux (running natively)
Linux (running on Windows, via WSL)
Other

With that, let’s get into how the next version of Windows intends to integrate AI agents at the OS level.

At the operating system level, agents increasingly look like regular users. Their sessions can last hours, use multiple programs, and use OS resources like the UI and clipboard. Windows allows developers to build agentic programs that are distinguishable from users.

Agent identification is enabled through Entra ID , Microsoft’s tool for centralized identity management in Windows. When an agent has a local identity, it acts as if it is another user on the system as visible in the Task Manager. Below, tasks are grouped by user, and there are two users in the task manager: kirupach (a human) and V9-G4 (an agent). All observability of human users is now available for agent users as well.

Task Manager showing human & agent users (kirupach and V9-G4). Source: Microsoft

Agents must be built with this agent identification capability in mind. Consider how a rogue agentic application can impersonate a user and not self-register as an agent – which in a sense is how viruses operate by using legitimate OS functionality for illegitimate purposes. That’s why Defender, Microsoft’s antivirus software, is becoming “agent aware” and scanning Windows for known local agent activity like it scans for viruses.

Microsoft Defender’s AI Assets feature can scan for known agent activity. Source : Microsoft

Agent discovery: agents are only as useful as the tools they’re able to interact with, and Windows On Device Agent Registry ( ODR ) is the centralized place where agents register and discover available MCP tools. ODR manages and runs MCP servers locally, and also comes with connectors for core operating system components like the File Explorer or System Settings.

Let’s say you want to build a specialized Photo Organizer Agent; the organizer agent organizes photos in the user’s Photos folders by theme and starts by running an image classifier to understand photos’ themes, then puts them in the relevant folder, such as ‘parties ’, ‘outdoors ’, ‘baby photos ’, etc.

As such, the Photo Organizer Agent needs the ability to read and change the user’s local files, which starts with Windows ODR finding an MCP connector for the file access capability. After discovering File Explorer, the agent uses File Explorer MCP to access and modify files in the user’s Photo directory.

ODR and built-in MCPs when agents access dependencies

ODR is still in development and available to beta testers, so little is known about its internals, but preliminary research by Origin Technology indicates an interesting implementation detail of ODR. By reverse engineering ODR behavior, researchers proved ODR puts itself as a proxy between the MCP client and its server. Here’s what that would look like in our updated Photo Organizer Agent example:

More than a registry: ODR also acts as a proxy between MCP clients and the server

The Photo Organizer agent discovers the File Explorer MCP capability in the same way as in the dependencies diagram above. However, the process ID that ODR sends back to the agent is ODR itself: the Photo Organizer Agent talks to the File Explorer MCP through ODR!

By putting itself between the MCP client and server, ODR can inspect payloads going back and forth. This is a useful choke point because it enables Windows to detect potentially dangerous behavior. However, this proxying behaviour is as yet unconfirmed by Microsoft, so it remains to be seen how they use it.

Tools are external programs that an agent uses to perform actions on the operating system, such as native OS tools, third-party app calls, or even programs written by an agent. Granting agents rights to execute tools potentially means code execution, so it’s essential the agent tools run in isolation to ensure the user’s files and session aren’t reachable by agentic tools.

Microsoft Execution Containers ( MXC ) is a new, in-development agent containment technology by Windows. Developers can use MXC to spawn agentic tools in isolated environments called sandboxes, and agent access within the sandbox environment is configured through MXC containment policies .

Containment policies are JSON-based config files that describe network, filesystem, UI, and execution restrictions. The image below illustrates how MXC works:

Agents run code with MXC in contained environments, with support for different containment tech & OSes

Let’s say you’re building an agentic application like OpenClaw. This application will necessarily have to use tools to do its job and can use file operations, web operations, shell commands, and more. These are potentially dangerous operations, as OpenClaw executes them through MXC’s spawnSandboxFromConfig() method, which spawns a new contained process in which said tools can run safely. The process containment technology used depends on the containment policy config. MXC doesn’t do the containment itself and uses multiple existing containment technologies for that; for example, on a Mac, it would use seatbelt , a process containment layer built into the Mac OS.

Speed versus safety tradeoff: some containment mechanisms are faster to start and cheaper to run than others. For example, starting a new process is faster than starting a new Windows session or a new virtual machine. At the same time, running a new process in the existing user’s Windows session potentially exposes the user’s filesystem to the new process.

MXC allows developers to adapt the containment level to the sensitivity of the operation being run. Multiple containment levels are available in Windows:

Process containment
Session containment
Running WSL containers
Lightweight Hyper-V containers
Full virtual machines

Choosing the right level of containment is a trade-off between the blast radius and speed of execution. The simplified example below shows how it all comes together:

How developers can use MXC to contain agents

The policy object specifies how the containerized workload should be constrained. It specifies network, UI, filesystem restrictions
The createConfigFromPolicy step configures the whole container. It takes the containment policy object, the chosen isolation level (“process”), and names the container.
The app then manipulates what gets executed in the “WHAT RUNS” step
Finally, spawnSandboxFromConfig() runs the container and processes its output.

OpenClaw for Windows ships as a native app and is one of the first agents to adopt MXC for isolation. Configuring the MXC containment in OpenClaw for Windows is packed as a standard windows settings screen.

OpenClaw Windows app containment config screen. Source: Microsoft

MXC abstracts away process isolation for agents, so the OpenClaw agent in this example isn’t aware of all the restrictions its tools are running within. It can therefore encounter situations where it tries to do something its sandbox disallows.

OpenClaw agent describes lack of access to Windows resources like installed programs. Source: Microsoft

Combined with agent identities, MXC will give enterprises fleet-wide control of their agents. For admins, Microsoft will offer MXC policy management through its Intune product for corporate device fleet management.

MXC enforcement isn’t broadly adopted yet, so we asked Microsoft for examples of how they use MXC internally to isolate autonomous agents. They shared some use cases with us:

How the Windows team uses MXC to isolate autonomous agents

These cases illustrate how Microsoft is automating many internal developer chores, and how MXC provides finegrained control over agent isolation. MXC is under development, but already runs on Windows, Linux, and macOS. Developers therefore get a unified way of containing external tools across operating systems.

Windows ML is a hardware-agnostic layer for running AI models locally on Windows. Its goal is to do for AI what DirectX did for computer graphics: abstract away hardware complexity, regardless of the underlying model’s architecture.

Windows ML is Microsoft’s second attempt at building a hardware-agnostic layer for running ML models. DirectML , the first attempt, is a library for running machine learning tasks on GPUs built on top of DirectX 12 , Microsoft’s graphics library used for running games on Windows, released in 2019.

So, an obvious question is how is Windows ML different? They say that Windows ML offers two improvements over DirectML:

#1: Higher level abstraction & better hardware coverage . DirectML is a low-level library where developers have to manually build the inference pipelines from simple mathematical operations, worry about memory layout, etc. Being based on DirectX, DirectML only worked with GPUs.

In contrast, Windows ML is based on the Open Neural Network Exchange format (ONNX), an open format for representing and running neural networks. Neural networks built with all popular machine learning frameworks can be converted into the ONNX format called ONNX graph and run on ONNX runtime .

ONNX runtime is an open source, cross-platform system for training and running neural nets defined in the ONNX format. Windows ML provides developers a framework that runs on a broader range of silicon, including GPUs, NPUs, and CPUs from all major vendors.

#2: No driver-update bottleneck: Windows ML wraps the ONNX runtime and abstracts away the hardware-specific runtime optimizations by providing hardware-agnostic APIs. Each hardware vendor implements this API for its own hardware through the Execution providers (EPs) concept. With DirectML, new runtime optimizations had to be done via driver updates which took six months to get meaningful adoption.

Windows ML loads EPs when they are needed, depending on the user’s hardware. For example, if a user has an NVIDIA GeForce RTX GPU, the NVIDIA Tensor RTX execution provider is downloaded and used to maximize GPU performance. Hardware vendors build and submit PEs to Microsoft for certification and testing before they can be used with Windows ML. Most execution providers available today are also available on GitHub.

Windows ML ecosystem layers

Small language models (SLMs) are embedded within the operating system. Without them, developers would have to ship models with their apps, or call cloud-based models. Even if running local models should be easier with Windows ML, it’s still not worth the effort if you want to add simple functionalities to an app.

For example, running a sentiment analysis on some text in an application used to involve calling a hosted AI model in the cloud. On top of incurring app running costs, it can also add lag to the app, whereas with embedded models developers can assume the models are already there, and use them as any other local library.

For example,

the Aion-1.0-Instruct model is built into Edge, Microsoft’s web browser. Accessed through the Prompt API , Aion model allows developers to build apps like sentiment analysis with a few lines of code. Here’s an example from Microsoft’s website :

Local language model usage example in JavaScript ( Source )

Even a simple web application can now have LLM-powered features like sentiment analysis without having to continuously pay for tokens. The demo is JavaScript-only, but versions for native apps and other languages are also expected to be available when it’s released.

Teams being able to run their own models is becoming increasingly important, especially in the corporate sector, driven by escalating costs, mode availability questions such as those recently raised with Fable , and compliance issues.

Running models locally also unlocks a hybrid approach, which is well aligned with a trend among tech companies we recently covered , where tech companies have successfully shifted their LLM workloads to cheaper models for simpler requests, and only use frontier models for advanced reasoning. By running models locally, some agentic jobs could be executed with local LLMs, and more complex jobs delegated to a cloud-hosted frontier model.

Running local models on a next-gen Surface laptop with NVIDIA GPUs is impressive. At Microsoft, Scott Hanselman showed us a pre-release Surface laptop with NVIDIA GPUs running Qwen as a local model, hooked up to GitHub Copilot. This local model churned out tokens at a rate of ~40 tokens per second! Once such laptops become widely available, coding-related use cases could become a lot more viable, running locally.

Besides Windows being agent-friendly, Microsoft is also investing in the agent building toolchain to simplify agentic app development by providing a rich agent building framework.

AI Skills with Matt Pocock

Gergely Orosz — Thu, 17 Sep 2026 11:29:00 +0000

Listen and watch now on YouTube , Apple , and Spotify . See the episode transcript at the top of this page, and timestamps for the episode at the bottom.

• turbopuffer – Here’s a crazy idea: what if AI agents could search its entire history, but without RAG or other clever workarounds? turbopuffer’s cheap storage, combined with fast performance makes this surprisingly practical, for any and all agent memory use cases.

• Linear – Agents are only as good as what they know about your product, and for teams like OpenAI and Coinbase, Linear is where that lives. You can use Cursor, Codex, Claude Code, Linear Agent, or the agents you built; then delegate work to them right inside Linear, and everyone can see what they did.

• WorkOS – The fastest AI-native teams have to slow down for the hard problems. WorkOS makes sure auth, for your app and your agents, is never one of them.

Why is the “grill-me” skill so popular, and why does its creator swear by the importance of software fundamentals? Matt Pocock created this widely-used skill – and many others – alongside being an educator, content creator, and engineer. His latest course is AI Hero , and he previously created the Total TypeScript course that generated more than $2.5 million in total sales.

In this episode, Matt and I discuss his unconventional path from working as a voice teacher to becoming a developer and going all-in on technical education. He reveals how communication skills helped him break into tech, why he took an unusual three-days-a-week contract at Vercel, and how he built Total TypeScript through workshops, courses, and lots of free tutorials.

We also explore “strategic coding,” and how he uses skills like “grill me” and “wayfinder” to plan, delegate, and course-correct with AI agents. Matt explains his “day shift” and “night shift” approach, why splitting context up can keep agents in their “smart zone,” and how concepts from classic software engineering books can guide agents to do better. In this episode, there’s also local versus cloud workflows, whether agents need TDD, how AI is changing the ways that engineers learn the fundamentals, and why humans are still essential in teaching.

1. Matt hacked together a web app for students while working as a voice coach. He taught himself JavaScript to build a web audio spectrogram analyzer for singing students, while he was teaching singing, accents, and Shakespeare at drama schools. He also ran his own company at university, studied for a Master’s qualification in voice, and coached consulting firms on public speaking.

2. Matt took a Vercel job in case Total TypeScript did not work out. Matt produced a lot of two-minute TypeScript tip videos which became popular. When he received a fulltime job offer from Vercel, Matt negotiated to work three days per week as a contractor so he could develop his TypeScript course on the other days. When it took off, Matt quit Vercel to focus entirely on his educational projects.

3. Matt does not work on weekends and dislikes the concept of ‘9-9-6.’ He explained that everything he does is an effort to build a lifestyle where he can spend most of his time with his family. That’s the goal.

4. The ‘grill-me’ skill was inspired by from Anthropic’s Thariq Shihipar. The grill-me skill is surprisingly simple and short : it instructs an agent to interview the user relentlessly. Thariq shared his approach of an agent interviewing its user about a topic – and how it was surprisingly useful –, and Matt created a skill around the novel concept.

5. Is “strategic programming” the future of devs’ work? Borrowing from John Ousterhout’s “tactical programming” vs “strategic programming” categorization, Matt believes that agents are more than capable of tactical programming, leaving us engineers to spend more time at the “strategic” level.

6. Using the right “leading words” makes agents produce better results. Matt noticed that agents try to build software layer by layer and that this causes bugs between the layers. Reading The Pragmatic Programmer, he discovered the concept of a “tracer bullet.” When he instructed the agent to use “tracer bullets” to build an app (aka implement a “golden path”), the agent started to produce better code! Since then, Matt has started reading classic software engineering books to discover other “leading words” that guide agents efficiently.

7. Cloud-based agent setups make more sense than running agents locally. Matt is moving his coding sessions to cloud agents because doing so means agents run when he closes his laptop and cloud agents can be made “multiplayer” (collaborative) in ways that local agents cannot.

8. Memento-driven development: optimize your codebase for a colleague who wakes up with no memory every morning. Matt says:

“Imagine you have a human who wakes up every morning and cannot remember who they are, like the guy from Memento. We are trying to optimize our codebases for new starters, so we want the healthiest codebase we’ve ever had. A human can work around a bad codebase and they develop a memory about it, but an agent can’t do that; it starts afresh every single session. So, you need to optimize your codebase for that agent. And it turns out that software fundamentals have been trying to do that [optimize codebases for readability/understandability for someone who sees the code for the first time] for the entire time.”

9. Matt has mixed feelings on Test-Driven Development (TDD) with agents. He believes TDD is helpful for devs as we tend to have shorter working memories, and so a failing test will remind a distracted human of what is still failing. However, agents have longer context windows, so Matt has started to ask agents to produce proof that their code works – with or without TDD. We cover another take on TDD in our episode with Kent Beck.

• What is “loop engineering?”

• The Philosophy of Software Design – with John Ousterhout

• Context engineering with Dex Horthy

• Are AI agents actually slowing us down?

• The AI Engineering Stack

• How Codex is built

• How Claude Code is built

• How Uber uses AI for development: inside look

00:00 Intro

05:48 How Matt got into tech

10:14 How Matt got into open source

12:58 Joining Vercel

18:39 Total TypeScript

23:21 AI’s impact on technical education

30:32 Building reusable skills for AI coding agents

40:46 The “smart zone” vs the “dumb zone”

45:02 The wayfinder skill

47:52 Why agents excel at software engineering

50:54 “Leading words”

1:01:10 Learning the fundamentals

1:09:17 Local vs. cloud agents

1:12:36 Planning vs. course-correcting

1:18:13 TDD and agents

1:23:06 Living in the UK

1:24:21 Teaching: the human part

1:28:36 Advice for junior engineers

1:31:07 Gardeners and great engineers

1:34:01 Book recommendation

Where to find Matt Pocock:

• X: https://x.com/mattpocockuk

• LinkedIn: https://www.linkedin.com/in/mapocock/

• YouTube: https://www.youtube.com/c/mattpocockuk

• Website: https://www.mattpocock.com

• AI Hero: https://www.aihero.dev

Mentions during the episode:

• Microsoft Build: https://build.microsoft.com

• David Khourshid on X: https://x.com/DavidKPiano

• XState: https://stately.ai/docs/xstate

• Mateusz Burzyński on X: https://x.com/AndaristRake

• Stately: https://stately.ai

• Vercel: https://vercel.com

• Lee Robinson on LinkedIn: https://www.linkedin.com/in/leeerob

• Jared Palmer on LinkedIn: https://www.linkedin.com/in/jaredlpalmer

• Cognition: https://cognition.com

• Turbopack: https://vercel.com/blog/turbopack

• Total TypeScript: https://www.totaltypescript.com

• Joel Hooks on X: https://x.com/joelhooks

• Everything is a Ralph loop: https://ghuntley.com/loop

• Ship working code while you sleep with the Ralph Wiggum technique:

• What is “loop engineering?”: https://newsletter.pragmaticengineer.com/p/what-is-loop-engineering

• “Software Fundamentals Matter More Than Ever” — Matt Pocock:

• Context engineering with Dex Horthy: https://newsletter.pragmaticengineer.com/p/context-engineering-with-dex-horthy

• Formal methods with Hillel Wayne: https://newsletter.pragmaticengineer.com/p/formal-methods-with-hillel-wayne

• The Pragmatic Programmer: Your Journey to Mastery: https://www.amazon.com/Pragmatic-Programmer-journey-mastery-Anniversary/dp/0135957052

• A Philosophy of Software Design: https://www.amazon.com/Philosophy-Software-Design-2nd/dp/173210221X

• The Philosophy of Software Design – with John Ousterhout: https://newsletter.pragmaticengineer.com/p/the-philosophy-of-software-design

• Domain-Driven Design: Tackling Complexity in the Heart of Software: https://www.amazon.com/Domain-Driven-Design-Tackling-Complexity-Software/dp/0321125215

• TDD, AI agents and coding with Kent Beck: https://newsletter.pragmaticengineer.com/p/tdd-ai-agents-and-coding-with-kent

• Uncle Bob Martin on X: https://x.com/unclebobmartin

• Matt’s post on X, “I’m moving away from my local dev setup”:

[

Matt Pocock @mattpocockuk

I'm moving away from my local dev setup Makes zero sense to me now

4:02 PM · Aug 22, 2026 · 951K Views

467 Replies · 56 Reposts · 3.54K Likes](https://x.com/mattpocockuk/status/2091194428639621284)

• The third golden age of software engineering – thanks to AI, with Grady Booch: https://newsletter.pragmaticengineer.com/p/the-third-golden-age-of-software

• Software architecture with Grady Booch: https://newsletter.pragmaticengineer.com/p/software-architecture-with-grady-booch

• Jared Friedman’s post on X, “tech debt..”:

[

Jared Friedman @snowmaker

Technical debt used to be something you just had to live with in a sufficiently large codebase. No longer.

2:18 AM · Aug 21, 2026 · 501K Views

205 Replies · 77 Reposts · 1.44K Likes](https://x.com/snowmaker/status/2090624581275152835)

• Lauren’s post on X, “every team needs a gardener...”:

[

lauren @poteto

every team needs a gardener. someone quietly watching the stream of PRs flowing into your codebase, noticing the smells: the third isRecord this week, the lint suppressions creeping like ivy across your carefully planned garden. a steady hand tending the weeds that would engulf…

lauren @poteto

organic architecture good codebases have always had strong foundations and constraints. these files go here. this kind of code goes there. the codebase is held together either by catering to the lowest common denominator developer using conventional frameworks and languages, or

9:08 PM · Aug 20, 2026 · 193K Views

38 Replies · 44 Reposts · 765 Likes](https://x.com/poteto/status/2090546476464451907)

• Lars Grammel on X: https://x.com/lgrammel

—

Production and marketing by Pen Name .

Inside OpenAI’s agentic software factory

Gergely Orosz — Tue, 15 Sep 2026 15:41:25 +0000

It’s rare to work with an unlimited token budget, but at OpenAI, that’s what all engineers, researchers, finance colleagues, and marketing folks do. Recently, I visited one of the world’s leading frontier labs to find out how OpenAI operates today – and for a glimpse at where software engineering might be headed as a profession.

Plenty has changed since I visited OpenAI’s headquarters last year. Within a year, Codex has gone from a “nice-to-have” tool to being the backbone of pretty much everything at the company.

To learn more, I talked with seven engineering leaders and engineers there: Venkat Venkataramani (VP of Engineering, Applied Infra), Sulman Choudhry (Head of Engineering, ChatGPT), Andrew Ambrosino (Lead, Desktop), Joe Gershenson (Lead, Core Agent team), Akshay Nathan (Engineering Lead, Productivity), Ahmed Ibrahim (Engineer, Codex) and Steve Coffey (Engineer, Responses API). Thanks to all for taking part!

Today, we cover:

Codex takes over at OpenAI. In a matter of months, nearly all OpenAI’s non-engineers moved over to Codex and ChatGPT Work without a mandate from above for it.
Death of the IDE & pull requests. IDE usage has been down since January when Codex usage started to surge. PRs and code reviews need to be rethought.
OpenAI’s agentic software factory. OpenAI has built a “software factory” with several automated, agentic feedback loops: for example, Perf Factory monitors production and kicks off Codex agents to automatically fix performance issues.
How engineering tooling & practices are changing. Hand-built internal tools are slowly being replaced by Codex, which is increasingly preferred for debugging over specialized tools. Harness efficiency is critical in software factories.
Engineering for a billion users: how OpenAI scales up its infra. They buy first and take it in-house later. Also, geographic infra distribution, capacity planning tactics and challenges.
Making OpenAI’s API more reliable and performant. CPUs are becoming a bottleneck, doing slower deployments on purpose, and solving load challenges.
How the software engineering job is changing. Engineering specializations are disappearing, judgment and agency are more important, and it only takes one or two engineers for previously “impossible” rewrites and migrations to succeed.

Before we start, a scheduling update: I’m in New York for the week, attending the LDX3 conference and visiting a few startups and tech companies in the city, so there will be no edition of The Pulse on Thursday. Normal service resumes next week!

The bottom of this article could be cut off in some email clients. Read the full article uninterrupted, online.

Read the full article online

The takeaway from my visit to the company’s headquarters which really sticks out is that Codex – and lately Codex and ChatGPT Work – have taken over everything there, starting in around January. Desktop lead, Andrew Ambrosino, told me:

“The big theme of the past months has been that everything is now a coding agent. Whether the visible code is your output or not, agents write your artifacts.

Think of it like this: your entire life is via software. You have these powerful tools (agents) in your computer, and the ability to loop and reason and write code is the ability to do everything.”

The token usage chart below shows this sudden adoption surge:

Codex usage since August 2025 at OpenAI by department. Source: OpenAI

In a four-month period, non-engineering orgs like finance, recruitment, and legal went from ~0% usage of Codex to 90% usage. Now, almost all OpenAI employees use Codex and ChatGPT Work weekly. So, what happened?

OpenAI released the Codex app for Mac in February and for Windows in March, and ChatGPT Work (powered by the Codex harness) in July. Following that, non-engineers there moved all their workflows over to Codex and then to Work.Caveat: OpenAI’s internal version of Codex is a lot more advanced than its external counterpart because it’s plugged into pretty much every OpenAI system – similar to how Ramp’s Inspect AI agent has been wired up.

The fascinating part of this is that OpenAI got close to 40% adoption across non-engineering teams at a time when the Codex app was hostile to non-engineering users (hard to use). Between February and April, the Codex app still showed the code on-screen, but even so, non-technical colleagues outside of engineering still used it because it could do complex work like researching and creating a presentation, document, spreadsheet, or tasks that produce rich output. Today, those folks are very heavy users of it.

Being able to work for longer on more complex things drove adoption. OpenAI added the /goal setting to Codex, where you can set up a goal for the agent and it keeps working until it is complete. Between April and May, usage surged from 60% to 90%. Andrew believes improvement in the harness’s handling of long-running tasks was one cause of this:

“The number one thing that is changing is that people are starting to use threads for much longer, and this longer usage has been a breakthrough. It’s surprising to see the sheer length of time that people spend on a thread – even days! They often set a goal and then have the model crank.

Codex being good at longer-running tasks seems to cause people to do fewer things in parallel. This is because a long-running agent often spins off other agents to do other things, reducing the surface area that you, as a human, have to manage.”

“Awareness overhang” is another cause of the rapid adoption, the Codex team believes. As Akshay Nathan, Engineering Lead, Productivity team, told me:

“For a long time, we had a ‘capability overhang’: the models were capable but the products didn’t fully bring that out. Now, we’re seeing an awareness gap. Some people have figured out they can use Codex to monitor Slack, update Airtable, or create onboarding materials. But many others still use it for one task and then discover more uses from teammates via word-of-mouth.

But there’s still so much more, under the surface, that you can do with Codex.”

Role-specific and team-specific plugins are created and distributed. Another thing that sped up adoption is that each group started to distribute useful role-specific workflows as plugins. Andrew explained why it’s important to not just offer a generic coding agent:

“If you build a product that can do anything, teams need a way to make it their own. You can’t just give everyone an empty box. Skills and plugins let teams adapt the agent to their work. Sometimes, we also need a new app capability, like a browser that the agent can use alongside those skills. But the same building blocks already cover a lot of different roles.”

Subject matter experts are embedded in ChatGPT Work engineering teams. The models have become “smarter” than developers in some domains, so devs cannot channel “taste” into the harness in those areas. So, people who are domain experts are onboarded onto engineering teams. This is one outcome of ChatGPT Work being used by so many non-engineering domains: experts embedded with engineering advise developers on things like what a good slide deck, spreadsheet, or business report looks like.

Of course, domain experts being in engineering teams is a decades-old best practice for building quality products. It seems like this gets rediscovered in different contexts every few years!

OpenAI is fully dependent on Codex and Work. This is so much the case that in the event of even a minor outage, internal messages from colleagues alert the Codex and Work teams at the same time as – or before – automated alerts.

Basically, work happens through Codex and Work, and pretty much nothing else. From the outside, this dependence on a single shared harness is particularly eye-catching; two years ago, there were no AI agents, only advanced AI autocomplete!

Late last year, the Codex team was torn about whether to release the Codex desktop app. Andrew recalls the hesitation:

“In December 2025, we weren’t entirely sure if we would release the Codex app. We had the Codex CLI as a terminal, and there are large, feature-rich IDEs out there. So, would there be space for a dev tool that is between a terminal and an IDE? In my head, there was this future where it would not work out, and be the kind of ‘misfit’ like the iPad was.

A lot of people buy iPads and then never use them: they either use their smaller, more portable smartphone (which could be the equivalent of the CLI in this metaphor), or their feature-rich laptop (the equivalent of the IDE).

Also, don’t forget that in November, Antigravity came out as a VS Code fork. This added to the feeling that perhaps we should have also forked VS Code for the Codex app. But still, we dismissed the temptation and went with our gut feeling that as AI agents get better, IDEs will matter less.”

Indeed, since January, IDE usage has gone down and OpenAI’s bet looks like a good one. However, the Codex app is becoming a little more akin to an IDE: for example, the ability to edit files inside the app was shipped in June.

CI/CD systems are seeing massive load increases. One sign of productivity gains from Codex is the amount of additional code flowing through OpenAI’s dev infra systems. More code being created and pushed leads to new scaling challenges which the team is currently heads-down on solving. Venkat Venkataramani, VP of Engineering, Applied Infra, said:

“The number of pull requests (PRs) per engineer is growing like a hockey stick (at a very high, accelerating rate). Every part of the build-test-deploy pipeline is seeing dramatically more load.

We’re talking about roughly a 10x increase in load on some systems. At most companies, that kind of growth might happen over two or three years. At OpenAI, we see it in about six months.

That level of acceleration exposes bottlenecks everywhere: version control has to handle far more code being written and pushed, CI/CD systems have to scale with it, and production release processes have to absorb a much higher rate of change.

Every month, we wake up to a new set of infrastructure scaling challenges to solve. Just when we think we’ve created enough capacity for the next phase of growth, the model unlocks another wave of capabilities, which creates a new set of bottlenecks somewhere else in the system.”

In this context, PRs and code reviews are being rethought. They have “core primitives” in software engineering, but this level of development acceleration is an opportunity to reimagine them. Again, from Venkat:

“The question we ought to ask ourselves in the middle of all this development acceleration is how do we reimagine many things we took for granted. For example, how do we reimagine the CI (continuous integration) and CD (continuous deployment) process? What does observability mean in this world, and how should people interact with pull requests?

If you ask me, the way we do code review today makes less and less sense, and the same is true for pull requests.

We’re now seeing agentic code reviews that look at code changes through a series of different lenses. In the past, it would have been impractical for a cloud infrastructure engineer and a security engineer to review every single code change. With agents, that becomes possible.

We can rethink how code is deployed with agents, too. We are building an agent that “handholds” a change all the way to production — whether it’s a code change or a change behind a feature flag. It observes the relevant monitoring graphs, but can also build its own dashboard to monitor important signals. More of our code changes are going to production with this kind of agent monitoring.”

An increasingly painful bottleneck is in deploying native mobile apps. When there are ten times more pull requests, it’s challenging to deploy on the backend or the web and more infrastructure is needed to do so. Then, after you rework a CI/CD system and make sure there’s enough capacity to run them, you’ll be deploying that many more PRs to production.

This arises in the shipping of updates to native iOS and Android apps because every app update needs to go through Apple’s and Google’s manual approval processes which take hours or days to complete.

Talking with Sulman Choudhry, Head of Engineering, ChatGPT, he explained how the app review bottleneck is affecting iteration speed. Sulman used to work at Facebook and remembers how the social media company sped up shipping mobile releases:

“Back in the 2010s, Facebook had a pretty important breakthrough in how to ship native mobile code faster. App Store releases went from monthly to bi-weekly to weekly. At the same time, experimentation and feature flags let teams ship code before it was ready to launch, then turn features on remotely when they were.

That model brought a lot more velocity to mobile.

In the age of Codex, I think we’re hitting the next version of this problem. Code generation is getting dramatically faster, but getting that code into users’ hands on native mobile is not. For Codex in particular, where usage is heavily mobile-first, that gap is already becoming painful for us and users.

I expect the pressure here to increase quickly. If software can be written in minutes, waiting days or weeks to get it onto a phone starts to look increasingly absurd.

We should be aiming for a world where shipping code on native mobile is as fast as shipping on the web. Getting there will probably require some creative rethinking of what we ship, when we ship it, and what can be activated remotely. Today, we’re nowhere close.”

There’s some irony in how shipping a native iOS or Android app has the exact same challenges today as in 2008, when the App Store was launched. In 18 years, not much has changed! Apple still does not officially allow apps to bypass the App Store review process to ship meaningful experience changes.

The idea of a “software factory” is similar to a physical factory where robots and humans produce autos together. In the software context, it is AI agents and humans producing software. Some manufacturing sites are fully automated “dark factories” where illumination isn’t needed because there are no humans. Could the same fully automated process emerge in software engineering? At OpenAI today, there’s a “software factory” running and it’s all built around Codex.

Here’s how the “traditional” software development pipeline used to look, compared to what OpenAI’s agentic infra pipeline looks like today, as described by VP of Engineering, Applied Infra, Venkat Venkataramani:

OpenAI’s “agentic software factory”

The pipeline:

1. A human builder defines the desired outcome. A software engineer or product manager specifies the problem and desired outcome. Judgment, prioritization, and taste are becoming more important for this phase. Interestingly, Venkat told me that engineers at OpenAI are becoming more like product managers than traditional systems engineers.

2. Codex gathers context. OpenAI has moved all its documentation inside of the source code, which makes it easier for agents to understand more of the code. Codex also has access to:

Git repositories and GitHub
Slack and Notion
Internal data sources: Databricks, Datadog, internal logs, etc
Internal Codex skills – some of which are maintained by OpenAI’s Codex implementation itself!

Codex is so “plugged” into OpenAI that new engineers are directed to ask Codex any questions they have during onboarding because it has a surprising amount of context.

3. Codex implements code changes. This part is trivial enough: Codex gets to work and makes a series of code changes until it reaches its goal, and then verifies that the software works as it should.

4. Build & test, then CI. The agent builds the code, runs the tests, fixes the code when it breaks the test, and then creates a pull request. This pull request triggers the continuous integration (CI) server to run and execute a more thorough suite of linters and tests. The agent babysits the PR until it’s “green”, fixing any CI failures and automatically updating the PR.

New: a “perf harness”: the agent also uses a perf harness to send problematic PRs to the Synthetics A/B framework for evaluating performance implications. As mentioned above, the load upon CI systems has increased greatly in the past six months.

5. Agentic code review. Instead of using one generic AI code reviewer, OpenAI spins off multiple agents, each with a “domain specialist” configuration. Venkat told me they see this as equivalent to having a human domain expert from each relevant infrastructure team review every change.

Note from Gergely: I was skeptical about the claim that an agent that’s told to be a cloud infra specialist would produce a different review from a generic agent. However, all Codex agents have full access to OpenAI’s code and docs, so this “cloud infra expert” agent likely has gathered a lot of context about cloud infra setup and best practices, meaning it should provide highly targeted feedback. The important thing is how these “domain specialist” agents are set up, the context they have access to, and how they focus only on their own domain to make best use of their limited context window.

Code changes are classified by risk. High-risk changes can be sent through stricter processes; for example, they might invoke more AI code reviews, or mandate that a human reviews it after the AI agents finish. Low-risk changes follow an easier path; areas of the codebase can opt in to an agent that will auto-approve low risk PRs, removing human acceptance as a bottleneck and improving velocity.

A neat thing about risk assessment is that OpenAI can automate when additional compliance input is needed: either automated (via another agent) or human review. With the quantity of PRs being produced, it simply wouldn’t be possible for humans to review all code without assistance.

Just like with CI, the coding agent babysits the comments and updates the PR to fix issues surfaced.

6. Agentic deploy. After a human approves a change to go to production, it is assigned its own agent with an instruction that could be summarized as:

“Handhold this change until it is safely and fully rolled out into production.”

Agents handhold both the code changes and the changes behind feature flags. For example, in the case of a change behind the feature flag, the agent will:

Read the codebase and figure out where the feature flag lives
Understand what the change does
Decide which signals indicate success and failure
Builds its own monitoring dashboard to use – this is a pretty impressive improvement and something that’s new to me
Watches relevant production signals and its dashboard(s)

OpenAI’s long term goal is to have something like a “per-change autonomous SRE” (site reliability engineer) in the form of an agent that can deploy pretty much autonomously.

7. Observe production. Tools which track the production system:

Dashboards generated by the agent during previous steps
OpenAI’s internal observability stack, including a bunch of custom tools that generate logs, metrics, trace & wide event data

One big change at OpenAI since my previous visit, pre-Codex, is that back then, engineers created dashboards to monitor services whereas now, agents do this at the granularity of per-change deployment.

8. Production monitoring feeds back into development. OpenAI’s “Perf Factory” uses agents to sift through alerts and dashboards, de-duplicate signals, identify real latency regressions, root-cause them and propose fixes. This helps catch performance issues introduced by ongoing code changes, extending the workflow beyond deployment into continuous improvement.

9. Respond to outages. Sevbot is OpenAI’s internal incident response agent; unsurprisingly, it’s also built on top of Codex. When an incident is detected, the bot “wakes up.” Here’s what it does:

Collects context about the incident
Determines possible mitigations (but never executes any)
Answers devs’ questions (it’s part of the Slack channel)
An engineer can tell it to apply a specific mitigation

OpenAI’s goal is to get to the point where Sevbot can take autonomous action when mitigating some outages. The dream is that no humans be woken up outside of their working hours during an outage because Sevbot can handle “routine” outages autonomously, with humans reviewing its actions when they return to work. But as of now, oncall duty is not a thing of the past at the company.

Unsurprisingly, Codex is changing how easy it is to build internal tools and having an impact on standard engineering practices like debugging. Here’s what I gathered from talking with folks at OpenAI.

The Pulse #191: a new trend of CPU shortages

Gergely Orosz — Thu, 10 Sep 2026 17:13:44 +0000

For compute-intensive services: it’s worth reserving more compute now. Also: growth dreams end for more COVID-era unicorns, engineers losing touch with systems when AI handles incidents, and more

Sep 10, 2026

∙ Paid

The Pulse is a series covering events, insights, and trends within Big Tech and startups.

Today, we cover:

New trend of CPU shortages: after a GPU shortage and memory shortage driven by AI companies, we’re now experienceding a CPU shortage, thanks to AI agents using a lot more CPU with tool usage. If you will need more compute in the future: secure it now…

Building Codex with Tibo Sottiaux

Gergely Orosz — Wed, 9 Sep 2026 15:57:24 +0000

Listen and watch now on YouTube , Apple , and Spotify . See the episode transcript at the top of this page, and timestamps for the episode at the bottom.

• turbopuffer – a vector and full-text search engine built on object storage. It’s fast, cheap, and extremely scalable. The teams building the smartest AI products out there — Anthropic, Cognition, Notion, Harvey — all run on turbopuffer. Check it out

• Antithesis – use AI agents to work on critical systems without worrying about correctness. Antithesis runs your complete system in a hostile environment, analyzes its behavior, and reproduces every issue perfectly. Learn more

Tibo Sottiaux is one of the engineers who created Codex, and today, he heads up the Core Products & Platform org at OpenAI which also includes Codex. He’s also one of the most public faces of Codex due to his frequent – and generous – usage reset announcements, like this one yesterday .

In this episode of the Pragmatic Engineer Podcast, Tibo and I discuss how Codex was built and continues to be iterated upon. We explore why the Codex CLI is written in Rust and was released as open source, how the harness and models have evolved, and why Codex supports models from multiple providers.

Tibo also shares details about how the OpenAI team uses Codex throughout the software development lifecycle, including code reviews, maintenance, and system rearchitecture. We look into how AI is lowering the cost of changing code – and some interesting side effects of this – the merger of ChatGPT and Codex, and also how Tibo uses the tools in his own work.

1. A shock cancellation proved to be an important lesson. While at Google in London, Tibo spent two enjoyable years working on a project in the Ads organization – right up until a VP flew in from California and abruptly cancelled the whole thing. Tibo was shocked: there were hundreds of users and he was having a great time solving engineering challenges. Reflecting later, he realized he’d been blind to the fact that the product had no product-market fit and the feedback loop from users was poor. Tibo also learned that just because a product manager says a project is going well, doesn’t mean it is! That experience means he now always questions the impact of his work, and the importance of projects to which he contributes. Looking at Tibo’s career since, it seems like a well-timed lesson!

2. Google had a “ChatGPT-like” project a year before OpenAI. At the start of the decade, DeepMind was largely focused on games and reinforcement learning, but some members, including Tibo, reckoned that scaling language models might lead to artificial general intelligence (AGI). Their “chat with an LLM” project spread like wildfire internally, but for some reason the product was never launched. Who knows what would have happened if Google had beaten OpenAI to the public launch of AI chatbots in 2022.

3. Tibo was drawn to OpenAI because only 20 people worked on ChatGPT. Tibo had a great run at Google, but missed working somewhere where Research and Product collaborated more closely than at the search giant. When he learned in 2023 that ChatGPT was built and maintained by around 20 engineers – despite its massive popularity – he was very surprised and wanted to join.

4. Codex is built using Rust, even though the AI models were much better at writing Python and TypeScript at the time. The Codex team had a vision of Codex instances running on millions of cloud machines, which meant performance, security, and engineering for efficiency and scale were the first design principles. This led to Rust, despite AI models then not being the strongest at writing Rust. The team’s decision to choose a performant language upfront and avoid a rewrite later reminds me of Casey Muratori’s point about the need to architect for performance, upfront.

5. Codex is open source, which has upsides and downsides. Codex’s biggest competitor, Claude Code, is closed source, so I find it inspiring that the Codex team chose the open source path. Tibo says the upsides are trust and the community of contributors who are an energizing influence. A less discussed downside of open source is that the Codex team’s work sometimes gets copied and released in other tools before Codex. Tibo told me this stings, but it’s the price of working in the open.

6. Open source is a big reason why Codex supports working with other AI models. Claude Code can only be used with Anthropic’s models, whereas Codex is usable with any model, not just OpenAI’s. Being open source means that even if Codex were locked down to a model, anyone could still fork the harness and change a few lines of code to support a different model. Tibo believes in winning by letting users use the best models; the Codex team themselves also try other models in the same harness. Personally, I appreciate this approach of encouraging competition from a frontier lab!

7. Cloud development environments (CDEs) never took off outside of Big Tech because of setup costs – but AI agents could change this. Tibo predicts a resurgence of fully cloud-orchestrated machines, where agents like Codex can configure and stay in sync with your local machine setup.

8. The Codex harness is always slightly ahead of OpenAI’s latest model. The harness provides the model with crutches: guardrails, safety, efficiency, steerability, and the developer message injected into context at the start of each turn. As models improve, some “crutches” are discarded and the harness shrinks. This has been the development cycle between Codex and OpenAI’s new models to date.

9. Codex “knows” a staggering amount because it’s plugged into pretty much every OpenAI system. I asked Tibo what pointers he’d give a new joiner on the Codex team. His answer: “have you asked Codex?” At OpenAI, it’s plugged into Slack, every document, and all code, by default. New starters are surprised about being able to ask it anything, including who’s working on something, or why a decision was made. The team purposely work in public channels and open documents with broad permissions.

10. Correctness checks and security reviews will be automated with AI. With code review changing under AI’s influence, Tibo believes that discussion about the intent of code doesn’t have to happen inside a code review, which is mostly about correctness, information exchange, and providing a forcing function for conversations that should’ve happened sooner. With AI code review, conversations about what the system should do still matter, and are probably best had before the code is written. This echoes the theme of this week’s Tuesday article about what is happening with code reviews .

11. Maintaining and re-architecting code is becoming very cheap with agents, especially time-wise. Maintenance tasks like dependency upgrades can be handled by a model blasting through the codebase within a couple of hours. Meanwhile, re-architecting for new tradeoffs that used to take years can now take days, at most. Tibo adds a caveat: quality code, good abstractions, and good test suites greatly affect how easy – or not – a codebase change is to make.

12. Being “in the zone” is history; Tibo sees code as a tool for solving problems. During the podcast recording, we bonded over remembering the “good old days” of pulling late nighters and staying “in the zone” to solve difficult problems with code. These days, Tibo has adapted, like most people at OpenAI. He still opens an editor and writes a little code because it feels nice, but says that an upside of AI agents is being able to gather more data faster – meaning there’s less need for the lengthy coding sessions of yore. Instead of making gut calls, he can fire off an agent and get the data within a minute to make much better decisions with.

• How Codex is built

• How Claude Code is built

• How Cursor was built

• What is “loop engineering?”

• How Uber uses AI for development: inside look

• Why Ramp built its own in-house coding agent, Inspect

• “I ship code I don’t read”: with Peter Steinberger, the creator of OpenClaw

00:00 Intro

07:21 Working at Google

12:41 What drew Tibo to OpenAI

15:19 The early days of Codex

18:20 Why Codex was built in Rust

21:15 Why Codex is open source

25:50 Codex plays nice with other models: why?

32:09 How the harness works

36:44 Harness and model improvements

41:19 The SDLC behind Codex

46:39 Code reviews at Codex

52:09 Maintenance and architecture

56:43 How AI tools expand what engineers can do

1:02:30 The Merge: ChatGPT + Codex

1:07:16 How Tibo uses Codex and ChatGPT

1:10:44 Advice for engineers who want to work in AI

Where to find Tibo Sottiaux:

• X: https://x.com/thsottiaux

• LinkedIn: https://www.linkedin.com/in/thibault-sottiaux-27195366

Mentions during the episode:

• How Codex is built: https://newsletter.pragmaticengineer.com/p/how-codex-is-built

• Slow down to speed up: so much has changed in 6 months’ time: https://newsletter.pragmaticengineer.com/p/slow-down-to-speed-up

• N-Side: https://www.n-side.com

• Google DeepMind: https://deepmind.google

• AlphaFold: https://deepmind.google/science/alphafold

• Tibo’s reply on X about Google’s canceled bot, LMChat:

[

Tibo @thsottiaux

@_chenglou I was part of that team. Basically ChatGPT one year before it came out. Called LMChat and then another codename. Google was too nervous to release it and DeepMind was blocked from shipping products that could disrupt Google. I think about this a lot.

4:53 PM · Aug 1, 2026 · 1.97M Views

250 Replies · 566 Reposts · 12.6K Likes](https://x.com/thsottiaux/status/2083596911060324570)

• Greg Brockman on X: https://x.com/gdb

• Sam Altman on X: https://x.com/sama

• Python: https://www.python.org

• Rust: https://rust-lang.org

• The creator of Clawd: “I ship code I don’t read”: https://newsletter.pragmaticengineer.com/p/the-creator-of-clawd-i-ship-code

• Using Goals in Codex: https://developers.openai.com/cookbook/examples/codex/using_goals_in_codex

• ChatGPT is now a partner for your most ambitious work: https://openai.com/index/chatgpt-for-your-most-ambitious-work

• ChatGPT Work: https://chatgpt.com/work

—

Production and marketing by Pen Name .

What is happening with code reviews?

Gergely Orosz — Tue, 8 Sep 2026 16:32:01 +0000

One question haunting the minds of CTOs and heads of engineering whom I’ve been talking with, is how to deal with large quantities of code review which have only been growing now that AI agents generate most code at many tech companies.

Since the end of 2025, it has seemed that the era of devs writing code by hand is over at startups and in Big Tech. AI agents work faster and generate more pull requests (PRs) than devs ever did, and the size of those pull requests is also increasing.

Today’s article summarizes some approaches to code review at various workplaces in this new paradigm, covering:

Humans review the AI code reviews. The most popular approach: AI code review tools go through code changes, and devs review the review itself.
Triage by “blast radius” & decide an approach. Low-risk changes don’t need human review, and high-risk ones do. Adopted by OpenAI, Anthropic, and others.
Review the plan/tests/database schema, but not implementation. Focus on reviewing the “before” and “after” states of an implementation, rather than the implementation itself.
Produce less code. Set up AI agents to produce smaller PRs that are easier to review and reason about.
Review everything by hand. Not everyone has adopted AI code review tools – even those that have sometimes still expect devs to read through all the new code, before allowing it to go to prod.
No human code review? There’s more talk about dropping human code reviews than there is evidence of this actually happening, so far. The most I could find was AI startups doing it and building additional layers for safer production rollouts.
Why do we review code, anyway? Before figuring out whether or not code review should stay, it’s worth going back to the fundamental technical, team, and organizational reasons for code reviews.

Unsurprisingly, it’s clear there’s no one-size-fits-all solution to the question of how to handle a deluge of AI-generated code review. Please leave a comment below about how your team or company deals with this new, pressing issue!

A snapshot of what’s going on in code review at this stage of AI development is provided by the graphic from GitHub, below. The background context it provides is pretty stark. It shows the stats for the number of PRs and commits over the course of three years on the popular platform:

Change in number of PRs, commits, and new repos across three years. Source: GitHub

Over that time, the number of PRs opened has increased fivefold, which is a lot! And growth sped up from the end of 2025, when PRs and commits nearly doubled just in that period alone! So, how are teams dealing with this avalanche of extra work? To find out more, I asked around.

The most common approach is to add an AI code review step to every pull request in a variety of ways:

Use one or more vendors to review PRs. There are dozens of vendors offering this functionality – ones like CodeRabbit, Gitar, Greptile, GitHub Copilot Code Review, Qodo, Claude Code Review, Ellipsis and more. Many teams choose one or more, and the bots then review PRs, leaving comments for devs. For example, the Bun project by Anthropic has CodeRabbit, GitHub Code Review, and Claude Code Review all generating comments on PRs.
Multi-agent code review. Build a custom solution which triggers several models/agents to review the code and suggest fixes.
Agents update PRs with fixes. Vendors and home-grown solutions can instruct agents to update PRs with fixes and then re-trigger reviews – if you trust agents to make sensible fixes, that is!

Typical processes:

AI code reviews increasingly part of the development cycle

In the above cases, engineers typically review the review itself, and not usually the code. Here’s Etienne Dilocker, cofounder and CTO at AI database software, Weaviate, explaining why he likes their approach:

“It’s very hard for agents to get the balance [of the code review] right. If you ignore human code review entirely and leave it to agents, every PR will either suffer from scope creep or ship critical issues. But, of course, you can’t review everything by hand. So my current favorite setup is:

1. an (adversarial) agent does a review

2. a human makes a scope decision

3. an agent implements the feedback

4. either repeat or break the loop (likely a human decision)

So basically, 90% is left to agents, with humans in the loop for critical scope decisions and exit criteria.”

Noise is a big problem with AI code reviews. WeTravel, a Series C travel tech company, decided to not use AI for code reviews because of the amount of noise it generated. In June, they did an updated evaluation which showed lots of improvement, but still not enough to justify adopting AI for the task.

As things stand, custom tooling is probably needed to reduce code-review noise. Uber built a clever approach for this; an agentic pipeline called uReview:

What uReview does:

Bots generate lots of code review comments
Comments are graded, and low-confidence comments removed
Comments are merged, categorized, and unimportant ones removed
… in the end, the AI review results in important comments being shown to devs

Another common approach is to decide whether to review code by hand or with AI, based on how “risky” a change is:

Low-risk change: only AI, without human review. It can ship to production once AI agents are happy
High-risk change: mandatory human review

This is the approach that Anthropic and OpenAI follow, which I confirmed by talking with both companies. At Anthropic, Jarred Sumner told me that a human merges even low-risk changes, but that their goal is eventually to get another Claude instance to merge low-risk changes.

And it’s not just at leading AI labs: fifteen-person startup, Duckbill, a cloud and AI compute and contracts management company, changed their process, as explained by cofounder and CEO Mike Julian:

“We ditched code review at Duckbill Group (mostly)

About a month ago, we found ourselves with 60 open PRs for a team of five. They had been accumulating for a few weeks and we all had the sudden realization we were looking at two days of just code review.

I had been tossing around the idea for a while about having AI do all code review and so I just asked the team: what if we just didn’t review the PRs?

We decided to do a couple of things:

Switch to a risk-based system. With a risk-based system, we agreed that if your change touched the public API/MCP, auth, design system, non-additive database schema changes, or agent skills, it needed a human review. We then enforced that with a shell script to add a GitHub label.

Improve our guardrails (unit and end-to-end testing, post-deploy observability, stricter linting and type checking, etc). Improving guardrails was pretty easy, just expensive in tokens and attention. We enabled nearly every rule in ruff/prettier/eslint/ty, and we improved our unit test coverage to a floor of 85%.

Results before vs after:

PRs merged: 353 → 684 (80/wk → 154/wk, +94%)

Merged within 1h: 28% → 45%; within 24h: 76% → 80%

Human-reviewed PRs median merge time: 26h

No human-review median merge time: 1h.”

Here’s how I’d visualize this approach:

Selecting a code review approach by “blast radius”

Some companies have built additional tooling to make it easier for devs to know which reviews to focus on. For example, Uber’s custom-built Code Review Inbox highlights high-impact changes, so devs know to spend more time and effort on them:

Some devs and teams have stopped reviewing the code (the implementation), and instead review the “before” and “after” states:

Review the plan: spend a lot more time on the plan than before, to get a much more detailed spec. Using The /grill-me skill by Matt Pocock is a popular method, and I’m also a fan of it for thorough upfront planning, as is Andrea Francesco Speziale, Principal Engineer at Musixmatch:

“After 3 hours of /grill-me, it better one-shot the implementation. I’m not spending a single minute on any review!”

Review the tests: via Test Driven Development (TDD) – which is much easier with agents when writing the tests upfront is a chore – or by focusing the review to ensure the software is tested.

One argument for this approach is that customers and users of software usually don’t care about the code. There’s a caveat that automated tests can verify a lot of different software – and are great at verifying business logic – but they don’t do a good job at verifying whether a UI looks and feels good.

Review the database schema. Jackie Luo, cofounder and CEO of AI startup Sigil, and formerly an engineer at Square, says:

“My current take is that all that really matters is the database schema. Speaking from a fast-moving startup perspective:

1. Everything, besides data, is fluid and recoverable.

2. The schema is the “hard” representation of what’s been built and reveals the riskiest changes, so it’s a good attention/impact tradeoff.

3. Business logic only matters because product behavior matters—so ideally align on that before interacting with a coding agent at all. Then, once the code is written, use abstractions to understand any other significant decisions made.

Understand the product over the code. Use abstractions to translate the latter to the former – except in the case of schemas!”

Jackie’s point is that data (that is, the state) is the most “rigid” part of any system. Stateless business logic is now easy to change because it’s “just” code, and code is easy and fast to generate and regenerate. For startups, it’s worth getting the data schema – and thereby your state machine – right. Then, everything else will be easy and fast to iterate on.

My sense is this approach makes perfect sense for a startup iterating to get product-market fit. However, once you have a business, you’ll want to “guard” the business logic with tests: else your product could break, and existing users will be unhappy when this happens!

The Pulse: tech companies move to open AI models

Gergely Orosz — Thu, 3 Sep 2026 17:00:14 +0000

The Pulse is a series covering events, insights, and trends within Big Tech and startups.

Today, we cover:

New trend: tech companies moving to open models. Uber, Pinterest, Stripe, Coinbase, Ramp, and AT&T are making large savings on their AI bills by dropping proprietary models and using smart model routing.
Automatic software maintenance experiments by Linear and Anthropic. Both startups are experimenting with how far they can push AI agents to automatically fix bugs and remove tech debt. It’s working better than anyone might’ve expected in the recent past, but not producing code that can be merged without review.
Frontier AI lab wars: OpenAI pulls models from SpaceX / Cursor. With SpaceX now a frontier model and rival to OpenAI and Anthropic, OpenAI has pulled its GPT models from Cursor. This isn’t an option for Anthropic which is dependent on the SpaceX compute they rent to serve Claude.
HR tech startup’s one-dev-per-project approach. A full-remote HR startup with 70 engineers has a single engineer run each project, and says the approach works well. Will this approach be adopted elsewhere, especially at other full-remote startups?
Industry Pulse. Meta moved over to Slack for better agent interoperability, layoffs at Uber and PagerDuty, Anthropic upsets users by calling a rate limit decrease an “increase”, token usage explodes on OpenRouter, AI drives surging demand for Apple’s Mac Mini & Mac Studio, and more.

Update: a week after publishing this article, Ara Krahzian at Ramp has confirmed that AI spend in August, has, indeed, declined at the top 1% of businesses by 10%, based on Ramp data. I’d wager those companies are not spending fewer tokens, but they are optimizing cost, in ways outlined below.

In May, I covered an emerging trend of companies wanting to cut back their AI spending, starting with engineering departments. Different approaches were being tried:

The Pragmatic Engineer: Five years

Gergely Orosz — Tue, 1 Sep 2026 17:14:34 +0000

Wow, has it already been five years?! The newsletter hits a big milestone this week, and it wouldn’t have been possible without subscribers. Thanks to everyone who’s read an article or listened to a podcast episode during that time!

Before we get into the latest issue, I’d like to point you to two upcoming events in New York City which I’ll be at. It would be great to meet some of you! They’re free to attend:

Wednesday, 16 September, 6pm: turbopuffer talks . What happens when you mix a whiteboard, two senior turbopuffer engineers, and myself, with the spotlight on database internals? We’ll draw out turbopuffer’s database architecture and geek out over it, then hang out afterwards. Join us! Sign up here .
Thursday, 17 September, 5:30pm: An evening with The Pragmatic Engineer and WorkOS. A fireside chat between WorkOS founder, Michael Grinich, and me, with him asking the questions. We’ll get into stories about the tech industry – including a few which probably won’t get published! Sign up here.

If you’re at LDX3 New York on 15-16 September (Tue & Wed), I’ll be doing a keynote on Tuesday morning and hanging out around the Antithesis booth during the day. See the full agenda and get tickets.

I checked the calendar and it is indeed half a decade – almost to the day in 2021 – since I published the first-ever issue of The Pragmatic Engineer:

On launch, the paid version of The Pragmatic Engineer cost $100/year, or $10/month (this has since increased to $150/year or $15/month). As a special offer, I’m “resetting” the price of the publication to annual subscribers for $100/year: claim this offer here. The offer ends in a week, on 8 September. Get this offer here.

My personal expectations weren’t high back then; the subscription model for newsletters on platforms like Substack was starting to take off, but the focus was strongly on politics, business, and finance. It wasn’t clear if there was any demand for a publication about software engineering, written by a software engineer.

I soon found out there was demand that surpassed all my expectations – and then some! Fast forward to today; the newsletter has more than 1.1M readers, tens of thousands of paid subscribers, a podcast, and more than 500,000 YouTube subscribers.

But the numbers aren’t the most validating thing for me; that would be the feedback sent in by readers. Via email, in DMs, or in-person at events, it’s great to hear how an article or a podcast we published helped someone try a new approach, or to gain confidence that theirs was the right one, or that an article helped convince a team to change things. It also means a lot to learn that The Pragmatic Engineer helps people feel more confident about keeping up in this fast-changing industry.

Thanks again for your support! It’s the reason why The Pragmatic Engineer is a viable business and a growing publication, and means we can “scale up” our coverage to deliver ever-more detail about how software gets built, today.

Today’s issue covers:

Diving deeper, year after year. The evolution of the Pragmatic Engineer’s coverage over five years, getting in through the “front door” instead of the “back door” for deepdives, launching the podcast, The Pragmatic Summit, and growing our team.
What’s next? What we’re excited about, the second Pragmatic Summit, and how you can expect us to stay focused on how building software is changing, and the ways that successful engineers, teams, and companies adapt.
Cash-prize writing contest: Software is changing faster than ever. Send us your essay about how things are changing for you, for the chance to win cash prizes worth up to $10,000. Read more on how to take part .

The Pragmatic Engineer was 15 years in the making and not an overnight success; I started to write a blog in 2007 about software development, which was “ rebooted ” as “The Pragmatic Engineer Blog” in 2015. At the time, it was read by almost nobody!

In 2019 , I launched an email digest (“v0” of the newsletter), and a year later, I decided to focus on the newsletter fulltime. This was, after I resigned at Uber, as a manager, following job cuts in 2020. The pandemic hit Uber hard: during those cuts, a quarter of my team was laid off and the rest were disbanded due to the pandemic. I took an employment break, planning to finish ‘ The Software Engineer’s Guidebook ’ in six months and then start a VC-funded startup.

In the end, finishing the book took another two years, and I didn’t kick off that VC-funded startup I was originally planning to do. Instead, I decided to go all-in on writing a newsletter targeted exclusively at software engineers and engineering leaders.

The Pragmatic Engineer started off with one in-depth article on an interesting topic per week, including these ones:

The Platform and Program Split at Uber : platform teams were starting to go “mainstream” across the industry
How Big Tech runs tech projects and the curious absence of Scrum : this article made a splash
The perfect storm causing an insane tech hiring market : the job market in late 2021 was one of the hottest ever seen
Growing a junior-heavy team : this was published when it was hard to hire seniors and many teams were junior-heavy

This time, the newsletter took off and crossed 1,000 paid subscribers six weeks after launch. By the end of 2021 , it was the #1 paid technology newsletter on Substack (!!), with 2,700 paid subscribers and 30,000 free subscribers. Most surprising was that the growth happened via word-of-mouth, and from me posting about issues on social media, without spending on ads and marketing. Growth has continued since then; for example, as per the Brex Benchmark, The Pragmatic Engineer is the third most expensed newsletter at startups , globally, in 2026, and remains one of the fastest growing ones .

If you have a learning & development budget or something similar in your workplace, you can probably expense the newsletter. Here’s an email template to send to your manager. And you can also get the newsletter on “launch” price, at a 33% discount.

In the second year, I added a Thursday article called ‘The Scoop’ – now ‘ The Pulse ’ – alongside the research-heavy Tuesday articles. Unlike those articles, The Scoop covered tech news and developments on a weekly basis and wasn’t ‘evergreen’ material. But as I talked with more techies, I got to spot new trends and patterns, often months before mainstream publications covered them.

Recent examples have included us reporting the odd tokenmaxxing trend a month before the Financial Times did so, or covering the extreme work patterns at AI startups months before The Wall Street Journal picked up the story , or The Economist republishing my Trimodal Nature of Software Engineering Compensation diagram four months later across their digital and print editions.

Until mid-2023, I mostly wrote deepdives on engineering topics without the involvement of companies that weren’t interested in showing me how they did things from the inside.

At the end of 2022, I heard from former colleagues at Uber that the ridesharing giant was moving off its own data centers, and moving onto the cloud, and onboarding to GCP and Oracle. I gathered plenty of details from current and ex-Uber workers, but the infra leadership didn’t engage with me via official channels.

So, I went ahead and published ‘ Inside Uber’s move to the Cloud ’, which got the majority of details right, except for a few. Inside Uber, the article was criticized for not getting everything correct, but that would’ve involved me talking on the record to Uber’s infra leadership, which they didn’t do!

In 2023, the newsletter had 350,000 readers and was starting to make a name for itself, which began to open some previously closed doors. I increasingly got details from engineers at companies via the “front door” rather than the “back door” route of informal contacts and scraps of information. Uber’s cloud migration was the final deepdive of its type; after that, going through the “front door” became easier, with companies sharing details with me about what they were doing. This change led to more accurate and detailed articles, including:

Since then, deepdives about interesting, cutting-edge tech companies have become a regular part of our publishing schedule. And as a bonus, we’ve figured out a “recipe” for getting access to folks at tech companies who are usually off-limits to the media, which leads to more exclusive content for readers.

In the fall of 2024, I launched The Pragmatic Engineer Podcast with two intentions:

Share previously “private” conversations. When doing a deepdive about an interesting company or technology, I usually did a call with engineers. These fascinating, one-hour-long conversations got summarized in a paragraph or two in articles, but I always felt there was more that readers would be interested in.
Meet interesting people. I was spending most of my days behind a keyboard writing deepdives, with the occasional video call when researching engineering teams. I hoped that doing a podcast would just allow me to meet more people!

Simon Willison, one of the most grounded voices in AI engineering, was the first podcast guest , and feedback was warm and positive. I always look forward to talking with guests due to their experience and significant industry contributions; the likes of Grady Booch , Nicole Forsgren , and Mitchell Hashimoto – or for their unusually deep expertise – like context engineering with Dex Horthy , developer productivity with Laura Tacho , and building software without looking at the code with Peter Steinberger . These days, when I go to a conference, more people talk to me about the podcast than the written articles, which is interesting.

Over time, I’ve developed a preference for in-person podcast conversations instead of video calls. When the podcast launched, I had a home studio set up for the remote recording of on-screen meetings:

My podcast studio in Amsterdam. On the left wall: a noise-absorbing panel and a city map

But you might have noticed that these days, podcast episodes are in-person conversations more often than they are video calls. I find that conversation flows better in person, and that the format is more engaging for everyone involved than a conversation on a screen is. It also offers an opportunity to hang out before and after the recording! Of course, the logistics of in-person recording are more complicated due to travel, and when there’s someone I’d really like to get on the show but it’s tricky to arrange, we stick with the remote option.

I’d like to hear your suggestions about future guests for the podcast. Let us know who you’d like to hear from and why. Send suggestions here.

In summer 2025, I attended the LeadDev Conference in London and enjoyed it so much that I asked if we could organize a conference for The Pragmatic Engineer, with deepdives and podcast guests for readers and listeners.

We spent the second half of last year organizing the first-ever Pragmatic Summit , which took place in February this year in San Francisco, with great help from the excellent Statsig team (many of whom now work at OpenAI). With 500 attendees, 15 standout speakers – and with it being the first-ever conference I’d organized – it was a smashing success. A one-minute video recap of the event:

Based on the feedback from attendees, there will be another Pragmatic Summit in San Francisco next year. I’ll share details in the coming weeks; we’re in the middle of putting this event together now!

Predictably, the most time-consuming task in The Pragmatic Engineer is working on deepdives. We often spend up to two months on a single deepdive, educating ourselves on the topic being covered, talking with expert engineers, and generally getting deep into a topic in order to deliver a deepdive worth reading.

I say “us” because this year, Jessica Salmon and Ivan Klaric joined the team. Both are software engineers with startup and Big Tech experience, who enjoy spending time getting to understand topics and contributing to longform articles.

With a larger team than before, we’ve continued to produce deepdives for readers. A few recent ones:

I’ve found there are no shortcuts for producing an in-depth article on a relevant topic for readers. You simply have to put the time and effort into it. This involves spending a lot of time on thinking and understanding things, going directly to engineers who build what we want to learn about, and then spending even more time on organizing research material into a lengthy article that’s informative and hopefully not dull to read.

Needless to say, we’ve experimented with the analytical powers of AI tools in parts of the research process. The technology is good at gathering publicly-available sources and does a decent job of summarizing them, but that’s been more or less the limits of AI’s usefulness for our purposes to date – except as a spelling and grammar checking tool after a full draft is written by a person.

If anything, AI can easily lead you down the wrong track by theorizing about non-existent connections and confidently espousing theories which some basic critical thinking could easily debunk!

Aside from research and correcting (most) typos, we don’t use AI in this publication. That’s because we are writing for a readership of humans and believe in the value of human voices. What you read, hear, and see in The Pragmatic Engineer comes from me and other engineers, and plenty of thought goes into each sentence.

As a company that doesn’t have any venture funding, we can be ambitious without chasing any “growth goals.” In the near future, our goal is to keep doing what we do, and do it better. This means:

More ambitious deepdives. We’ll keep bringing you deepdives from inside companies and teams building cutting-edge, fascinating software; how do they do it, what is working for them, and why? The more you understand about why engineering approaches work in certain places and situations, the more likely you’ll be able to effectively apply them in your own setting.
In-person events. The Pragmatic Summit returns to San Francisco in February 2027. It’s my long-term goal to have a second summit in Europe, as well. Meanwhile, I’ll keep attending in-person events, like the upcoming shows in New York City with turbopuffer and WorkOS , and let you know in the newsletter when these happen.
Interesting podcast conversations. Recording conversations with software professionals and leaders is something that keeps filling my bucket. Expect more great conversations coming your way.
Keeping up with The Pulse. I spend most of the week talking with engineers on various channels and at companies as part of my efforts to check ‘the pulse’ of the tech business: what’s happening and what’s changing. Writing The Pulse on Thursdays remains a personal highlight of my week.
Building more of our own software stack – on the side. One thing that AI has made easier is context switching between writing and building software. In the past six months, I’ve found myself building more parts of The Pragmatic Engineer backend stack, such as landing pages, through to API endpoints used for group subscriptions, refunds, and more. This year, I’ve probably built more software scratching my own itch at the publication than in previous years combined! Of course, our software stack is not our top focus, but it’s a welcome distraction to work on.

A big theme for the rest of 2026 is how software engineering is changing: tools used for decades like IDEs are falling out of style, and processes long considered as best practices, like code reviews, are becoming optional. Of course, the biggest change is that we’re spending little to no time on typing out the code, which has never been the case since computing existed. Even before computer keyboards, programmers were writing programs on punch cards!

The good news is that we see that fundamentals still matter: many software engineers who were considered standout devs before seem to be even more in demand than ever, while picking up AI engineering appears to be easier than learning a new programming language.

Nonetheless, this change is destabilizing, fast-paced, and no one has figured out the “right” way to build software with AI. We’ll keep reporting on cases of teams and individuals that adapt well, while also paying attention to those things that don’t change, such as how teams, as a “core” unit of a business, seem to be just as important at leading AI labs as they were pre-AI.

To finish, we’re delighted to announce an essay challenge on the prescient topic of how software engineering is changing for professionals.

As mentioned, the pace of change in software engineering is only accelerating, with rapid industry-wide adoption of LLMs, AI tooling, and AI infrastructure. At The Pragmatic Engineer, we aim to cover much of what’s going on, and as part of that, we’d love to pull in more perspectives than what our team can cover alone.

Send us an article no more than 10,000 words long on how you see things changing at your startup or tech company. We’ll award $10,000 for the best essay we read, and other leading entries can win smaller prizes. Articles sent to us will be eligible for publication in future editions of the Pragmatic Engineer. See more details here.

So, tell us what’s new, different, better, or worse in your part of the tech industry since AI has been in your workflow.

Submissions close 4 October at midnight (PST). Read all details of the challenge here, and we look forward to reading your article about interesting and consequential changes in your part of the industry.

If you’re thinking of upgrading to the paid version, you can do so for the “launch” price, with a 33% discount on annual plans. This offer ends in a week, on 8 September. Get it here. If you have an L&D budget to expense from, here’s an email template to send to your manager.

Grab this limited time offer

Thank you for being a reader of The Pragmatic Engineer; we value your attention and support, and never take it for granted. With that, onwards to the next five, exciting years!

– Gergely and The Pragmatic Engineer Team

No posts

The Pulse: Meta wanted to reduce teams by 60% because of AI

Gergely Orosz — Thu, 27 Aug 2026 17:59:17 +0000

We find out why Meta destroyed its standout engineering culture: it feared AI-native startups doing more with less. Also: thoughts on Ramp’s AI infra, GitHub’s load doubles in four months, and more

Aug 27, 2026

∙ Paid

The Pulse is a series covering events, insights, and trends within Big Tech and startups.

Today, we cover:

Did Meta really decide to reduce team sizes by 60% because of AI? An in-depth report by Reuters details how Meta’s leadership decided to slash team sizes by 60%, hatching plans in January to execute the social media giant’s largest-ever layoffs. But …

Why performant code matters (but gets widely ignored), with Casey Muratori

Gergely Orosz — Wed, 26 Aug 2026 15:59:59 +0000

Listen and watch now on YouTube , Apple , and Spotify . See the episode transcript at the top of this page, and timestamps for the episode at the bottom.

• Antithesis – turbocharge testing of your systems by running your whole system under aggressive fault injection. There’s good reason teams like Jane Street, Fly.io, and the etcd community rely on Antithesis. Learn more.

• Sentry – application monitoring software built by developers, for developers. Sentry’s Seer AI agent is one of their new, neat tools, which I’ve used as a way to quickly fix errors on my backend. Check out Sentry.

• turbopuffer – A vector and full-text search engine built on object storage. It’s fast, cheap, and extremely scalable. I met their team in San Francisco, and am a fan of their “hardcore and whimsical” engineering culture, and how pragmatic their engineering philosophy is. Check them out.

There can be few people around who care about software performance more than today’s pod guest, Casey Muratori . He’s a programmer and videogame developer, founder of Molly Rocket , and creator of Handmade Hero – a long-running series about building a game from scratch. He also evangelizes about performance on his Substack, Computer, Enhance .

We got to know each other about three years ago, first via messages, including this one from Casey:

“Why does the industry zeitgeist place so little emphasis on software performance when there seems to be overwhelming evidence that performance is critical to their bottom line?

Like you, I run a Substack for professional programmers, but I focus exclusively on software performance. Although we are quite large by Substack standards, so a certain subset of programmers must believe performance is important, I nonetheless hear lots of dismissive excuses when I post on social media. This happens so frequently, I devoted an entire article to cataloging the extensive pro-performance evidence we already have from the world’s leading software companies: Performance Excuses Debunked .

Strangely, nobody has a rebuttal to why performance is important. When I point people to this, they actually tend to agree. But the prevailing attitude nonetheless stays the same.”

I’m delighted we finally have Casey on the podcast because it’s overdue! In this episode, we discuss why software performance matters, why it’s overlooked, and how developers can get better at writing performant code. We explore why performance should be considered during design, the value of learning to read assembly & understanding how CPUs work, Casey’s critique of ‘clean code’, and why he believes testing shouldn’t drive software design.

We touch on how videogame development has changed, and influential game engines. Casey also tells us why he prefers to write code by hand, not with AI, and more.

1. DirectX might not exist without an “unauthorized” internal Microsoft project. DirectX is a very popular Microsoft library that standardized rendering on top of GPUs, used mostly for games. Casey tells how Chris Hecker built a library for fast on-screen rendering at Microsoft called WinG, which was never authorized; it was a total “ Skunk Works ” project. DirectX’s roots go back to WinG, which its three founders were testers on.

2. Is performance starting to matter to businesses? Enterprise software buyers care mainly about cost, compliance, and capabilities – but not performance. Even so, there are some products gaining major popularity and market share due to their performance, such as File Pilot (next-gen file explorer) and the Blick video editor . Is the tide turning?

3. Profiler-driven performance optimization is the wrong way to optimize. The standard way of optimizing is to profile the application, tweak hotspots, then check if the stats have improved. But this only finds a local minimum; Casey says every engineer he’s worked with who was a great “optimizer” began by establishing what the hardware could theoretically do, and then did not stop until they’d closed the gap to that performance level.

4. If you care about performance, learn to read assembly (no need to write it). There are about 20-30 instructions you need to learn to be able to read basic assembly. For example, here’s a program that calculates the value of 5 + 3 - 1 (which is 7), then prints it out:

An assembly program calculates 5+3-1 (the first 3 lines after _start), then prints the result to stdout

5. Take a grain of salt with conventional wisdom that premature optimization is the “root of all evil”. Many devs use it as an excuse to delay performance optimization, but Casey says that not optimizing in time could mean that only performance hotspots can be fixed later, and not the architectural issues that create poor performance. Architect your system to be performant, or you’ll have trouble solving problems without a rewrite!

6. Only three things are needed to understand how CPUs work. Casey believes that knowing them means you’ll be able to tell from any CPU announcement roughly how well it performs. Those three pillars of understanding:

How data moves in and out: load/store units and L1–L3 caches
How instructions flow through the pipes: branch prediction, i-cache
Execution unit scheduling: raw throughput per operation type

7. Why are game studios so secretive? Before licensable videogame engines existed, the game engine was a studio’s “core” intellectual property (IP), and every studio built rendering, pathfinding, and other tools from scratch. This is how Blizzard rolled Warcraft 1’s engine into Warcraft 2. Any competitor making a rival game had to start from scratch, which was a reason for game studios to closely guard the secrets of how their own game engines worked.

8. The games industry already had its “AI moment” – and it wasn’t pretty. When game engines became licensable, pretty much any developer could build and publish a game with the likes of Unity and Unreal, on a platform like Steam.

Initially, this change empowered new devs to build interesting games. But soon enough, the market was flooded with tens of thousands of releases per year, which destroyed organic discovery. Without a marketing strategy, the chances of a game gaining traction today are basically zero, says Casey.

9. Old games don’t look dated anymore, and that’s a problem. For decades, graphics were a vital barometer for showing how videogames improved over time; a new release in 1995 was guaranteed to be visually superior to one from 1990. But a new game in 2026 likely doesn’t look much different from one that’s nine years old, and new releases face ongoing competition from older games.

10. Casey’s problem with test-driven development is the “test” bit. Casey believes tests should be a cost/benefit decision, and not put in place by default. For some projects, doing tests upfront – or doing any tests at all, in some cases – is simply a bad choice.

11. One trait of almost every great engineer: refusing to accept programming wisdom untested in the real world. As Casey puts it:

“I find there’s a lot of received programming wisdom that’s just nonsense. Clearly, no one’s ever tested it. In order for something to be received wisdom, you should have to at least demonstrate concrete upsides, but often this cannot be done. I would say focusing on what actually works in practice is a huge plus.”

12. No AI in Casey’s upcoming game. He acknowledges that many developers will disagree, but insists there’s nothing wrong with being outside of mainstream tastes, just like some people chose handmade furniture over the flatpack kind. His reasoning for omitting AI is straightforward:

“I want to program things in a game because I want to program them. If I only wanted output, I’d just get the Unreal Engine.”

• Pushing software engineering limits with “napkin math”withSimon Eskildsen

• How Games Typically Get Built : prototyping, game engines, and a different type of QA

• Game Development Basics : deepdive on how game studios differ from standard software teams

• Inside Linear’s Engineering Culture : building a performant product with a tiny team

• Building a best-selling game with a tiny team – with Jonas Tyroller. A two-person team built a game that sold 1M+ copies

More on premature optimization: read or watch Casey’s extended take on “premature optimization is the root of all evil”:

00:00 Intro

05:17 Games at Microsoft

12:52 Building games

16:00 Why performance matters

27:12 Why you should learn to read assembly

30:36 Designing for optimization

42:51 How to get better at writing performant software

49:04 Understanding how the CPU works

55:53 Building games then and now

1:05:56 How game engines changed building games

1:10:48 Why new games compete with old games

1:13:25 GTA 6: why is it taking so long?

1:16:59 Casey’s critique of clean code

1:21:48 Casey’s take on TDD

1:24:30 What is good code?

1:27:32 What makes a good software engineer?

1:33:56 Why Casey doesn’t code with AI

1:39:01 AI’s impact on the game industry

1:44:43 AI and burnout

1:50:21 Why you should read papers

Where to find Casey Muratori:

• X: https://x.com/cmuratori

• Website:

• Substack: https://substack.com/@cmuratori

Mentions during the episode:

• Digital Equipment Corporation: https://en.wikipedia.org/wiki/Digital_Equipment_Corporation

• VAX 9000: https://en.wikipedia.org/wiki/VAX_9000

• Intel: https://www.intel.com

• Chris Hecker’s website: https://www.chrishecker.com/Homepage

• Doom: https://en.wikipedia.org/wiki/Doom_(franchise)

• Wolfenstein 3D: https://en.wikipedia.org/wiki/Wolfenstein_3D

• WinG: https://en.wikipedia.org/wiki/WinG

• Ron Gilbert: https://en.wikipedia.org/wiki/Ron_Gilbert

• Humongous Entertainment: https://en.wikipedia.org/wiki/Humongous_Entertainment

• The Secret of Monkey Island: https://en.wikipedia.org/wiki/The_Secret_of_Monkey_Island

• DirectX: https://en.wikipedia.org/wiki/DirectX

• Todd Laney on Tumblr: https://toddla.tumblr.com

• Craig Eisler on LinkedIn: linkedin.com/in/craigeisler

• Eric Engstrom: https://en.wikipedia.org/wiki/Eric_Engstrom

• Dungeon Siege: https://en.wikipedia.org/wiki/Dungeon_Siege

• RAD Game Tools: https://www.radgametools.com

• Alex St. John: https://en.wikipedia.org/wiki/Alex_St._John

• Molly Rocket: https://mollyrocket.com

• File Pilot: https://filepilot.tech

• Bun: https://bun.com

• npm: https://www.npmjs.com

• Napkin math: https://github.com/sirupsen/napkin-math

• Fortnite: https://www.fortnite.com

• Minecraft: https://www.minecraft.net

• Roblox: https://www.roblox.com

• GTA online: https://www.rockstargames.com/gta-online

• Unreal Engine: https://www.unrealengine.com

• Ken Silverman’s website: https://advsys.net/ken

• id software: https://www.idsoftware.com

• Bullfrog Productions: https://en.wikipedia.org/wiki/Bullfrog_Productions

• Thief: The Dark Project: https://en.wikipedia.org/wiki/Thief:_The_Dark_Project

• Death Rally: https://en.wikipedia.org/wiki/Death_Rally

• Grand Theft Auto V: https://www.rockstargames.com/gta-v

• “Clean” Code, Horrible Performance:

• TDD, AI agents and coding with Kent Beck: https://newsletter.pragmaticengineer.com/p/tdd-ai-agents-and-coding-with-kent

• Python, Go, Rust, TypeScript and AI with Armin Ronacher: https://newsletter.pragmaticengineer.com/p/python-go-rust-typescript-and-ai

—

Production and marketing by Pen Name .

Why Ramp built its own in-house coding agent, Inspect

Gergely Orosz — Tue, 25 Aug 2026 15:20:23 +0000

At a select few tech companies, they write most of their code with their own, custom-built, internal AI coding agents. This is different from most of the industry which uses AI coding agents and harnesses like Codex, Claude Code, Cursor, OpenCode, GitHub Copilot, etc. At Ramp, their own version is called Inspect , while at Block it’s Goose (open source), at Stripe it’s Minions , and River at Shopify.

But why not just use what frontier labs and coding harness AI startups already offer; why take the time and effort?

We reached out to Ramp, a fintech company big on building its internal AI infrastructure, and sat down with the founding team of Inspect and engineering leadership. We talked with CTO Rahul Sengottuvelu , Head of Engineering Hamid Dadkhah , and Zach Bruggeman , principal engineer and founding engineer of Inspect.

Today, we cover:

What is Inspect? Imagine an AI coding agent running on remote sandboxes with access to most internal data sources, and verifying all backend and frontend changes on the remote machine.
Why build your own background coding agent? Engineers and designers at Ramp were dissatisfied with third-party harnesses: they wanted to run more than a few agents in parallel – which local machines don’t support – to have better frontend tooling, and also faced demand for remote development environments.
How Ramp uses Inspect: coding, bugfixing in Slack, debugging, and building internal agents like code review and incident management on top of the Inspect platform
Tech stack and architecture: React/Vite, Cloudflare Durable Objects, SQLite, Cloudflare Agents SDK, Modal sandboxes.
What makes Inspect so popular? The machine in the cloud is a developer machine, plus it has access to numerous internal integrations via API and MCP.
Inside the sandbox. OpenCode, services for development (e.g. Postgres, Redis, RabbitMQ, Temporal), Chromium, and VS Code Server. Plus, we check out smart tricks to make sandboxes spin up in five seconds or less(!!)
Collaboration & feedback. All Inspect sessions are public and open to collaboration, with no opt-outs allowed. More than 150 people at Ramp have contributed to the project.

If you’re like us, you might wonder what the point would be of building your own harness and investing the time and resources in it, given all the choices already out there. This article sets out to answer that question, to understand why other places chose a similar path, and how a non-AI frontier lab can build more efficient tooling than what the frontier AI labs offer. It looks like the “buy, don’t build” tooling convention might not apply to AI tools!

Let’s get into it.

Inspect is Ramp’s internal background coding agent, shipped and opened internally last November. Engineers at Ramp can use any tool they want, but 75% of merged PRs are now raised by Inspect; a clear indication that many engineers prefer the tool over others:

Inspect’s home page: showing sessions started by the user

Inspect: how the UI looks for engineers inside of Ramp

A couple of things make Inspect different from coding agents like Claude Code and Cursor:

Remote sandboxes: Inspect spins up a sandboxed remote development environment which unlocks unlimited session concurrency, centralized setup configuration, and cross-functional session collaboration.
Internal integrations: Inspect is integrated across the org with the same tools and context that a Ramp engineer has; the only constraint on agents’ ability is model intelligence, not missing tools or access.

Inspect verifies all its changes. As a remote development environment with full tooling access, it can “close the loop” and confirm the changes it makes work:

Backend verification: Inspect runs tests, reviews telemetry and queries feature flags
Frontend work verification: Inspect visually verifies its own work by providing screenshots and live previews to users.

At present, most third-party AI harnesses cannot do these kinds of verifications ‘out of the box’ because they lack internal integrations with things like telemetry and feature flag systems. Also, almost a year ago, Ramp built screenshot verification before it was supported by third-party vendors. Things like this placed Ramp months ahead of nearly all AI coding harnesses, and they could also build a far better feedback loop in their own harness.

The v1 of Inspect was a Chrome extension for designers to prompt AI to make minor website changes. A few months later, the v2 version with background agents followed.

History of adoption numbers

By January of this year, just two months after the v2 launch, around 60% of PRs at Ramp were authored by Inspect, which increased to 75% by May. At Anthropic, Claude Code won rapid adoption after an internal release, as covered in the deepdive How Claude Code is built.

Then Inspect hit a neat milestone in July, crossing the one million total sessions mark:

Milestone: one million Inspect sessions

There are a few reasons why Ramp decided to turn down tried-and-tested products and create their own:

Local machines are limited in how many agents they can run. Ramp found third-party products below expectations; they liked Claude Code on day 1, but were constrained by only being able to run one or two sessions on local machines.
Better frontend tooling. The web engineering team wanted to improve their frontend tooling so designers could make small UI tweaks. There was an opportunity to use AI to automate themselves out of that loop.
Need for remote dev environments. As Ramp scaled, so did the complexity, and with it there was more work at the intersection of systems, like debugging backward compatibility, and broken API contracts. The solution was to create remote dev environments.

Inspect started as a designer’s frontend tool, and a good part of its team were frontend engineers with interests in UX and speedy performance. The v1 was a Chrome extension for visual edits, where a user could highlight an area and tell the AI what minor website changes to make, like copy edits and button placements. The task of building a tool for making UI edits with AI was given to two frontend engineers, Zach Bruggeman and Jason Quense, who aside from their frontend domain knowledge, brought a welcome adversarial perspective, as they were less than fully convinced by AI at that time.

People liked v1 but it wasn’t adopted because engineers already knew how to go to a file and edit a single line of code, so didn’t have a reason to use it, and it also required setting up a local development environment, making it too complicated for non-devs.

For the current iteration of Inspect (released November 2025) the team pivoted. They built Inspect v2 as a remote development environment with a coding agent on top. Setting it up as a remote environment that they could configure centrally removed the need for local setup on each machine. They were also encouraged by seeing that OpenCode, the open-source coding agent which serves as Inspect’s harness, exposed an HTTP API which made it straightforward to set up, and was open-source, good enough, and importantly, offered model agnosticism.

Check out the episode of The Pragmatic Engineer podcast with OpenCode creator, Dax Raad .

After pivoting, adoption skyrocketed to where it is today:

Daily unique human Inspect users

Adoption numbers today:

75% of all merged PRs come from Inspect sessions
~90% share of PRs merged into the Inspect repo come from an Inspect session
Under 5 seconds to spin up a fully provisioned remote dev environment
5.5 people in the Inspect team: four engineers, a director, and part-time PM
150+ engineers at Ramp who have contributed to the Inspect codebase

Having built it, Ramp uses Inspect for a few things:

Coding: obvious use case; engineers prompt Inspect with small and medium-sized coding tasks that can often be one-shot passes. For larger, more complex tasks, devs often use Inspect to kick-start an idea and then take over developing it locally.
Bugfixing in Slack: the @inspect fix this prompt in Slack. Inspect reads all the thread context and raises a pull request (PR) with a fix.
Debugging: Inspect can do things like debug the code (stepping through the code in debugger mode), query the sanitized read-only prod DB replica, find business logic/data mismatches.
Using Inspect to build Inspect: Inspect is used to build itself, and more than 80% of Inspect is written in Inspect sessions.
Platform for agents: Engineers at Ramp have built more than 200 agents running on top of the Inspect platform

Here’s an example of how debugging works. Devs can ask the agent to investigate an issue, and Inspect goes off and pulls data from the correct sources:

Debugging with Inspect: asking the agent about an incorrect allocation. Debugging is done via the web chat interface

The tool goes and makes database or Snowflake queries when helpful:

Making database and Snowflake queries

The debug agent can be long-running while it gathers data from various sources. Finally, it presents its findings:

The debug agent found the root cause: in this case, it was a routing/policy decision, discovered by querying relevant data sources

This debugging example illustrates how much more capable agents can be with the correct access to tools, data, and context.

Some internal agents built on top of Inspect:

ReviewBuddy: Ramp’s own code review system, customizable per team. The difference from third-party AI code review tools is that it’s very aware of Ramp’s context, and the team found it to work better than third-party tools. Built by a single engineer in a week.
Oncall Assistant: connected to all production and observability systems. When the agent detects an incident, it gathers all relevant context and tries to determine the cause. The oncall engineer can choose to join the Inspect session and prompt against this proposed fix.
Testo: a frontend QA tool and browser-based agent that clicks around like a user would, and creates Playwright tests.
Ramp Research: the company’s agentic “data analyst” is connected to all Ramp’s data sources, like Looker , Snowflake and dbt tables. Ping it from Slack about any topic and it gets answers. Before Ramp Research, engineers and data analysts had to know which data tables to query and join. Ramp previously shared more about its Research .
Voice of the Customer: connects to several customer feedback sources like chat, email, App Store reviews, etc. It collects feedback from the last 90 days, and allows prompting against them as a Slack bot
Error automations: automatically create draft pull requests based on alerts from Sentry or Datadog.

Visualized:

Most agentic automations inside Ramp are built on top of Inspect

It’s clever that the Ramp team extended Inspect into a platform, and made it easy to build additional agentic tools, without engineers having to worry about the cloud backend for those tools. Not bad for a tool that started as a simple Chrome extension almost exactly a year ago!

Inspect’s core principle is that agents should have access to the same context and tools as software engineers.Hooking up Inspect to the data sources that engineers would browse with the same tools seems to be a key difference between Inspect and third-party AI harnesses.

The Pulse: We need to talk about migrations with AI

Gergely Orosz — Thu, 20 Aug 2026 17:53:01 +0000

The Pulse is a series covering events, insights, and trends within Big Tech and startups.

Today, we cover:

More on the “great engineering leader career break.” The industry is changing fast, and the VPE and CTO roles also need to adapt. And don’t forget that these are the roles from which you can drive change that reorganizes engineering in ways that work better.
We need to talk about migrations with AI. Asana needed to migrate off testing framework Enzyme, but it meant doing a massive rewrite of test cases. With AI, the project was completed in two weeks: without AI, this work would surely have been kicked down the road. Airbnb and Uber share similar stories, and AI seems like a superb fit for framework migrations.
Are AI startups making the Gartner Magic Quadrant irrelevant? Gartner ranked AWS, Microsoft and IBM above Anthropic, Cursor and OpenAI in their “AI code modernization tools” ranking. This is most likely because the first three pay large sums of money to Gartner, but AI labs and vendors refuse to pay this “Gartner tax.”
Industry Pulse. Another hours-long GitHub outage, GitHub alternatives are here and fighting for market share, Slack launches Slack Code, text generated by Claude to be watermarked, and Uber open sources SubmitQueue.

Before we start: apologies for the numerous typos last week. My editor, Dominic, was on vacation, and numerous typos made it through the spellchecker. A reader asked for cute puppy pictures to accept my apology, and so I updated the post with pictures of our 3-month old puppy.

From Chrome DevTools to AI Engineering, with Addy Osmani

Gergely Orosz — Wed, 19 Aug 2026 16:53:57 +0000

Listen and watch now on YouTube , Apple , and Spotify . See the episode transcript at the top of this page, and timestamps for the episode at the bottom.

• Antithesis – verify your system’s correctness without human review or traditional integration tests – and avoid bugs or outages. Teams like Jane Street, Fly.io, and the etcd community use Antithesis to ship better code, faster. Learn more.

• Google Cloud Run – run untrusted agent code without the security anxiety. Cloud Run sandboxes deliver hyper-isolated, ephemeral execution environments that spin up in milliseconds. Check out Cloud Run sandboxes.

Addy Osmani spent more than 14 years at Google, working on Chrome, DevTools, Core Web Vitals, and most recently, AI developer experience.

If you’ve ever opened Chrome DevTools, or optimized a page for Core Web Vitals, you’ve used software built by Addy Osmani. In this episode, I sit down with Addy and we talk about his path from building a web browser aged just 16 to becoming a director at Google. We discuss what he learned from building tools for millions of developers, Google’s engineering culture, and why he continued doing hands-on coding work as a manager. We also get into how he works with AI agents today, the risks of ‘cognitive surrender,’ his approach to ‘loop engineering,’ and why it’s good to develop skills in product management, go-to-market, and other areas.

Here are eleven interesting points from the chat with Addy:

1. Addy built a web browser from scratch, aged just 16. Back then, a pain point was that Addy had to carry floppy disks to his local library to download data. To speed up browsing, he built a browser that opened multiple connections when fetching webpages.

2. Publishing free educational materials helped Addy land a job at Google. A documentary about Google which he watched as a youngster made Addy want to work somewhere like it. Later, Google noticed his work in publishing educational resources about frontend and JavaScript development. The company reached out about a DevRel-and-builder role, and Addy was hired to join the Chrome team.

3. Chrome DevTools was an effort by Google to meet web developers in the browser. Today, DevTools is one of the closest things Google has to an IDE (not counting Antigravity, that is), but the project started as a way to add tools to the browser to help debug web applications. As web engineers started to use more complex frameworks and build chains, DevTools added capabilities like source-map-aware debugging, hiding library code, mobile device emulation, tooling for service workers, and more.

4. Most developers don’t understand memory management. Addy says this is because memory debugging tooling has not advanced in a decade, and remains a hard problem to solve. This is despite making improvements in runtime performance debugging in Chrome DevTools ( flame graphs and deep tracing ).

5. Becoming accountable on a weekly basis for a top company goal is the biggest difference in a director of engineering at a major tech company. Addy worked his way up from engineer to Director of Engineering at Google, and I asked what the biggest change was when he made it to that level. Being on the hook and reporting regularly on a top company goal was something he found entirely new, Addy said.

6. A big culture shift at Google in the last two years has been VPs and SVPs coding on weekends. Naturally, this is because AI tools make coding much easier. During his last two years at Google, it was common for these folks to talk about their weekend side projects and tools they used to build them.

7. A big risk of AI-assisted development is cognitive surrender. Addy defines cognitive surrender as the erosion of your comprehension of the problems being worked on, and of your own memory of what’s going on. He recommends pushing back against this by understanding every major decision an LLM makes. Unfortunately, his former method of reading the AI’s entire reasoning process is no longer practical given how much output agents can generate, but you’ll still want to understand the most important decisions.

8. Aim for mutual amplification when using AI tools. The aim is to do two things simultaneously:

Help the agent improve throughout the task by having it log its decisions and key learnings
You also improve by reviewing, understanding, and internalizing what the agent does and how you can learn from it

9. Addy believes software engineers will always be important because an AI model cannot be accountable. Accountability for code and software is possible even if the accountable party didn’t write the code, as is the case in projects like Chromium, where designated engineers own parts of the codebase. They’re responsible for approving and rejecting contributions, and for shaping that part of the codebase. Addy reckons that a “what am I accountable for?” mindset will be adopted by many software engineers.

10. Addy is bullish about software engineering’s outlook. Every time the profession has made it easier to create software, we’ve created exponentially more software. Addy predicts the same will happen with AI, and that the total addressable market of people building software will get much bigger.

11. Advice on where to invest efforts as engineers in the coming years. In his words:

“What we are very likely to see happen next with engineering careers (as well as product and other roles) is the unbundling of them, so that an engineer also has product sense, while a product person also has engineering sense, or UX sense.

[You should] think about the non-engineering things if you don’t [usually] have the time to think about product or technical evangelism, or go-to-market approaches, or any other parts of how businesses are successful.

If you can show employers that you are not just a builder, but someone that can help them as roles start to become a little bit fuzzier, then I think that you can be successful in these times. Don’t be just an engineer.”

• What is loop engineering?

• Inside Google’s engineering culture

• How AI-assisted coding will change software engineering: hard truths

• Are AI agents actually slowing us down?

• How Claude Code is built

• How Codex is built

• From IDEs to AI Agents with Steve Yegge

• Google’s engineering culture : the podcast

00:00 Intro

02:50 Addy’s current workflow

05:11 Addy’s path into tech

15:04 Addy’s work on jQuery

16:44 TodoMVC

21:44 Getting hired at Google and working on Chrome

27:17 Building dev tools

40:15 Core Web Vitals

45:42 Google’s engineering culture

51:03 Addy’s career trajectory at Google

57:55 The director role at Google

1:01:40 Cognitive debt and cognitive surrender

1:03:03 Working with agents

1:05:52 Loop engineering

1:12:55 The changing role of the software engineer

1:18:15 How Addy uses AI in writing

1:27:40 What’s next for Addy

1:28:47 Career advice

Where to find Addy Osmani:

• X: https://x.com/addyosmani

• LinkedIn: https://www.linkedin.com/in/addyosmani

• Website: https://addyosmani.com

Mentions during the episode:

• Beyond Vibe Coding with Addy Osmani: https://newsletter.pragmaticengineer.com/p/beyond-vibe-coding-with-addy-osmani

• Borland: https://en.wikipedia.org/wiki/Borland

• jQuery: https://jquery.com

• John Resig on X: https://x.com/jeresig

• AngularJS: https://angularjs.org

• Backbone.js: https://backbonejs.org

• YUI: https://github.com/yui/yui3

• Ext JS: https://en.wikipedia.org/wiki/Ext_JS

• Sindre Sorhus’s website: https://sindresorhus.com

• Speedometer: https://browserbench.org/Speedometer3.0

• Next.js: https://nextjs.org

• Grunt: https://en.wikipedia.org/wiki/Grunt_(software)

• Firebug: https://en.wikipedia.org/wiki/Firebug_(software)

• Pavel Feldman on LinkedIn: https://www.linkedin.com/in/pavel-feldman-24b0041

• Paul Irish on LinkedIn: https://www.linkedin.com/in/paulirish

• Paul Bakaus on LinkedIn: https://www.linkedin.com/in/paulbakaus

• Impeccable: https://impeccable.style

• Visual Studio: https://visualstudio.microsoft.com

• Yang Gao on LinkedIn: https://www.linkedin.com/in/yang-gao-08567b51

• Understanding Core Web Vitals and Google search results: https://developers.google.com/search/docs/appearance/core-web-vitals

• Google’s engineering culture: https://newsletter.pragmaticengineer.com/p/googles-engineering-culture

• Inside Google’s Engineering Culture: Part 1: https://newsletter.pragmaticengineer.com/p/google

• Inside Google’s Engineering Culture: the Tech Stack (Part 2): https://newsletter.pragmaticengineer.com/p/google-part-2

• Simon Hørup Eskildsen’s website: https://sirupsen.com

• Pushing software engineering limits with “napkin math”: https://newsletter.pragmaticengineer.com/p/pushing-software-engineering-limits

• Loop engineering: https://addyosmani.com/blog/loop-engineering

• What is “loop engineering?”: https://newsletter.pragmaticengineer.com/p/what-is-loop-engineering

• Peter Steinberger on X: https://x.com/steipete

• Boris Cherny on X: https://x.com/bcherny

• Ryan Dahl’s post on X:

[

Ryan Dahl @rough__sea

This has been said a thousand times before, but allow me to add my own voice: the era of humans writing code is over. Disturbing for those of us who identify as SWEs, but no less true. That's not to say SWEs don't have work to do, but writing syntax directly is not it.

4:02 PM · Jan 19, 2026 · 7.33M Views

943 Replies · 2.67K Reposts · 19.8K Likes](https://x.com/rough__sea/status/2013280952370573666)

• The Effective Software Engineer: How ICs at Every Level Can Leverage AI, Prioritize High-Value Work, and Lead Beyond Their Role: https://www.amazon.com/Effective-Software-Engineer-Prioritize-High-Value/dp/B0FMJ5XVSD

• Leading Effective Engineering Teams: Lessons for Individual Contributors and Managers from 10 Years at Google: https://www.amazon.com/Leading-Effective-Engineering-Teams-Contributors/dp/109814824X

• Beyond Vibe Coding: From Coder to AI-Era Developer: https://www.amazon.com/Beyond-Vibe-Coding-AI-Era-Developer/dp/B0F6S5425Y

• Michael Novati on LinkedIn: linkedin.com/in/michaelnovati

• “The Coding Machine” at Meta with Michael Novati: https://newsletter.pragmaticengineer.com/p/the-coding-machine-at-meta

—

Production and marketing by Pen Name .

Headed for the Exit: the Great Engineering Leader Career Break

Gergely Orosz — Tue, 18 Aug 2026 16:21:06 +0000

In my ~20 years in this industry, I’ve not seen as many capable engineering leaders opting out or taking prolonged breaks as now, with some high-ranking engineering leaders – CTOs, VPs of Engineering, heads of engineering, etc. – quitting their high-status roles and departing, if not into the sunset, then at least with nothing lined up.

To find out what might be behind this spate of sign-outs, I talked with almost 20 engineering leaders currently on a career break – or seriously considering one – and they let me into their personal reasons for deciding to jam the brakes on their careers. Thanks to everyone who shared their input!

Today, we cover:

Ten of the most common reasons for quitting, sometimes without the next gig lined up:
- 1. The job got (much) worse
- 2. The startup is “losing” and becoming worthless
- 3. Not being AI-native enough for other skills to be relevant
- 4. Their predecessor saw the “writing on the wall”
- 5. Long hours – rarely decisive
- 6. Smaller teams mean less need for leaders
- 7. Fractional CTO work preferred over fulltime positions
- 8. AI startups pay ICs more than non-AI startups pay executives
- 9. Quitting to launch their own business
- 10. Burnout
“Founder mode” looks here to stay, so how to deal with it? And has it made the CTO and VPE roles become “low ROI”?
‘Work at companies that truly want to drive change’. A personal account from someone who took the VP of Engineering role at Gitpod (later, Ona, now acquired by OpenAI) and enjoyed a rewarding experience. Matt Boyle says he interviewed the employer beforehand on whether their business truly leans into the changes brought by AI.

I was recently messaged by a head of engineering in San Francisco, who said:

“I’m talking to four startups in San Francisco about the head of engineering roles. Pretty normal.

But one interesting pattern is how founding CTOs/heads of engineering are stepping away to take a full career break. We’re talking about two of these four startups. And these are good startups!

Have you seen this trend? I have a small number of data points here, so you might have a broader view.”

I asked around privately, and it turns out a majority of the CTO-level folks I spoke to are considering the very same thing, or are actually in the process of leaving the office for a long spell away; 6/10 engineering leaders said they’re on the way out.

Unrealistic expectations, including about AI, by founders and CEOs are the leading cause of jobs turning bad for CTOs and VPEs right now in 2026:

CTO expected to magically transform the company to be “AI-native”
CTO must make significant engineering cost cuts of up to 20-50%, including morale-sapping job cuts
“Do more with less” equals shipping more with fewer people (e.g., no backfills)
CTO faces pressure on business results as AI coding bills rack up
Founder slop: they want wonky AI prototypes shipped as full-blown products within weeks

Hands-on founders with “AI psychosis” make the job predictably harder, according to one CTO who just signed out of his job:

“Managing ‘AI psychosis’ with founders and executive peers has become very difficult. For example, what do you do when a founder ships a 60,000-line pull request into the product, gleaming with joy at how much more productive they’ve become with AI? They won’t see all the issues with that PR, and how do you bring up that they’ve created a massive amount of tech debt? Especially without looking like a ‘Debbie Downer’.”

Founder slop issues begin when top leadersgetexcited about AI’s capability, then get hands-on and start issuing PRs, and shipping code to production. It can cause issues across the board:

Accountability. Who’s oncall when founder-shipped code breaks? In the “you build it, you own it” culture of startups, it’s confusing when a founder gets hands-on while not owning their work.
Quality out the door: if a founder’s half-baked features are accepted, it sends the wider message that quality does not matter. Some people may adopt this attitude to their own work.
A founder can overrule whatever was previously agreed with the CTO or VPE about what to build next. Vibes the founder has or feels are reason enough.

Another way that leadership roles have diminished is that craft and quality are less important, says a VP of engineering who’s in the process of signing out of their job:

“Shipping software became all about speed. Finding differentiation with your product in the market is brutal, and speed / go-to-market becomes the biggest differentiator. Craft, quality, and care going into the product are taking a backseat.”

Things also go bad when companies don’t ‘get’ AI+engineering, except as a way to cut jobs. CTOs I talked to mentioned the likes of Ramp, Stripe, and Notion as places that understand how to integrate AI into the engineering culture with a growth mindset without forsaking quality. Elsewhere, bad vibes dominate at places where going all-in on AI leads to the cynical conclusion that product management, design, and engineering leadership are irrelevant.

Director+ roles have a few differences from individual-contributor engineering ones:

Larger equity stake in the business. Base salary at these levels is often similar to a staff engineer’s, but usually with more generous equity grants – especially at the VP of Engineering and CTO levels. A good financial outcome depends on the company becoming more valuable, and – in the case of private companies – having a good exit by being acquired or selling shares.
Understanding of the business and competition is a baseline. At Director+ level, a big part of the job is making strategic decisions that grow the business and help the company get ahead. It’s a nice-to-have for an engineer to possess business acumen, but director-and-above folks use it much more than most individual contributors (ICs). Great engineering leaders are good at understanding business performance and outlook.

A company that adopts AI rapidly usually falls into one of three buckets:

“AI-native”, building & selling AI products. The large AI labs and a select few “AI-native” startups are thriving, but many AI startups with VC funding struggle. Engineering leaders know this, and that their equity – usually issued as options – could end up worthless.
Software startups threatened by AI-native businesses. Good businesses in the pre-AI world can be threatened by AI today, like SaaS startups selling seat-based products in areas where agents are taking over the functionality. They have to pivot their businesses or seek an exit. Bending Spoons buying Airtable for less than the company raised is an example of a business threatened by AI and choosing to sell, instead of pivoting the whole business.
Unaffected by AI. Usually stable businesses which do more than software, such as with a real-world side to the operation like manufacturing or distribution.

The majority of software startups fall into one of the first two buckets of being AI-native or under threat. Senior leaders at such companies are in a good position to evaluate whether their company is a “winner” worth staying with.

A CTO who quit their startup told me:

“My company would have needed a massive exit for me to realize any upside. I had an equity grant that was 2% of the common shares. However, this equity was behind an already steep preference stack for investors, post Series A.”

This CTO had a very generous equity grant at 2% of shares, so what made him leave it behind? They laid out how it will be difficult to get any benefit from them because the shares are most likely rendered worthless by rules about the order in which different investors get their share of the pie:

Assume that this company raised a $10M seed round at a $50M valuation, then a $100M Series A at a $500M valuation. So, a total of $110M was raised across two rounds.
Investors typically have a 1x preference. 1x preference would mean that upon any sale, they get the first $110M of the sale.
But in this company, the Series A investors negotiated a 2x preference: so upon a sale, $210M goes to investors first ($10M to the Seed, and $200M to the Series A investors).
The company now needs to sell for at least $210M for common shareholders (like the CTO) to make any money!
If the CTO does not believe a $200M+ exit could happen, then their equity is worthless. A $200M+ exit is typically an acquisition, because a stock market flotation rarely happens at below a $10B+ valuation, these days.

If a VC-funded company does not have the revenue or customers to grow at a fast tick (circa 20-50% per year), then it’s often a struggle to raise the next round of funding, and the business’s actual value usually shrinks to 3-5x of annual revenue. So, if a startup is making $10M per year after raising $110M in funding, and growing 30% year-on-year, then the company is likely worth around $30-50M. Perhaps the right buyer would pay $100M, but if growth slows, the value is likely to drop.

An experienced CTO who takes a step back and assesses things can realize when there’s a high chance of their equity turning into smoke, removing a reason to not sign out of the job. It’s what happened to the CTO above, and when they couldn’t turn the business around, they quit.

When a VC-funded startup’s business stops growing, the prognosis can be dire in the sense that it’s unlikely to be worth as much as in the previous funding round. This is true even when the startup becomes profitable: this might mean it could theoretically go on forever; but with slow or no growth, it won’t win in another VC funding round.

Here’s a VP of Engineering who saw their startup stop growing, partly due to wrong bets by the CEO:

“My founder/CEO was nontechnical, and was both moving too slow and too fast with AI.

Too slow, as in they did not take the time to understand what our customers wanted. We built a TON of AI stuff, it totally confused them, they churned, growth stalled, word-of-mouth growth was gone. Heck, I don’t think our customers ever wanted or needed anything with AI!

Too fast, as in they deprioritized core systems’ reliability in favor of shipping AI work to prod which did not have any commercial potential. So, our core offering started to have more outages and we lost customers because of this as well.”

I’d add that deprioritizing reliability in favor of building features may be sensible in the early days. The problem seemed to be that this company had not found product-market fit, and the new AI features didn’t resonate with customers. Basically, the CEO lacked customer understanding, business intuition, or both.

So, good on the VPE for getting out when they saw the direction of travel. If the CEO won’t accept input from the VPE – who would’ve at least prioritized reliable operation – then there isn’t much left to stick around for!

The top-paying engineering leadership positions have one thing in common: experience of leading AI-native organisations is expected, and leaders are sought who have turned their current company AI-native, or work at such a place.

It’s new to see people signing out of large companies for feeling like they’re lagging behind in adopting new AI workflows. An ex-engineering director at a large bank told me they quit their job to accelerate their career:

“I was not getting the opportunity to ‘close the loop’ on hypotheses enough. [...] To stay relevant in the industry, I feel like I need to pull out into the “fast lane.”

Like many others, I see the future of software development is with AI. If you don’t get hands-on with your team, working with AI tools day-in, day-out, you’re falling behind.

My plan is to get on the cutting edge of things through a mix of academia and consulting AI companies. I am not saying the plan is perfect, but I need more time to do things differently than I had in my job.”

Consider this: if you stay in your job for two more years, do you expect to find career opportunities at cutting-edge companies in the future? If the answer is “no”, then there’s a risk in just staying put. Joining an uncertain startup or taking a career break to develop AI expertise is also risky, but the outcomes may be more controllable than letting your skillset become outdated, relatively quickly.

But it might actually be necessary to quit in order to get AI experience: you might be able to get this by transferring to an IC role. As Charity Majors, co-founder and CTO of Honeycomb, said in last week’s episode of The Pragmatic Engineer podcast:

“You’ve got to get AI on your resume. You just have to. If you don’t, this is a huge career risk. If you’re working somewhere where you’re not getting these skills, I would do whatever I could to change that [including taking an IC role within the company].”

There are companies where moving from Director+ to individual contributor is possible, even if these companies are the minority. If you happen to work at a place like this: consider if you can and will take advantage of this opportunity.

Claire Vo – founder of ChatPRD and host of ‘How I AI’ podcast, and the former Chief Product & Technology Officer at LaunchDarkly – says most companies will never become “AI native” simply because most VP of Engineering or CTO folks don’t have what it takes to pull off such a transformation. In her words :

“The VPE role used to be primarily about deploying the dark arts to defend engineers from the roadmap, and now everyone thinks that’s BS and leaders are under tremendous pressure to inflect velocity or GTFO (get the f*** out).

Engineers are unhappy (don’t make me tokenmaxx, bro!), product and design sending slop PRs, and everyone good has left for a lab.

Most of these companies’ EPD (Engineering, Product, Design) orgs will never go AI-native, not even close. Most VPEs aren’t good enough at change management to pull it off.”

It looks like there’s a deadlock:

The current engineering org is frustrated by how AI is making engineering culture worse, morale is down, and people are frustrated and confused
To resolve this, drastic changes are needed to how everyone (engineers, product, designers) works
To pull it off, a VP of Engineering or CTO is needed who’s capable of this; someone excellent at change management, who’s ideally done it before.
But most VPEs and CTOs are not experts at large-scale change management, nor have done it before.

According to this, many VPEs and CTOs are doomed to fail at making the change they want, and it’s hard to know if that’s because organizations didn’t support them properly or resisted change.

There’s (usually) a honeymoon period in a new job, when we believe in the business we’ve joined and in its direction. But when this phase passes, a fraction or all of the problems described above may emerge, and there’s a decent chance that some of them are why your predecessor signed out:

Has AI helped make the role worse?
Is the equity on course to be worthless?
Is getting AI-native experience actually possible, or is the organization resisting change?

I’ve talked with a CTO who replaced their predecessor and founding CTO. A few years into the job, the predecessor CTO realized their equity in the business was worth almost nothing due to stalled growth, all while they were also being out-competed by AI-native rivals. So, the new CTO also resigned after a short, six-month tenure.

Two engineering leaders – a CTO and a VP of Engineering – mentioned “insane working hours” as a factor that contributed to them finally quitting. But there were other things as well:

The business struggling for growth
Their equity grant’s value shrinking to nothing before their eyes
CEO/founder ignoring or overriding efforts to help the business succeed

My sense is that at a thriving business during chaotic times like these, it’s unlikely that long hours alone would spur people to leave, if their contribution to current success counts and is valued. When things are going well, it’s possible to delegate more and take time to recharge batteries. But when things are going badly, it feels like every waking hour needs to be spent on working to turn things around.

Several engineering leaders are stepping back into IC roles for more stability because engineering teams are smaller now.

Karthik Hariharan, engineering leader at DoorDash, notes :

“Expectations have been shifting a lot in these roles, and a lot of folks qualified for them have consciously been stepping back into IC roles or joining bigger companies for stability and better compensation.

Engineering teams are also smaller now. A VPE isn’t needed until the team is large enough to require it. A technical founder can run the team for a lot longer these days.”

Some reasons why engineering teams have shrunk:

“Fullstack engineer” is mainstream, and was even before AI. Fullstack engineering was becoming relevant a few years ago in terms of a single engineer working on both the front and backends, instead of having a frontend engineer building the UI, and a backend engineer working on backend services. Fullstack frameworks like Next.js or Ruby on Rails made all this pretty easy before AI. Today with AI coding agents, you can rely on them to write decent code on platforms you’re unfamiliar with. There’s now little to no reason why a project would need multiple devs with different specializations.

It’s normal for one, or a maximum of two fullstack engineers, to be working on any given project at Anthropic as well. Head of Claude Platform, Katelyn Lesse, shared how it works at Anthropic:

“On an individual project, you often cannot have more than two people working on it.

This is because each engineer is already running several agents. And so as an engineer, you’re already fighting against your agents, which are stepping on each other’s toes on implementation. And in this setup, you just cannot have that many humans, who also come with all their agents!”

Frontend-only and native mobile teams are also getting smaller or disappearing. Even at companies where iOS and Android are a big part of the business, more places are building using cross-platform technologies where one engineer can do the work that used to need several. For example, social media app Bluesky had a single engineer build its web, iOS, and Android apps for launch by using React Native and Expo. Bluesky later hired more people to work on the web and apps, but they all work across these three platforms. It’s not the same as hiring separate web engineers, iOS engineers, and Android engineers.

We cover this in more detail in the deepdives Cross-platform mobile development and Is there a drop in native iOS and Android hiring at startups? We also observed a steep drop in frontend engineers and native mobile engineers in our latest state of the tech jobs market report:

Demand for frontend engineers and native iOS+Android engineers keeps dropping with the trend of smaller engineering teams. Source: The tech jobs market in 2026

Tech companies have been flattening their org structures for three years now. We first covered the trend for fewer middle managers back in 2023, when Meta drastically reduced manager positions. The trend has not stopped, and many – if not most – companies have increased the number of reports each engineering manager has, while reducing the number of layers in their organization.

The Pulse: Meta’s self-inflicted resignation-wave

Gergely Orosz — Fri, 14 Aug 2026 16:55:55 +0000

The Pulse is a series covering events, insights, and trends within Big Tech and startups.

Today, we cover:

Meta can’t stop the “resignation-wave” it triggered. In what was predictable: Meta’s layoffs and forced reassignments pushed engineers not impacted by either to look for a new job. Meta is now offering large equity retainers to keep these folks, and it doesn’t seem to be working.
Grok Bot: the “OpenClaw moment” for managed AI agents? The Cursor team built and released a generic AI harness that feels like the “Codex experience, but for knowledge work.” I tried it out, automated a lot of my daily workflows, and am a massive fan. More AI vendors will surely copy this harness.

Apologies for this week’s The Pulse arriving a day later than usual – our family got a puppy this week – who is beyond adorable –, but has kept me up a few nights, indirectly delaying this week’s The Pulse. We’re getting into a rhythm, so things should be back to normal, looking ahead.

Stop being skeptical about AI for development with Charity Majors

Gergely Orosz — Wed, 12 Aug 2026 16:45:07 +0000

Listen and watch now on YouTube , Apple , and Spotify . See the episode transcript at the top of this page, and timestamps for the episode at the bottom.

• Antithesis – turbocharge testing of your systems by running your whole system under aggressive fault injection. There’s good reason teams like Jane Street, Fly.io, and the etcd community rely on Antithesis. Learn more.

• Buildkite – the CI platform trusted by OpenAI, Anthropic, Cursor, Meta, Uber, NVIDIA , Airbnb and many more. When CI volume becomes an architecture problem, you deserve better CI. Engineered to reliably manage whatever your coding agents throw at the build queue: today, next year, and beyond. Learn more.

• WorkOS – make your app and agents Enterprise Ready, with SSO, SCIM, RBAC, and more. Get started .

In 2025, it was rational to be skeptical about AI, but in 2026 it’s clear that AI is changing all of the industry, and there’s less and less place for skepticism. This take is from one of my favorite voices in software reliability and observability: Charity Majors, CTO and cofounder of Honeycomb, co-author of Observability Engineering . (Note: the second edition of Observability Engineering is out, and it’s pretty much a full rewrite of the book, I recommend grabbing it if you’re building reliable systems)

In this episode, I sat down with Charity to discuss how her thinking on AI has evolved, why she believes it is becoming a foundational part of software engineering, and what that means for how teams build, review, and ship software.

We explore how AI is changing the economics of code generation, why reliability and verification are increasingly the bottlenecks, and why the rise of non-deterministic systems requires more engineering discipline. Charity shares her views on code reviews, observability, DevOps, leadership, and why both AI skeptics and enthusiasts are getting important things right.

Here are 13 parts I found especially interesting, talking with Charity:

1. In March 2025, Charity told the audience at SREcon to try vibe coding, and back then, the response was grumbling. Charity’s point was that people who are skeptical of AI should still learn to use it, because you can complain better if you’ve learned it. At this time, Charity still saw AI having a bigger impact than a new programming language, but was skeptical that it would have a generational impact.

2. Charity’s turning point in seeing AI as a generational change was in November 2025. This was due to Opus 4.5, but Charity argues that the coding harness (Claude Code) made the bigger difference. Because thanks to Claude Code, harnesses went from being more of a shell script to serious infrastructure.

3. The impact of AI on the industry in 2025 was similar to the impact of the cloud in 2010. Looking back, Charity is comfortable saying this: in 2010, it became clear that cloud computing was certainly going mainstream and would change the infra-layer. After 2025, it’s also clear that AI will have a similar impact on the infrastructure of building software.

4. Engineers who were skeptical of AI up to 2025: they had good reason to be so. This was because we’ve seen plenty of technologies and innovations in the past that all promised to transform the software industry, but later fell short. Examples include COBOL (a technology promising that programmers would no longer be needed to create software), neural nets, no-code and low-code tools.

5. The question engineers need to answer: what would it take for you to be fully comfortable shipping code you have not read? Charity believes it is a “when” and not an “if” that professional software engineers will ship code they never looked at – and thus do not understand – to production. Engineering is building the systems that validate this code, and allow shipping with full confidence.

6. AI could have the software industry go through the “pets” to “cattle” change that compute infra went through in the 2010s. Up to now, writing software from scratch was far more expensive than editing existing software. But now, generating hundreds of variants of a function can be done faster than how long it would take you to hand-write it once.

Charity believes that we might be at the beginning of the transition from “pets” to “cattle” that happened at the hardware infrastructure layer. Before the 2010s, configuring and repairing individual servers was commonly done. But with tools like Terraform and Kubernetes, individual servers having issues are no longer fixed up: they are re-created instead. Charity thinks the same might happen with code, sooner rather than later. When there’s an issue with the code, generate new code that solves it, and is verifyably correct.

7. Her contrarian take: code review is overrated, and the least valuable part of what humans add to software engineering. Charity says that humans are good at conversations and deciding what to build, not reading code to check for correctness, syntax and bugs.

8. Charity’s verdict of 20 years of DevOps: it failed. The DevOps feedback was about trying to create a feedback loop that connected people writing the code to the code running in production. She thinks that the “ops people: learn to code!” wave worked, but the “software engineers: understand your code in production” failed, to this day.

9. Non-deterministic systems require more engineering discipline versus before. With code written by AI, we’re reducing the trust in the code (because we no longer wrote it), so we need to increase trust at the other part of the development process. Specifically, at validation: with things like tests, evals, and conformance testing.

10. Charity’s career advice for engineering directors: run towards the waves, and get AI on your resume, immediately. It’s an anxious time to work in tech, thanks to all the change, driven by AI. Charity reminds us that anxiety and excitement are physiologically almost the same, but the difference is agency. When you have no agency, you’re more likely to get anxious, and when you do, you’re more likely to get excited.

So her advice to anxious engineering directors: consider going back to IC work, where you’ll have far more agency. IC work is well-respected, getting back to it has never been easier, but the window to do so is closing. As she put it:

“The next time you’ll have a job interview, you’ll be filtered out if you don’t have AI experience.”

11. On AI fatigue: take back control with small acts! We talked about various types of AI fatigue: reviewing AI slop, getting tired of the AI hype, and getting worn down by “doom trolling” by AI CEOs. Charity finds small acts of taking control back in your work from AI tools help. For example, none of the Honeycomb team uses AI on Wednesdays.

12. Charity would like to see both the “AI-pilled” and the “anti-AI” camps tell the stories better. As she put it:

“There are some really incredible things happening in software right now, for example, with rewrites and with automating away toil. Not a single person that I’ve talked to would give up using AI.

But half of the people are seeing the wins, and they’re not connecting it to the cost, which makes them think that their coworkers are just afraid of getting automated out of existence.

So that’s my beg to everyone who listens to this: tell the whole story! Talk about the costs as well. We’re all in it together.”

13. Charity’s rule on AI writing: do not send any message/email to a human that you yourself have not read in full. She also says that it would take them longer to read whatever you send than it took you to produce it: it’s probably slop!

• Shipping to production

• Deepdive: How 10 tech companies choose the next generation of dev tools

• Why is Meta destroying its engineering organization?

• When AI writes almost all code, what happens to software engineering?

• Are AI agents actually slowing us down?

• Observability: the present and future , with Charity Majors

• The third golden age of software engineering – thanks to AI , with Grady Booch

00:00 Intro

02:56 How Parse led to Honeycomb

06:00 The limits of individual productivity metrics

09:08 How Charity’s perspective on AI has evolved

13:50 Rewriting code vs. editing code

19:20 Production as a stage of development

22:14 Code reviews

26:56 Non-deterministic systems

31:11 Sensible uses of AI

37:41 The two AI camps

44:40 Why AI works so well for building software

49:42 DevOps

55:13 Modern observability

1:00:40 Handling context overload

1:01:56 What’s new in Observability Engineering’s 2nd edition

1:07:45 What effective leadership looks like

1:10:25 Engineering management: what is changing?

1:16:31 Junior engineers

1:18:01 AI fatigue

1:21:39 Book recommendations

Where to find Charity Majors:

• X: https://x.com/mipsytipsy

• LinkedIn: https://www.linkedin.com/in/charity-majors

• Website:

Mentions during the episode:

• Observability Engineering, 2nd Edition: https://www.oreilly.com/library/view/observability-engineering-2nd/9781098179915

• Honeycomb: https://www.honeycomb.io

• Linden Lab: https://lindenlab.com

• Second Life: https://secondlife.com

• Parse: https://en.wikipedia.org/wiki/Parse,_Inc .

• Scuba: https://research.facebook.com/publications/scuba-diving-into-data-at-facebook

• Can You Really Measure Individual Developer Productivity? - Ask the EM: https://blog.pragmaticengineer.com/can-you-measure-developer-productivity

• Let’s Talk Agentic Development: Spotify x Anthropic Live: https://engineering.atspotify.com/2026/4/anthropic-agentic-development

• Questionable Advice: Can Engineering Productivity Be Measured?:

• 2025 was for AI what 2010 was for cloud:

• AI demands more engineering discipline. Not less:

• The Phoenix Architecture: https://aicoding.leaflet.pub

• The third golden age of software engineering – thanks to AI, with Grady Booch: https://newsletter.pragmaticengineer.com/p/the-third-golden-age-of-software

• Software architecture with Grady Booch: https://newsletter.pragmaticengineer.com/p/software-architecture-with-grady-booch

• TypeScript, C# and Turbo Pascal with Anders Hejlsberg: https://newsletter.pragmaticengineer.com/p/typescript-c-and-turbo-pascal-with

• David Poll on LinkedIn: https://www.linkedin.com/in/depoll

• Intercom: https://www.intercom.com

• AI is approving our pull requests: Here’s how we made it safe: https://www.intercom.com/blog/ai-is-approving-our-pull-requests-heres-how-we-made-it-safe

• How AI will change software engineering – with Martin Fowler: https://newsletter.pragmaticengineer.com/p/martin-fowler

• HackerRank open sourced its ATS. My resume scored 90/100. Oh wait 74. No – 88: https://news.ycombinator.com/item?id=48713832

• AI enthusiasts are in a race against time, AI skeptics are in a race against entropy:

• Ep. #89, Software is the Killer App with Bryan Cantrill of 0xide Computer: https://www.honeycomb.io/resources/podcasts/ep-89-bryan-cantrill-software-is-the-killer-app

• Eric Riddoch’s post on LinkedIn: https://www.linkedin.com/posts/eric-riddoch_the-observability-engineering-book-has-share-7475807056285814785-pw4J

• Why traditional observability misses AI agent failure: https://www.dataiku.com/blog/traditional-observability-misses-ai-agent-failure

• Charity’s LinkedIn post on effective leaders: https://www.linkedin.com/posts/charity-majors_the-most-effective-leaders-are-kind-caring-share-7477160924928233472-qcLw

• Catastrophe Ethics: How to Choose Well in a World of Tough Choices: https://www.amazon.com/dp/0593471970

• More Everything Forever: AI Overlords, Space Empires, and Silicon Valley’s Crusade to Control the Fate of Humanity: https://www.amazon.com/More-Everything-Forever-Overlords-Humanity/dp/1541619595

—

Production and marketing by Pen Name .

Software engineering at a proprietary trading company: Optiver

Gergely Orosz — Tue, 11 Aug 2026 16:17:39 +0000

Before we start: I’ll be in New York, on 15 September, presenting the keynote at LDX3 New York , doing a book signing, and hanging out with attendees. The focus of the conference is engineering leadership at a time when things are moving very fast. See the full agenda and get tickets . If you’ll be around – hopefully catch you there!

The Pragmatic Engineer is back from our summer break. We resume with a detailed deepdive about the trading industry, and interesting engineering challenges that come when working at a company that has no external customers, but where a single, unfortunate enough software bug could wipe out the whole company.

In tech recruitment, proprietary trading companies have a particularly high bar and typically offer compensation on a par with, or even exceeding, Big Tech; right at the top of the market. That’s because for these market makers, success is all about gaining a competitive edge over rivals. Such competitive advantages today includes software that is superior to that at their competitors.

Software engineers tend to know little about trading companies – and this piece aims to change that. Trading companies build bespoke hardware stacks and have larger platform engineering teams than most workplaces. For software engineers, it’s a lucrative niche in terms of compensation, full-stack (hardware to software) work and for engineering challenges, and so we decided to go deeper in this interesting area.

In order to find out more, The Pragmatic Engineer sat down with a leading proprietary trading firm, Optiver . Headquartered in Amsterdam, they also have a large engineering presence in the US and globally. We met engineers and engineering leaders to learn in depth how engineering works in a modern trading business, with contributions from:

Alex Itkin : CTO, Optiver US
Pat Cooney : Head of Global Platform Engineering
David Gross : Technology Lead, Options

Thanks to everyone at Optiver for taking part in this report which lifts the lid on how software engineering is done when even nanoseconds can count. In this article, we look into a software engineering environment that’s distinct from what you expect at most startups and Big Tech. For example:

No external customers. Usually, companies have consumer customers (B2C), business customers (B2B), or both. But not trading houses like Optiver, where their own business is the customer. This is a different reality: there’s no external deadlines and related pressures, but personal motivation to improve is highly valued.
Latency: “enemy number one”. Nearly every major engineering decision at Optiver is made in the interest of minimizing latency – the amount of time between a request and response. This approach is present across the software stack and in kernel-level work. It’s why Optiver manufactures its own hardware.
Today, latency is the floor, and AI models are becoming a differentiator. Gone are the days of having lower latency than the competition allowing for arbitrage opportunities to make risk-free profits. Instead, information models are becoming a differentiator: slow models with a fast trigger sending signals to execute trades, and fast models running at the edge of the network making trade decisions realtime.
Haunted by a bug that nearly killed a business. Among trading houses, there’s a cautionary tale of when a peer company, Knight Capital , nearly went bankrupt after a single bug in a high-frequency trading system triggered a $440M loss.
Different incentives. The business is incentivized to move very fast, but with a high premium on caution in order to avert potential financial disasters on the market. This cautious attitude to risk in concert with chasing speed feels pretty distinct in tech.

I this deepdive, we cover:

Overview of trading & hedge funds. Categories of trading companies, high-frequency trading (HFT), plenty of ML & math, and AI labs poaching HFT talent
Engineering organization. How trading-specific roles work together, platform engineering, the “build and own” culture, and more.
Software tech stack. The three-layer tech stack, languages and tools, CI/CD stack and the data layer.
Hardware engineering, FPGAs and Silicon. Latency progression, custom FPGAs, custom hardware, AMD hardware partnership, and more.
Network & physical infrastructure. Physical infrastructure, dedicated fiber & wavelength leasing, optical cable, radio, data centers & co-locations, and why AI models matter more than ever before.
Engineering practices. Risk vs speed, knowledge-sharing culture, testing culture, monitoring & incident detection, risk management.
AI at Optiver. AI tooling stack, future of agentic coding, details about adoption, and how it all looks in practice.
Hiring, career development & culture. Engineering levels at Optiver, goingfrom hiring mostly juniors to hiring experienced engineers today, competition during hiring, and the onboarding feedback loop.

We’re delighted to publish this report, including details never shared before. Let’s dive in!

Here’s a summary of the world of ‘prop shops’; another name for firms like Optiver that invest their own funds in trading financial assets. Below are some useful mental models for understanding the sector.

Buy side: companies invest money and earn returns. Examples: hedge funds, asset managers, pension funds.
Sell side: firms sell services or products such as advice, underwriting, research, execution, etc. These are usually investment banks and broker-dealers.

Optiver is on the “buy side”, as a prop shop.

Trading categories based on capital source

Based on whose money is being traded, there are three main capital sources:

Investment banks serve corporate and institutional clients by raising capital, advising on deals, and executing trades on their behalf. Examples: Goldman Sachs, JPMorgan, Morgan Stanley.
Hedge funds raise money from external investors and trade it on their behalf, charging management & performance fees. Examples: Citadel, Millennium, Two Sigma, Bridgewater.
Proprietary trading firms trade only their own capital, with no clients or external funding. Examples: Optiver, Jane Street, Jump Trading, DRW, Hudson River Trading.

Optiver’s CTO US Alex Itkin pictures the evolution of trading as having unfolded across four eras to date:

Pre-electronic (pre-1990s). Trading was done face-to-face on noisy trading floors and by phone. Prices were shared on reels of ticker tape and printed in newspapers. Investors contacted brokers to place orders.
First wave of electronification (early/mid 1990s). Financial markets moved onto computer screens but orders were still entered manually.
Automated trading (late 1990s to ~2015). Computers did the same as human traders, but faster and at scale. This was the “mechanical” automation era of building automated workflows without data-driven decision-making.
Quantitative trading (~2015 to present). Data-driven decision-making with machine learning models and inference compute, with human decision-making in some key areas.

Each era “weeded” the market. Some companies excelled at automated trading but never made the leap to quantitative trading. According to Itkin, competition has got tougher over time, while the number of serious players has decreased. Today, there are only a handful of really big firms, and one reason for this is cost: investment in research clusters – which serious prop shops all do – requires hundreds of millions of dollars.

Optiver turned 40 years old in March 2026, launching in 1986 at the European Options Exchange. Today, the company has:

~2,200 employees
~950 engineers and ~1,000 traders and researchers
11 offices: Amsterdam (HQ), Chicago (US HQ), Austin, New York (2025), London, Sydney, Shanghai, Hong Kong, Singapore, Taipei, and Mumbai.
10M+ trades executed per day, across 100 exchanges
€4.5B ($5.1B) in trading income, and €1.7B ($1.95B) profit, as per 2025 financial results

Optiver is a mix of:

Market maker: providing liquidity on exchanges by quoting ‘buy’ and ‘sell’ prices of financial products and earning the spread between the two.
High-frequency trader: executing automated trading strategies at very low latency

High-frequency trading involves placing high volumes of orders at lightning speed in an effort to take advantage of extremely rapid market movements. In this domain, speed is the biggest advantage, and achieving it obviously involves high-performance computing. The basic trading loop is run millions of times a day. It’s made up of three steps:

Watch the market for new information like price changes
Decide what the information means and the right trade to make
Send a trade to the exchange before competitors do

In trading, timing is everything, and for some types of trade even nanoseconds count. Optiver’s fastest trading system operates in the realm of sub-nanosecond, where measurement noise becomes a challenge in itself. Software, hardware, and physics are all involved, along with microwave and shortwave links between data centers, and custom-manufactured chips.

We go deep into this in the “Hardware Engineering” section below.

However, in this niche, even ultra-low latency is no longer a competitive moat in itself. As competitors have squeezed performance out of their systems, focus has shifted towards fine-tuning of trading strategies. Today, Optiver invests substantially more in building better models than it does in lowering latencies. More on this in the “Network and physical infrastructure” section below.

HFT evolves faster than other industries. Profitable strategies don’t last long, opportunities are fleeting, and innovation is a constant. In this environment, a tool like AI is relatively straightforward to implement because trading houses like Optiver are well used to change in their daily business environment. More on this topic in the ‘Optiver & AI’ section.

There’s a big role for machine learning (ML) and mathematics in quantitative trading. A good chunk of Optiver’s business is the buying and selling of options , and the pricing of these rests on mathematical theorems like the Black-Scholes model . Traders, quants, and even software engineers building option-pricing strategies must understand the math of this problem space.

Over time, machine learning is becoming more important than math models, but it’s worth keeping in mind that trading is not purely an ML pursuit.

AI infra providers are heavily involved. NVIDIA, Groq, and Cerebras are actively courting trading firms, due to how much money they spend on GPUs. For example, see Hudson River Trading discussing Blackwell deployments at NVIDIA’s GTC conference, or Jump Trading being among the first to deploy next-gen Vera Rubin systems. HFT companies have very clear monetization paths for GPUs and spend large sums on hardware, hence why NVIDIA and other suppliers are keen to partner with them.

One new trend is AI labs like Anthropic and OpenAI recruiting from prop shops, defying the assumption that AI labs mostly recruit from Big Tech. There are a few reasons why AI labs seek out talent from the trading world:

Infra expertise. Prop shops like Optiver have spent decades operating their own data centers and deploying on-prem hardware at co-location facilities.
Custom, high-performance hardware. Prop shops also often build their own hardware and their kernel stacks achieve very low latencies. That’s a talent AI labs seek!
Skillsets. The highest-paying destinations for CS majors out of standout colleges are often prop shops, paying top-of-market compensation for standout talent. Outside of select colleges prop shops recruit from, however, there tends to be little awareness about these companies for new grads, or across the industry.

Optiver’s history can be seen as two distinct ages:

Regional systems (“unblock yourself”: 1986-2020): internal systems and platforms were built to serve local needs, such as building support for a market. Systems built exclusively for the US, Europe, or Asia were common.
Global platforms (“build for the whole company”: 2020-present): Optiver recently started to build new systems to work globally across their platform. This global focus is also why the company is investing a lot more in its platform engineering arm. A globalization push started around 2023, and its momentum has been growing.

The benefit of the old “unblock yourself” approach of local teams building whatever they needed, was that it enabled them to move fast and not get held up by dependencies. But this became problematic because of fragmentation and duplication, and the downsides became more visible over time:

Fragmentation: different teams use different technologies, frameworks, and infrastructure
Duplication: teams in different parts of the business independently build the same or very similar services

The career trajectory of Pat Cooney, Optiver’s head of platform engineering, mirrors the shift to a global platform: he was the CTO of Optiver in Europe in the mid-2010s when the business was split by region, and was appointed head of platform engineering in 2025 when that approach was replaced.

Optiver’s approach to continuous integration (CI) has also evolved. Previously, the company had several regional CI services, but from 2025, it started to rebuild its CI system with two new goals:

Build for scale: create a CI system built to scale across regions and stand the test of time
Use from any region: standardize deployment pipelines, so that code built in one location can run anywhere without friction

At Optiver, there are three main areas for tech roles:

Engineering: build and own the full trading-platform stack
Research: quantitative scientists who build models and predictive signals to create and improve trading algorithms. Typically, their background is in math, physics, economics, and statistics
Trading: quantitative traders who watch live markets, adjust trading system parameters in response to conditions, and build tools to automate decisions

In reality, the boundaries between these areas are porous. Yes, people do the job they were hired for, but it’s common to also see researchers roll up their sleeves and take part in implementing a trading strategy, or software engineers conducting research.

At Optiver, folks aren’t tied to one task

Cross-functional collaboration between roles is very common. For example, when developing market signals and associated trading strategies, it’s normal for engineers, researchers, and traders to collaborate on most, if not all, projects.

End-to-end ownership, plus autonomy, is a given. Engineers have autonomy in how they get things done, and they own and solve problems from the ideas stage through to implementation. There is a limited amount of guidance for trading, and it’s down to engineers to find the right solution.

In many ways, this approach to software engineering is pretty similar to startups’: software engineers get limited guidance and lots of autonomy. In order to succeed at tech startups, engineers typically need to understand the business, as well as being excellent at building production-ready software. It’s the same at Optiver, where understanding the business means understanding markets.

Before Optiver’s globalized platform efforts started seriously in ~2023, regions duplicated effort:

Multiple implementations of identical core logic
Each region had its own systems, frameworks, and infrastructure
Local teams built whatever they needed in an “unblock yourself” culture

But that’s all changed. An obvious sign of global platform efforts is the appointment of Optiver’s first global CTO, Lance Braunstein , who joined with a mandate to scale the platform.

Roughly 30-40% of Optiver’s 950 engineers work on the platform. In contrast, a more typical ratio at other large tech companies is for 15-20% of engineers to be dedicated to platform work.

Prior to the global platform, there was a lot more tolerance of development experience friction; new engineers could spend weeks checking out the codebase and getting their build system to work. This mindset has changed, with the platform team stressing user empathy and reducing friction on engineers’ journeys, like by setting up build pipelines for their software.

Now, the platform is beginning to reimagine itself as built for AI. As agents proliferate at Optiver, users are both humans and automated systems. The goal of this shift is to empower people to decompose work into workstreams and orchestrate agents. Two projects were launched earlier this year by the platform team for agentic work:

AI gateway: gives Optiver engineers access to models
MCP hosting platform: makes it easy for engineers to access internal systems and tools via agents

Trading teams at Optiver have three roles:

Traders decide strategy and make risk decisions
Researchers and quantitative analysts (“quants”) build hypotheses, pricing models, and run evaluations
Engineers build production systems

In practice, these roles overlap. This was true before the AI era, but it seems to be accelerating with AI adoption. Most traders and quants have STEM backgrounds without recent production coding experience. AI enables quantitatively-minded people to automate workflows with agents and to implement strategies.

Trading teams are organized by asset class and strategy. For example (asset classes in italic):

A large team is focused on a broad area like options
A team focused on cash markets and building strategies for exchange-traded funds (ETF) and stocks.
A team focused on machine learning (ML) and trading in the cash market.

Within larger teams, there are horizontal and vertical sub-teams. Horizontal teams take on challenges that impact any trading desk; for example, pricing is a horizontal team as the underlying mechanism is the same whether a soybean or an index fund being priced.

Vertical teams are similar to “tiger teams”, accelerators, and program teams at other companies. They focus on short-term goals attached to a few different desks in a location like the US, Amsterdam, Mumbai or Sydney.

Each team has a trading or research lead and a tech lead, who identify work for the team to do. The overall direction is set by a partnership structure, similar to an investment bank, but partners are not necessarily in charge of teams. At Optiver, partners are collections of senior people responsible for overall strategy.

Regardless of asset class or vertical, every trading team builds a version of a trading loop with four components.

Retrieval of market-related information
Collecting signals to work out which trades to execute
Execution of strategies (sending orders to market)
Intervention via a feedback loop, enabling a trader to monitor the system.

Optiver runs on an ownership culture, with the principle that the best engineers take work personally and care deeply about Optiver’s systems, decisions, and outcomes. Leaders want engineers to treat their projects as if they were CEOs of a company, and be responsible for design, build, rollout, shipping, or support. There is no notion of throwing work over the wall to a QA team.

Optiver’s ownership model:

Traders and engineers define problems together. Engineers design, build, test, deploy, and monitor a solution. There are hundreds of production changes daily
Design reviews for architectural decision-making. When an engineer has a project that entails architectural change to the stack, the engineer is responsible for bringing multiple options with the pros and cons to the team for consultation. The goal is to share information and knowledge, and to make decisions

Optiver pushes new hires and interns to develop ownership. From day one, engineers have something they own and are assigned a real project with mentoring support. Production code changes are an expectation for new hires. Within a year, a new hire becomes the experienced person in their domain, ramping up the next engineer. This is explicitly emphasized in Optiver’s onboarding materials:

Ownership is also baked into the interview process, with explicit questions about problem-solving, talking through trade-offs, and implementation.

Optiver started life with options trading. The word ‘Optiver’ is actually a Dutch portmanteau of “options” and “trader”, so it’s unsurprising that the options team is among the most developed parts of the operation, with engineers split across multiple locations. The organization is composed of both vertical and horizontal teams.

One of the technical systems for which the Options organization is responsible is the retreat system. When Optiver trades an option, that trade itself changes the price of the next quote on offer. The retreat system has to reprice the entire option surface (i.e., all options related to the one just traded). This is called a ‘retreat’.

In the case of S&P options, the option surface can consist of thousands of options that have to be updated. Ten years ago, the retreat process took seconds; now, through optimizations at every level of the stack, it’s down to nanoseconds.

How the ‘retreat system’ works, at a high-level

Retreat speed matters because everything changes as soon as a trade occurs: the original quote is stale and a trader needs to remove the bid from the exchange before anyone can exploit it. Faster firms can take advantage of others’ stale prices, leading to an adversarial market dynamic.

Vertical teams work on specific tactical problems related to local trading desks with a focus on immediate impact. But they are not short-term or temporary teams, even if they work on short-term problems. They’re empowered to solve the most important current problems, end-to-end. On the other hand, horizontal teams serve most desks, and have longer time horizons because they work on cross-cutting problems like pricing, market connectivity, or auto-trading.

Most trading software applications or services (aka “apps”) at Optiver can be simplified to the basic trading loop. The exchange where the trading takes place is part of the outside world from which signals are extracted:

The three layers of trading: signals, strategy and execution

This is the information-gathering phase where services collect market data such as prices and order book information, and also run various data calculations, such as pricing algorithms and machine learning pipelines. These signals are made available to strategy applications/services which decide how to trade.

A single trading strategy typically focuses on a particular class of assets and trades, and many different strategies run concurrently. The strategy sets what and how to trade, but doesn’t execute the trade; that’s the next step.

All strategies are enveloped by a risk management system that can block trades and stop individual strategies. To be effective, it has a broader view of the combined risk level of multiple strategies.

Risk mechanisms can include human oversight, with traders tweaking strategy parameters, and also automated monitoring that checks if apps are outputting orders within expected parameters, regardless of what the algorithm wants. The latter approach is essential in low latency strategies where faster-than-human reaction speeds are needed.

The execution step involves executing trades on exchanges. There’s a hard ‘separation of concerns’ principle where execution steps are only permitted to execute the trade. No additional logic is meant to run there.

In some market-making use cases where nanosecond-level latencies matter, much of this process may run within a single chip (FPGA or ASIC) where the strategy part can be memoized with precomputed responses for all expected input patterns. This is then burned into the hardware to minimize latency from when market information arrives until a trading order is issued.

All apps implementing the trading loop sit on top of a multi-layered internal platform:

Basic infrastructure layer: the stuff you’d see at most tech companies (CI/CD pipelines, k8s, Kafka, Postgres, etc), but they’re also customizing their stack. They run their own data centers, have custom hardware, custom Linux kernels, customized CI tooling, and databases.
Domain-specific infrastructure contains core trading-specific services such as trading data dictionaries, metadata on securities, and the trade booking system.

The three layers of Optiver’s tech stack. The ‘basic infra platform’ is similar to infrastructure at most other tech companies

Historically, most of this infrastructure was duplicated at each local office level when teams prioritized moving fast and independently over avoiding duplication. A centralized platform team has started consolidating these efforts in recent years.

Roughly 30-40% of the engineering headcount is allocated to the Platform team. This level of investment in the platform is beyond what you’d typically see in a tech company. That’s likely to remain the case for a while longer as they focus on improving the development experience, consolidating duplicated functionality, and catering to the specifics of their tech stack.

At a glance:

Language choices at Optiver are fairly standard for a financial institution: C++ for low latency applications, and Python for modeling, prototyping and internal tooling work.

However, looking closely at Optiver’s contributions to the Python ecosystem reveals that this language is not just a prototyping tool:

optiver-asyncpg : Optiver’s fork of a performance-focused async Python lib for Postgres
vulcan-py : Optiver’s own dependency manager for Python allows more granular control over indirect dependencies
opti-napalm : Optiver’s fork of a library for automating and simulating various network equipment

Optiver’s internal tooling also has strict performance requirements because traders use internal dashboards and tools to make time-sensitive trading decisions. Avoiding hand-offs between traders and engineers for reimplementation in C++ saves time, and empowers non-engineers to solve their problems directly, in line with the “unblock yourself” ethos.

Rust is starting to play a significant role in research tooling and service orchestration, likely driven by the performance requirements. It’s interesting to see Rust used in areas such as Python, as opposed to it replacing C++, which would be obvious given its focus on performance. It’s likely due to Optiver’s decades’ worth of investment in the low-latency C++ ecosystem, its deep integration with existing internal hardware, and being able to directly control things like memory allocation with C++.

Other languages used in some niche use cases include:

C# for building data-intensive trader-facing GUIs,
VHDL and SystemVerilog for FPGA development.

Much of the software that Optiver builds interacts with custom hardware, custom Linux kernels, and requires predictable compute performance for predictable results in performance tests. These are all constraints that the CI/CD stack has to operate within.

Optiver’s CI/CD runs on bare metal machines, with custom hardware installed, the right OS tweaks, and a well-understood performance profile. Interestingly, this means Optiver needs to plan capacity in advance for its CI/CD clusters in the same way as it plans capacity for production systems. This is tricky since AI-coding tools started boosting the number of builds an average engineer does in a day.

They chose GitHub Actions as their CI Platform for the seamless development experience with GitHub. Unfortunately, Actions doesn’t provide overall, system-level metrics like queue times and utilizations, which are critical information for planning CI cluster capacity. Therefore, they had to build a bespoke observability layer over GitHub Actions pipelines with GitHub webhooks.

When it comes to databases and storage systems in general, Optiver is a big user of Kafka, Postgres, and Databricks (the company built its entire data platform around this).

A few interesting details show the role of Postgres:

They contributed a new timestamp type to Postgres, allowing timestamps to be expressed with nanosecond precision. Few Postgres applications care about nanosecond-level precision, and this wasn’t available “out of the box”.
They built their own internal version of the NOTIFY - LISTEN mechanism called ‘PG Feed,’ based on Postgres’ write-ahead log. This is used for distributing high-fanout, latency-sensitive messages to clients like pricing and configuration data, whereas using something like Kafka may involve additional disk reads and writes, which imply unwanted latency.

Optiver generally picks industry-standard tooling, but heavily tweaks it to fit their specific performance needs. Not many tech companies of this size tweak Postgres or GitHub Actions, let alone Linux kernels!

Formal methods with Hillel Wayne

Gergely Orosz — Wed, 29 Jul 2026 16:22:31 +0000

Listen and watch now on YouTube , Apple and Spotify . See the episode transcript at the top of this page, and timestamps for the episode at the bottom.

• Antithesis — Turbocharge testing of your systems by running your whole system under aggressive fault injection. There’s good reason teams like Jane Street, Fly.io, and the etcd community rely on Antithesis. Learn more.

• WorkOS – make your app and agents Enterprise Ready, with SSO, SCIM, RBAC, and more. Get started .

• turbopuffer – A vector and full-text search engine built on object storage. It’s fast, cheap, and extremely scalable. I met their team in San Francisco, and am a fan of their “hardcore and whimsical” engineering culture, and how pragmatic their engineering philosophy is. Check them out.

There’s a popular theory that AI will finally make formal verification mainstream because mathematical proof of correctness will be needed when machines write most or all of the code. But will this happen? Today, I’m talking with one of the best people to tackle the prediction. Hillel Wayne is a formal methods consultant, educator, and author (his most recent book being Logic for Programmers ), who’s deeply interested in software history.

In this episode of The Pragmatic Engineer podcast, I sit down with Hillel to compare software engineering with traditional engineering, discuss where formal methods fit into modern software development, and we explore why they are essential for some of the world’s most complex systems. We cover the formal specification language, TLA+, walk through several formal verification tools, examine why distributed systems are so difficult to reason about, and look into whether AI will make formal methods accessible to more engineering teams.

1. Are we “real” engineers? After thorough research, Hillel has an answer. For The Crossover Project , Hillel interviewed ~20 people in different fields of traditional engineering and software engineering, and found plenty of similarities and differences. He concluded that the rigor needed in software engineering means we earn the right to the title of “engineer.”

2. Version control is unique to software engineering. Other fields of engineering have change management, but “traditional” engineers wish the concept of version control in software engineering existed in their fields because it’s far more sophisticated.

3. TLA+ is a formal specification language created by Leslie Lamport for designing and verifying systems. Lamport is a mathematician and creator of LaTeX, who wanted to create a language for modeling complex systems. The language represents the state machine of the system and every possible state it can transition to. From the initial state, the system enumerates to get to every reachable state and checks whether properties defined upfront apply to those states. In this episode, Hillel walks us through a demo with TLA+.

4. Amazon used TLA+ to find a bug almost impossible to locate without formal methods. In the paper How AWS uses formal methods , the AWS team shared that they’d found a complicated bug for which the shortest error trace to exhibit was 35 steps (!!). The bug passed unnoticed through extensive design review, code reviews, and testing. AWS concluded they wouldn’t have uncovered it if they’d stuck to conventional testing approaches.

5. Lack of practice makes most engineers bad at dealing with concurrency problems and race conditions. When a system has a race condition due to your code, you usually don’t find out until a few months later – if ever! In contrast, a system modeled in TLA+ can tell you about race conditions as soon as the tool is run, making it a fast feedback loop.

6. Why not use formal verification for everything, then? It’s because specs in the real world are a nightmare to write. Even a simple problem like “find the file in a directory that has the most lines” gets complicated when modeled with formal methods. We would have to answer questions like: ‘do we look at ASCII or UTF-8 new line characters, what about unreadable files, and Symlinks ?’ Without formal methods, we can write a simple verification that is right in 99%+ of cases. Formal methods require a lot of extra effort for the less than 1% of exotic use cases!

7. Hillel recommends most engineers adopt property-based testing, and stop there. Property-based tests mean defining properties which the test then throws thousands of inputs at, in order to stress test a system. Hillel is convinced that formal methods are a niche tool for most engineers, whereas property-based testing is the most practical approach for building robust software with this lightweight formal method.

8. AI won’t make formal verification mainstream, but will increase its use. As Hillel says, “AI bringing formal verification up from maybe 0.1% to 0.3% across the industry would still be huge!” He also finds that people who succeed at using AI to generate formal specs are often formal verification experts.

9. Hillel worries about the time-of-check vs time-of-use bug. It makes Hillel want to pull his hair out when he sees a time gap between the time of checking something (e.g., whether a bank account contains sufficient funds for withdrawals), and the action itself (e.g., withdrawing money). This category of bug is hard to defend against and can cause annoying issues in real-world systems.

10. Hillel worries less about job losses from AI and more about software becoming an “ordinary” job. Revisiting his 2025 predictions of the impact of AI on the tech industry, one of Hillel’s concerns is that software engineering in the future will be lower-paid and lower-prestige than today. At present, the range of software careers available is pretty magical, especially compared to “traditional” engineering roles. But will this last?

11. One of Hillel’s coolest projects: verifying train transponders. Beyond databases and distributed systems, he has also formally verified device firmware. One cool project was working on the electric beacons between rail tracks that pass traffic information to the control system. He found a really odd bug in one transponder system, and fixing it made the real-world system more reliable and safe.

12. One thing that software engineering could take from “traditional” engineering: books on “the fundamentals” which every professional in the field should know. One of Hillel’s favorite books is The First Snap-Fit Handbook , a nearly 500-page tome on those little clips that hold battery covers in place. He observes that while most industries have copious documentation for the most mundane topics, within software engineering there’s not even a book on how to version an API! We could learn from other fields about the value of documenting our own craft.

13: The “materials” in software engineering are freakishly consistent. All other engineering professions have to worry about the consistency of their materials; for example, electrical engineers work with resistors that offer resistance within 20% of 100 ohms across a thousand units, and only when operated within a given temperature range. In contrast, a program runs identically on any given computer in software engineering. Hillel argues that the variability we deal with in software, like versions, APIs, bugs with integrations, etc, are largely battles of our own making.

• How to debug large, distributed systems: Antithesis

• How AWS S3 is built

• Paying down tech debt

• How Big Tech does quality assurance (QA)

• Bug management that works

• Resiliency in distributed systems

00:00 Intro

04:32 The Crossover Project

11:37 What software engineering does better

15:30 What traditional engineering does better

18:17 Formal methods

29:32 TLA+: what it is and demo

36:58 TLA+ at Amazon

38:10 Ways distributed systems break

41:03 Formal methods and systems thinking

46:20 The value of learning math

50:23 What TLA+ is good for and isn’t

52:50 Alloy: a declarative language for software modeling

58:53 Other formal methods tools

1:01:24 Property-based testing

1:05:31 AI and the need for formal verification

1:12:29 Logic for programmers

1:14:35 Hillel’s 2025 prediction on AI’s impact

1:21:30 Book recommendation

Where to find Hillel Wayne:

• LinkedIn: linkedin.com/in/hillel-wayne

• Newsletter: https://buttondown.com/hillelwayne

• Website: https://www.hillelwayne.com

Mentions during the episode:

• The Crossover Project: https://www.hillelwayne.com/tags/crossover-project

• Blog Series: Real Software Engineering: https://vanderburg.org/blog/series/real-software-engineering

• Software Art Thou: Glenn Vanderburg — Real Software Engineering:

• New Austrian tunneling method: https://en.wikipedia.org/wiki/New_Austrian_tunneling_method

• The Design of Everyday Things: https://www.amazon.com/dp/0465050654

• The First Snap-Fit Handbook: Creating Attachments for Plastics Parts: https://www.amazon.com/dp/1569902798

• NuSMV: https://nusmv.fbk.eu/

• TLA+: https://github.com/tlaplus

• Use of Formal Methods at Amazon Web Services: https://lamport.azurewebsites.net/tla/formal-methods-amazon.pdf

• Common Sense Computing: From the Society of Mind to Digital Intuition and beyond: https://link.springer.com/chapter/10.1007/978-3-642-04391-8_33

• Alloy: https://alloytools.org

• Time-of-check to time-of-use: https://en.wikipedia.org/wiki/Time-of-check_to_time-of-use

• P: Formal Modeling and Analysis of Distributed Systems: https://github.com/p-org/P

• Quint: https://quint.sh

• PRISM: https://www.prismmodelchecker.org

• NuSMV: a new symbolic model checker: https://nusmv.fbk.eu

• I formally modeled Dreidel for no good reason: https://buttondown.com/hillelwayne/archive/i-formally-modeled-dreidel-for-no-good-reason

• Formally modeling dreidel, the sequel: https://buttondown.com/hillelwayne/archive/formally-modeling-dreidel-the-sequel

• Event-B: https://eventb-soton.github.io/en-us

• MCRL2: https://mcrl2.org/web/index.html

• KeYmaera X: https://keymaerax.org

• Dafny: https://dafny.org

• JML: https://www.openjml.org

• Frama-C: https://frama-c.com

• Ada SPARK: https://www.adacore.com/languages/spark

• The Coming AI Revolution in Distributed Systems: https://zfhuang99.github.io/github%20copilot/formal%20verification/tla+/2025/05/24/ai-revolution-in-distributed-systems.html

• CRAQ.tla: TLA+ specification of CRAQ (lamport-agent): https://github.com/zfhuang99/lamport-agent/blob/main/spec/CRAQ/CRAQ.tla

• My EuroSys 2026 paper is obsolete: https://claudiacauli.com/2026/03/08/my-eurosys-2026-paper-is-obsolete

• Situated Software — Clay Shirky (2004): https://gwern.net/doc/technology/2004-03-30-shirky-situatedsoftware.html

• Lamport Agent - AI-assisted Formal Specification: https://zfhuang99.github.io/github%20copilot/formal%20verification/tla+/2025/11/14/lamport-agent.html

• LLMs are bad at vibing specifications: https://buttondown.com/hillelwayne/archive/llms-are-bad-at-vibing-specifications

• Logic for Programmers: https://logicforprogrammers.com

• Engineering a Safer World: Systems Thinking Applied to Safety: https://www.amazon.com/dp/0262533693

• The following can all be true: https://www.linkedin.com/posts/hillel-wayne_the-following-can-all-be-true-1-vibe-coders-share-7341040573711073281-3V8C

• The third golden age of software engineering – thanks to AI, with Grady Booch: https://newsletter.pragmaticengineer.com/p/the-third-golden-age-of-software

• Data and Reality: A Timeless Perspective on Perceiving and Managing Information in Our Imprecise World: https://www.amazon.com/dp/1935504215

• Debugging: The 9 Indispensable Rules for Finding Even the Most Elusive Software and Hardware Problems: https://www.amazon.com/Debugging-Indispensable-Software-Hardware-Problems/dp/0814474578

—

Production and marketing by Pen Name .

How building software is changing at Anthropic

Gergely Orosz — Tue, 28 Jul 2026 15:49:30 +0000

Much-improved AI tooling is changing how we build software, and I want to take a peek into how the future of software engineering may unfold under its influence. What better place for that than with tech’s most “AI-pilled” teams: the AI labs themselves.

So, I visited the two leading AI labs to see how teams and engineers do things day to day. In this article and in an upcoming follow-up, I’ll share what I learned about how AI is reshaping software engineering principles many of us are accustomed to – and what’s stayed mostly the same despite the AI wave.

In a later article, we’ll compare findings from Anthropic and OpenAI to see what their ways of working might mean for the overall direction of software engineering.

Inside Anthropic’s HQ (left). AI development milestones framed on the wall (right)

Thanks to Anthropic for showing me inside their lab in San Francisco. I talked with four people:

Katelyn Lesse, Head of Engineering for Claude Platform, whose organization owns the infrastructure that Claude runs on
Jarred Sumner, creator of Bun, now at Anthropic on Bun and Claude Code
Thariq Shihipar, who works across Claude Code engineering and education
David Hershey, at Anthropic’s Applied AI organization in a role resembling a sales engineer, working with customers like Cursor, Cognition, and Perplexity

Thanks to them, I got a sense of where things are headed at the leading AI lab – and possibly for the wider industry.

Before we continue, The Pragmatic Engineer will be on summer break for the next week and a half. This means no Thursday article this week, and no articles next week. I appreciate your understanding and support!

Back to today’s deepdive, we cover:

Complex & long: Claude Managed Agents. One of the most complicated projects took the Claude Platform team six months to ship, and created a new primitive to use at the agent infra level. Infra projects still need re-architecting mid-way through and take time to get right.
Twelve-month project done in 11 days: Bun rewrite to Rust. Migrating a 500K+ line project to another language used to take a small team a year, making it impractical. With Fable and $165K of tokens, it recently took the creator of the project less than two weeks.
Changing engineering practices. Inside the AI lab with more than 3,500 employees, prototyping is more fluid, verification is more time-consuming than implementation, code review and testing are increasingly done by AI.
Team-level changes. Design is more ongoing and less upfront, teams work on more projects, a maximum of two engineers per project, and more.
Still the same: two-pizza teams, planning is important, PRDs are relevant in complex projects, context switching is a challenge, the ratio of time spent on coding vs testing not changing that much.
Changing the “standout” software engineer archetype? Deep understanding, including of a layer below what you work on, is valuable, along with the ability to coordinate work.
Will AI replace software engineering? The more hands-on software engineers get with AI at the lab, the less they fear their jobs are going away.

The Claude Platform team’s most complex project in the past year was building Claude Managed Agents, a pre-built harness for production agents that runs in the cloud on infrastructure managed by Anthropic, or on your team’s own infrastructure, with any sandbox you choose. The project took around six months from idea until launch in April. Katelyn Lesse, head of engineering for Claude Platform, shared the story.

With Katelyn Lesse, at Anthropic

This team sits between the model/accelerator layer (Claude models operate on GPUs) and the product/application layer (with products like Claude Code and Claude Cowork):

Where Claude Platform sits inside of Anthropic

Katelyn on what the Platform team does:

“We’re on the ‘token hot path.’ The prompt comes in, then we tokenize it. Then, things like safeguards and billing all happen within our layer.”

What the Claude team calls “Platform,” I think more of as “API.” Claude Platform operates the API, and owns responsibilities an API would have. Of course, the team does more than that, and Claude Managed Agents is one case we cover here.

The platform layer is being migrated from Python to Rust. Originally, this layer was written for Python for the “usual” reasons at AI companies: it’s a convenient language and AI researchers use Python already, which enables quick iteration. But Python is single-threaded, and at scale, when the API is under high load, it’s not as performant as Rust.

The project came together due to customers wanting their own “harness infrastructure”, says Katelyn:

“We started with a model where you get an API to define an agent, then you get an API to start a session with an agent. The reality of what the world wants and needs right now is people running their own infrastructure. So, we started to build a self-hosted sandbox.

But then, what we started to hear from lots of customers is that they’re trying to hack harnesses together, running their own “harness infrastructure,” and this gave us the idea for Claude Managed Agents.”

In this project, Katelyn said the single biggest matter was planning:

“There are products you can jump straight to prototyping, but then there are ones where you need to start by architecting it properly. For example, if we build a TypeScript CLI – which is pretty trivial for what needs to be built – we could go straight to prototyping. But with Claude Managed Agents, we needed to first figure out what we are doing.

Of course, we did some upfront prototyping for Managed Agents: hacking and spiking things. But prototyping itself was more about understanding the requirements.

Our planning process looked more like a typical pre-AI planning process. You know how every team has the project, where everyone comes up with some version of the same idea and people keep floating and circling it around until you finally do it? Managed Agents was this for our team. When we started the project, we had documents dating back up to two years about ideas and suggestions.

Post-planning, when the project officially kicked off, a PRD (product requirements document) was created:

“In the end, it was the Product Manager and the Tech Lead on our API Agents team who decided to pull the trigger and kick off this project. We’d get in a room, go through it, and get aligned. But it wasn’t just us: we’d have to align with teams around the business, other cloud providers, and other engineering teams. For example, we have a sandboxing team inside of the Platform org: and so this team was consulted on the design of Managed Agents, given this product would spawn a lot of sandboxes.

Just like before, we had a PRD, it was a Google Doc. We used a Google Doc because we needed to coordinate all interested people. This has not gone away.

Similarly, my product counterpart and I run product reviews.”

Some processes from before AI, like the PRD, are still useful in complex projects today, for getting large groups of people on the same page.

With planning complete, the team decided to do a “spike” and stress-test the idea and architecture, by building the backend of Claude Code on the web. Remote execution of code with an agent harness was a similarly shaped problem to the one they wanted to solve for customers. The thinking was to start by solving it for the Claude Code team before tackling it in a more generic way for customers.

Internal teams are more fluid than before AI. Katelyn:

“Pre-AI, we might have hit the Claude Code team up with a bunch of big requirements documents, and they would have then hit us back with another set of documents. Now it was much easier: someone on our team built a few components, took it over to the Claude Code team, and they started to hack around it. We could figure out how this component plugs into this part of their product, and the other way around. It was just a faster and easier process, getting this first internal version of the product up and running.

Aligning with other teams on interfaces remains important, and it’s easier.Back in the day, you’d have to come with a fully spec’d interface to use. Now, we could do it a lot more fluidly: we could stand up a stub service that shadowed traffic to start with, and iron out the interfaces with the Claude Code team as we went. They did some hacking on it and gave feedback, we made changes while building out the service under the interface, then went back to make it work.”

They launched a service for Claude Code’s mobile app to spin up a sandbox, boot up Claude Code and run it. The service went to production and the Claude Platform team took the learnings.

It’s likely a familiar scenario many engineers can relate to, that after planning a project and getting underway, you see that you’re going to need to change the architecture. It happened on this project, too.

The platform team ended up re-architecting Managed Agents based on learnings from the Claude Code “spike.” Re-architecting meant decoupling the “brain” of Claude and its harness from the “hands” (sandboxes & tools that perform actions) and the “session” (the log of events). Each became an interface that made few assumptions about each other.

High-level architecture of Claude Managed Agents after the re-architecture

The team also built an abstraction around vaults and credentials. Credentials can safely be stored inside a vault . All calls using credentials are made via a proxy which has a session token. It is the proxy that fetches the right credentials from the vault: the credentials are never seen by the agent, sandbox, or session. Credentials are only injected at the egress boundary when the service is invoked:

Adding credentials the harness never sees

Internal “dogfooding” helped surface hard problems to solve. A few examples:

Reliability and scalability: these are really hard to do well for agents because if connection to the sandbox is lost, the whole agent dies and you lose state
Credentials and access control: also hard and problematic, especially when first building the service

The Managed Agents team shared more about this re-architecting project .

The project took about six months, by no means a rapid process. Katelyn emphasized that pre-AI, a project like this would have probably been in the realm of two years. Managed Agents is one of the biggest projects the Claude Platform team has built, and more complex than it looks: for example, adding support for running agents on AWS, GCP and Azure.

As covered before, Jarred Sumner is the creator of Bun, a popular JavaScript runtime with 22 million monthly downloads currently and Claude Code as a dependency.

With Jarred Sumner (left), creator of Bun

Bun is written in Zig, a performant, productive language. However, it’s not memory safe and memory issues kept coming up . Jarred thought that rewriting the project to an also-performant, memory-safe language like Rust could be an option – except that rewrites like this turned out as follies in the past. Jarred (emphasis mine:)

“Historically, rewrites are a terrible idea. Excluding comments, Bun is 535,496 lines of Zig. A rewrite in another language would take a small team of engineers a full year. It would mean freezing bugfixes, security fixes or feature development for that time. The least risky approach to getting something shippable would be a mechanical port from Zig to Rust, with the minimal number of behavioral changes, using the exact same test suite we already use for testing Bun.

Fortunately, Bun’s own test suite is written in TypeScript which means it doesn’t depend on the runtime’s programming language.

A year of zero user-facing impact was not an option we could consider. So, enforcement through code style to fix stability issues was our best bet, and was our plan when we added Rust-inspired smart pointers to Bun’s codebase.

But honestly, I didn’t want to do it. Homegrown smart pointers offer worse ergonomics than Rust, with none of the guarantees.”

But then, Jarred asked if AI could do the heavy lifting and wondered how much the migration could be sped up. In the end, he completed the rewrite from start to merge in 11 days, using 64 parallel agents and $165,000 in tokens at API price. Here’s Jarred on how his AI-heavy rewrite compared:

“By hand, I think this would’ve taken three engineers with full context on the codebase about a year, during which time we wouldn’t be able to improve Node.js compatibility, fix bugs, fix security issues or implement new features. We never would’ve done that. The realistic alternative was to do nothing and keep fixing the bugs at the top of this post forever.”

There was a lot more to the project than typing out the “...make zero mistakes” prompt:

Jarred made a detailed plan and style guide on how to migrate
He set up the project so agents would not use Git worktrees which he found slow, but worked on different files in the same codebase
He created an orchestration system where each AI agent came up with suggestions of what to change, but did not make a change to the file to avoid conflicts; an orchestrator AI agent created the commits
The most time and tokens went on fixing the compile bugs, tests, and verifying that things worked
Bun itself has a very robust test harness: when all tests pass, it’s a high-confidence signal that the rewrite works
Crucially, Jarred is the ultimate domain expert in Bun: he created the project and knows the codebase better than anyone

The rewrite has been shipped to production and powers Claude Code today.

We cover a lot more on this in What can we learn from Bun’s rapid Rust rewrite with AI?

So, what has changed in how teams build software at Anthropic, compared to the pre-AI days? That’s the question of this article, and it seems that many things are different. Let’s go through it:

Some things as normal as breathing at AI labs like Anthropic stand out as different with an outside perspective:

Everyone runs multiple AI agents all the time. Running 3-10 parallel agents is a given. Folks I talked with had their agents running in the background or cloud.
No token budget, usage not tracked. One major difference between AI labs and everyone else is that there really is no token limit or token leaderboards that promote tokenmaxxing; people already use agents all the time.
Very high autonomy. Work is becoming more structured inside AI labs, but there’s still massive autonomy compared to Big Tech and most startups. When everyone has unlimited tokens, it’s pretty easy to prototype any idea.

It was several times faster to prototype early approaches for Claude Managed Agents. Similarly, “spiking” the Claude Code mobile backend implementation was much faster than pre-AI, Katelyn told me.

Jarred made a point about the split between implementation and validation in his 11-day rewrite to Rust. Roughly, it was:

Implementation of the Rust rewrite took far less time than fixing it up, then validating that it works as expected

The “implementation” part of rewriting the code from Zig to Rust took about 15% of the time, while 85% went on fixing things up: getting it to compile, fixing tests, verifying that it worked.

Thariq:

“We see that few tokens are spent on actual implementation. Most are spent on discovery of unknowns, prototyping, mocking, and then in verification and testing.”

Jarred’s Bun rewrite echoes this: he spent more tokens on fixing up the implementation and verifying that it worked than on the implementation itself!

Jarred:

“Critiquing the code and testing it with agents is a new approach we do a lot more of**.** I think a lot about trust when you merge a lot of code. How do you merge 100+ PRs a day, and make sure the code works? At this pace, you need to trust the code without the ability to read it all yourself. And I think it’s a few things:

Code review: it needs to be really good and automated. I’m clearly tooting our own horn here, but I find Claude’s code review to be really good. Claude’s code review catches bugs that would take me an hour of closely reading the code to figure out. The caveat is that it’s expensive!

Security scanning: for this Rust rewrite we did 11 runs of the Claude Security Scanner.

Fuzz testing: we’ve also been doing different types of fuzzing ( fuzz testing ), where we had Claude write a fuzzer for things like parser fuzzing.

Running out-of-process testing, where it happens in a different process/session from coding, is one way to build trust in the code. I expect more of this.”

Jarred described a new way he works:

“A new approach I’m using is fanning out a lot of the work to many Claudes at the same time. I did this with the Bun rewrite, but I use it for other work. This approach works very well for me, and I feel it’s pretty underused.”

Jarred listed several time-saving automations set up by the Bun team to run an active open-source project with a small team, while the team works on Claude Code:

Every time someone files an issue, Claude runs to try and reproduce the issue. If it succeeds, it starts another container, which then tries to fix the issue and submit a PR.
The agent tasked with submitting a PR has to write a test that fails in the system version (the one without the patch) of Bun, and passes in the debug build with the patch, before it is allowed to submit a PR
There are other automations, like if there is no test, the PR is auto-rejected; all linters are run: Claude Code review is run, CodeRabbit’s code review is run, and the agents go back and forth on the GitHub pull request

Pull requests are merged manually when all quality gates pass, but this could become automatic at some point. Once all the above checks pass, all (AI) code review comments are addressed, tests are added to new code, etc. As an interesting aside, a lot of GitHub activity is Claude talking to Claude!

Claude talking to Claude. Source: Bun

But manual merging may vanish in low-risk cases, at least for the Bun project. Jarred told me:

“Today, a person presses ‘merge’ but within a few months, I expect:

Automated reviewer LGTMs

→ another Claude with a fresh context window judges if it’s simple and low blast-radius

→ if it is: auto-merge!”

Inside Anthropic, the team keeps testing their priors. Thariq gave an interesting example:

“The thing with agents is that you have to revisit any assumptions you have made because it can change with a new model generation. For that reason, we deleted 80% of the Claude Code system prompt recently because the model has gotten smarter.

Using HTML is another assumption we needed to re-examine. HTML is one of those things which Claude is a lot smarter at than many of us expected. I’ve started preferring HTML as an output format over Markdown, and see this being used by others on the Claude Code team.

HTML can convey much richer information compared to markdown, HTML documents are easier to read and share.”

At Anthropic, there are also changes in how engineering teams operate, compared to pre-AI.

The Pulse: Quitting Spotify Podcasts over reliability

Gergely Orosz — Thu, 23 Jul 2026 15:59:37 +0000

The Pulse is a series covering events, insights, and trends within Big Tech and startups. Notice an interesting event or trend? Hit reply and share it with me.

Today, we cover:

Moving my video podcast off Spotify due to constant reliability issues. Spotify’s podcast platform has become chronically unreliable since the company’s leadership started boasting about high AI adoption. But competitors haven’t had similar issues, and so I have offboarded from Spotify.
Will Kimi K3 trigger US push for closed-source AI models? Moonshot AI’s latest open model, Kimi K3, is on par with Anthropic’s Fable 5. Could it lead to the US government regulating or banning Chinese open models to protect US labs?
AWS laughs off “heart attack” billing error. AWS customers were billed trillions more than they should have been, due to what was likely a conversion error. But instead of sharing an incident report, AWS saw the funny side.
Industry pulse. OpenAI’s unreleased model tried to hack HuggingFace to improve its test scores, X took more than a year to develop its new Android app, Google’s new AI model flops, and more.

Pushing software engineering limits with “napkin math”

Gergely Orosz — Tue, 21 Jul 2026 16:52:07 +0000

Hi, this is Gergely with the monthly, free issue of the Pragmatic Engineer Newsletter. In every issue, I cover Big Tech and startups through the lens of senior engineers and engineering leaders. If you’ve been forwarded this email, you can subscribe here .

After I recently interviewed Simon Eskildsen, co-founder and CEO of turbopuffer, on the main stage at the AI Engineer’s World Fair, many people at the event told me they found him relatable and inspiring for his choices to stick with one company for close to a decade, his belief in the power of “napkin math” to reveal why products run slowly or cost too much, and for his insight about how too much of VC funding is about ego, not business needs.

This article contains the most interesting parts from that conversation; in particular, the concept of “napkin math” – doing quick calculations to get rough answers – as a way to challenge existing systems to improve. The full 55-minute-long video of the discussion at the AI Engineer’s World Fair is available to watch:

Watch the full interview

Today, we cover:

Algorithmic programming speedrun. While in high school, Simon competed in the International Olympiad for Informatics (IOI) which pushed him to learn about writing correct programs that are fast and memory-efficient, and more.
Eight years of infra at Shopify. There are many upsides to longer tenure: Simon learned infrastructure concepts, dug deep into databases running across regions, and learned to write software that ages well.
“Napkin math” as a superpower. Simon became obsessed with finding the theoretical limits of compute operations, such as sending over data and reading bytes. This held him in good stead at Shopify, and then at his startup.
Origins of turbopuffer. When ChatGPT took off, context windows were small, and so stuffing them with the right information was key for AI applications. Fast search was needed, but the search solutions were surprisingly expensive. Using “napkin math”, Simon discovered they were far more expensive than necessary.
A new product without VC funding & Cursor as customer no.1. After raising $8M in seed funding, Cursor rolled the dice on the new turbopuffer team after Simon helped with their search & database needs.
Reasons to raise venture capital. Fund R&D, fund growth, stroke founders’ egos, and more.

Disclaimer: turbopuffer is a season sponsor of the podcast , but as with all our deepdives, this article is independent of podcast sponsorships.

A self-taught professional, Simon skipped college to work at Shopify and spent nearly a decade there building a variety of systems. His interest in computers started after initially getting into building websites aged 12. Growing up in Denmark, he dabbled in HTML with tools like Microsoft FrontPage and Adobe’s Dreamweaver.

While still a teenager, Simon “hit a wall” by exhausting the Danish-language part of the internet for learning programming, and got into the World of Warcraft MMO game, which helped him acquire English.

After discovering the International Olympiad for Informatics (IOI), he decided to enter, despite a very competitive, multi-stage selection process open to all Danish high schools. IOI problems are pretty similar to Leetcode problems: algorithmic challenges that value correctness, speed, and memory usage.

Simon cleared the online qualification round and was invited to the Danish Nationals. This was more than a competition: a weekend-long bootcamp to teach participants more advanced programming techniques such as recursion (which Simon already knew), the divide-and-conquer algorithm , and dynamic programming ; one of the more tricky concepts to master for algorithmic programming.

I tip my hat to the organizers for creating a challenging bootcamp and offering the opportunity for people like Simon, who wasn’t aware of what “NP complete” meant when he entered. He recalls :

“The routine was that every four hours we’d be introduced to a new “programming concept”, and receive ~2-6 tasks where this, combined with previously introduced concepts, had to be applied. All the solutions had to be submitted to the same site I submitted my qualification solutions to, as it was all part of the final evaluation. The tasks were incredibly challenging, like nothing I had ever tried before.

Sometimes in extreme desperation combined with tiredness from the trip, I’d think about taking the next train home. This feeling would disappear with the utter joy and confidence that arose whenever I would finally solve a task, and creep back once again when I found myself still struggling after an hour on a new problem. But this kept me going. By Saturday afternoon, I had almost managed to get up to speed with the others, and was doing the same tasks as them.”

Ultimately, Simon claimed one of six spots in the Danish national team, making it through to the regional finals.

Aware of how little he knew about programming, Simon doubled down to catch up. He realized that most other participants were better prepared and that he had to do something to survive the next qualifying round. So, he got to work:

Simon’s desk with Donald Knuth’s “The Art of Computer Programming,” and the training week he created

As Simon recalled:

“I armed myself with a borrowed copy of “The Art of Computer Programming”, worked through the exercises, read up on common algorithms on Wikipedia, completed tasks on USACO, and memorized the critical parts of my Vim config for the competition computers. I managed to create quite an intense training weekend for myself.”

The regional finals were even more challenging than the Nationals, and participants struggled to write performant solutions to the problems. Despite putting in the effort, Simon was pretty sure he was out of the competition, and recorded his learnings:

“[From the programming competition] I learned that you must avoid digging holes. Repeatedly, I found myself so fixated on getting a particular idea to work that I’d get absolutely nowhere. Sometimes, you have to bite the bullet, delete your program, find a new sheet of paper, and start from scratch. A good case of this is when you start working around a general solution to solve specific edge-cases. I learned that there is almost always a simple way to solve a problem without explicitly handling edge-cases. If there are two edge-cases, there’s almost certainly two more. The simple solution will handle edge-cases automatically – even those you might not have considered.”

But as it turned out, two months later he was told he had been selected to represent Denmark in the final phase of the competition. Simon continued to push himself out of the comfort zone of web development, using HTML, a bit of PHP, and getting into algorithmic programming.

Another decision that would later bear fruit career-wise was starting a blog while at high school – though he didn’t know it at the time. In 2010, Simon launched his English-language blog but posted only one or two articles per year; mostly short ones describing problems he’d solved:

A five-line script to take screenshots and upload them to Imgur, all from the command line (2010)
Setting up a Ruby HTTP server on NGINX using the library Unicorn (2010)
Why Simon favors SQL over ORMs: Object-Relational Mappings (ORMs) are frameworks to abstract SQL, these were very popular around that time (2011)

Two posts he published would go on to have an impact on his life. One was about his IOI experience and learnings . The other only came about after he suffered the disaster of fatally dropping his iPhone during his final year in high school.

With his smartphone dead, Simon was forced to switch to an old-school Nokia “dumb” phone. He wrote a short article about the experience titled Why I’m glad my iPhone broke , which went viral on Hacker News , making it to the front page of the site with many comments.

It seems that someone at Shopify, in Canada, read the article and others by him – including the summary of his impressive performance at the International Olympiad for Informatics – as a recruiter from Shopify flew Simon out to interview in Ottawa, Canada, where he was offered a software engineering position at the company.

In his new job on the infrastructure team, Simon made notes about things he didn’t understand and read up on new concepts. As he told me:

“When I started at Shopify, I was insecure about having not studied computer science and my biggest exposure to programming had been the IOI. If nothing else, preparing for the IOI taught me that you can sit down, read a paper, and figure it out if you spend enough time on it. So, I did that repeatedly.

In my first year at Shopify, every time I heard something I didn’t know, I noted it down on a piece of paper. Then, that evening, I would read up about it.

For example, if someone at work mentioned TCP, I assumed that surely they would know exactly what’s in the three-way handshake and how TLS is layered on top. And I also assumed they’d looked at Wireshark and all of that. I don’t think that’s true, but that’s what I thought at the time! So, I dug deep into everything I encountered.”

Working in infrastructure meant solving interesting engineering problems at a time when Shopify was growing 120-140% in load year-on-year; Simon was exposed to problem domains like:

Ruby on Rails and databases: Shopify was already one of the world’s largest Ruby on Rails monolith applications, and working on infrastructure meant being close to the database layer
Sharding and cutting over: as Simon’s manager used to say, “you cannot cache writes”, so Shopify had to move from running on a single group of machines to a shard (partitioned) setup of machines. The team did the cut-over (moving to the shard) just a week before Black Friday, the busiest time of the year.
Multi-data centers: expanding database footprint from one data center (DC) to multiple DCs
Splitting up key services: Shopify had a 128GB machine (massive for the time) running Redis as a key-value store. Nobody dared touch it until one day the service went down. Then, the team split responsibilities up into separate services.

Simon built a framework to simulate networking conditions called toxiproxy . The idea came to him while attempting to write a more thorough, systems-level test to see how Shopify’s application held up during partial outages. He created a matrix of Shopify’s services, and wanted to write a complete test suite for this matrix, to see if the system was resilient enough to handle some parts of the system being down. For example:

An example of a test case Simon wanted to run

As Simon wrote about this problem at the time:

“Having tests for the matrix was a must; otherwise we couldn’t guarantee the state of the matrix wouldn’t degrade over time. Since the tools mentioned previously require root access, we investigated proxies to simulate latency and downtime at the TCP level, but didn’t find one that suited our needs. We needed an online API to edit proxies and to support deterministic latencies, which made it suitable for integration testing.”

This was the inspiration that led to toxiproxy. As Simon told me on stage:

“Toxiproxy is a proxy that sits in between the application and the databases. With the proxy in place, you could do things like make an API call to the proxy, instructing it to simulate taking the database down, or making it slow.

Over time, we added a bunch of other ways to inject failures. With this proxy, we did not have to mock low-level drivers, but we could test failure handling really well.”

Toxiproxy was open sourced in 2014, and apparently still runs in Shopify’s CI system 12 years later, to Simon’s knowledge!

The biggest benefit of a long tenure at a company was learning to write software that ages well. Simon told me that this was the lesson that made it so worthwhile, and how often the simple solution someone put together in a week or two outlasted the big, multi-team RFC-driven solutions. That was a lesson he still uses today.

While at Shopify, Simon became interested in figuring out the theoretical limits of certain computer operations. For example, how much bandwidth does DRAM have for memory transfer? How long does a round-trip operation to AWS S3 take, and how much does it cost? What does a gigabyte of memory cost?

He sought the actual numbers, wrote a script to collect them, and created a table with the data:

Simon started to memorize key numbers with flash cards. He wanted to be able to instantly recall all important numbers. As he told me:

“Napkin math was essentially just this table that I maintain on GitHub. There’s probably like 50 of these numbers and then a script that generates them all.

For example, what does a gigabyte of memory cost? $2. What does a gigabyte of S3 cost? Two cents. What does a gigabyte of this cost? 10 cents. What does it cost on spot? What does it cost on a three-year commit? I had a massive table, then I created flashcards for almost every single cell so I know all these numbers.

This was a project I started taking on at Shopify because I would review projects and these numbers would be helpful.”

Napkin math enabled him to challenge design decisions based on benchmarks that had issues. As he told me:

“When reviewing a project, a product team would tell me that they chose Database A over Database B because they benchmarked both, and Database A was better. I hate benchmarks because making design decisions based on benchmarks is not a satisfying answer to me.

For example, you’re saying that with Database A, as per the benchmark, it takes 10 seconds to do this one thing. But it should take 10 milliseconds if you do the napkin math. Say, it’s a search query. Okay, you’re searching for three terms then. Each term has this many documents that match it. That’s this many megabytes. We intersect this many lists. You have DRAM bandwidth on multiple cores, at 100 GB/sec. So if you do the math, it should be 10 milliseconds.

So, now you’re telling me that the benchmark for the same thing takes 10 seconds, which means one of us is wrong! Either there’s a gap in my understanding – which is possible! – or the benchmark measures the wrong thing.

Often, it would be things like the person doing the benchmark and not realizing that the query would be a distributed one, running across a hundred different nodes. And in this case, of course the p99 is going to be very high!”

Simon went even deeper into “napkin math” after leaving Shopify. He investigated whether MySQL’s maximum transactions per second is equivalent to fsyncs per second (the number of file writes per second a system can handle). He discovered that MySQL can handle more writes than the operating system could sync to the file, which was a surprise:

“It takes ~3 seconds to perform 16,000 insertions, or ~5,300 insertions per second. This is 5x more than the 1,000 fsyncs per second our napkin math told us would be the theoretical maximum transactional throughput!

Typically, with napkin math we aim for being within an order of magnitude, which we are. But when I do napkin math, it usually establishes a lower bound for the system, i.e., from first-principles, how fast could this system perform in ideal circumstances?

Rarely is the system 5x faster than the napkin math says. When we identify a significant-looking gap between the real-life performance and the expected performance, I call it the “first-principle gap.” This is where curiosity sets in. It typically means there’s (1) an opportunity to improve the system, or (2) a flaw in our model of the system. In this case, only (2) makes sense, because the system is faster than we predicted.”

Simon started to investigate , and learned that MySQL does grouping of transactions (not doing an fsync for every write), and it also does smart merging of multiple fsync operations that would be processed in parallel, effectively doing a “group commit” to further improve performance. This is a learning he’d later use: grouping writes together on top of S3, to reduce latency and improve cost.

Personally, I find it amazing how much you can learn with some measurements and by asking questions about how a system achieves results that go against what the “napkin math” suggested should be its limit!

“Napkin math” also played a part in the creation of turbopuffer. Simon told me that three things had to combine for the project to come to life.

1. Search was a difficult project at Shopify. Building search was one of Simon’s final projects at the company, and he did not have a good time with it. He used a popular database vendor but struggled to have it perform napkin math. The query plans were not exposed easily by the database, so he could not figure out what was missing and why it wasn’t performing the napkin math without reading massive amounts of source code. Also, the search infra was difficult to operate.

2. Napkin math became a surprisingly efficient reasoning tool. As a toolkit, napkin math gave Simon a way to reason about what could be achievable with a machine, if utilized perfectly.

3. His friends’ startups needed fast search badly because of AI. In early 2023, Simon helped a few friends with infra at their startups, many of whom had the same problem: LLM context windows were very small (4-8 KB), and they needed to fill them with the right parts of documents, but this demanded very fast search. One startup budgeted that their new search vendor would cost $30K/month, while the existing infra bill was only $5K/month! For a bootstrapped Canadian company, the cost was too high, so they didn’t ship the AI feature.

Simon could not stop thinking about why search was so expensive, and why it didn’t line up with what the “napkin math” predicted the cost should be. Eventually, he laid out a basic architecture to make it fast and cheap:

Store the data to search in AWS S3
Do some clustering and organize the files
Get latency down. S3 is cheap and reliable, but has high latency

Driven by curiosity about how that could work, he set out to build it. As he told me:

“I just became fully obsessed that summer [of 2023] with building it. The first version was the simplest possible thing. I’m a very pragmatic person, so I didn’t get buried in detail. I barely read the literature on log-structured merge ( LSM ).

The simplest way to do this is to run a clustering algorithm on the vectors. You get the clusters, and then put the clusters in files. The files are called ‘cluster_01’, ‘cluster_02’, ‘cluster_03.’ Then you have a file called centroids of the clusters. Then, you search by downloading centroids, looking into them, then downloading the closest clusters.

There were a few optimizations around merging clusters that were adjacent in files, just to control costs and boost performance. But this was the core of it.”

What about performance? Simon:

“For the first version, I didn’t even implement a dedicated caching layer. I just put the reverse proxy (NGINX) in front of S3, and that was it! The performance improvement came from caching all of the S3 objects. It was the simplest possible layer at the NGINX level.”

I first heard of turbopuffer last year, when I interviewed Cursor cofounder Sualeh Asif about how they built the AI coding harness. From our previously published Cursor deepdive :

“The ongoing need to re-shard as usage grew became error-prone and frustrating. Sualeh told me the biggest lesson the team learned was to avoid sharding where possible in future. So, they looked for a vector database that could support multi-tenancy without requiring manual sharding.

Turbopuffer was a startup that promised this, so Cursor tried it and migrated a good part of their vector search use cases over.”

So, how did a billion-dollar startup come to bet on a tiny, unproven infra product with zero customers? There were a few factors:

First, Cursor was not yet a billion-dollar company in the fall of 2023, but a relatively fresh startup. The company raised $8M in seed funding in the same month. In 2024, the company was valued at $400M, and then its valuation surged to $29.3B. It was sold to SpaceX for $60B this year.

Secondly, it involved a tweet Simon posted after having had enough of building all summer, and wanting to test the waters. As he told me:

“I was so sick of working on [turbopuffer]. I’d been working on this all summer and didn’t know if anyone cared. I only wanted to work on it if anyone cared. So, I decided to put it on Twitter.

At that point, I had a single TMUX instance running on an 8-core node somewhere in GCP. I was thinking: “if someone goes to prod, I’ll set it up properly on multiple nodes.

But first, let’s see if anyone cares.

Anyone who’s worked in the internals of databases would’ve had too much pride to ship anything like that; I was just releasing it like a SaaS project. Why can’t you work on a database like it’s SaaS?”

So, he put out this confident-sounding tweet:

Simon had the confidence to launch his product because he knew it was rock-solid and scalable:

“I knew turbopuffer was reliable. It upheld its invariants. For example, you shut down all the VMs and no data is lost. All the writes are committed directly to blob storage.”

Cursor reached out. The company was an eight-person team at the time, fresh from seed funding and growing fast, but with a search problem that they were looking to partner with a startup on. Simon recounts:

“Knowing Cursor’s founders now, I’m sure they must have sat at the dinner table one day and were like: ‘the unit economics of what we have right now, where all the vectors are in DRAM, are not working.’

They probably were asking why someone had not built a solution where you can put all the vectors from the codebase into S3 (to make it cheap), and move the part of the codebase used in memory (to make it fast). Then, everything sits in blob stores and you just hot load it all into cache. When you open the codebase, after a few seconds it’s all in RAM and the queries are as fast as everything else.

Aman (one of the cofounders) was already talking about using S3 as a key-value cache, which at the time, barely anyone was thinking of for unit economics.”

Simon knew nothing about B2B sales at that point, and was not trying to “sell” anything to Cursor. He exchanged a few emails with the Cursor team about their use case of searching many small local codebases. He wanted to help Cursor with the unit economics, while also proving that turbopuffer works. So, he hopped on a flight from Canada to San Francisco, arrived at Cursor’s office and got down to debugging their database provider:

“When I showed up at Cursor’s office, they were having some Postgres problems. I asked, ‘do you guys have pganalyze ?’ And they didn’t, so I was like, ‘Okay, let’s get that going. Let’s look at it. “

And the problem was the same thing as it always is with Postgres: autovacuum hadn’t run enough. And so they had all of these going to heap , when they should be doing index scans, etc. So, we were talking about all of that.

I was just helping them: my “database genes” just kicked in. I think this built enough trust with them to believe that if I know enough to help them with their database, maybe I also know how to build one.”

It was around when turbopuffer’s other cofounder, Justine Li , joined that the work with Cursor kicked off. Simon describes Justine as the best engineer he ever worked with at Shopify, and together, they made more performance optimizations. Cursor migrated their local code-base search over to this brand new product.

Simon promised to reduce Cursor’s bill by 95%, and delivered – thanks to napkin math! Cursor was spending around $80K/month on indexing and search; with turbopuffer in place, this dropped to $4K/month. Simon was confident about making this prediction thanks to napkin math! He did the calculations based on the fundamentals; only counting resource usage that Cursor was using, and not taking a margin for turbopuffer’s operating cost at the time.

After securing Cursor as the first customer, Simon was not convinced that turbopuffer should raise VC money. He explained his thinking:

“I understood that if you take venture capital, no matter how many smiles there are in the room, everyone’s expecting to earn a big return on some timeline that makes sense to everyone involved. And ‘everyone involved’ are pension funds in Canada.

But at the time, I did not know if turbopuffer could be a billion dollar company. It felt like a very niche kind of search engine. And that was completely fine with me!

So, I just looked at what we invoiced and I looked at my GCP bill. And I was making this equation that:

Customer

*Invoices >= GCP bill [the turbopuffer costs]*Justine and I were going to optimize the system until these numbers were roughly equal. And if we could get some other workloads over to turbopuffer, we could then start paying ourselves.

But either way, I didn’t know if I could even go and raise a bunch of money. I didn’t have any relationships. I was an outsider who grew up in Aarhus, Denmark, and moved to Canada.”

A few months later, turbopuffer ended up raising a total of $700K, just to be able to hire two engineers until the end of the year on real salaries. Simon found that most VCs did not take them seriously for wanting to raise too little money, and assumed this signified a lack of ambition! Two years later, with less than $1M in initial funding raised, turbopuffer crossed $100M in annual run rate. So, there evidently was ambition!

Since our conversation, Simon has reflected on the topic of VC capital and built a mental model of six reasons that justify raising external funding:

Reason #1: Research and development. This is what turbopuffer raised for: they needed the money to hire two more engineers until the end of the year, in order to build out more of the platform. Then, they would either get more customers and generate enough revenue to not need funding, or shut down the project if it failed to gain traction.

Reason #2: Growth. When you’ve built something and want to tell the world about it, often, you have to spend money on doing that.

Reason #3: Massaging founders’ egos. Simon:

“This is a very, very popular reason! You see big numbers, you get lots of press. But I think it’s a really dangerous reason to raise money.

I wish this reason was talked about more, because you are diluting all of your employees when you do it. And for some people, it can become a status game. It’s not what we are about: we’re here to build a big business together.”

Reason #4: Employee rewards. A startup is a long journey, and everyone wants to work with the best people. But there are necessarily few of these folks, so you want to reward the standout ones by raising money so that employees can sell their shares. This was why, in December 2025, turbopuffer raised a round where they let employees sell some of their equity: it meant they would not have to wait for a future financial event like an IPO.

Reason #5: Strategic partnerships. It might be the case that raising from a VC whose network of connections is key to your business succeeding, or that taking funding from a strategic investor gives you access to their platform or services.

Reason #6: Mergers & acquisitions (M&A). Purchase another company with the funding in order to expand the business.

Simon encourages founders to be honest about why they raise capital funding – and to watch out if the reason is egocentric!

Watch the full interview here .

It’s inspiring to see how far it’s possible to get with “napkin math” and by understanding the bottom layers. It didn’t sit right with Simon to make decisions about vendors based on hastily written benchmarks that might measure the wrong thing. By understanding the constraints of data transfer latency and storage cost, he found a way to estimate the theoretical lower bounds of the system.

This helped lead to more informed design decisions at Shopify, and showed him there was an opportunity to build a faster, cheaper search product than the status quo.

Getting “lucky” in business requires a lot of skill, and in-person greatly helps with first impressions. Turbopuffer’s first customer being Cursor sounds almost too good to be true. But the account from Cursor cofounder Sualeh Asif – and now from Simon – reveals the ingredients in more detail:

Spotting a new business need: AI-native startups like Cursor saw their search bills explode, but needed search functionality to offer usable AI products
Offering a product with a magnitude of lower pricing: what people made suddenly pay attention to turbopuffer was the promise of not “just 20-50%” cost savings, but a seemingly radical, 90%+ reduction in costs. When you’re late to enter a market (like search), you need major differentiation, then deliver on it! This is also what Cursor’s attention.
Building trust before a sale: Simon helped the Cursor team fix their existing database before discussing using his product
In-person impressions: if Simon had not flown to San Francisco to meet the Cursor team in person, would they have taken a bet on turbopuffer?
Launching at the earliest opportunity: none of this would have happened if Simon did not announce the first version of the product when he knew it could work, but was still in a pretty unpolished state!

It’s always helpful to be aware of the dynamic of venture funding – some of which are rarely mentioned. Simon knew investors expect returns and growth on timelines which they set. Raising money is helpful in many situations, but problems arise from egotistical reasoning, and Simon believes too many founders prioritize them – knowingly or not.

No posts