{
  "@context": "https://schema.org",
  "@id": "https://aggr.aymericbeaumet.com/items/newsletter-pragmaticengineer-com/2026-09-30-distributed-databases-with-peter-mattis/#webpage",
  "@type": [
    "WebPage",
    "ArchiveComponent"
  ],
  "articleBody": "**Listen and watch now on [YouTube](https://youtu.be/0GzwuYGvKA4), [Apple](https://podcasts.apple.com/us/podcast/the-pragmatic-engineer/id1769051199), and [Spotify](https://open.spotify.com/show/2Bho9xCbOQMWMJ7UKmqCzD).** See the episode transcript at the top of this page, and timestamps for the episode at the bottom.\n\n[![](https://substackcdn.com/image/fetch/$s_!Qhi0!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1228ff6a-514b-4643-9360-02cda14dc5ac_800x70.png)](https://substackcdn.com/image/fetch/$s_!Qhi0!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1228ff6a-514b-4643-9360-02cda14dc5ac_800x70.png)\n\n• **[turbopuffer](http://turbopuffer.com/pragmatic)** – The turbopuffer engineering team is doing something really cool: they are completely redesigning their storage architecture from first principles to make search faster, cheaper, and more reliable at scale. And they’re documenting all of it! Follow along their rewrite at [turbopuffer.com/v3](http://turbopuffer.com/v3)\n\n• **[Linear](https://linear.app/pragmatic)** – most of us work with agents in a “single-player” setting. Linear’s take is agent work should be teamwork: your teammates can follow the session of the agent, check out the PR it produces, and join the review. Works with Codex, Cursor, Linear’s own agent, or custom agents. [Check it out](https://linear.app/pragmatic)\n\n**• [WorkOS](https://workos.com/)** – how do you deal with agents and permissions? WorkOS built [Airlock](http://workos.com/airlock), the authorization layer for AI agents. It evaluates every request against the agent’s intent and your rules and either allows it, denies it, or routes it to a human for approval. Works with Claude Code, Codex and MCP gateways. [Give it a spin](http://workos.com/airlock).\n\nHow is it that a software veteran who regularly shipped ~100K of database-grade code to production each year, pre-AI, feels like he’s even more productive today, with no drop in quality? [Peter Mattis](https://x.com/petermattis) is co-founder and CTO of [Cockroach Labs](https://www.cockroachlabs.com), and an original creator of GIMP. He also worked on Gmail and distributed storage at Google.\n\nIn this episode, Peter reflects on his journey from open source to Google to founding a database company, and we explore how to keep systems fast, reliable, and correct at scale, from Gmail’s early storage challenges to the tradeoffs in building distributed databases.\n\nPeter tells us how AI has brought him back to writing code after his work shifted toward management, and why he believes AI can improve quality and multiply the impact of domain experts. We also consider the future of code review, and Peter has some advice about how to level up our engineering skills.\n\n[www.youtube.com](https://www.youtube.com/watch?v=0GzwuYGvKA4)\n\n**1\\. Peter initially turned down Sergey Brin’s offer to join Google because of the commute**. The very first version of the famous Google logo was made in GIMP, the free, open-source raster graphics and image editing software created by Peter and his college roommate Spencer Kimball.\n\n[![](https://substackcdn.com/image/fetch/$s_!jw61!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F906855ba-aa27-4c43-9e76-ee966ca9cb9e_548x288.png)](https://substackcdn.com/image/fetch/$s_!jw61!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F906855ba-aa27-4c43-9e76-ee966ca9cb9e_548x288.png)\n\n*The first version of the Google logo, created by GIMP*\n\nIn 2001, Sergey reached out to Peter, invited him to an interview and then made an offer. But Peter said “no” because he lived in San Francisco and did not fancy the hour-long commute to Mountain View. Instead, he joined another startup, but that didn’t go anywhere. When Google reached out again, he took the chance.\n\n**2\\. Peter almost did not ship GIMP after learning about an even more ambitious photo editor**. Peter and Spencer worked hard on GIMP, but a few weeks before launching it, they saw an announcement about another program that promised to do everything GIMP did – and then some! Peter and Spencer felt discouraged but shipped anyway – and the rest is history. They never heard about the other ambitious project again. “There’s always going to be someone else working on your idea,” says Peter. “You can’t get dissuaded if they pre-announce it. To founders: assume that dozens of people have the same idea you have, but most won’t ship it! Know that it’s a competition; you’ve got to enjoy that aspect of it – don’t be afraid of it.”\n\n**3\\. B-trees were used in the first version of Gmail**. Gmail launched on 1 April 2004, offering 1GB of email storage for free, an offer seen as so ridiculously generous at the time that most people assumed it was an April Fool’s joke! B-trees played a role in Gmail’s storage layer, thanks to storing email threads. Under the hood, each incoming message was matched to a thread using the search index. Threads and their unread counts were tracked by B-trees.\n\n**4\\. Colossus reduced Google’s file storage overhead by 33%, while increasing redundancy**. Before Colossus, Google File System (GFS) stored three full copies of data. For Colossus, which became GFS’s successor, Peter and the team pioneered [Reed–Solomon erasure coding](https://en.wikipedia.org/wiki/Reed%E2%80%93Solomon_error_correction) in a distributed file system. Data was stored twice, but redundancy increased!\n\n**5\\. The fastest way to send a packet around the globe is through space!** Peter keeps “speed of light numbers” in his head – similar to what turbopuffer founder Simon Eskildsen does with [”napkin math”](https://newsletter.pragmaticengineer.com/p/pushing-software-engineering-limits) calculations. For example, a network round trip within a zone went from milliseconds when Colossus was built to about 100 microseconds today (10x faster!). But sometimes the bottleneck is the speed of light, and as light speed is faster in a vacuum than anywhere else, that means the fastest global route is straight up to Starlink, across by laser, and then back down. Find out more in ‘ [Delay is not an option: low latency routing in space](https://discovery.ucl.ac.uk/id/eprint/10062262/7/Handley_hotnets.pdf) ’.\n\n**6\\. Peter twice “beat” standard library data structures in performance terms.** At Google, one of his colleagues noticed that std::map showed up in memory profiles. Peter looked closer and figured out that the std::map implementation is a [red-black tree](https://en.wikipedia.org/wiki/Red%E2%80%93black_tree), meaning every node has two pointers. Peter built a B-tree with nearly the same semantics which was both faster, due to spatial locality, and smaller because it used fewer pointers. They used this data structure inside Google. Peter also did something similar with Go’s map: it was performant, but he built a [Swiss Table](https://abseil.io/about/design/swisstables) implementation that was faster. That implementation later made it into the Go library, with the Go team helping to finish it!\n\n**7\\. If you squint hard, everything in distributed databases and storage systems starts looking like a B-tree**. These B-trees are a recurring theme in this podcast episode: the backend of Gmail, the std::map replacement, CockroachDB’s range index, etc. There’s even a paper on this phenomenon, [The Ubiquitous B-Tree](https://dl.acm.org/doi/10.1145/356770.356776).\n\n[![](https://substackcdn.com/image/fetch/$s_!7W2s!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc1ead9b0-c1ba-4418-8712-e1cd39f7bef6_960x267.png)](https://substackcdn.com/image/fetch/$s_!7W2s!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc1ead9b0-c1ba-4418-8712-e1cd39f7bef6_960x267.png)\n\n*A B-tree: a very useful data structure. Source: [Wikipedia](https://en.wikipedia.org/wiki/B-tree)*\n\n**8.** **For consensus in a distributed database, at least three replicas are needed**. With only one, recovery after it crashes isn’t possible. With primary and secondary replicas, neither one will know if the other received the last write following a crash. So, CockroachDB uses three replicas by default – and up to five for some system tables – and customers can add more, although this will result in higher latency.\n\n**9\\. CockroachDB’s design was inspired by Google’s Colossus data storage system and Spanner**. Colossus was append-only, meaning you could only add to files and not mutate them. Google’s distributed database, Spanner, was built on top of Colossus, and so Spanner’s design decisions came from Colossus’ append-only nature. But if you cannot update files and only append to them, it’s not possible to use B-trees as the database’s design structure since B-trees need the files to be updated. An ideal data structure for append-only filesystems is the [Log-structured merge tree](https://en.wikipedia.org/wiki/Log-structured_merge-tree) (LSM tree), where writes are appended to a log file for durability:\n\n[![](https://substackcdn.com/image/fetch/$s_!0D2M!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F46bf5715-c7d5-487a-b90a-1979fb5c10a8_1395x700.png)](https://substackcdn.com/image/fetch/$s_!0D2M!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F46bf5715-c7d5-487a-b90a-1979fb5c10a8_1395x700.png)\n\n*A log-structured merge tree. Source: [Wikipedia](https://en.wikipedia.org/wiki/Log-structured_merge-tree)*\n\nGoogle popularized LSMs with [LevelDB](https://github.com/google/leveldb), and [RocksDB](https://rocksdb.org/) later forked LevelDB. RocksDB was the underlying storage engine that CockroachDB used in its early years. In 2019, Peter wrote and open sourced [Pebble](https://github.com/cockroachdb/pebble), which is now CockroachDB’s storage engine.\n\n**10\\. Peter, the former C++ readability reviewer at Google, predicts we will stop reviewing code**: At Google, every code change needs to be signed off for “readability” by a language readability reviewer, and Peter was a C++ readability reviewer for years. But these days, he’s reviewing less code and foresees this continuing. As he puts it: “What I’m finding is the agents are getting better; you’re having to give less and less scrutiny. I don’t know if it’s going to be this year or next, \\[but\\] we’re materially going to stop looking at the code in the same way we don’t look at assembly anymore.”\n\n**11\\. Non-engineers at Cockroach Labs built ~1,000 internal apps (!!) in a couple of months.** Earlier this year, Cockroach Labs launched an internal platform for non-devs to build apps (think of it as an “internal Lovable”). Folks jumped at it; HR professionals built new tools they’d only dreamt of, while the CFO created the dashboards they’d always longed for, and many others too. Peter and the engineering team were surprised at this uptake, but it chimes with how OpenAI saw nearly all its non-engineers move almost their entire token spend from ChatGPT to Codex within just 4 months.\n\n**12\\. Does “flow” still exist with AI**? I asked Peter this, and he believes it does, but it’s different from the “coding flow” state: “It definitely feels a bit different. It’s maybe a bit less intense, but you’re managing more things cognitively. I’m thinking of ideas that normally would’ve taken me a week to experiment with. And I think of multiple of these experiments and then fire them off all simultaneously. \\[It feels as\\] if I was a college professor with a whole swarm of research assistants, and they’re all off doing things and it’s coming back *really* rapidly.”\n\n• [Inside Google’s Engineering Culture](https://newsletter.pragmaticengineer.com/p/google)\n\n• [Resiliency in distributed systems](https://newsletter.pragmaticengineer.com/p/resiliency-in-distributed-systems)\n\n• [How to debug large, distributed systems: Antithesis](https://newsletter.pragmaticengineer.com/p/antithesis)\n\n• [Pushing software engineering limits with “napkin math”](https://newsletter.pragmaticengineer.com/p/pushing-software-engineering-limits)\n\n• [Designing Data-intensive Applications with Martin Kleppmann](https://newsletter.pragmaticengineer.com/p/designing-data-intensive-applications)\n\n• [Formal methods with Hillel Wayne](https://newsletter.pragmaticengineer.com/p/formal-methods-with-hillel-wayne)\n\n[00:00](https://www.youtube.com/watch?v=0GzwuYGvKA4) Intro\n\n[02:42](https://www.youtube.com/watch?v=0GzwuYGvKA4&t=162s) Peter’s path into tech\n\n[04:00](https://www.youtube.com/watch?v=0GzwuYGvKA4&t=240s) Building GIMP\n\n[09:30](https://www.youtube.com/watch?v=0GzwuYGvKA4&t=570s) Working on Gmail at Google\n\n[14:51](https://www.youtube.com/watch?v=0GzwuYGvKA4&t=891s) Google’s infra: google3, build files, Bazel, and Colossus\n\n[21:30](https://www.youtube.com/watch?v=0GzwuYGvKA4&t=1290s) Distributed storage bottlenecks\n\n[23:59](https://www.youtube.com/watch?v=0GzwuYGvKA4&t=1439s) Latency, throughput, and availability\n\n[30:04](https://www.youtube.com/watch?v=0GzwuYGvKA4&t=1804s) Contributing to libraries\n\n[41:52](https://www.youtube.com/watch?v=0GzwuYGvKA4&t=2512s) Google Spanner\n\n[46:10](https://www.youtube.com/watch?v=0GzwuYGvKA4&t=2770s) CockroachDB\n\n[52:00](https://www.youtube.com/watch?v=0GzwuYGvKA4&t=3120s) Manual vs. automatic sharding\n\n[55:28](https://www.youtube.com/watch?v=0GzwuYGvKA4&t=3328s) Consistency models and strong consistency\n\n[1:00:03](https://www.youtube.com/watch?v=0GzwuYGvKA4&t=3603s) Raft consensus\n\n[1:06:15](https://www.youtube.com/watch?v=0GzwuYGvKA4&t=3975s) How AI brought Peter back to coding\n\n[1:19:12](https://www.youtube.com/watch?v=0GzwuYGvKA4&t=4752s) Peter’s tools and agentic workflows\n\n[1:23:08](https://www.youtube.com/watch?v=0GzwuYGvKA4&t=4988s) How AI can improve quality\n\n[1:26:39](https://www.youtube.com/watch?v=0GzwuYGvKA4&t=5199s) Code reviews: are they done?\n\n[1:29:17](https://www.youtube.com/watch?v=0GzwuYGvKA4&t=5357s&pp=0gcJCWMAwfN6Pr3D) 100x engineers\n\n[1:35:33](https://www.youtube.com/watch?v=0GzwuYGvKA4&t=5733s) Peter’s advice for leveling up your engineering skills\n\n**Where to find Peter Mattis:**\n\n• X: [https://x.com/petermattis](https://x.com/petermattis)\n\n• LinkedIn: [https://www.linkedin.com/in/peter-mattis-46549144](https://www.linkedin.com/in/peter-mattis-46549144)\n\n**Mentions during the episode:**\n\n• Cockroach Labs: [https://www.cockroachlabs.com](https://www.cockroachlabs.com)\n\n• GIMP: [https://www.gimp.org](https://www.gimp.org)\n\n• Usenet: [https://en.wikipedia.org/wiki/Usenet](https://en.wikipedia.org/wiki/Usenet)\n\n• Larry Page: [https://en.wikipedia.org/wiki/Larry_Page](https://en.wikipedia.org/wiki/Larry_Page)\n\n• Sergey Brin: [https://en.wikipedia.org/wiki/Sergey_Brin](https://en.wikipedia.org/wiki/Sergey_Brin)\n\n• Paul Buchheit on X: [https://x.com/paultoo](https://x.com/paultoo)\n\n• Y Combinator: [https://www.ycombinator.com](https://www.ycombinator.com)\n\n• Delay is Not an Option: Low Latency Routing in Space: [https://discovery.ucl.ac.uk/id/eprint/10062262/7/Handley_hotnets.pdf](https://discovery.ucl.ac.uk/id/eprint/10062262/7/Handley_hotnets.pdf)\n\n• Bazel: [https://opensource.google/projects/bazel](https://opensource.google/projects/bazel)\n\n• Colossus under the hood: a peek into Google’s scalable storage system: [https://cloud.google.com/blog/products/storage-data-transfer/a-peek-behind-colossus-googles-file-system](https://cloud.google.com/blog/products/storage-data-transfer/a-peek-behind-colossus-googles-file-system)\n\n• Pushing software engineering limits with “napkin math”: [https://newsletter.pragmaticengineer.com/p/pushing-software-engineering-limits](https://newsletter.pragmaticengineer.com/p/pushing-software-engineering-limits)\n\n• turbopuffer: [https://turbopuffer.com](https://turbopuffer.com)\n\n• Trimodal Nature of Tech Compensation in the US, UK and India: [https://newsletter.pragmaticengineer.com/p/trimodal](https://newsletter.pragmaticengineer.com/p/trimodal)\n\n• Quicksort: [https://en.wikipedia.org/wiki/Quicksort](https://en.wikipedia.org/wiki/Quicksort)\n\n• B-tree: [https://en.wikipedia.org/wiki/B-tree](https://en.wikipedia.org/wiki/B-tree)\n\n• Go: [https://go.dev](https://go.dev)\n\n• Google Goggles: [https://en.wikipedia.org/wiki/Google_Goggles](https://en.wikipedia.org/wiki/Google_Goggles)\n\n• Spencer Kimball on LinkedIn: [linkedin.com/in/spencerwkimball](http://linkedin.com/in/spencerwkimball)\n\n• Ben Darnell: [https://en.wikipedia.org/wiki/Ben_Darnell](https://en.wikipedia.org/wiki/Ben_Darnell)\n\n• The Ubiquitous B-Tree: [https://dl.acm.org/doi/pdf/10.1145/356770.356776](https://dl.acm.org/doi/pdf/10.1145/356770.356776)\n\n• Paxos: [https://martinfowler.com/articles/patterns-of-distributed-systems/paxos.html](https://martinfowler.com/articles/patterns-of-distributed-systems/paxos.html)\n\n• Raft is so fetch: The Raft Consensus Algorithm explained through “Mean Girls”: [https://www.cockroachlabs.com/blog/raft-is-so-fetch/](https://www.cockroachlabs.com/blog/raft-is-so-fetch/)\n\n• DoorDash: [https://www.doordash.com](https://www.doordash.com)\n\n• RocksDB: [https://rocksdb.org](https://rubberduckdebugging.com)\n\n• Rubber duck debugging: [https://rubberduckdebugging.com](https://rubberduckdebugging.com)\n\n• Claude Code: [https://claude.com/product/claude-code](https://claude.com/product/claude-code)\n\n• Codex: [https://chatgpt.com/codex](https://chatgpt.com/codex)\n\n• Building Claude Code with Boris Cherny: [https://newsletter.pragmaticengineer.com/p/building-claude-code-with-boris-cherny](https://newsletter.pragmaticengineer.com/p/building-claude-code-with-boris-cherny)\n\n• Programming TypeScript: [https://www.oreilly.com/library/view/programming-typescript/9781492037644](https://www.oreilly.com/library/view/programming-typescript/9781492037644)\n\n• Building Codex with Tibo Sottiaux: [https://newsletter.pragmaticengineer.com/p/building-codex-with-tibo-sottiaux](https://newsletter.pragmaticengineer.com/p/building-codex-with-tibo-sottiaux)\n\n• Terence Tao Digests the Jacobian Conjecture Counterexample: How Claude Fable 5 Broke an 87-Year-Old Math Problem: [https://www.developersdigest.tech/blog/jacobian-conjecture-counterexample-fable](https://www.developersdigest.tech/blog/jacobian-conjecture-counterexample-fable)\n\n• Jeff Dean on LinkedIn: [https://www.linkedin.com/in/jeff-dean-8b212555](https://www.linkedin.com/in/jeff-dean-8b212555)\n\n• Sanjay Ghemawat on LinkedIn: [https://www.linkedin.com/in/sanjay-ghemawat-763485428](https://www.linkedin.com/in/sanjay-ghemawat-763485428)\n\n—\n\nProduction and marketing by [Pen Name](https://penname.co/).\n",
  "authors": [
    "Gergely Orosz"
  ],
  "captured_at": "2026-09-30T20:48:45.272500234+00:00",
  "category": "programming",
  "content_html": "<p><strong>Listen and watch now on <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://youtu.be/0GzwuYGvKA4\">YouTube</a>, <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://podcasts.apple.com/us/podcast/the-pragmatic-engineer/id1769051199\">Apple</a>, and <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://open.spotify.com/show/2Bho9xCbOQMWMJ7UKmqCzD\">Spotify</a>.</strong> See the episode transcript at the top of this page, and timestamps for the episode at the bottom.</p>\n<p><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://substackcdn.com/image/fetch/$s_!Qhi0!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1228ff6a-514b-4643-9360-02cda14dc5ac_800x70.png\"><picture class=\"article-picture article-picture-fallback\" style=\"--image-ratio:16 / 9\"><img src=\"https://substackcdn.com/image/fetch/$s_!Qhi0!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1228ff6a-514b-4643-9360-02cda14dc5ac_800x70.png\" alt=\"\" loading=\"eager\" decoding=\"async\" fetchpriority=\"high\" referrerpolicy=\"no-referrer\" class=\"progressive-image\"></picture></a></p>\n<p>• <strong><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"http://turbopuffer.com/pragmatic\">turbopuffer</a></strong> – The turbopuffer engineering team is doing something really cool: they are completely redesigning their storage architecture from first principles to make search faster, cheaper, and more reliable at scale. And they’re documenting all of it! Follow along their rewrite at <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"http://turbopuffer.com/v3\">turbopuffer.com/v3</a></p>\n<p>• <strong><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://linear.app/pragmatic\">Linear</a></strong> – most of us work with agents in a “single-player” setting. Linear’s take is agent work should be teamwork: your teammates can follow the session of the agent, check out the PR it produces, and join the review. Works with Codex, Cursor, Linear’s own agent, or custom agents. <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://linear.app/pragmatic\">Check it out</a></p>\n<p><strong>• <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://workos.com/\">WorkOS</a></strong> – how do you deal with agents and permissions? WorkOS built <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"http://workos.com/airlock\">Airlock</a>, the authorization layer for AI agents. It evaluates every request against the agent’s intent and your rules and either allows it, denies it, or routes it to a human for approval. Works with Claude Code, Codex and MCP gateways. <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"http://workos.com/airlock\">Give it a spin</a>.</p>\n<p>How is it that a software veteran who regularly shipped ~100K of database-grade code to production each year, pre-AI, feels like he’s even more productive today, with no drop in quality? <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://x.com/petermattis\">Peter Mattis</a> is co-founder and CTO of <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://www.cockroachlabs.com\">Cockroach Labs</a>, and an original creator of GIMP. He also worked on Gmail and distributed storage at Google.</p>\n<p>In this episode, Peter reflects on his journey from open source to Google to founding a database company, and we explore how to keep systems fast, reliable, and correct at scale, from Gmail’s early storage challenges to the tradeoffs in building distributed databases.</p>\n<p>Peter tells us how AI has brought him back to writing code after his work shifted toward management, and why he believes AI can improve quality and multiply the impact of domain experts. We also consider the future of code review, and Peter has some advice about how to level up our engineering skills.</p>\n<p><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://www.youtube.com/watch?v=0GzwuYGvKA4\">www.youtube.com</a></p>\n<p><strong>1. Peter initially turned down Sergey Brin’s offer to join Google because of the commute</strong>. The very first version of the famous Google logo was made in GIMP, the free, open-source raster graphics and image editing software created by Peter and his college roommate Spencer Kimball.</p>\n<p><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://substackcdn.com/image/fetch/$s_!jw61!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F906855ba-aa27-4c43-9e76-ee966ca9cb9e_548x288.png\"><picture class=\"article-picture article-picture-fallback\" style=\"--image-ratio:16 / 9\"><img src=\"https://substackcdn.com/image/fetch/$s_!jw61!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F906855ba-aa27-4c43-9e76-ee966ca9cb9e_548x288.png\" alt=\"\" loading=\"lazy\" decoding=\"async\" fetchpriority=\"low\" referrerpolicy=\"no-referrer\" class=\"progressive-image\"></picture></a></p>\n<p><em>The first version of the Google logo, created by GIMP</em></p>\n<p>In 2001, Sergey reached out to Peter, invited him to an interview and then made an offer. But Peter said “no” because he lived in San Francisco and did not fancy the hour-long commute to Mountain View. Instead, he joined another startup, but that didn’t go anywhere. When Google reached out again, he took the chance.</p>\n<p><strong>2. Peter almost did not ship GIMP after learning about an even more ambitious photo editor</strong>. Peter and Spencer worked hard on GIMP, but a few weeks before launching it, they saw an announcement about another program that promised to do everything GIMP did – and then some! Peter and Spencer felt discouraged but shipped anyway – and the rest is history. They never heard about the other ambitious project again. “There’s always going to be someone else working on your idea,” says Peter. “You can’t get dissuaded if they pre-announce it. To founders: assume that dozens of people have the same idea you have, but most won’t ship it! Know that it’s a competition; you’ve got to enjoy that aspect of it – don’t be afraid of it.”</p>\n<p><strong>3. B-trees were used in the first version of Gmail</strong>. Gmail launched on 1 April 2004, offering 1GB of email storage for free, an offer seen as so ridiculously generous at the time that most people assumed it was an April Fool’s joke! B-trees played a role in Gmail’s storage layer, thanks to storing email threads. Under the hood, each incoming message was matched to a thread using the search index. Threads and their unread counts were tracked by B-trees.</p>\n<p><strong>4. Colossus reduced Google’s file storage overhead by 33%, while increasing redundancy</strong>. Before Colossus, Google File System (GFS) stored three full copies of data. For Colossus, which became GFS’s successor, Peter and the team pioneered <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://en.wikipedia.org/wiki/Reed%E2%80%93Solomon_error_correction\">Reed–Solomon erasure coding</a> in a distributed file system. Data was stored twice, but redundancy increased!</p>\n<p><strong>5. The fastest way to send a packet around the globe is through space!</strong> Peter keeps “speed of light numbers” in his head – similar to what turbopuffer founder Simon Eskildsen does with <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://newsletter.pragmaticengineer.com/p/pushing-software-engineering-limits\">”napkin math”</a> calculations. For example, a network round trip within a zone went from milliseconds when Colossus was built to about 100 microseconds today (10x faster!). But sometimes the bottleneck is the speed of light, and as light speed is faster in a vacuum than anywhere else, that means the fastest global route is straight up to Starlink, across by laser, and then back down. Find out more in ‘ <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://discovery.ucl.ac.uk/id/eprint/10062262/7/Handley_hotnets.pdf\">Delay is not an option: low latency routing in space</a> ’.</p>\n<p><strong>6. Peter twice “beat” standard library data structures in performance terms.</strong> At Google, one of his colleagues noticed that std::map showed up in memory profiles. Peter looked closer and figured out that the std::map implementation is a <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://en.wikipedia.org/wiki/Red%E2%80%93black_tree\">red-black tree</a>, meaning every node has two pointers. Peter built a B-tree with nearly the same semantics which was both faster, due to spatial locality, and smaller because it used fewer pointers. They used this data structure inside Google. Peter also did something similar with Go’s map: it was performant, but he built a <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://abseil.io/about/design/swisstables\">Swiss Table</a> implementation that was faster. That implementation later made it into the Go library, with the Go team helping to finish it!</p>\n<p><strong>7. If you squint hard, everything in distributed databases and storage systems starts looking like a B-tree</strong>. These B-trees are a recurring theme in this podcast episode: the backend of Gmail, the std::map replacement, CockroachDB’s range index, etc. There’s even a paper on this phenomenon, <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://dl.acm.org/doi/10.1145/356770.356776\">The Ubiquitous B-Tree</a>.</p>\n<p><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://substackcdn.com/image/fetch/$s_!7W2s!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc1ead9b0-c1ba-4418-8712-e1cd39f7bef6_960x267.png\"><picture class=\"article-picture article-picture-fallback\" style=\"--image-ratio:16 / 9\"><img src=\"https://substackcdn.com/image/fetch/$s_!7W2s!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc1ead9b0-c1ba-4418-8712-e1cd39f7bef6_960x267.png\" alt=\"\" loading=\"lazy\" decoding=\"async\" fetchpriority=\"low\" referrerpolicy=\"no-referrer\" class=\"progressive-image\"></picture></a></p>\n<p><em>A B-tree: a very useful data structure. Source: <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://en.wikipedia.org/wiki/B-tree\">Wikipedia</a></em></p>\n<p><strong>8.</strong> <strong>For consensus in a distributed database, at least three replicas are needed</strong>. With only one, recovery after it crashes isn’t possible. With primary and secondary replicas, neither one will know if the other received the last write following a crash. So, CockroachDB uses three replicas by default – and up to five for some system tables – and customers can add more, although this will result in higher latency.</p>\n<p><strong>9. CockroachDB’s design was inspired by Google’s Colossus data storage system and Spanner</strong>. Colossus was append-only, meaning you could only add to files and not mutate them. Google’s distributed database, Spanner, was built on top of Colossus, and so Spanner’s design decisions came from Colossus’ append-only nature. But if you cannot update files and only append to them, it’s not possible to use B-trees as the database’s design structure since B-trees need the files to be updated. An ideal data structure for append-only filesystems is the <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://en.wikipedia.org/wiki/Log-structured_merge-tree\">Log-structured merge tree</a> (LSM tree), where writes are appended to a log file for durability:</p>\n<p><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://substackcdn.com/image/fetch/$s_!0D2M!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F46bf5715-c7d5-487a-b90a-1979fb5c10a8_1395x700.png\"><picture class=\"article-picture article-picture-fallback\" style=\"--image-ratio:16 / 9\"><img src=\"https://substackcdn.com/image/fetch/$s_!0D2M!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F46bf5715-c7d5-487a-b90a-1979fb5c10a8_1395x700.png\" alt=\"\" loading=\"lazy\" decoding=\"async\" fetchpriority=\"low\" referrerpolicy=\"no-referrer\" class=\"progressive-image\"></picture></a></p>\n<p><em>A log-structured merge tree. Source: <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://en.wikipedia.org/wiki/Log-structured_merge-tree\">Wikipedia</a></em></p>\n<p>Google popularized LSMs with <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://github.com/google/leveldb\">LevelDB</a>, and <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://rocksdb.org/\">RocksDB</a> later forked LevelDB. RocksDB was the underlying storage engine that CockroachDB used in its early years. In 2019, Peter wrote and open sourced <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://github.com/cockroachdb/pebble\">Pebble</a>, which is now CockroachDB’s storage engine.</p>\n<p><strong>10. Peter, the former C++ readability reviewer at Google, predicts we will stop reviewing code</strong>: At Google, every code change needs to be signed off for “readability” by a language readability reviewer, and Peter was a C++ readability reviewer for years. But these days, he’s reviewing less code and foresees this continuing. As he puts it: “What I’m finding is the agents are getting better; you’re having to give less and less scrutiny. I don’t know if it’s going to be this year or next, [but] we’re materially going to stop looking at the code in the same way we don’t look at assembly anymore.”</p>\n<p><strong>11. Non-engineers at Cockroach Labs built ~1,000 internal apps (!!) in a couple of months.</strong> Earlier this year, Cockroach Labs launched an internal platform for non-devs to build apps (think of it as an “internal Lovable”). Folks jumped at it; HR professionals built new tools they’d only dreamt of, while the CFO created the dashboards they’d always longed for, and many others too. Peter and the engineering team were surprised at this uptake, but it chimes with how OpenAI saw nearly all its non-engineers move almost their entire token spend from ChatGPT to Codex within just 4 months.</p>\n<p><strong>12. Does “flow” still exist with AI</strong>? I asked Peter this, and he believes it does, but it’s different from the “coding flow” state: “It definitely feels a bit different. It’s maybe a bit less intense, but you’re managing more things cognitively. I’m thinking of ideas that normally would’ve taken me a week to experiment with. And I think of multiple of these experiments and then fire them off all simultaneously. [It feels as] if I was a college professor with a whole swarm of research assistants, and they’re all off doing things and it’s coming back <em>really</em> rapidly.”</p>\n<p>• <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://newsletter.pragmaticengineer.com/p/google\">Inside Google’s Engineering Culture</a></p>\n<p>• <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://newsletter.pragmaticengineer.com/p/resiliency-in-distributed-systems\">Resiliency in distributed systems</a></p>\n<p>• <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://newsletter.pragmaticengineer.com/p/antithesis\">How to debug large, distributed systems: Antithesis</a></p>\n<p>• <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://newsletter.pragmaticengineer.com/p/pushing-software-engineering-limits\">Pushing software engineering limits with “napkin math”</a></p>\n<p>• <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://newsletter.pragmaticengineer.com/p/designing-data-intensive-applications\">Designing Data-intensive Applications with Martin Kleppmann</a></p>\n<p>• <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://newsletter.pragmaticengineer.com/p/formal-methods-with-hillel-wayne\">Formal methods with Hillel Wayne</a></p>\n<p><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://www.youtube.com/watch?v=0GzwuYGvKA4\">00:00</a> Intro</p>\n<p><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://www.youtube.com/watch?v=0GzwuYGvKA4&amp;t=162s\">02:42</a> Peter’s path into tech</p>\n<p><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://www.youtube.com/watch?v=0GzwuYGvKA4&amp;t=240s\">04:00</a> Building GIMP</p>\n<p><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://www.youtube.com/watch?v=0GzwuYGvKA4&amp;t=570s\">09:30</a> Working on Gmail at Google</p>\n<p><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://www.youtube.com/watch?v=0GzwuYGvKA4&amp;t=891s\">14:51</a> Google’s infra: google3, build files, Bazel, and Colossus</p>\n<p><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://www.youtube.com/watch?v=0GzwuYGvKA4&amp;t=1290s\">21:30</a> Distributed storage bottlenecks</p>\n<p><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://www.youtube.com/watch?v=0GzwuYGvKA4&amp;t=1439s\">23:59</a> Latency, throughput, and availability</p>\n<p><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://www.youtube.com/watch?v=0GzwuYGvKA4&amp;t=1804s\">30:04</a> Contributing to libraries</p>\n<p><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://www.youtube.com/watch?v=0GzwuYGvKA4&amp;t=2512s\">41:52</a> Google Spanner</p>\n<p><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://www.youtube.com/watch?v=0GzwuYGvKA4&amp;t=2770s\">46:10</a> CockroachDB</p>\n<p><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://www.youtube.com/watch?v=0GzwuYGvKA4&amp;t=3120s\">52:00</a> Manual vs. automatic sharding</p>\n<p><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://www.youtube.com/watch?v=0GzwuYGvKA4&amp;t=3328s\">55:28</a> Consistency models and strong consistency</p>\n<p><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://www.youtube.com/watch?v=0GzwuYGvKA4&amp;t=3603s\">1:00:03</a> Raft consensus</p>\n<p><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://www.youtube.com/watch?v=0GzwuYGvKA4&amp;t=3975s\">1:06:15</a> How AI brought Peter back to coding</p>\n<p><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://www.youtube.com/watch?v=0GzwuYGvKA4&amp;t=4752s\">1:19:12</a> Peter’s tools and agentic workflows</p>\n<p><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://www.youtube.com/watch?v=0GzwuYGvKA4&amp;t=4988s\">1:23:08</a> How AI can improve quality</p>\n<p><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://www.youtube.com/watch?v=0GzwuYGvKA4&amp;t=5199s\">1:26:39</a> Code reviews: are they done?</p>\n<p><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://www.youtube.com/watch?v=0GzwuYGvKA4&amp;t=5357s&amp;pp=0gcJCWMAwfN6Pr3D\">1:29:17</a> 100x engineers</p>\n<p><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://www.youtube.com/watch?v=0GzwuYGvKA4&amp;t=5733s\">1:35:33</a> Peter’s advice for leveling up your engineering skills</p>\n<p><strong>Where to find Peter Mattis:</strong></p>\n<p>• X: <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://x.com/petermattis\">https://x.com/petermattis</a></p>\n<p>• LinkedIn: <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://www.linkedin.com/in/peter-mattis-46549144\">https://www.linkedin.com/in/peter-mattis-46549144</a></p>\n<p><strong>Mentions during the episode:</strong></p>\n<p>• Cockroach Labs: <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://www.cockroachlabs.com\">https://www.cockroachlabs.com</a></p>\n<p>• GIMP: <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://www.gimp.org\">https://www.gimp.org</a></p>\n<p>• Usenet: <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://en.wikipedia.org/wiki/Usenet\">https://en.wikipedia.org/wiki/Usenet</a></p>\n<p>• Larry Page: <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://en.wikipedia.org/wiki/Larry_Page\">https://en.wikipedia.org/wiki/Larry_Page</a></p>\n<p>• Sergey Brin: <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://en.wikipedia.org/wiki/Sergey_Brin\">https://en.wikipedia.org/wiki/Sergey_Brin</a></p>\n<p>• Paul Buchheit on X: <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://x.com/paultoo\">https://x.com/paultoo</a></p>\n<p>• Y Combinator: <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://www.ycombinator.com\">https://www.ycombinator.com</a></p>\n<p>• Delay is Not an Option: Low Latency Routing in Space: <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://discovery.ucl.ac.uk/id/eprint/10062262/7/Handley_hotnets.pdf\">https://discovery.ucl.ac.uk/id/eprint/10062262/7/Handley_hotnets.pdf</a></p>\n<p>• Bazel: <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://opensource.google/projects/bazel\">https://opensource.google/projects/bazel</a></p>\n<p>• Colossus under the hood: a peek into Google’s scalable storage system: <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://cloud.google.com/blog/products/storage-data-transfer/a-peek-behind-colossus-googles-file-system\">https://cloud.google.com/blog/products/storage-data-transfer/a-peek-behind-colossus-googles-file-system</a></p>\n<p>• Pushing software engineering limits with “napkin math”: <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://newsletter.pragmaticengineer.com/p/pushing-software-engineering-limits\">https://newsletter.pragmaticengineer.com/p/pushing-software-engineering-limits</a></p>\n<p>• turbopuffer: <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://turbopuffer.com\">https://turbopuffer.com</a></p>\n<p>• Trimodal Nature of Tech Compensation in the US, UK and India: <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://newsletter.pragmaticengineer.com/p/trimodal\">https://newsletter.pragmaticengineer.com/p/trimodal</a></p>\n<p>• Quicksort: <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://en.wikipedia.org/wiki/Quicksort\">https://en.wikipedia.org/wiki/Quicksort</a></p>\n<p>• B-tree: <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://en.wikipedia.org/wiki/B-tree\">https://en.wikipedia.org/wiki/B-tree</a></p>\n<p>• Go: <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://go.dev\">https://go.dev</a></p>\n<p>• Google Goggles: <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://en.wikipedia.org/wiki/Google_Goggles\">https://en.wikipedia.org/wiki/Google_Goggles</a></p>\n<p>• Spencer Kimball on LinkedIn: <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"http://linkedin.com/in/spencerwkimball\">linkedin.com/in/spencerwkimball</a></p>\n<p>• Ben Darnell: <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://en.wikipedia.org/wiki/Ben_Darnell\">https://en.wikipedia.org/wiki/Ben_Darnell</a></p>\n<p>• The Ubiquitous B-Tree: <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://dl.acm.org/doi/pdf/10.1145/356770.356776\">https://dl.acm.org/doi/pdf/10.1145/356770.356776</a></p>\n<p>• Paxos: <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://martinfowler.com/articles/patterns-of-distributed-systems/paxos.html\">https://martinfowler.com/articles/patterns-of-distributed-systems/paxos.html</a></p>\n<p>• Raft is so fetch: The Raft Consensus Algorithm explained through “Mean Girls”: <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://www.cockroachlabs.com/blog/raft-is-so-fetch/\">https://www.cockroachlabs.com/blog/raft-is-so-fetch/</a></p>\n<p>• DoorDash: <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://www.doordash.com\">https://www.doordash.com</a></p>\n<p>• RocksDB: <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://rubberduckdebugging.com\">https://rocksdb.org</a></p>\n<p>• Rubber duck debugging: <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://rubberduckdebugging.com\">https://rubberduckdebugging.com</a></p>\n<p>• Claude Code: <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://claude.com/product/claude-code\">https://claude.com/product/claude-code</a></p>\n<p>• Codex: <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://chatgpt.com/codex\">https://chatgpt.com/codex</a></p>\n<p>• Building Claude Code with Boris Cherny: <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://newsletter.pragmaticengineer.com/p/building-claude-code-with-boris-cherny\">https://newsletter.pragmaticengineer.com/p/building-claude-code-with-boris-cherny</a></p>\n<p>• Programming TypeScript: <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://www.oreilly.com/library/view/programming-typescript/9781492037644\">https://www.oreilly.com/library/view/programming-typescript/9781492037644</a></p>\n<p>• Building Codex with Tibo Sottiaux: <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://newsletter.pragmaticengineer.com/p/building-codex-with-tibo-sottiaux\">https://newsletter.pragmaticengineer.com/p/building-codex-with-tibo-sottiaux</a></p>\n<p>• Terence Tao Digests the Jacobian Conjecture Counterexample: How Claude Fable 5 Broke an 87-Year-Old Math Problem: <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://www.developersdigest.tech/blog/jacobian-conjecture-counterexample-fable\">https://www.developersdigest.tech/blog/jacobian-conjecture-counterexample-fable</a></p>\n<p>• Jeff Dean on LinkedIn: <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://www.linkedin.com/in/jeff-dean-8b212555\">https://www.linkedin.com/in/jeff-dean-8b212555</a></p>\n<p>• Sanjay Ghemawat on LinkedIn: <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://www.linkedin.com/in/sanjay-ghemawat-763485428\">https://www.linkedin.com/in/sanjay-ghemawat-763485428</a></p>\n<p>—</p>\n<p>Production and marketing by <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://penname.co/\">Pen Name</a>.</p>\n",
  "content_markdown": "**Listen and watch now on [YouTube](https://youtu.be/0GzwuYGvKA4), [Apple](https://podcasts.apple.com/us/podcast/the-pragmatic-engineer/id1769051199), and [Spotify](https://open.spotify.com/show/2Bho9xCbOQMWMJ7UKmqCzD).** See the episode transcript at the top of this page, and timestamps for the episode at the bottom.\n\n[![](https://substackcdn.com/image/fetch/$s_!Qhi0!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1228ff6a-514b-4643-9360-02cda14dc5ac_800x70.png)](https://substackcdn.com/image/fetch/$s_!Qhi0!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1228ff6a-514b-4643-9360-02cda14dc5ac_800x70.png)\n\n• **[turbopuffer](http://turbopuffer.com/pragmatic)** – The turbopuffer engineering team is doing something really cool: they are completely redesigning their storage architecture from first principles to make search faster, cheaper, and more reliable at scale. And they’re documenting all of it! Follow along their rewrite at [turbopuffer.com/v3](http://turbopuffer.com/v3)\n\n• **[Linear](https://linear.app/pragmatic)** – most of us work with agents in a “single-player” setting. Linear’s take is agent work should be teamwork: your teammates can follow the session of the agent, check out the PR it produces, and join the review. Works with Codex, Cursor, Linear’s own agent, or custom agents. [Check it out](https://linear.app/pragmatic)\n\n**• [WorkOS](https://workos.com/)** – how do you deal with agents and permissions? WorkOS built [Airlock](http://workos.com/airlock), the authorization layer for AI agents. It evaluates every request against the agent’s intent and your rules and either allows it, denies it, or routes it to a human for approval. Works with Claude Code, Codex and MCP gateways. [Give it a spin](http://workos.com/airlock).\n\nHow is it that a software veteran who regularly shipped ~100K of database-grade code to production each year, pre-AI, feels like he’s even more productive today, with no drop in quality? [Peter Mattis](https://x.com/petermattis) is co-founder and CTO of [Cockroach Labs](https://www.cockroachlabs.com), and an original creator of GIMP. He also worked on Gmail and distributed storage at Google.\n\nIn this episode, Peter reflects on his journey from open source to Google to founding a database company, and we explore how to keep systems fast, reliable, and correct at scale, from Gmail’s early storage challenges to the tradeoffs in building distributed databases.\n\nPeter tells us how AI has brought him back to writing code after his work shifted toward management, and why he believes AI can improve quality and multiply the impact of domain experts. We also consider the future of code review, and Peter has some advice about how to level up our engineering skills.\n\n[www.youtube.com](https://www.youtube.com/watch?v=0GzwuYGvKA4)\n\n**1\\. Peter initially turned down Sergey Brin’s offer to join Google because of the commute**. The very first version of the famous Google logo was made in GIMP, the free, open-source raster graphics and image editing software created by Peter and his college roommate Spencer Kimball.\n\n[![](https://substackcdn.com/image/fetch/$s_!jw61!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F906855ba-aa27-4c43-9e76-ee966ca9cb9e_548x288.png)](https://substackcdn.com/image/fetch/$s_!jw61!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F906855ba-aa27-4c43-9e76-ee966ca9cb9e_548x288.png)\n\n*The first version of the Google logo, created by GIMP*\n\nIn 2001, Sergey reached out to Peter, invited him to an interview and then made an offer. But Peter said “no” because he lived in San Francisco and did not fancy the hour-long commute to Mountain View. Instead, he joined another startup, but that didn’t go anywhere. When Google reached out again, he took the chance.\n\n**2\\. Peter almost did not ship GIMP after learning about an even more ambitious photo editor**. Peter and Spencer worked hard on GIMP, but a few weeks before launching it, they saw an announcement about another program that promised to do everything GIMP did – and then some! Peter and Spencer felt discouraged but shipped anyway – and the rest is history. They never heard about the other ambitious project again. “There’s always going to be someone else working on your idea,” says Peter. “You can’t get dissuaded if they pre-announce it. To founders: assume that dozens of people have the same idea you have, but most won’t ship it! Know that it’s a competition; you’ve got to enjoy that aspect of it – don’t be afraid of it.”\n\n**3\\. B-trees were used in the first version of Gmail**. Gmail launched on 1 April 2004, offering 1GB of email storage for free, an offer seen as so ridiculously generous at the time that most people assumed it was an April Fool’s joke! B-trees played a role in Gmail’s storage layer, thanks to storing email threads. Under the hood, each incoming message was matched to a thread using the search index. Threads and their unread counts were tracked by B-trees.\n\n**4\\. Colossus reduced Google’s file storage overhead by 33%, while increasing redundancy**. Before Colossus, Google File System (GFS) stored three full copies of data. For Colossus, which became GFS’s successor, Peter and the team pioneered [Reed–Solomon erasure coding](https://en.wikipedia.org/wiki/Reed%E2%80%93Solomon_error_correction) in a distributed file system. Data was stored twice, but redundancy increased!\n\n**5\\. The fastest way to send a packet around the globe is through space!** Peter keeps “speed of light numbers” in his head – similar to what turbopuffer founder Simon Eskildsen does with [”napkin math”](https://newsletter.pragmaticengineer.com/p/pushing-software-engineering-limits) calculations. For example, a network round trip within a zone went from milliseconds when Colossus was built to about 100 microseconds today (10x faster!). But sometimes the bottleneck is the speed of light, and as light speed is faster in a vacuum than anywhere else, that means the fastest global route is straight up to Starlink, across by laser, and then back down. Find out more in ‘ [Delay is not an option: low latency routing in space](https://discovery.ucl.ac.uk/id/eprint/10062262/7/Handley_hotnets.pdf) ’.\n\n**6\\. Peter twice “beat” standard library data structures in performance terms.** At Google, one of his colleagues noticed that std::map showed up in memory profiles. Peter looked closer and figured out that the std::map implementation is a [red-black tree](https://en.wikipedia.org/wiki/Red%E2%80%93black_tree), meaning every node has two pointers. Peter built a B-tree with nearly the same semantics which was both faster, due to spatial locality, and smaller because it used fewer pointers. They used this data structure inside Google. Peter also did something similar with Go’s map: it was performant, but he built a [Swiss Table](https://abseil.io/about/design/swisstables) implementation that was faster. That implementation later made it into the Go library, with the Go team helping to finish it!\n\n**7\\. If you squint hard, everything in distributed databases and storage systems starts looking like a B-tree**. These B-trees are a recurring theme in this podcast episode: the backend of Gmail, the std::map replacement, CockroachDB’s range index, etc. There’s even a paper on this phenomenon, [The Ubiquitous B-Tree](https://dl.acm.org/doi/10.1145/356770.356776).\n\n[![](https://substackcdn.com/image/fetch/$s_!7W2s!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc1ead9b0-c1ba-4418-8712-e1cd39f7bef6_960x267.png)](https://substackcdn.com/image/fetch/$s_!7W2s!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc1ead9b0-c1ba-4418-8712-e1cd39f7bef6_960x267.png)\n\n*A B-tree: a very useful data structure. Source: [Wikipedia](https://en.wikipedia.org/wiki/B-tree)*\n\n**8.** **For consensus in a distributed database, at least three replicas are needed**. With only one, recovery after it crashes isn’t possible. With primary and secondary replicas, neither one will know if the other received the last write following a crash. So, CockroachDB uses three replicas by default – and up to five for some system tables – and customers can add more, although this will result in higher latency.\n\n**9\\. CockroachDB’s design was inspired by Google’s Colossus data storage system and Spanner**. Colossus was append-only, meaning you could only add to files and not mutate them. Google’s distributed database, Spanner, was built on top of Colossus, and so Spanner’s design decisions came from Colossus’ append-only nature. But if you cannot update files and only append to them, it’s not possible to use B-trees as the database’s design structure since B-trees need the files to be updated. An ideal data structure for append-only filesystems is the [Log-structured merge tree](https://en.wikipedia.org/wiki/Log-structured_merge-tree) (LSM tree), where writes are appended to a log file for durability:\n\n[![](https://substackcdn.com/image/fetch/$s_!0D2M!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F46bf5715-c7d5-487a-b90a-1979fb5c10a8_1395x700.png)](https://substackcdn.com/image/fetch/$s_!0D2M!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F46bf5715-c7d5-487a-b90a-1979fb5c10a8_1395x700.png)\n\n*A log-structured merge tree. Source: [Wikipedia](https://en.wikipedia.org/wiki/Log-structured_merge-tree)*\n\nGoogle popularized LSMs with [LevelDB](https://github.com/google/leveldb), and [RocksDB](https://rocksdb.org/) later forked LevelDB. RocksDB was the underlying storage engine that CockroachDB used in its early years. In 2019, Peter wrote and open sourced [Pebble](https://github.com/cockroachdb/pebble), which is now CockroachDB’s storage engine.\n\n**10\\. Peter, the former C++ readability reviewer at Google, predicts we will stop reviewing code**: At Google, every code change needs to be signed off for “readability” by a language readability reviewer, and Peter was a C++ readability reviewer for years. But these days, he’s reviewing less code and foresees this continuing. As he puts it: “What I’m finding is the agents are getting better; you’re having to give less and less scrutiny. I don’t know if it’s going to be this year or next, \\[but\\] we’re materially going to stop looking at the code in the same way we don’t look at assembly anymore.”\n\n**11\\. Non-engineers at Cockroach Labs built ~1,000 internal apps (!!) in a couple of months.** Earlier this year, Cockroach Labs launched an internal platform for non-devs to build apps (think of it as an “internal Lovable”). Folks jumped at it; HR professionals built new tools they’d only dreamt of, while the CFO created the dashboards they’d always longed for, and many others too. Peter and the engineering team were surprised at this uptake, but it chimes with how OpenAI saw nearly all its non-engineers move almost their entire token spend from ChatGPT to Codex within just 4 months.\n\n**12\\. Does “flow” still exist with AI**? I asked Peter this, and he believes it does, but it’s different from the “coding flow” state: “It definitely feels a bit different. It’s maybe a bit less intense, but you’re managing more things cognitively. I’m thinking of ideas that normally would’ve taken me a week to experiment with. And I think of multiple of these experiments and then fire them off all simultaneously. \\[It feels as\\] if I was a college professor with a whole swarm of research assistants, and they’re all off doing things and it’s coming back *really* rapidly.”\n\n• [Inside Google’s Engineering Culture](https://newsletter.pragmaticengineer.com/p/google)\n\n• [Resiliency in distributed systems](https://newsletter.pragmaticengineer.com/p/resiliency-in-distributed-systems)\n\n• [How to debug large, distributed systems: Antithesis](https://newsletter.pragmaticengineer.com/p/antithesis)\n\n• [Pushing software engineering limits with “napkin math”](https://newsletter.pragmaticengineer.com/p/pushing-software-engineering-limits)\n\n• [Designing Data-intensive Applications with Martin Kleppmann](https://newsletter.pragmaticengineer.com/p/designing-data-intensive-applications)\n\n• [Formal methods with Hillel Wayne](https://newsletter.pragmaticengineer.com/p/formal-methods-with-hillel-wayne)\n\n[00:00](https://www.youtube.com/watch?v=0GzwuYGvKA4) Intro\n\n[02:42](https://www.youtube.com/watch?v=0GzwuYGvKA4&t=162s) Peter’s path into tech\n\n[04:00](https://www.youtube.com/watch?v=0GzwuYGvKA4&t=240s) Building GIMP\n\n[09:30](https://www.youtube.com/watch?v=0GzwuYGvKA4&t=570s) Working on Gmail at Google\n\n[14:51](https://www.youtube.com/watch?v=0GzwuYGvKA4&t=891s) Google’s infra: google3, build files, Bazel, and Colossus\n\n[21:30](https://www.youtube.com/watch?v=0GzwuYGvKA4&t=1290s) Distributed storage bottlenecks\n\n[23:59](https://www.youtube.com/watch?v=0GzwuYGvKA4&t=1439s) Latency, throughput, and availability\n\n[30:04](https://www.youtube.com/watch?v=0GzwuYGvKA4&t=1804s) Contributing to libraries\n\n[41:52](https://www.youtube.com/watch?v=0GzwuYGvKA4&t=2512s) Google Spanner\n\n[46:10](https://www.youtube.com/watch?v=0GzwuYGvKA4&t=2770s) CockroachDB\n\n[52:00](https://www.youtube.com/watch?v=0GzwuYGvKA4&t=3120s) Manual vs. automatic sharding\n\n[55:28](https://www.youtube.com/watch?v=0GzwuYGvKA4&t=3328s) Consistency models and strong consistency\n\n[1:00:03](https://www.youtube.com/watch?v=0GzwuYGvKA4&t=3603s) Raft consensus\n\n[1:06:15](https://www.youtube.com/watch?v=0GzwuYGvKA4&t=3975s) How AI brought Peter back to coding\n\n[1:19:12](https://www.youtube.com/watch?v=0GzwuYGvKA4&t=4752s) Peter’s tools and agentic workflows\n\n[1:23:08](https://www.youtube.com/watch?v=0GzwuYGvKA4&t=4988s) How AI can improve quality\n\n[1:26:39](https://www.youtube.com/watch?v=0GzwuYGvKA4&t=5199s) Code reviews: are they done?\n\n[1:29:17](https://www.youtube.com/watch?v=0GzwuYGvKA4&t=5357s&pp=0gcJCWMAwfN6Pr3D) 100x engineers\n\n[1:35:33](https://www.youtube.com/watch?v=0GzwuYGvKA4&t=5733s) Peter’s advice for leveling up your engineering skills\n\n**Where to find Peter Mattis:**\n\n• X: [https://x.com/petermattis](https://x.com/petermattis)\n\n• LinkedIn: [https://www.linkedin.com/in/peter-mattis-46549144](https://www.linkedin.com/in/peter-mattis-46549144)\n\n**Mentions during the episode:**\n\n• Cockroach Labs: [https://www.cockroachlabs.com](https://www.cockroachlabs.com)\n\n• GIMP: [https://www.gimp.org](https://www.gimp.org)\n\n• Usenet: [https://en.wikipedia.org/wiki/Usenet](https://en.wikipedia.org/wiki/Usenet)\n\n• Larry Page: [https://en.wikipedia.org/wiki/Larry_Page](https://en.wikipedia.org/wiki/Larry_Page)\n\n• Sergey Brin: [https://en.wikipedia.org/wiki/Sergey_Brin](https://en.wikipedia.org/wiki/Sergey_Brin)\n\n• Paul Buchheit on X: [https://x.com/paultoo](https://x.com/paultoo)\n\n• Y Combinator: [https://www.ycombinator.com](https://www.ycombinator.com)\n\n• Delay is Not an Option: Low Latency Routing in Space: [https://discovery.ucl.ac.uk/id/eprint/10062262/7/Handley_hotnets.pdf](https://discovery.ucl.ac.uk/id/eprint/10062262/7/Handley_hotnets.pdf)\n\n• Bazel: [https://opensource.google/projects/bazel](https://opensource.google/projects/bazel)\n\n• Colossus under the hood: a peek into Google’s scalable storage system: [https://cloud.google.com/blog/products/storage-data-transfer/a-peek-behind-colossus-googles-file-system](https://cloud.google.com/blog/products/storage-data-transfer/a-peek-behind-colossus-googles-file-system)\n\n• Pushing software engineering limits with “napkin math”: [https://newsletter.pragmaticengineer.com/p/pushing-software-engineering-limits](https://newsletter.pragmaticengineer.com/p/pushing-software-engineering-limits)\n\n• turbopuffer: [https://turbopuffer.com](https://turbopuffer.com)\n\n• Trimodal Nature of Tech Compensation in the US, UK and India: [https://newsletter.pragmaticengineer.com/p/trimodal](https://newsletter.pragmaticengineer.com/p/trimodal)\n\n• Quicksort: [https://en.wikipedia.org/wiki/Quicksort](https://en.wikipedia.org/wiki/Quicksort)\n\n• B-tree: [https://en.wikipedia.org/wiki/B-tree](https://en.wikipedia.org/wiki/B-tree)\n\n• Go: [https://go.dev](https://go.dev)\n\n• Google Goggles: [https://en.wikipedia.org/wiki/Google_Goggles](https://en.wikipedia.org/wiki/Google_Goggles)\n\n• Spencer Kimball on LinkedIn: [linkedin.com/in/spencerwkimball](http://linkedin.com/in/spencerwkimball)\n\n• Ben Darnell: [https://en.wikipedia.org/wiki/Ben_Darnell](https://en.wikipedia.org/wiki/Ben_Darnell)\n\n• The Ubiquitous B-Tree: [https://dl.acm.org/doi/pdf/10.1145/356770.356776](https://dl.acm.org/doi/pdf/10.1145/356770.356776)\n\n• Paxos: [https://martinfowler.com/articles/patterns-of-distributed-systems/paxos.html](https://martinfowler.com/articles/patterns-of-distributed-systems/paxos.html)\n\n• Raft is so fetch: The Raft Consensus Algorithm explained through “Mean Girls”: [https://www.cockroachlabs.com/blog/raft-is-so-fetch/](https://www.cockroachlabs.com/blog/raft-is-so-fetch/)\n\n• DoorDash: [https://www.doordash.com](https://www.doordash.com)\n\n• RocksDB: [https://rocksdb.org](https://rubberduckdebugging.com)\n\n• Rubber duck debugging: [https://rubberduckdebugging.com](https://rubberduckdebugging.com)\n\n• Claude Code: [https://claude.com/product/claude-code](https://claude.com/product/claude-code)\n\n• Codex: [https://chatgpt.com/codex](https://chatgpt.com/codex)\n\n• Building Claude Code with Boris Cherny: [https://newsletter.pragmaticengineer.com/p/building-claude-code-with-boris-cherny](https://newsletter.pragmaticengineer.com/p/building-claude-code-with-boris-cherny)\n\n• Programming TypeScript: [https://www.oreilly.com/library/view/programming-typescript/9781492037644](https://www.oreilly.com/library/view/programming-typescript/9781492037644)\n\n• Building Codex with Tibo Sottiaux: [https://newsletter.pragmaticengineer.com/p/building-codex-with-tibo-sottiaux](https://newsletter.pragmaticengineer.com/p/building-codex-with-tibo-sottiaux)\n\n• Terence Tao Digests the Jacobian Conjecture Counterexample: How Claude Fable 5 Broke an 87-Year-Old Math Problem: [https://www.developersdigest.tech/blog/jacobian-conjecture-counterexample-fable](https://www.developersdigest.tech/blog/jacobian-conjecture-counterexample-fable)\n\n• Jeff Dean on LinkedIn: [https://www.linkedin.com/in/jeff-dean-8b212555](https://www.linkedin.com/in/jeff-dean-8b212555)\n\n• Sanjay Ghemawat on LinkedIn: [https://www.linkedin.com/in/sanjay-ghemawat-763485428](https://www.linkedin.com/in/sanjay-ghemawat-763485428)\n\n—\n\nProduction and marketing by [Pen Name](https://penname.co/).\n",
  "dateCreated": "2026-09-30T20:48:45.272500234+00:00",
  "date_created": "2026-09-30T16:30:05+00:00",
  "date_modified": "2026-09-30T16:30:05+00:00",
  "date_published": "2026-09-30T16:30:05+00:00",
  "description": "Cockroach Labs co-founder Peter Mattis discusses building reliable distributed systems and how AI helps him write more code without sacrificing quality.",
  "external_url": "https://newsletter.pragmaticengineer.com/p/distributed-databases-with-peter",
  "git": {
    "commit": "ebb0941c5c06a2ab2377342d607535262a3e969e",
    "path": "items/newsletter-pragmaticengineer-com/2026/09/2026-09-30-distributed-databases-with-peter-mattis.md",
    "permalink": "https://github.com/aymericbeaumet/aggr-instance/blob/ebb0941c5c06a2ab2377342d607535262a3e969e/items/newsletter-pragmaticengineer-com/2026/09/2026-09-30-distributed-databases-with-peter-mattis.md"
  },
  "id": "urn:aggr:item:b843ab50b50c25cccd86bfd6df0cc86b9b7bb0de",
  "isBasedOn": {
    "@id": "https://newsletter.pragmaticengineer.com/p/distributed-databases-with-peter"
  },
  "mainEntity": {
    "@id": "https://newsletter.pragmaticengineer.com/p/distributed-databases-with-peter",
    "@type": "CreativeWork",
    "archivedAt": {
      "@id": "https://aggr.aymericbeaumet.com/items/newsletter-pragmaticengineer-com/2026-09-30-distributed-databases-with-peter-mattis/#webpage"
    },
    "author": [
      {
        "@type": "Person",
        "name": "Gergely Orosz"
      }
    ],
    "dateModified": "2026-09-30T16:30:05+00:00",
    "datePublished": "2026-09-30T16:30:05+00:00",
    "headline": "Distributed databases with Peter Mattis",
    "timeRequired": "PT9M",
    "url": "https://newsletter.pragmaticengineer.com/p/distributed-databases-with-peter",
    "wordCount": 1994
  },
  "name": "Archived snapshot: Distributed databases with Peter Mattis",
  "preview": null,
  "reading_minutes": 9,
  "replicated_at": null,
  "source": "newsletter-pragmaticengineer-com",
  "source_name": "The Pragmatic Engineer",
  "summary": "Cockroach Labs co-founder Peter Mattis discusses building reliable distributed systems and how AI helps him write more code without sacrificing quality.",
  "tags": [],
  "temporalCoverage": "2026-09-30T20:48:45.272500234+00:00",
  "title": "Distributed databases with Peter Mattis",
  "url": "https://aggr.aymericbeaumet.com/items/newsletter-pragmaticengineer-com/2026-09-30-distributed-databases-with-peter-mattis/",
  "word_count": 1994
}