{
  "description": "Browse aymeric's, an independent, searchable archive of readable snapshots from followed feeds, preserved in Git with aggr.",
  "feed_url": "https://aggr.aymericbeaumet.com/sources/jxnl.co/feed.json",
  "home_page_url": "https://aggr.aymericbeaumet.com/sources/jxnl.co/",
  "items": [
    {
      "authors": [
        {
          "name": "Jason Liu"
        }
      ],
      "content_html": "<p>There are songs that make me cry that you'll never hear about. There are poems I read while my mother lay dying in front of me, and unless I tell you, you'll never know which ones. My taste in music, in food, in art, all of it is private by default. You'd have to eat with me to know I'd happily live on fried chicken. Nobody reads my diary unless I show them. Nobody comes into my home unless I invite them in.</p>\n<p>But the moment I step outside, they're looking at you. Before I speak, before you know what I do, before you know anything about how I think, you've already seen my taste. Personal style is the most front-loaded expression of taste there is. Every other form you get to keep to yourself. This one you have to subject everyone to. Some days I feel terrible and I put on a fucking blazer. Some days I want to wrap myself up and hide in the shadows, and some days I want to command the room. The clothes go out ahead of me.</p>\n<p>So style is where I want to start, because it's the honest lab. But this essay is really about taste, and why I think it's become the thing most people are missing.</p>\n<h2>Bottlenecks</h2>\n<p>As AI has allowed more and more people to create things, the biggest differentiation, the biggest skill people are lacking, is taste itself.</p>\n<p>What does that mean? Taste is a little bit of aesthetic judgment, knowing what's beautiful. But I think taste is really your ability to model out what people will like. It's curation versus consumption. And with AI there's so much more volume of work now that it's really easy to regress to the mean. In which case it's not really taste. Taste is a kind of risk-taking. You're choosing to deviate from the safe average.</p>\n<h2>Art School</h2>\n<p>I felt the old version of this in art school. My taste exceeded my ability. I could see what good work looked like, I just couldn't make it yet, and I didn't have the words to describe what was going on.</p>\n<p>Now AI flips that. Ability is basically free, so for most people, ability exceeds taste. You can make anything, and you don't know what's worth making.</p>\n<h2>Eating</h2>\n<p>If you want to be a good chef, you have to eat at people's restaurants. You can't just look at the menu.</p>\n<p>Same with clothes. So many people I know refuse to try things on because they're not going to buy them. This is exactly the wrong way to approach it. Go into the store. Try on a bunch of shoes. Try on the weird jacket. Move around in it. Some days you learn something and buy nothing, and that's a fine day.</p>\n<h2>Vocabulary</h2>\n<p>In art school I didn't have the words. This is how you get them.</p>\n<p>As you eat, start asking why you like what you like. Do I like the way this touches my skin, or the way it drapes over my skin? Those are different things. I rarely think about items of clothing in isolation. I'm thinking about materials and draping first, structure and form first. Then lines. Color, for me, is simple: gray, blue, black, red. That's it. Your hierarchy will be different. You only find it by putting things on your body and paying attention.</p>\n<p>That's the whole method, honestly. Eat widely, notice what you notice, and keep a record of why.</p>\n<h2>Transcribing</h2>\n<p>Walter Benjamin talked about mechanical reproduction stripping the aura from art. I think AI does something worse. It gives you the destination without the hours of noticing that used to get you there.</p>\n<p>In music, transcribing a solo forces you to listen deeply. You sit there rewinding the same four bars over and over. AI can just give you the chart. But it gives you the chart, not the ear. So AI doesn't just strip the aura from art, it strips the aura from learning. The felt distance that made a skill feel earned.</p>\n<p>This has been true for how I learned pottery. I can't mechanistically learn pottery. I just have to feel and listen to my intuition.</p>\n<h2>Wandering</h2>\n<p>There are video games that don't allow minimaps, because the designers felt that with waypoints, players would just walk from one area to another and not really notice where anything is.</p>\n<p>We do the same thing to ourselves. We don't wander down streets anymore. We take Ubers that take us there directly. We used to learn places by wandering. Now we follow the blue line.</p>\n<p>If taste grows with attention, and the tools keep removing the attention, then you have to put it back on purpose.</p>\n<h2>Authorship</h2>\n<p>The same collapse happened to the things we consume.</p>\n<p>Think about what TikTok did. On YouTube, on early Instagram and Twitter, you followed a person. The person made things, and you saw the things because you'd chosen the person. TikTok inverted it. You don't follow anyone. The content just floods in, and the creator is secondary to the content. It's like never reading books, only screenshots of pages. Songs are a minute long now because the algorithm optimized them down, and people listen to forty-second soundbites without ever learning the name of the song, let alone who's playing on it.</p>\n<p>It's not just content. Nobody's eating food from their mom anymore. Nobody's thinking about who's cooking the food. And look at how people talk about clothing trends. It's cowboy boots, it's jorts, it's the Chanel glasses. A kind of object. Nobody's asking about the creative director, the house, who actually made this thing and why.</p>\n<p>Here's what making things yourself teaches you: everything, someone had to make. How deep a pocket goes, how wide the pant leg is, even the curvature of a seam. Someone had to think about that. When you have low consciousness about this, you go, &quot;these are pants, I'm going to put on these pants.&quot; The app is the way it is because it is the way it is. But nothing is the way it is by accident. Every object is a pile of decisions somebody made, and taste is mostly learning to see the decisions.</p>\n<h2>Eating Slowly</h2>\n<p>So the practice I'd recommend goes past consuming the object. Go find the human.</p>\n<p>With music, this is the clearest version. I don't just listen to the single the algorithm hands me. I listen to the whole album, on record. And then I go read. Who's the drummer? Who's the bassist? If I love the drums on this record, what else has this drummer played on? Who produced it? When was it made? Oh, this was made right after his father died. Now I'm not just consuming a sound. I'm in a relationship with the circumstances that produced it.</p>\n<p>I do the same thing with clothes. I buy from very few brands, and I think about the designer first, then explore within those constraints. Yohji Yamamoto when I want all black and drapey. Kapital when I want a little more fun. Extreme Cashmere when I want to be elevated and relaxed. Rick Owens when I want to be more aggressive and intense. Most of my shoes are just leather shoes. That's basically the whole roster. It's a narrow bet, and I'm fine with that, because taste was never about covering the whole map.</p>\n<p>And I've developed relationships. When I go to New York I'll text someone at the Yohji store, someone at the Extreme Cashmere store, tell them I'm coming. I'll try things on for forty minutes, an hour, chit-chat. Some days I buy nothing. Some days a couple of pieces. But that friction, the texting ahead, the conversation, the hour of just being in the clothes, is what makes it rewarding. We've spent twenty years optimizing friction out of consumption, and the friction was where the taste was. Knowing who made a thing is just eating slowly.</p>\n<h2>Voting</h2>\n<p>Yes, the things I buy are expensive. That's not a knock on thrifting, it's probably a critique of myself. But I don't think I'm buying material value. When I spend money, I'm voting. For a designer, for a way of making things, for a set of decisions I want more of in the world. That money came from my time, my attention, my work. The trade feels fair to me, because I think I'm expensive too.</p>\n<h2>Posters</h2>\n<p>And once you've paid attention like this for long enough, you start composing in the other direction. I think of dressing like a poster. From across the street, an impression. Come closer and you see the fabrication, the detail in a seam. Talk to me and you get something else, and eventually I'm telling you some random fact about my free diving. Clothing built in layers, for people who come closer.</p>\n<p>You spend years learning to notice what others made, and then you make something worth noticing.</p>\n<h2>Rikyū</h2>\n<p>There's a story in the Book of Tea about Kobori Enshū, the tea master who came after Rikyū. A student told him, your taste must be even better than Rikyū's, because everything you collect, everyone agrees is beautiful. Rikyū's collection, only Rikyū could appreciate. And Enshū said no, that's exactly why I'm the lesser man. I was just picking what most people would like. Rikyū had the courage to love things only he could see.</p>\n<p>That's the whole thing, in one story. Taste isn't consensus. It's not judgment first, either. It's easy to be a critic. Curiosity has to come first, because without curiosity you never broaden your horizons. What's left after the curiosity is courage: being honest enough to love what actually resonates with you, even when nobody else is wearing it or listening to it or buying it.</p>\n<p>AI shifted the bottleneck from making to noticing. Ability is free. Attention is not. If the tools remove the attention, apprenticeship has to be reinvented around attention, and you have to put it back yourself. Eat at the restaurants. Try on the clothes. Transcribe the solo. Learn the drummer's name. Wander without the blue line.</p>\n<p>If you want taste, you're gonna have to eat.</p>\n",
      "date_modified": "2026-07-05T00:00:00+00:00",
      "date_published": "2026-07-05T00:00:00+00:00",
      "external_url": "https://jxnl.co/writing/2026/07/05/taste/",
      "id": "urn:aggr:item:be36e46a82b9a734fa3e09e567ea6c48afe7a0bc",
      "summary": "A reflection on taste, style, attention, and why AI makes the ability to notice matter more than ever.",
      "tags": [
        "programming",
        "ai",
        "attention",
        "personal",
        "style",
        "taste"
      ],
      "title": "If You Want Taste, You're Gonna Have to Eat",
      "url": "https://aggr.aymericbeaumet.com/items/jxnl-co/2026-07-05-if-you-want-taste-you-re-gonna-have-to-eat/"
    },
    {
      "authors": [
        {
          "name": "Jason Liu"
        }
      ],
      "content_html": "<p>Most automation language is more complicated than the job.</p>\n<p>You want Codex to do something later, or keep checking something until it changes. That sounds like one feature. It is actually two different kinds of work, and the difference is simple:</p>\n<ul>\n<li><strong>Scheduled Tasks</strong> create a new thread every time they run.</li>\n<li><strong>Scheduled Messages</strong> use the same existing thread every time they run.</li>\n</ul>\n<p>That is the whole model.</p>\n<h2>Use a Scheduled Task when every run can start fresh</h2>\n<p>A Scheduled Task is best when the job makes sense without the conversation that created it.</p>\n<p>For example:</p>\n<pre><code class=\"language-text\"><span class=\"code-snippet\">Every morning at 9 AM, summarize what I need to catch up on from my email, calendar, and team messages.\n</span></code></pre>\n<p>Tomorrow's summary does not need to remember today's summary. It needs the same instructions, current information, and a fresh place to report the result.</p>\n<p>This is also useful when one task should run across several projects, or when you want each run to appear separately in Triage. Each run is its own thread. The schedule is the thing that connects them.</p>\n<h2>Use a Scheduled Message when the next check needs the thread</h2>\n<p>A Scheduled Message, sometimes called a thread automation, returns to the same existing thread each time it runs.</p>\n<p>For example:</p>\n<pre><code class=\"language-text\"><span class=\"code-snippet\">Check this PR every 30 minutes. If there are comments, address them and keep CI green. Stop when the PR merges.\n</span></code></pre>\n<p>The next check depends on the work that already happened. The thread knows which PR you mean, which comments were addressed, what failed in CI, and what has changed since the last check.</p>\n<p>This is the right shape for:</p>\n<ul>\n<li>polling for updates</li>\n<li>checking for a status change</li>\n<li>ongoing research or triage</li>\n<li>work with a clear stopping condition</li>\n</ul>\n<p>The thread is the thing that connects the runs.</p>\n<h2>The decision rule is context</h2>\n<p>Ask one question:</p>\n<blockquote>\n<p>If this ran tomorrow, would it need the earlier conversation?</p>\n</blockquote>\n<p>If no, use a Scheduled Task. Give it a durable prompt and let each run start clean.</p>\n<p>If yes, use a Scheduled Message. Keep the work in one thread so each check can build on the decisions and results that came before it.</p>\n<p>The schedule matters, but it is not the main decision. The main decision is where the useful context should live.</p>\n<h2>Make your own loop skill</h2>\n<p>I do not want to remember the setup questions every time, so I would make a small skill that turns a rough request into the right scheduled workflow.</p>\n<p>Give Codex this prompt:</p>\n<pre><code class=\"language-text\"><span class=\"code-snippet\">Create a reusable loop skill for scheduled work.\n\nWhen I give it a request, first decide whether each run can start fresh or whether the next check needs the current thread&#39;s context.\n\nIf each run can start fresh, help me create a Scheduled Task.\nIf the next check needs the current thread, help me create a Scheduled Message.\n\nInfer what you can from the conversation. Ask only the missing questions that materially change the workflow:\n- What should Codex do each time?\n- How often should it run?\n- What change is important enough to report?\n- When should it stop?\n- When should it ask me for input?\n\nThen create the scheduled workflow with a short, durable prompt that will still make sense on a later run.\n</span></code></pre>\n<p>The best automations do not start with a complicated system. They start with knowing whether the next run needs a clean slate or the same thread.</p>\n",
      "date_modified": "2026-06-28T00:00:00+00:00",
      "date_published": "2026-06-28T00:00:00+00:00",
      "external_url": "https://jxnl.co/writing/2026/06/28/two-kinds-of-scheduled-work-in-codex/",
      "id": "urn:aggr:item:7fe0189116ee42625a9a08ff2501c462ef43e38a",
      "summary": "A simple rule for choosing between Scheduled Tasks and Scheduled Messages in Codex.",
      "tags": [
        "programming",
        "applied ai",
        "codex",
        "personal",
        "personal software",
        "workflows"
      ],
      "title": "Two kinds of scheduled work in Codex",
      "url": "https://aggr.aymericbeaumet.com/items/jxnl-co/2026-06-28-two-kinds-of-scheduled-work-in-codex/"
    },
    {
      "authors": [
        {
          "name": "Jason Liu"
        }
      ],
      "content_html": "<p>Recently I had a package stolen. Amazon told me it would take about 25 minutes to connect me to a person, so I gave the thread Computer Use and told it to check every five minutes. Once someone joined, check every minute and do its best to get me a refund.</p>\n<p>I went to take a shower. When I came back, the refund was done.</p>\n<p>That was when Computer Use clicked for me. Codex is not just a coding agent. Most of my work is messages, forms, browser tabs, and apps that do not connect neatly to anything else.</p>\n<p>Codex now has three ways to work there:</p>\n<table>\n<thead>\n<tr>\n<th>Use</th>\n<th>Best for</th>\n<th>Start with</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>Computer Use</td>\n<td>Native apps and work across apps</td>\n<td><code>@Computer</code>, <code>@AppName</code>, an Appshot, or Remote Control</td>\n</tr>\n<tr>\n<td>Chrome</td>\n<td>Signed-in sites and multiple tabs</td>\n<td><code>@Chrome</code></td>\n</tr>\n<tr>\n<td>In-app browser</td>\n<td>Pages, artifacts, and visual review</td>\n<td><code>@Browser</code></td>\n</tr>\n</tbody>\n</table>\n<p>My rule is to use the narrowest tool that can finish the job. If a plugin works, use it. If the task needs a screen, choose among these three.</p>\n<h2>1. Computer Use: almost any app</h2>\n<p><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://developers.openai.com/codex/app/computer-use\">Computer Use</a> is the most powerful option and usually the slowest. It can operate Spotify, finance apps, System Settings, or iPhone Mirroring. But it has to look at the screen, click, wait, and look again.</p>\n<p>On a Mac, it can work in the background while you keep using your computer. On Windows, it takes over the active desktop.</p>\n<p>Install it from <strong>Settings &gt; Computer Use</strong>. On macOS, grant Screen Recording and Accessibility permissions. Then mention <code>@Computer</code>, mention an app directly with <code>@AppName</code>, or just ask Codex to use Computer Use.</p>\n<p>The stolen-package example also shows the difference between a Goal and a thread automation. The Goal was to get the refund. The automation woke the thread every five minutes, then every minute when the agent appeared. One supplied the finish line; the other let Codex wait without forgetting the job.</p>\n<h3>People are already using it for stranger things</h3>\n<p>I asked people for <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://x.com/jxnlco/status/2066547199983473013\">the most ambitious things they had done with Computer Use</a>. Someone <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://x.com/thevedlabs/status/2066577349273874617\">applied for a Schengen visa</a>. Someone else <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://x.com/GabGarrett/status/2066584487161467090\">paid monthly Airbnb taxes through a county website</a>. One person <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://x.com/craigsdennis/status/2066550916158009401\">repaired a virus-filled PC through a reboot loop</a>. Another used iPhone Mirroring for <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://x.com/ChristianInProd/status/2066682841249796481\">iOS Screen Time</a>. I immediately wondered whether the same setup could <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://x.com/jxnlco/status/2066618356958896428\">operate a dating app</a>.</p>\n<p>I have also used Computer Use as the last mile when another tool stopped short. One thread read feedback from Slack, changed a video, and rendered a new version. The Slack integration could not upload files, so Computer Use clicked <strong>Add file</strong> and posted it.</p>\n<h3>Appshots let me point</h3>\n<p>An <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://developers.openai.com/codex/appshots\">Appshot</a> sends the frontmost Mac window to a Codex thread. Press both Command keys, then say what you want.</p>\n<p>I use this for emails, Slack messages, errors, designs, settings panels, and forms. Sometimes the Appshot is enough. If Codex needs to act, I tell the same thread to continue with Computer Use.</p>\n<p>Appshots are how I point. Computer Use is how Codex acts.</p>\n<h3>Remote Control lets me leave</h3>\n<p><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://developers.openai.com/codex/remote-connections\">Remote Control</a> lets me start or steer the same work from my phone. In the Codex app, click <strong>Set up Codex mobile</strong>, scan the QR code, and connect ChatGPT mobile. The host stays awake and online; the work still runs there with its projects, plugins, Chrome setup, and Computer Use permissions.</p>\n<p><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://x.com/Dimillian\">Thomas Ricouard</a> has been pushing this part of Codex forward. He started with <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://x.com/Dimillian/status/2010330190510273016\">Codex Monitor</a>, where every active and past session was visible so you could jump back into the work. Codex Mobile now has <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://x.com/Dimillian/status/2057452433777807486\">notifications when work finishes and better reconnection</a>, followed by <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://x.com/Dimillian/status/2060215793971786098\"><code>/side</code> conversations</a>. Remote Control is becoming less like a status screen and more like Codex in your pocket.</p>\n<p>I let Computer Use research and draft. I stay present for payments, credentials, account changes, and security settings. Follow <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://x.com/AriX\">Ari Weinstein</a> for Computer Use and Appshots, and <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://x.com/Dimillian\">Thomas</a> for Codex Mobile and Remote Control.</p>\n<h2>2. Chrome: your signed-in browser</h2>\n<p>Use the <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://developers.openai.com/codex/app/chrome-extension\">Chrome extension</a> when Codex needs your Chrome profile, cookies, extensions, or an authenticated site. It can control several tabs in a thread and keep working in its tab group while you use Chrome elsewhere.</p>\n<p>Open <strong>Plugins</strong>, add <strong>Chrome</strong>, and install the <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://chromewebstore.google.com/detail/codex/hehggadaopoacecdllhhajmbjkdcmajg\">Codex Chrome extension</a>. Once it says <strong>Connected</strong>, mention <code>@Chrome</code>.</p>\n<h3>Make the music more interesting</h3>\n<p>In one thread, I gave Codex an open Strudel Composer tab and told it to make the music more interesting. It rewrote the harmony and arrangement, changed the tempo, saved the track, and left it playing.</p>\n<h3>A thread that watches Twitter</h3>\n<p>I also keep a Twitter thread that uses Chrome to check DMs, read news, and collect product feedback. It writes anything durable to my vault but does not post or send messages.</p>\n<p>That is my default boundary for Chrome: research and draft freely; ask before sending, publishing, purchasing, or submitting.</p>\n<h2>3. The in-app browser: review and annotate</h2>\n<p>The <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://developers.openai.com/codex/app/browser\">in-app browser</a> lives inside a Codex thread. I use it for public pages and visual artifacts I want to review with Codex. It does not carry your Chrome profile or login state.</p>\n<p>Open <strong>Plugins</strong>, add <strong>Browser</strong>, and mention <code>@Browser</code>.</p>\n<h3>The page becomes the specification</h3>\n<p>The annotation tools are my favorite part. I often ask Codex to create a single <code>index.html</code>, open it in the browser, and wait for comments. Then I can click an element and say “this hierarchy is backwards,” “make this less card-heavy,” or “these controls need more room.” The comment includes the screenshot and element context, so I do not have to describe the whole page again.</p>\n<h3>Hand the work to another tool</h3>\n<p>The browser can also establish context and hand the job to another tool. I once opened an X post and asked Codex to investigate it. The page showed which post I meant; the Twitter CLI then retrieved 38 replies, including nested replies that were not visible on the page.</p>\n<p>The Atlas browser team is working to make the in-app browser more powerful: not just a preview, but a place where you and Codex can work on the same thing together. They are cooking. Follow <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://x.com/JamesZmSun\">James Sun</a> to see what they are working on.</p>\n<h2>Put the routing rule in AGENTS.md</h2>\n<p>I do not want to remember this routing every time, so I put it in <code>AGENTS.md</code>:</p>\n<pre><code class=\"language-md\"><span class=\"code-snippet\" data-language=\"Markdown\"><span class=\"syntax-text syntax-html syntax-markdown\"><span class=\"syntax-meta syntax-block-level syntax-markdown\"><span class=\"syntax-markup syntax-heading syntax-2 syntax-markdown\"><span class=\"syntax-punctuation syntax-definition syntax-heading syntax-begin syntax-markdown\">##</span> </span><span class=\"syntax-markup syntax-heading syntax-2 syntax-markdown\"><span class=\"syntax-entity syntax-name syntax-section syntax-markdown\">Browser and desktop routing</span><span class=\"syntax-meta syntax-whitespace syntax-newline syntax-markdown\">\n</span></span></span>\n<span class=\"syntax-markup syntax-list syntax-unnumbered syntax-bullet syntax-markdown\"><span class=\"syntax-punctuation syntax-definition syntax-list_item syntax-markdown\">-</span></span><span class=\"syntax-markup syntax-list syntax-unnumbered syntax-markdown\"> <span class=\"syntax-meta syntax-paragraph syntax-list syntax-markdown\">Prefer a plugin when one can finish the task.\n</span></span><span class=\"syntax-markup syntax-list syntax-unnumbered syntax-bullet syntax-markdown\"><span class=\"syntax-punctuation syntax-definition syntax-list_item syntax-markdown\">-</span></span><span class=\"syntax-markup syntax-list syntax-unnumbered syntax-markdown\"><span class=\"syntax-meta syntax-paragraph syntax-list syntax-markdown\"> Use an Appshot when showing the frontmost Mac window is easier than describing it.\n</span></span><span class=\"syntax-markup syntax-list syntax-unnumbered syntax-bullet syntax-markdown\"><span class=\"syntax-punctuation syntax-definition syntax-list_item syntax-markdown\">-</span></span><span class=\"syntax-markup syntax-list syntax-unnumbered syntax-markdown\"><span class=\"syntax-meta syntax-paragraph syntax-list syntax-markdown\"> Use <span class=\"syntax-markup syntax-raw syntax-inline syntax-markdown\"><span class=\"syntax-punctuation syntax-definition syntax-raw syntax-begin syntax-markdown\">`</span>@Browser<span class=\"syntax-punctuation syntax-definition syntax-raw syntax-end syntax-markdown\">`</span></span> for public pages, visual review, and annotations.\n</span></span><span class=\"syntax-markup syntax-list syntax-unnumbered syntax-bullet syntax-markdown\"><span class=\"syntax-punctuation syntax-definition syntax-list_item syntax-markdown\">-</span></span><span class=\"syntax-markup syntax-list syntax-unnumbered syntax-markdown\"><span class=\"syntax-meta syntax-paragraph syntax-list syntax-markdown\"> Use <span class=\"syntax-markup syntax-raw syntax-inline syntax-markdown\"><span class=\"syntax-punctuation syntax-definition syntax-raw syntax-begin syntax-markdown\">`</span>@Chrome<span class=\"syntax-punctuation syntax-definition syntax-raw syntax-end syntax-markdown\">`</span></span> for signed-in sites, existing tabs, extensions, or multiple tabs.\n</span></span><span class=\"syntax-markup syntax-list syntax-unnumbered syntax-bullet syntax-markdown\"><span class=\"syntax-punctuation syntax-definition syntax-list_item syntax-markdown\">-</span></span><span class=\"syntax-markup syntax-list syntax-unnumbered syntax-markdown\"><span class=\"syntax-meta syntax-paragraph syntax-list syntax-markdown\"> Use <span class=\"syntax-markup syntax-raw syntax-inline syntax-markdown\"><span class=\"syntax-punctuation syntax-definition syntax-raw syntax-begin syntax-markdown\">`</span>@Computer<span class=\"syntax-punctuation syntax-definition syntax-raw syntax-end syntax-markdown\">`</span></span> for native apps or work across apps.\n</span></span><span class=\"syntax-markup syntax-list syntax-unnumbered syntax-bullet syntax-markdown\"><span class=\"syntax-punctuation syntax-definition syntax-list_item syntax-markdown\">-</span></span><span class=\"syntax-markup syntax-list syntax-unnumbered syntax-markdown\"><span class=\"syntax-meta syntax-paragraph syntax-list syntax-markdown\"> Use Remote Control to start or steer work from my phone.\n</span></span><span class=\"syntax-markup syntax-list syntax-unnumbered syntax-bullet syntax-markdown\"><span class=\"syntax-punctuation syntax-definition syntax-list_item syntax-markdown\">-</span></span><span class=\"syntax-markup syntax-list syntax-unnumbered syntax-markdown\"><span class=\"syntax-meta syntax-paragraph syntax-list syntax-markdown\"> Ask before sending, publishing, purchasing, submitting, or changing account settings.\n</span></span></span></span></code></pre>\n<h2>Follow the work</h2>\n<p>Follow <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://x.com/jxnlco\">me (@jxnlco)</a> for the ways I am using these tools, <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://x.com/AriX\">Ari Weinstein</a> for Computer Use, <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://x.com/Dimillian\">Thomas Ricouard</a> for Codex Mobile, <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://x.com/JamesZmSun\">James Sun</a> for browser work, <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://x.com/ajambrosino\">Andrew Ambrosino</a> for Codex app updates, and <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://x.com/OpenAIDevs\">OpenAI Developers</a> for broader news.</p>\n<p>The interesting part is not that Codex can click buttons. A thread can start with an Appshot, move across apps and browsers, wait for someone else, and keep working after I leave.</p>\n<h2>What should I write about next?</h2>\n<p>If you want me to go deeper on Goals, thread automations, Appshots, Remote Control, or how I route work between these tools, leave a comment. Tell me what you are trying to do; I will use the best questions for the next post.</p>\n",
      "date_modified": "2026-06-16T00:00:00+00:00",
      "date_published": "2026-06-16T00:00:00+00:00",
      "external_url": "https://jxnl.co/writing/2026/06/16/three-ways-codex-can-use-a-computer/",
      "id": "urn:aggr:item:7a80421703446e961dc2382b30364e77b99976ee",
      "summary": "When to use Computer Use, Chrome, or the in-app browser in Codex.",
      "tags": [
        "programming",
        "applied ai",
        "codex",
        "computer use",
        "personal",
        "personal software",
        "workflows"
      ],
      "title": "Three Ways Codex Can Use a Computer",
      "url": "https://aggr.aymericbeaumet.com/items/jxnl-co/2026-06-16-three-ways-codex-can-use-a-computer/"
    },
    {
      "authors": [
        {
          "name": "Jason Liu"
        }
      ],
      "content_html": "<p>This is the easiest way I know to teach someone how to use AI.</p>\n<p>Not by starting with models, agents, or some abstract taxonomy of what the technology can do. Start with a job people already understand, then make that job quietly more powerful.</p>\n<p>The morning brief works because almost everyone already has one. They just assemble it badly.</p>\n<p>I think the morning brief is the first Codex workflow that normal people actually understand.</p>\n<p>You wake up, open Slack, check your calendar, click into email, forget why you opened email, go back to Slack, then realize you have a meeting in seven minutes and no idea what happened yesterday.</p>\n<p>The appeal is simple: help me remember what is going on.</p>\n<p>When we were talking about Codex onboarding, this was the first workflow that felt both boring enough to teach and strong enough to matter. It starts as a dumb little orientation prompt. If you keep pushing it, it turns into a pretty good model for how people actually graduate into using Codex seriously.</p>\n<p>Start with the thing a beginner can understand. Then add one real capability at a time until the shape of the whole system becomes obvious.</p>\n<p>I think there are six real levels.</p>\n<h2>Level 1: ask your actual day what is going on</h2>\n<p>The first version is almost embarrassingly small.</p>\n<p>Connect Slack, Gmail, and Calendar, then ask:</p>\n<p><picture class=\"article-picture article-picture-fallback\" style=\"--image-ratio:16 / 9\"><img src=\"https://jxnl.co/writing/img/codex-plugins-screen.png\" alt=\"Codex plugins screen showing Slack, Gmail, Calendar, and other installed tools\" loading=\"eager\" decoding=\"async\" fetchpriority=\"high\" referrerpolicy=\"no-referrer\" class=\"progressive-image\"></picture></p>\n<blockquote>\n<p>Using Slack, Gmail, and Calendar, what do I have going on today?</p>\n</blockquote>\n<p>Just see whether Codex can look across the three places your work usually hides and tell you something you actually care about.</p>\n<p>Maybe it notices the Slack thread where someone is waiting on you. Maybe it finds the meeting you forgot to prepare for. Maybe it catches the email that changes what that meeting is even about.</p>\n<p>If that gives you a cleaner first ten minutes of the day, it worked.</p>\n<h2>Level 2: add an agents file</h2>\n<p>The second level is adding a little bit of persistent instruction.</p>\n<p>You already have enough raw surface area for Codex to be useful. Now you need to stop it from being generically helpful every single morning.</p>\n<p>This is where an agents file becomes useful. Not as a piece of infrastructure trivia, but as a place to say how you want this work shaped by default.</p>\n<p>For the morning brief, that might look like:</p>\n<pre><code class=\"language-text\"><span class=\"code-snippet\">Give me a morning brief.\n\nFocus on:\n- people waiting on me\n- meetings that need preparation\n- commitments I made yesterday that are still open\n- anything that changed overnight and affects today&#39;s priorities\n\nUse clear headers and bullet points.\n\nOrganize the brief into:\n- To-dos\n- Needs replies\n- Project-level updates\n\nFor anything in &quot;Needs replies&quot;:\n- include the direct Slack link\n- say what the person is waiting on\n- include just enough context for me to decide whether to answer now\n\nKeep it short, structured, and easy to scan.\n</span></code></pre>\n<p>You do not need to worry about where the file lives. Just tell Codex:</p>\n<pre><code class=\"language-text\"><span class=\"code-snippet\">I want you to save this in your agents file so you use it every time you prepare my morning brief.\n</span></code></pre>\n<p>Then paste in the instructions above. The point is to define your taste once, then let every future brief start from there.</p>\n<p>You are not trying to sound sophisticated. You are trying to make tomorrow morning less annoying than this morning.</p>\n<p>For other jobs, the instructions change. Recruiters might want briefs grouped by candidate. Engineers might want blockers separated from code review. Comms people might want external scans split from internal asks. But the move is the same: first give Codex the real context, then give it your defaults.</p>\n<h2>Level 3: ask it to keep an eye on this</h2>\n<p>The third level is adding recurrence, but I do not think I would teach it as &quot;create an automation.&quot;</p>\n<p>I would say:</p>\n<blockquote>\n<p>Keep an eye on this every weekday morning.</p>\n</blockquote>\n<p>That is the instruction people actually remember.</p>\n<p>Under the hood, yes, you are turning the morning brief into a recurring automation. In practice, you are just saying: this is useful enough that I want it to come back on its own.</p>\n<p>The unit of value stops being, &quot;I remembered to ask.&quot; It becomes, &quot;the brief is already waiting.&quot;</p>\n<p>And because it lives in the same thread, you can keep tuning it without rewriting the prompt from scratch. If it overweights calendar events, say so. If it keeps missing open loops from Slack, say so. If it should separate <code>needs reply</code>, <code>needs prep</code>, and <code>worth knowing</code>, teach it once and let the thread carry the preference forward.</p>\n<p>This is why I like the thread version more than a generic scheduled report. The brief improves as you complain about it.</p>\n<p>The recurring prompt can stay simple:</p>\n<pre><code class=\"language-text\"><span class=\"code-snippet\">Keep an eye on this every weekday morning.\nCheck Slack, Gmail, and Calendar.\nTell me:\n- what needs my attention\n- what I should prepare for\n- what changed that I might miss\n- what seems blocked on me\n\nKeep it short.\nIf nothing important changed, say that.\n</span></code></pre>\n<p>At this point the morning brief starts paying rent. You wake up to something that already did the first pass through the sludge.</p>\n<h2>Level 4: split it into multiple project-level briefs</h2>\n<p>Eventually one brief gets mushy.</p>\n<p>This came up directly in the conversation. I do not really want one universal daily assistant forever. I usually have multiple project threads, and each one gives me its own morning brief.</p>\n<p>One thread for a launch.</p>\n<p>One for a project.</p>\n<p>One for open source.</p>\n<p>One for chief-of-staff-ish personal triage.</p>\n<p>Each thread has a different definition of &quot;important.&quot;</p>\n<p>The project thread cares about blockers, PRs, decisions, and whether some promised draft ever got finished.</p>\n<p>The recruiting thread might care about interview briefs for candidates coming in today and whether any context changed overnight.</p>\n<p>The personal thread cares about messages, scheduling, travel, and whatever low-grade obligation is quietly sitting in the background.</p>\n<p>This is where pinned threads start to make obvious sense. A recurring morning brief is not just a report. It is a way to keep each project warm on its own terms.</p>\n<p>The reason the prompt gets better is that the thread already knows which &quot;launch thing&quot; you meant.</p>\n<h2>Level 5: let the brief draft the work</h2>\n<p>At this level, the morning brief should not only tell me what happened. It should draft the obvious work that follows from the brief.</p>\n<p>Draft the Slack reply, but do not send it.</p>\n<p>Pull together the candidate brief.</p>\n<p>Write the meeting prep note.</p>\n<p>Summarize the one thread I should read before I reply.</p>\n<p>Tell me which decision looks stuck on me.</p>\n<p>A good Level 5 brief might end like this:</p>\n<blockquote>\n<p>Here are the three messages I would reply to first. I drafted responses for each. Here are the two meetings that deserve prep. Here is the one decision that seems to be stuck on you.</p>\n</blockquote>\n<p>I want the brief to leave a few obvious pieces of work half-done for me, not just hand me another summary to read.</p>\n<p>This is also where I start using the phone more. If the context gathering already happened before I opened my laptop, I can glance at the brief while I am moving around, approve one small thing, or just decide what actually deserves my attention when I sit down.</p>\n<h2>Level 6: save what matters in a memory vault</h2>\n<p>The sixth level is where the brief stops being only a morning ritual and starts feeding a memory system.</p>\n<p>If the morning brief keeps finding the same people, same projects, same open loops, and same decisions, some of that should leave the thread and become durable.</p>\n<p>Write the important bits down.</p>\n<p>Update the project note.</p>\n<p>Record the open loop.</p>\n<p>Add the decision that should not be forgotten.</p>\n<p>Preserve the context that will make next week's brief better.</p>\n<p>This is the point where I start reaching for a vault. The short version is simple: I like memory as files I can inspect, diff, and reuse across threads.</p>\n<p>The smallest useful version might look like this:</p>\n<pre><code class=\"language-text\"><span class=\"code-snippet\">vault/\n├── TODO.md\n├── people/\n├── projects/\n├── daily/\n├── notes/\n└── AGENTS.md\n</span></code></pre>\n<p>The outline is pretty simple:</p>\n<ul>\n<li><code>TODO.md</code> keeps open loops from disappearing into chat history.</li>\n<li><code>people/</code> stores durable context about recurring collaborators.</li>\n<li><code>projects/</code> stores the state of important workstreams.</li>\n<li><code>daily/</code> gives each day a place to leave behind the facts that mattered.</li>\n<li><code>notes/</code> is where looser research or one-off memos can land.</li>\n<li><code>AGENTS.md</code> tells Codex how to use the vault without inventing structure every time.</li>\n</ul>\n<p>You should also update your agents file so the brief reads from the vault before it speaks. Something like:</p>\n<pre><code class=\"language-text\"><span class=\"code-snippet\">Before writing my morning brief:\n- review TODO.md for open loops\n- review people/ for relevant collaborator context\n- review projects/ for active workstream state\n- review notes/ for recent background that may change today&#39;s priorities\n\nUse that vault context to make the brief more specific and less forgetful.\n</span></code></pre>\n<p>Once the brief has enough structure, you can also ask Codex to use subagents to search in parallel. One can review open to-dos, one can scan project notes, another can look at recent people context, then the main thread can assemble the actual brief and update the underlying documents when something material changed. You do not need that on day one, but it becomes useful once the brief has enough moving parts that a single serial pass starts to feel slow, shallow, or stale.</p>\n<p>You can also ask Codex to review the work you are already doing and ask you questions that help organize the vault better. It might notice that the same person appears across three projects, that one project has no clear note yet, or that your <code>TODO.md</code> is mixing personal errands with launch blockers. That back-and-forth is useful. The vault gets better when Codex is allowed to say, &quot;I think this wants a better home. Where should it live?&quot;</p>\n<p>The morning brief gets better when it can read from that vault before it runs and update the vault after something material changes. Tomorrow's project brief can remember the open decision from last week, the person who still needs a reply, or the status note that should not vanish just because the thread got long.</p>\n<h2>Why I like this ladder</h2>\n<p>Each level teaches a more serious way of using Codex without asking the user to jump straight into some grand agent architecture.</p>\n<p>Level 1 teaches the connectors.</p>\n<p>Level 2 teaches defaults through an agents file.</p>\n<p>Level 3 teaches recurrence by asking Codex to keep an eye on this.</p>\n<p>Level 4 teaches project-level briefs in durable threads.</p>\n<p>Level 5 teaches the trust boundary I actually care about: draft the work, do not impersonate me.</p>\n<p>Level 6 teaches the part that compounds: save durable context in a vault so tomorrow's brief starts smarter than today's.</p>\n<p>That is why I would teach Codex through the morning brief. It begins as a simple summary and ends as a miniature operating system for your work.</p>\n",
      "date_modified": "2026-05-18T00:00:00+00:00",
      "date_published": "2026-05-18T00:00:00+00:00",
      "external_url": "https://jxnl.co/writing/2026/05/18/six-levels-of-complexity-of-an-ai-powered-morning-brief-with-codex/",
      "id": "urn:aggr:item:0e578347b01e47ff0d8259d9dfad15b41d35d625",
      "summary": "A simple way to teach Codex through an AI-powered morning brief that grows into automations, project threads, drafted work, and memory vaults.",
      "tags": [
        "programming",
        "applied ai",
        "codex",
        "personal",
        "personal software",
        "workflows"
      ],
      "title": "Six levels of complexity in a Codex morning brief",
      "url": "https://aggr.aymericbeaumet.com/items/jxnl-co/2026-05-18-six-levels-of-complexity-in-a-codex-morning-brief/"
    },
    {
      "authors": [
        {
          "name": "Jason Liu"
        }
      ],
      "content_html": "<p>I was already using coding agents a lot before Codex. Mostly, though, I used them through interfaces built for coding work: making diffs, changing repos, and shipping code.</p>\n<p>Around November, I started pushing them into knowledge work too. I made presentations in <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://sli.dev/\">Slidev</a>, used agents more like note-takers with voice input, and kept looking for other artifacts a coding agent could help me produce: an <code>index.html</code>, a PDF, a spreadsheet, a slide deck.</p>\n<p>The latest Codex app upgrades are the first thing I've used that make that broader mode feel native. Codex is still excellent for coding, but the more interesting shift is that it gives my work somewhere to live.</p>\n<p>What changed my behavior was learning to give work an operating loop: a durable thread, shared memory, tools that can act on my computer, ways to steer and resume the task, and a surface where I can review the artifact itself.</p>\n<h2>Durable threads</h2>\n<p>The first thing that changed my behavior was compaction.</p>\n<p>Compaction</p>\n<p>Compaction</p>\n<p>Compressing a long-running thread so it can keep going without carrying every old message in full.</p>\n<p>I now keep a pinned thread for every important workstream I care about:</p>\n<ul>\n<li>my Chief of Staff thread</li>\n<li><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://openai.com/index/the-next-evolution-of-the-agents-sdk/\">the Agents SDK</a></li>\n<li><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://github.com/openai/openai-cli\">the OpenAI CLI</a></li>\n<li><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://openai.com/form/codex-for-oss\">Codex for open source</a></li>\n<li>one just to monitor Twitter</li>\n</ul>\n<p>These are not short chats. They are megathreads that I have been compacting for months. They accumulate history, preferences, and old decisions that I do not want to recreate every time I come back.</p>\n<p>Pinned thread shortcuts</p>\n<p>You can jump directly to pinned threads with <code>Command-1</code> through <code>Command-9</code>.</p>\n<p>There is a tradeoff here. Long-running threads are not free. If you revisit them later, the conversation is probably no longer in cache, so you can incur more cost than you would in a fresh short thread. For workstreams I care about, continuity is worth it.</p>\n<h2>Voice input</h2>\n<p>Voice input gets more of my actual thinking into Codex.</p>\n<p>The benefit is not speed. It is that the agent gets the unedited version of my thinking. Codex has built-in voice input, but I also use Wispr Flow because system-wide dictation changes how much context I can feed into everything else too. If I am planning a piece of work, I might say, &quot;I think there is some guy named Ben in Slack who mentioned this, I do not remember exactly what, just go look.&quot; That sentence is too vague and annoying to type, but completely natural to say.</p>\n<p>The same thing applies to transcripts. If I want to write a post, I can call someone, record the conversation, or talk to them in person with Granola on my phone, then use the transcript as the starting material. A lot of plans get better when the model has access to the messy version of what I think, not just the polished one.</p>\n<h2>Steering</h2>\n<p>Voice becomes more useful when combined with steering.</p>\n<p>Steering</p>\n<p>Steering</p>\n<p>Adding more direction while Codex is already working instead of waiting for the current step to finish.</p>\n<p>Steering lets you inject the next message after a tool call. If I am reviewing a website, I can keep talking while I look at it:</p>\n<ul>\n<li>make this smaller</li>\n<li>this copy is wrong</li>\n<li>the spacing between these two things feels off</li>\n<li>once this is done, open a PR</li>\n<li>wait for the preview deploy</li>\n<li>send the preview link to the person who needs to review it on Slack</li>\n</ul>\n<p>I do not need to wait for each step to finish before deciding the next one. I can keep adding intent while the agent is still working, then walk away with the queue already shaped.</p>\n<p>Later, Heartbeats can monitor the PR or Slack thread after I leave. The unit of work stops being &quot;one prompt, one answer.&quot; It becomes a small operating loop.</p>\n<h2>Memory</h2>\n<p>Once threads started lasting longer, they needed shared memory outside any one repo.</p>\n<p>The important move is saving things to disk. A long thread can remember a lot, but that memory is trapped inside the thread unless the useful parts get written somewhere durable. The point of the memory system is to turn what the thread learns into artifacts I can inspect, edit, diff, and reuse.</p>\n<p>Most of my long-running threads start in an Obsidian vault:</p>\n<pre><code class=\"language-text\"><span class=\"code-snippet\">vault/\n├── TODO.md\n├── people/\n├── projects/\n├── agent/\n└── notes/\n</span></code></pre>\n<p>At the top level, I keep <code>AGENTS.md</code> instructions that tell the model to write things down: as you learn more about people, make progress on projects, make a decision, or close an open loop, update the relevant pages in the vault.</p>\n<p>The vault is where the agent lives, separate from any one project. Repositories hold code. The vault holds rolling context around my work: people, decisions, open loops, daily notes, project state, and the bits of understanding that would otherwise get lost between threads.</p>\n<p>I also keep the vault as a GitHub repo. That buys me two things:</p>\n<ol>\n<li>it can work in the cloud</li>\n<li>diffs become a review surface for memory</li>\n</ol>\n<p>When the agent updates the vault, I can read the diff and see what it thought was important enough to remember. That review step matters. I do not want evergreen threads to quietly accumulate vibes in conversation history. I want them to write down what changed: this person prefers this, this project is waiting on that, this decision was made, this loop is closed.</p>\n<p>This is also why I like memory as files. Files force the agent to compress experience into a form that can survive the thread. If the thread dies, compacts badly, or becomes too expensive to keep leaning on, the useful knowledge is still there. The pattern is simple: keep important work in a repo, give Codex permission and instructions to update it, and review the diff like any other change.</p>\n<p>At that point, pinned threads start to feel less like chats and more like different workers reading from the same notebook.</p>\n<p>Codex also has first-party memory features in <code>Settings &gt; Personalization &gt; Memories</code>. I think of those as a recall layer on top of the explicit disk-backed system. The vault is the source of truth I can review and edit. Memories are what help Codex remember stable preferences, recurring workflows, project conventions, and known pitfalls when I start a new thread.</p>\n<p><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://developers.openai.com/codex/memories/chronicle\">Chronicle</a> is interesting for the same reason. I have not used it seriously yet, and the docs are clear that it is an opt-in research preview with real tradeoffs around permissions, rate limits, prompt injection, and unencrypted local memory files. But directionally it points at the same thing I care about: work should leave behind structured memory, not just a longer chat transcript.</p>\n<p>Shared memory</p>\n<p>Shared memory</p>\n<p>Context kept outside a single chat, such as notes in my vault that different threads can reuse.</p>\n<h2>Computer and Browser Use</h2>\n<p>Once a thread has memory, the next question is what it can touch.</p>\n<p>The most useful distinction in my own head is:</p>\n<ul>\n<li><code>$browser</code> is for local web surfaces I want to inspect and annotate</li>\n<li><code>@chrome</code> is for signed-in browser state and multiple tabs</li>\n<li><code>@computer</code> is for work that only exists as a GUI</li>\n</ul>\n<p>If I am iterating on a local app, I want <code>$browser</code>. If I need to work inside a logged-in browser session, I want <code>@chrome</code>. If the only way to do the task is to click through a desktop app, I want <code>@computer</code>.</p>\n<p><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://dub.sh/PbgvcAJ\">Appshots</a> is the lighter-weight version of this. Sometimes I do not want Codex to operate the app yet. I just want to show it the thing I am looking at. On macOS, you can press both Command keys and send the frontmost window into a Codex thread with a screenshot and whatever text the app makes available.</p>\n<p>That is useful for the annoying middle category of context that is easier to show than describe: an error modal, a settings panel, an API reference page, an email, a calendar view, a design preview, or the weird state of an app that only makes sense when you see it. Instead of typing a long setup prompt, I can point Codex at the current window and say, &quot;this is the thing I mean.&quot;</p>\n<p>On my work machine, Twitter is logged into Safari. If I have <code>@computer</code> read Twitter there, I lose Safari while it works. <code>@chrome</code> is better when I want the agent to use several authenticated tabs in parallel without taking over the whole app I am using.</p>\n<p>Connectors extend that reach into the rest of my actual work. The ones I use most are <code>$slack</code>, <code>$gmail</code>, and <code>$calendar</code>, because Slack threads, inboxes, and calendars are where a lot of work shows up before it ever becomes code.</p>\n<p>Skills make repeated workflows reusable. Skill Creator and Skill Installer are a good place to start. Skill Installer lets you add OpenAI-recommended skills directly from the composer. After <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://developers.openai.com/codex/app/settings#codex-pets\">Codex pets</a> launched, I used it to install the Hatch Pet skill, but the useful pattern is general: once you do something useful once, you can often package it so Codex can do it again without reteaching the workflow.</p>\n<h2>Remote control</h2>\n<p>Remote control makes these longer loops feel portable.</p>\n<p>Codex can keep working from the machine where your files, permissions, and local setup already live, while you check in from mobile, review what it found, answer a question, approve the next step, or change direction without being back at your desk. <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://openai.com/index/work-with-codex-from-anywhere/\">OpenAI describes it as a way to work with Codex from anywhere</a>.</p>\n<p>This matters most when Codex is already doing something long-running and you want to preserve momentum. You can start a task, walk away, then steer it from your phone when it reaches a decision point.</p>\n<p>That matters for the same reason pinned threads, voice, and Heartbeats matter. The work no longer has to pause just because I changed locations. A thread can keep going, and I can keep just enough attention on it to unblock the next move.</p>\n<h2>Heartbeats</h2>\n<p>Pinned threads are useful, but they still wait for you to say something. Heartbeats are what make them recur.</p>\n<p>Heartbeats</p>\n<p>Heartbeats</p>\n<p>Recurring checks a thread can schedule for itself, like watching Slack or a pull request for new activity.</p>\n<p>A Heartbeat is a thread-local automation. You can say, &quot;keep an eye on this every few hours,&quot; and the thread can schedule itself. A thread can have multiple schedules, run until some condition is met, and adjust its cadence over time.</p>\n<h3>Chief of Staff</h3>\n<p>My Chief of Staff thread runs every 30 minutes:</p>\n<pre><code class=\"language-text\"><span class=\"code-snippet\">Every 30 minutes, check Slack and Gmail for unanswered messages that need my attention.\nHelp me prioritize what matters most.\nIf someone asks me a question, research the answer as deeply as you can and draft a reply for me, but do not send it.\n</span></code></pre>\n<p>When I come back to Slack, replies are often already sitting in drafts. I still decide what gets sent, but the expensive part of gathering context is done.</p>\n<h3>Monitor for feedback</h3>\n<p>The same pattern works for review loops. A Heartbeat can monitor Google Docs comments, pull request comments, or Slack replies and keep work moving as feedback arrives.</p>\n<p>One of my favorite examples came from an animation project. I had posted a video in Slack and asked Codex to check the thread every 15 minutes for feedback, re-render a new version when comments came in, and reply back into the thread tagging the reviewer. The Slack MCP server could not upload files, so the agent used <code>@computer</code> to press the Add file button and post the revised render anyway.</p>\n<p>The interesting part is not just that it checked Slack every 15 minutes. The loop crossed tool boundaries: Slack for feedback, Remotion for the render, <code>@computer</code> for the upload. That is when Heartbeats, connectors, and computer use stop feeling like separate features. Together they become a feedback loop that keeps running without me sitting there.</p>\n<h3>Get a refund</h3>\n<p>Recently I had a package stolen. Amazon told me it would take about 25 minutes to talk to a person, so I created a thread with <code>@computer</code> and told it:</p>\n<pre><code class=\"language-text\"><span class=\"code-snippet\">Every 5 minutes, check whether the customer support agent has joined this thread.\nIf they have, do your best to get me a refund.\nOnce they reply, switch to checking every minute so you can respond faster.\n</span></code></pre>\n<p>By the time I got out of the shower, the refund was done.</p>\n<p>Many of my Heartbeats also update my Obsidian vault as a kind of explicit memory.</p>\n<h2>Goals</h2>\n<p>The newest thing I am still learning how to use well is Goals.</p>\n<p>Goals</p>\n<p>Goals</p>\n<p>Longer-running tasks with a real finish line Codex can keep pushing toward.</p>\n<p>You should be ambitious with them. A weak goal is &quot;implement the plan in this Markdown file.&quot; A strong goal has a real success criterion that the agent can keep pushing against.</p>\n<p>Last week I tried to migrate the Python <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://github.com/Textualize/rich\">Rich</a> library into Rust. Because the original project already had a large unit test suite, I could set a goal like: migrate Rich into Rust, but it must pass all the unit tests from the original library.</p>\n<p>That test suite gave the run a real oracle: the Rust port was not done until it passed the same tests as the Python library.</p>\n<p>This is different from having a long conversation with an AI, accumulating a Markdown plan, and then eventually saying, &quot;implement this.&quot; Execution is only as good as the goal and the verification you give it. Ambition without verification is just a wish.</p>\n<h2>The side panel</h2>\n<p>The part of Codex I am most excited about is the side panel.</p>\n<p>It is easy to think of this as a place where previews happen. That undersells it. The side panel is where Codex stops being only a chat app and starts becoming the place the work happens.</p>\n<p>For me it does three jobs: inspect artifacts, operate web surfaces, and review changes. In all three, I can look at and comment on the same object the agent is acting on.</p>\n<h3>Inspect artifacts</h3>\n<p>Markdown, spreadsheets, CSVs, PDFs, and slides can all live there.</p>\n<p>Markdown is commentable. Spreadsheets render formulas and support cell edits, which I use to manage Codex open-source plans. CSVs become tables instead of raw text. PDFs render directly, which is especially useful with LaTeX. Slides can be created and reviewed without leaving the app.</p>\n<p>The important thing is not merely that Codex can generate artifacts. It is that I can inspect and annotate them without breaking the loop.</p>\n<h3>Operate web surfaces</h3>\n<p>The in-app browser is even more interesting. The agent can see it, control it with JavaScript through <code>$browser</code>, and I can leave annotations directly on what I am looking at.</p>\n<p>There are a few web surfaces I now use this way all the time:</p>\n<ul>\n<li><code>index.html</code> for lightweight static artifacts</li>\n<li>Storybook for reviewing UI components</li>\n<li>Remotion Studio for programmatic animation</li>\n<li><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://sli.dev/guide/\">Slidev</a> for presentations</li>\n<li>Streamlit for data apps</li>\n</ul>\n<p>The smallest version is often the best. You can ask the model to make a single <code>index.html</code> file with JavaScript and CSS, open it in the side panel, and start interacting with it immediately. No server required. I have been experimenting with using Heartbeats to update a static <code>index.html</code> over time so that whenever I return to a thread, there is already a fresh artifact waiting for me.</p>\n<p><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://x.com/trq212/status/2052809885763747935\">Thariq has a very good post</a> about preferring HTML over Markdown as an output format. I think that instinct is right. Once the output is a small application instead of just a document, the relationship changes.</p>\n<p>If I need something heavier, I can use a Vite app, but then I need to keep a server running. A plain <code>index.html</code> is much more durable.</p>\n<p>For animation work, I often have Storybook and Remotion Studio open side by side. I can leave comments like &quot;make this bounce&quot; or &quot;this should be larger,&quot; and the agent can inspect the same browser state I am looking at, including the current frame in the animation.</p>\n<p>For presentations, I often use Slidev. Codex can inspect the slides, catch content that is cut off, switch between slides, and respond to annotations while I review.</p>\n<p>I also expect this to become more useful for tools like Streamlit and Jupyter over time. Different people already live inside different applications. Codex can increasingly meet them there.</p>\n<p>The more Codex gets places to remember, revisit, inspect, and act, the less my work dies between prompts. That is the change I care about. Not that an agent can write code for me, but that more of my work can keep moving after I leave.</p>\n",
      "date_modified": "2026-05-10T00:00:00+00:00",
      "date_published": "2026-05-10T00:00:00+00:00",
      "external_url": "https://jxnl.co/writing/2026/05/10/codex-maxxing/",
      "id": "urn:aggr:item:03eaf9a35b4c0f23b4698fecf521ff5541416dbd",
      "summary": "How I use Codex as a place where long-running work can live.",
      "tags": [
        "programming",
        "ai",
        "applied ai",
        "codex",
        "personal",
        "personal software",
        "workflows"
      ],
      "title": "Codex-maxxing",
      "url": "https://aggr.aymericbeaumet.com/items/jxnl-co/2026-05-10-codex-maxxing/"
    },
    {
      "authors": [
        {
          "name": "Jason Liu"
        }
      ],
      "content_html": "<p>People often ask me what kind of music I listen to, and in my mind I almost never remember. Today I wanted to write a little bit more about the music I listen to and share some of my favorite albums with you. A lot of this has been co-written with the Spotify API, so everything is as true as it can be.</p>\n<p>I think my couple of favorite artists are just <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://open.spotify.com/artist/3rxeQlsv0Sc2nyYaZ5W71T\">Chet Baker</a>, <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://open.spotify.com/artist/20wkVLutqVOYrc0kxFs7rA\">Daniel Caesar</a>, and <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://open.spotify.com/artist/7tr9pbgNEKtG0GQTKe08Tz\">Mk.gee</a>. There's a kind of longing that I really resonate with. Artists like <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://open.spotify.com/artist/57okaLdCtv3nVBSn5otJkp\">HYUKOH</a> as well, but I also like to listen to a lot of guitar and ambient, trumpet and a bunch of other things.</p>\n<hr />\n<h2>Ambient</h2>\n<p>Ambient is probably the genre I play the most, primarily because it plays in the background forever while I work for many hours without interrupting anything. It doesn't pull at your attention the way most music does — it just holds the room.</p>\n<h3><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://open.spotify.com/artist/2h5syT5XdsQgKLq8Yn1klO\">Nala Sinephro</a></h3>\n<p>London-based harpist and composer, Barbadian-Belgian. Part of the South London jazz scene alongside Floating Points and Yussef Dayes, but her sound is the most inward of any of them. She was honestly the first ambient artist I really got into — Space 1.8 especially has a special place in my heart. Worth noting that not all of it is easy listening: the first couple tracks are smooth, but things get pretty heavy at points. Which is actually part of what I love about it.</p>\n<h4>Space 1.8</h4>\n<p>The one that started it. Long arcs of harp and synth that feel like watching light move through water — and then, without warning, something heavier comes through. That contrast is what makes it worth returning to.</p>\n<h4>Endlessness</h4>\n<p>The follow-up, and she went even further inward. More orchestral, more patient. The title is accurate.</p>\n<hr />\n<h3><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://open.spotify.com/artist/2AR42Ur9PcchQDtEdwkv4L\">Floating Points</a></h3>\n<p>Sam Shepherd — British musician and neuroscientist. Came up through the London club scene making modular electronics, then pivoted toward jazz and orchestral work. One of the few artists who sounds equally at home in a warehouse and a concert hall.</p>\n<h4>Promises (with Pharoah Sanders &amp; London Symphony Orchestra)</h4>\n<p>One continuous piece across nine movements. Like Sinephro, it starts smooth and gets heavy — there's real weight in the later movements. Pharoah Sanders was 90 years old when this was recorded, and you can hear a lifetime in every note.</p>\n<hr />\n<h3><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://open.spotify.com/artist/2o7L9DNcmzocYll1o0GGTU\">Leon Vynehall</a></h3>\n<p>British electronic musician who makes conceptual records — the kind where there's a story underneath the sound even if it's never stated. His music has a warmth that most electronic music doesn't bother with.</p>\n<h4>Nothing Is Still</h4>\n<p>A concept album about his grandparents emigrating to New York. The opening track, <em>From The Sea/It Looms (Chapters I &amp; II)</em>, is one of my all-time favorite pieces of music — the way it builds and resolves feels like watching a city arrive out of fog. If you only listen to one thing on this list, make it that track.</p>\n<hr />\n<h3><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://open.spotify.com/artist/0Zoe6ljAJo85rggnN6OaOF\">92914</a></h3>\n<p>Japanese lo-fi and ambient artist. Minimal information available — which suits the music. Quiet guitar, soft percussion, the feeling of being somewhere late at night. I found the first record at 2am and it hasn't left my rotation.</p>\n<h4>By The Calm Water</h4>\n<p>Where I started. <em>Night</em> is the track. Quiet and unhurried in a way that feels earned rather than ambient-by-default.</p>\n<h4>Home</h4>\n<p>Companion record, same mood. <em>Someday</em> is the one — quiet guitar, late-night feeling, nothing explained.</p>\n<h4>Okinawa</h4>\n<p>Named after the island. The whole record sounds like being somewhere warmer than where you actually are.</p>\n<hr />\n<h2>Classic Jazz</h2>\n<h3><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://open.spotify.com/artist/3rxeQlsv0Sc2nyYaZ5W71T\">Chet Baker</a></h3>\n<p>American jazz trumpeter and vocalist, 1929–1988. My most-played artist, all time. His voice and trumpet tone are essentially the same instrument — soft, unhurried, and a little heartbroken. He is honestly the reason I learned to play the trumpet. <em>Time After Time</em> is a song I want to play at my future bride's wedding. <em>My Funny Valentine</em> I've memorized on trumpet and sing to my lover. There's something in what he does with space and silence that I keep trying to understand.</p>\n<h4>Chet Baker Sings</h4>\n<p>The one that started everything. Recorded 1954, still sounds like 3am. Every standard on here has been covered a thousand times, and his versions are still the ones I go back to.</p>\n<h4>I Can Dream, Can't I?</h4>\n<p>One of my most-played tracks. The restraint in his phrasing is remarkable — he knows exactly when not to sing, which is rarer than it sounds.</p>\n<hr />\n<h3><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://open.spotify.com/artist/4V7Ate3UISn8cy06xnAprh\">Ryo Fukui</a></h3>\n<p>Japanese jazz pianist, 1948–2016. Largely self-taught. Spent his whole life in Sapporo, barely toured, barely left Japan. For years his records were nearly impossible to find outside the country. The internet eventually found him and now he has the cult following he always deserved.</p>\n<h4>Scenery</h4>\n<p>Recorded in 1976 in Sapporo. Piano trio jazz that sounds like it was made in a snowstorm — clear, cold, and alive. Like Sinephro and Promises, it has a heaviness in places that catches you off guard. His touch on the keys is something else.</p>\n<hr />\n<h3><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://open.spotify.com/artist/1Mxqyy3pSjf8kZZL4QVxS0\">Frank Sinatra</a></h3>\n<p>American singer, 1915–1998. The voice of the 20th century, which is not an exaggeration. His phrasing — the way he breathes through a lyric, the way he sits behind the beat — is something every vocalist since has had to reckon with.</p>\n<h4>Nothing But The Best</h4>\n<p><em>Fly Me to the Moon</em> is the reason I'm here. There are songs that feel like wishing for snow in spring — nostalgic for something that hasn't quite passed yet. That's this song. It's impossible to hear it and not feel something.</p>\n<hr />\n<h3><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://open.spotify.com/artist/2FatNL7pp4gwt10zTQHzbA\">鈴木 弘 (Hiroshi Suzuki)</a></h3>\n<p>Japanese jazz trombonist and bandleader. His 1975 album <em>Cat</em> became a cult classic of Japanese jazz-funk decades after it was recorded — warm, melodic, and with a production quality that still holds up. I found him the same way most people do: through a YouTube rabbit hole at midnight.</p>\n<h4>キャット (Cat)</h4>\n<p>Jazz-funk from 1975, and one of the warmest-sounding records I know. <em>ロマンス</em> has one of the most immediately inviting opening bars of any album in my library.</p>\n<hr />\n<h2>R&amp;B</h2>\n<h3><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://open.spotify.com/artist/20wkVLutqVOYrc0kxFs7rA\">Daniel Caesar</a></h3>\n<p>Canadian R&amp;B singer-songwriter from Toronto. He makes records about intimacy — not romance exactly, but the feeling of being close to another person and not quite being able to say what you mean. Every album he's made is worth listening to start to finish. His voice is one of the great voices working right now.</p>\n<h4>Son Of Spergy</h4>\n<p>His latest. Every track does something interesting with space and silence. The features are perfect — Sampha, Bon Iver, Yebba. The longest album he's made and it doesn't drag anywhere.</p>\n<h4>NEVER ENOUGH</h4>\n<p>More polished, more commercial — but <em>Ocho Rios</em> is one of his best songs and that alone justifies it.</p>\n<h4>H.E.R.</h4>\n<p>Where he broke through. <em>Best Part</em> is still one of the most perfectly constructed R&amp;B songs of the last decade — quieter and more personal than anything he's done since.</p>\n<hr />\n<h3><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://open.spotify.com/artist/2h93pZq0e7k5yf4dywlkpM\">Frank Ocean</a></h3>\n<p>American singer-songwriter from Long Beach. Two studio albums in fifteen years, and both of them are landmarks. He writes about memory and longing and the distance between people in a way that doesn't feel like writing — it feels like something he couldn't help saying.</p>\n<h4>Blonde</h4>\n<p>The only thing I listened to for my entire first four months in New York City. Every time I hear it now I'm back in Bushwick — taking the L, walking to whatever I was walking to, doing all of it for the first time. I don't think there's another album that does that to me. It's permanently embedded in a specific place and time.</p>\n<h4>Moon River</h4>\n<p>A cover of the Audrey Hepburn classic, two minutes long. He does more with restraint here than most artists do with full albums.</p>\n<hr />\n<h3><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://open.spotify.com/artist/3e76yvk1gLZQhKZiUHkMsP\">daoud</a></h3>\n<p>Independent R&amp;B artist. His music is minimal and a little raw — bedroom recordings that stayed bedroom recordings because that's the right format for what he's saying.</p>\n<h4>soda</h4>\n<p>Lo-fi R&amp;B that sounds like a text you weren't supposed to send. The production is stripped back so you end up listening to the words whether you meant to or not.</p>\n<hr />\n<h3><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://open.spotify.com/artist/6iUs0Aijnu60VAAf8Aj2YM\">YELLOW黃宣</a></h3>\n<p>Taiwanese singer-songwriter and actor. Makes music that sits between indie pop and experimental R&amp;B — bilingual, with a production sensibility that doesn't fit neatly into any scene.</p>\n<h4>翅膀和泥土</h4>\n<p>The title translates to something like &quot;Wings and Soil.&quot; The title track loops in my head for days — melodically intricate without being fussy.</p>\n<hr />\n<h2>Indie Soul</h2>\n<h3><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://open.spotify.com/artist/7tr9pbgNEKtG0GQTKe08Tz\">Mk.gee</a></h3>\n<p>American guitarist and songwriter, real name Mike Sclafani. Just incredible guitar. He plays it the way Chet Baker plays trumpet — you're always aware of the silence around the notes. His sound is hard to place: too slow for indie rock, too guitar-driven for R&amp;B, too weird for either. I have nine tracks from <em>Two Star</em> in my all-time top 50.</p>\n<h4>Two Star &amp; The Dream Police</h4>\n<p>His debut album. Guitar-driven, slow, slightly off-kilter — sounds like it was recorded in a room where everyone was half-asleep in the best way. Put it on and let it run.</p>\n<h4>ROCKMAN</h4>\n<p>A single, but the title track is a statement of intent. Three minutes of exactly what he does, distilled.</p>\n<h4>Fool</h4>\n<p>Earlier EP, before the sound fully crystallized. The bones of <em>Two Star</em> are already here.</p>\n<h4>Lonely Fight</h4>\n<p>One song. Plays like a closing argument. I keep coming back to it.</p>\n<hr />\n<h3><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://open.spotify.com/artist/0knGpCTbmG4ctl1wzYRZs4\">Dijon</a></h3>\n<p>American singer-songwriter Dijon Duenas, from Los Angeles. His music sits at the intersection of R&amp;B, soul, and something that sounds almost like classic country — confessional, melodic, and not interested in being fashionable.</p>\n<h4>Baby</h4>\n<p>His debut album. R&amp;B that leans toward classic soul — <em>Baby!</em> opens the record and you know immediately what kind of artist he is.</p>\n<h4>Absolutely</h4>\n<p>Earlier EP with a rawer, more country-adjacent feel. <em>The Dress</em> and <em>Big Mike's</em> are two of his best songs. Worth going back to after <em>Baby</em>.</p>\n<hr />\n<h3><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://open.spotify.com/artist/1uiEZYehlNivdK3iQyAbye\">Tom Misch</a></h3>\n<p>British guitarist, producer, and singer from London. Started releasing music as a teenager from his bedroom — jazz guitar layered over hip-hop drums and soul vocals. The early recordings have a looseness that's hard to manufacture.</p>\n<h4>Geography</h4>\n<p>His debut album, and the one that showed he could sustain a full record. Features De La Soul, Loyle Carner, GoldLink. It runs through a lot of ground without losing its thread.</p>\n<h4>Six Songs</h4>\n<p>The early EP that put him on the map. <em>Better Days</em> is the track — jazz guitar over a hip-hop beat, effortless in a way that takes real skill to pull off.</p>\n<hr />\n<h3><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://open.spotify.com/artist/6qqNVTkY8uBg9cP3Jd7DAH\">FKJ</a></h3>\n<p>French multi-instrumentalist, full name French Kiwi Juice. He plays everything — piano, guitar, saxophone, bass — and most of his recordings are essentially live solo performances layered on top of each other. The result is something that sounds effortless and took enormous craft. He and Masego also recorded <em>Tadow</em> together, which is one of the best jazz-soul collaborations of the last decade.</p>\n<h4>French Kiwi Juice</h4>\n<p>His debut album — warm, live-feeling, and melodically generous. <em>Lying Together</em> and <em>Better Give U Up</em> are the ones to start with.</p>\n<hr />\n<h3><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://open.spotify.com/artist/2ow5cXN8oHK1KFkENaJ9ci\">Masego</a></h3>\n<p>American singer, saxophonist, and producer. He coined the term &quot;TrapHouseJazz&quot; for what he makes — jazz harmony and saxophone over trap production — and it's a better description than anything I could write. His voice is warm and a little playful, and his records feel like parties where everyone actually knows how to play.</p>\n<h4>Lady Lady</h4>\n<p>His debut album, and it's confident from the first track. <em>Tadow</em> (with FKJ) became the gateway for a lot of people — including me. The whole record holds up.</p>\n<hr />\n<h3><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://open.spotify.com/artist/30b9WulBM8sFuBo17nNq9c\">G-DRAGON</a></h3>\n<p>Korean rapper and artist, leader of BIGBANG. One of the most influential figures in K-pop and Korean hip-hop — not just as a performer but as a stylist and tastemaker. His solo records are weirder and more personal than the group work.</p>\n<h4>POWER</h4>\n<p>I know. But <em>POWER</em> is one of the most purely fun records I've heard. The production is absurd in the best way — maximalist, committed, and completely uninterested in being cool about it.</p>\n<hr />\n<p>If you want to trade recommendations, I'm <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://x.com/jxnlco\">@jxnlco</a> on X.</p>\n<hr />\n<h2>On Vinyl</h2>\n<p>Most of these are available to buy. Prices as of April 2026.</p>\n<table>\n<thead>\n<tr>\n<th>Album</th>\n<th>Price</th>\n<th>Buy</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>Floating Points &amp; Pharoah Sanders — <em>Promises</em></td>\n<td>$22.99</td>\n<td><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://elusivedisc.com/floating-points-pharoah-sanders-the-london-symphony-orchestra-promises-2025-pressing-lp/\">Elusive Disc</a></td>\n</tr>\n<tr>\n<td>Nala Sinephro — <em>Space 1.8</em></td>\n<td>$29.99</td>\n<td><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://www.roughtrade.com/en-us/product/nala-sinephro/space-1-8\">Rough Trade</a></td>\n</tr>\n<tr>\n<td>Nala Sinephro — <em>Endlessness</em></td>\n<td>$34.98</td>\n<td><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://www.amoeba.com/endlessness-lp-nala-sinephro/albums/4394241/\">Amoeba</a></td>\n</tr>\n<tr>\n<td>Leon Vynehall — <em>Nothing Is Still</em></td>\n<td>$32.98</td>\n<td><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://www.amoeba.com/nothing-is-still-lp-leon-vynehall/albums/4042465/\">Amoeba</a></td>\n</tr>\n<tr>\n<td>Frank Sinatra — <em>Nothing But The Best</em></td>\n<td>~$31–40</td>\n<td><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://www.discogs.com/sell/release/18742321\">Discogs</a> (secondary)</td>\n</tr>\n<tr>\n<td>Lester Young — <em>The Aladdin Sessions</em></td>\n<td>~$20–32</td>\n<td><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://www.discogs.com/release/2211720-Lester-Young-The-Aladdin-Sessions\">Discogs</a> (secondary)</td>\n</tr>\n<tr>\n<td>Daniel Caesar — <em>Son Of Spergy</em></td>\n<td>$39.99</td>\n<td><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://shop.sonofspergy.com/products/son-of-spergy-2lp\">Official Store</a></td>\n</tr>\n<tr>\n<td>Daniel Caesar — <em>NEVER ENOUGH</em></td>\n<td>$29.99</td>\n<td><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://shop.udiscovermusic.com/products/daniel-caesar-never-enough-alternate-cover-2-lp\">uDiscover</a></td>\n</tr>\n<tr>\n<td>Daniel Caesar — <em>Freudian</em></td>\n<td>$150–230+</td>\n<td><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://www.discogs.com/master/1275484-Daniel-Caesar-Freudian\">Discogs</a> (secondary)</td>\n</tr>\n<tr>\n<td>Khamari — <em>To Dry a Tear</em></td>\n<td>$35.98</td>\n<td><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://khamari.indiemerch.com/products/to-dry-a-tear\">Official Store</a> (ships ~April 2026)</td>\n</tr>\n<tr>\n<td>daoud — <em>ok</em> (contains &quot;soda&quot;)</td>\n<td>~$30–32</td>\n<td><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://www.roughtrade.com/en-us/product/daoud/ok\">Rough Trade</a></td>\n</tr>\n<tr>\n<td>Mk.gee — <em>Two Star &amp; The Dream Police</em></td>\n<td>$39.98</td>\n<td><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://www.amoeba.com/two-star-the-dream-police-green-vinyl-lp-mk-gee/albums/4403515/\">Amoeba</a></td>\n</tr>\n<tr>\n<td>Dijon — <em>Baby</em></td>\n<td>$34.98</td>\n<td><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://www.amoeba.com/baby-silver-vinyl-lp-dijon/albums/4455182/\">Amoeba</a></td>\n</tr>\n<tr>\n<td>Dijon — <em>Absolutely</em></td>\n<td>$29.99</td>\n<td><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://shop.darksiderecords.com/products/dijon-absolutely-explicit-content-indie-exclusive-140-gram-vinyl-brown-colored-vinyl\">Darkside Records</a></td>\n</tr>\n<tr>\n<td>Hiroshi Suzuki — <em>Cat</em></td>\n<td>~$39+</td>\n<td><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://www.roughtrade.com/en-us/product/hiroshi-suzuki/cat-2\">Rough Trade</a> (check stock)</td>\n</tr>\n<tr>\n<td>G-DRAGON — <em>Übermensch</em> (contains &quot;POWER&quot;)</td>\n<td>$35</td>\n<td><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://ubermensch.us/collections/vinyl\">Official Store</a></td>\n</tr>\n<tr>\n<td>92914 — <em>Sunset and HOME</em></td>\n<td>~$55–174</td>\n<td><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://www.discogs.com/release/24289301-92914-Sunset-and-HOME-\">Discogs</a> (secondary)</td>\n</tr>\n</tbody>\n</table>\n",
      "date_modified": "2026-04-02T00:00:00+00:00",
      "date_published": "2026-04-02T00:00:00+00:00",
      "external_url": "https://jxnl.co/writing/2026/04/02/what-music-do-you-listen-to/",
      "id": "urn:aggr:item:84519019a30d7bb4ddc29f145a943299a4c413b8",
      "summary": "My top albums and artists from Spotify — and what the mix actually says about taste.",
      "tags": [
        "programming",
        "personal"
      ],
      "title": "What Music Do You Listen To?",
      "url": "https://aggr.aymericbeaumet.com/items/jxnl-co/2026-04-02-what-music-do-you-listen-to/"
    },
    {
      "authors": [
        {
          "name": "Jason Liu"
        }
      ],
      "content_html": "<p>I am sunsetting 567 Labs. The courses close today, February 2, 2026.</p>\n<p>I am open sourcing the written content: course transcripts edited into ebook format. If you took the course, this is the canonical archive. If you did not, it is the cleanest way to read the material end to end.</p>\n<p>Highlights from the RAG playbook:</p>\n<ul>\n<li>Product mindset over one-off implementation. Measurement, feedback loops, and improvement cycles first.</li>\n<li>Synthetic eval data to break the cold start and drive systematic improvements.</li>\n<li>Feedback UX that actually works. Ask specific questions and make responses easy.</li>\n<li>Specialized retrieval and routing instead of one-size-fits-all search.</li>\n</ul>\n<p>Highlights from the consulting archive:</p>\n<ul>\n<li>Flywheel: content → calls → proof → more content.</li>\n<li>Value equation: <code>Value = (Dream Outcome × Probability of Success) ÷ (Time × Effort)</code>.</li>\n<li>Three-option pricing with clear tradeoffs, not a single price.</li>\n<li>Discovery + proposals: five-question discovery, situational assessment structure.</li>\n</ul>\n<h2>Read</h2>\n<p><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://567-labs.github.io/systematically-improving-rag/\">Read the RAG Playbook</a></p>\n<p><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://567-labs.github.io/consulting/\">Read the Consulting Archive</a></p>\n<h2>Follow</h2>\n<p><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://www.youtube.com/@jxnlco/videos\">Watch on YouTube</a></p>\n<p><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://www.linkedin.com/in/jxnlco/\">Connect on LinkedIn</a></p>\n<p>If you spot issues, please let me know.</p>\n",
      "date_modified": "2026-02-02T00:00:00+00:00",
      "date_published": "2026-02-02T00:00:00+00:00",
      "external_url": "https://jxnl.co/writing/2026/02/02/sunsetting-567-labs/",
      "id": "urn:aggr:item:8d59b3bb6e49280a20958e54484e9bcb36e32d37",
      "summary": "Sunsetting 567 Labs, closing the courses today, and open sourcing the written content from transcripts as ebooks.",
      "tags": [
        "programming",
        "567 labs",
        "courses",
        "open source",
        "rag",
        "writing and communication"
      ],
      "title": "Sunsetting 567 Labs and Open Sourcing the Course Content",
      "url": "https://aggr.aymericbeaumet.com/items/jxnl-co/2026-02-02-sunsetting-567-labs-and-open-sourcing-the-course-content/"
    },
    {
      "authors": [
        {
          "name": "Jason Liu"
        }
      ],
      "content_html": "<p><em>Some links may include affiliate attribution. Recommendations are based on personal use.</em></p>\n<p>In the past 2 decades I went from sharing a bed with my parents renting out the unfinished basement of some Canadian family to doing quite well for myself. This is all the stuff I use, plan to use, and what's on my upgrade roadmap. Each item includes why it works for me.</p>\n<h2>Sleep</h2>\n<p>I really miss my own bed after traveling so much, but I've always stuck with <strong>Purple Mattress</strong> — everyone's going for 8 Sleep, but I don't want to shell out for a subscription just to sleep. For better breathing, <strong><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://www.amazon.com/dp/B07XF91NZ4\">Nasal Strips</a></strong> help me breathe easier (not into mouth taping yet). <strong>Allergen-proof bedding</strong> keeps dust mites at bay. With long hair now, <strong><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://www.amazon.com/gp/product/B0C7GRYXLS\">silk pillowcases</a></strong> prevent tangles and breakage.</p>\n<p>For darkness: <strong><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://www.amazon.com/gp/product/B0D4QHRC7W?th=1\">door light blockers</a></strong> and <strong><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://www.amazon.com/gp/product/B08MZLSMHM\">blackout blinds</a></strong>. For travel, <strong><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://mantasleep.com/products/manta-sleep-mask\">Manta Sleep eye mask</a></strong> — they have hot and cold versions which I love.</p>\n<p>I've been building better morning habits: getting outside first thing for coffee and sunlight exposure, eating dinner earlier. Still working on reducing pre-bedtime screen time.</p>\n<hr />\n<h2>Air Quality</h2>\n<p>Large part of my poor sleep comes from allergies and air quality, so it's useful to get an <strong><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://www.airgradient.com/\">air quality monitor</a></strong>, <strong><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://www.amazon.com/gp/product/B01728NLRG/ref=ox_sc_act_title_2?smid=ATVPDKIKX0DER&amp;psc=1\">air purifier</a></strong>, and humidifier during winter.</p>\n<hr />\n<h2>Personal Care</h2>\n<p>I usually shower with just water, but recently started using skincare products. <strong><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://www.amazon.com/dp/B0C7HHKH3G\">CeraVe Ultra-Light Gel</a></strong> — lightweight moisturizer with ceramides and hyaluronic acid. After a week, I received compliments on my skin. For supplements: <strong><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://www.amazon.com/dp/B07TK5K5TQ\">NAD+</a></strong> and <strong><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://www.amazon.com/dp/B07TTDFXFV\">Tongkat Ali</a></strong> support overall hormone health.</p>\n<p><strong><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://www.aesop.com/us/p/body/hand/reverence-aromatique-hand-wash/\">Aesop Reverence hand wash</a></strong> — exfoliating with wonderful scent. <strong><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://www.ssense.com/en-us/everything-else/product/comme-des-garcons-parfums/monocle-edition-scent-one-hinoki-eau-de-toilette-50-ml/6659441\">Hinoki from Monocle</a></strong> — woody, mossy scent inspired by Kyoto.</p>\n<p>Planning to start <strong><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://www.amazon.com/gp/product/B004D2C4Q4/ref=ewc_pr_img_4?smid=ATVPDKIKX0DER&amp;psc=1\">retinol</a></strong> for better skin.</p>\n<h3>On My Roadmap</h3>\n<ul>\n<li>Collagen peptides</li>\n<li>Fish oil</li>\n<li>Magnesium</li>\n<li>Glucosamine &amp; chondroitin</li>\n<li>Electrolyte protocols</li>\n</ul>\n<p><strong>What these ingredients do:</strong></p>\n<ul>\n<li><strong>Retinol</strong> — Vitamin A that accelerates skin cell renewal, reduces wrinkles, fades dark spots</li>\n<li><strong>Ceramides</strong> — Natural fats that strengthen skin barrier and prevent moisture loss</li>\n<li><strong>NAD+</strong> — Coenzyme for cellular energy and mitochondrial function</li>\n<li><strong>Tongkat Ali</strong> — Southeast Asian herb for energy, mood, and hormone balance</li>\n<li><strong>Collagen Peptides</strong> — Improves skin elasticity, stronger nails and hair</li>\n<li><strong>Fish Oil</strong> — Omega-3s for brain function and reduced inflammation</li>\n<li><strong>Magnesium</strong> — Essential mineral for energy, sleep, and stress reduction</li>\n<li><strong>Glucosamine &amp; Chondroitin</strong> — Joint health and cartilage support</li>\n</ul>\n<h3>Teeth and Oral Health</h3>\n<p><strong><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://www.amazon.com/dp/B083JHCCV2\">Nano-hydroxyapatite toothpaste</a></strong> (n-HA) — biocompatible material that remineralizes tooth enamel. It's the same mineral that naturally makes up our tooth enamel.</p>\n<hr />\n<h2>Hair Care</h2>\n<p>With long, permed hair, moisture retention is everything. <strong><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://www.amazon.com/dp/B086JTYR1G\">Milbon Curl Mousse</a></strong> keeps curls hydrated and defined. Planning to add hair oils and invest in a <strong><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://www.amazon.com/Mason-Pearson-Popular-Mixture-Bristle/dp/B001N444JK\">Mason Pearson brush</a></strong>.</p>\n<blockquote>\n<p><strong>The Science:</strong> Perms chemically break and reform hair's natural bonds, making hair more porous. Hair absorbs moisture quickly but loses it fast. Without regular hydration, permed hair becomes straight, frizzy, and brittle.</p>\n</blockquote>\n<h3>On My Roadmap</h3>\n<ul>\n<li>High-quality hair dryer</li>\n<li>Hair oils for moisture retention</li>\n</ul>\n<hr />\n<h2>Audio</h2>\n<p>Relatively speaking, my tech is pretty basic: MacBook Pro and iPhone. <strong><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://www.amazon.com/dp/B07N3S4X3P\">Audio-Technica record player</a></strong> with <strong><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://www.amazon.com/dp/B088685QVJ\">Edifier speakers</a></strong>. I connect an Amazon Echo to my <strong><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://www.amazon.com/dp/B086FS4BSV\">Harman Kardon Aura Studio 3</a></strong> — the single best speaker I've ever owned, perfect for hosting parties.</p>\n<h3>On My Roadmap</h3>\n<ul>\n<li>Level up audio setup (still deciding what I'd use it for)</li>\n<li>JBL L100 speakers</li>\n<li>GIK 242 panels</li>\n<li>AirPods Pro</li>\n</ul>\n<hr />\n<h2>Office</h2>\n<p><strong><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://www.hermanmiller.com/products/seating/office-chairs/embody-chairs/\">Herman Miller Embody chair</a></strong> and <strong><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://kinesis-ergo.com/keyboards/advantage360/\">Kinesis Advantage 360</a></strong> keyboard. <strong><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://www.eamesoffice.com/products/lounge-chair-and-ottoman\">Eames lounge chair and ottoman</a></strong> from Eames Office.</p>\n<p><strong><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://www.amazon.com/dp/B0BN8C3MDQ\">Multiple charging stations</a></strong> where I can charge battery packs to take on the go.</p>\n<h3>Ergonomics &amp; Eye Health</h3>\n<ul>\n<li>Your body tells you when something's wrong — listen to it</li>\n<li>Good equipment pays off (hence the Herman Miller and Kinesis choices)</li>\n<li>Movement is key — shift positions throughout the day</li>\n<li>Proper monitor height matters (eyes align with top third)</li>\n</ul>\n<hr />\n<h2>Lighting</h2>\n<p>Passive ambient lighting is key. <strong>Philips Hue lights</strong> and <strong><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://shop.noguchi.org/collections/akari-light-sculptures\">Noguchi lamps</a></strong> — pricey but I love them.</p>\n<blockquote>\n<p>It's funny how a simple switch from ceiling-mounted lights to the soft scatter of floor lamps can transform your home from &quot;just another workspace&quot; into a cozy cave of creativity.</p>\n</blockquote>\n<hr />\n<h2>Storage</h2>\n<p><strong><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://www.muji.us/collections/storage-organizers?srsltid=AfmBOoqMRarrbP1eRCeWDj_Qx6XfFzLqcCSFLNUw7R7_76XsnNbxAQTL\">Muji storage</a></strong> for organization.</p>\n<h3>Grail Item</h3>\n<p><strong>USM Haller storage</strong> — long-term investment worth saving for.</p>\n<hr />\n<h2>Traveling</h2>\n<p><strong><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://www.rimowa.com\">Rimowa luggage</a></strong> with <strong><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://www.amazon.com/dp/B014VBI5MS\">Amazon packing cubes</a></strong>.</p>\n<p>I genuinely hate traveling, but a few nice things make it bearable: <strong><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://mantasleep.com/products/manta-sleep-mask\">Manta sleep mask</a></strong> and bringing hydrating face and hair products because airplane air is so dry.</p>\n<hr />\n<h2>Fashion</h2>\n<p><strong>Wyatt harness boots</strong> — the uniform that never ages. Black, resolable, built to last. Get them resoled right after purchase. I've worn my <strong><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://www.ysl.com/en-us/pr/wyatt-harness-boots-in-smooth-leather-810162485.html\">Wyatt harness boots</a></strong> for years and they're still pristine.</p>\n<p><strong><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://www.ssense.com/en-us/men/product/nanamica/khaki-straight-cut-cargo-pants/16372621\">Cargo pants</a></strong> made from CORDURA. <strong><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://www.tenthousand.com/\">Ten Thousand</a></strong> for shirts.</p>\n<p><strong><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://www.amazon.com/dp/B07V91437Y\">Clothing steamer</a></strong> — wrinkly clothes are not it.</p>\n<h3>Yohji Yamamoto — the philosophy</h3>\n<p>My favorite brand is <strong><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://www.theshopyohjiyamamoto.com/\">Yohji Yamamoto</a></strong>. His signature style combines Japanese design with European techniques, focusing on innovative draping and materials. Asymmetrical cuts and predominantly black palette, influenced by watching his mother work with fabrics in their small shop.</p>\n<p>One good Yohji piece annihilates three seasons of hype drops. Buy once, hem to your proportions, and you'll look like you care about form and have better things to do than chase trends.</p>\n<p>The trick: find a couple good brands that work for your body type, then stick with them as you learn about your proportions.</p>\n<h3>Extreme Cashmere</h3>\n<p><strong><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://extreme-cashmere.com/\">Extreme Cashmere</a></strong> — Dutch brand making oversized, genderless cashmere knits. Their pieces are numbered instead of named. The staff at their Soho store is wonderful, and the quality shows in the blend of materials. I have a gray set for loungewear and an outerwear jacket.</p>\n<hr />\n<h2>Home and Objects</h2>\n<p><strong>Curated essentials:</strong></p>\n<ul>\n<li>Chikuseiko Charcoal Incense</li>\n<li>Cast Iron Incense Holder</li>\n<li>Dried Eucalyptus Bundle</li>\n<li>Noren Curtain for entrances</li>\n<li>Bell for doorway</li>\n<li>Sori Yanagi kettle</li>\n<li>Hasami porcelain</li>\n<li>Epicurean cutting board</li>\n<li>Kettlebell</li>\n<li>Noguchi coffee table and coffee table books</li>\n</ul>\n<p>Get art for your walls, magnets for photos on your fridge, and frame everything. A calendar is a nice touch.</p>\n<p><strong><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://www.dyson.com/\">Dyson vacuum</a></strong> and <strong><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://www.amazon.com/dp/B0DCHBC8JJ\">Bissell steam cleaner</a></strong> for deep cleaning.</p>\n<hr />\n<h2>Kitchen</h2>\n<p><strong>Essential tools:</strong></p>\n<ul>\n<li><strong>Good knife</strong> — I'm Chinese about this. I only use a vegetable cleaver for every function</li>\n<li><strong>Epicurean cutting board</strong></li>\n<li><strong>Rice cooker</strong></li>\n<li><strong>Good blender</strong></li>\n</ul>\n<hr />\n<h2>Services</h2>\n<p>I'm constantly buying my time back. First hire: EA, then cleaners, then chef. Still looking for a meal prep chef who could come to my house and cook once a week.</p>\n<ul>\n<li><strong>Laundry &amp; Cleaning Service</strong> — save time, maintain tidy space</li>\n<li><strong>Massage Therapy (bi-monthly)</strong> — stay loose, reduce stress</li>\n<li><strong>Concierge Medical &amp; Personal Chef</strong> — focus on lean body goals, specialized care</li>\n<li><strong>EA Support</strong> — manage appointments, organize travel, optimize calendar</li>\n</ul>\n<hr />\n<h2>Expensive Things That Are Worth It</h2>\n<ul>\n<li>Lay-flat seat on a plane</li>\n<li>Cool jackets</li>\n<li>Nice speakers</li>\n<li>LASIK eye surgery</li>\n</ul>\n",
      "date_modified": "2026-01-24T00:00:00+00:00",
      "date_published": "2026-01-24T00:00:00+00:00",
      "external_url": "https://jxnl.co/writing/2026/01/24/things/",
      "id": "urn:aggr:item:5670a0e2f5cfc085ada26d01b3d76cca8e147873",
      "summary": "A list of things I've been using and plan to use.",
      "tags": [
        "programming",
        "personal",
        "recommendations",
        "things"
      ],
      "title": "Things",
      "url": "https://aggr.aymericbeaumet.com/items/jxnl-co/2026-01-24-things/"
    },
    {
      "content_html": "<p>I hosted Colin Flaherty, previously a founding engineer at Augment and co-author of Meta's Cicero AI, to discuss autonomous coding agents and retrieval systems. This session explores how agentic approaches are transforming traditional RAG systems, what we can learn from state-of-the-art coding agents, and how these insights might apply to other domains.</p>\n<p><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://maven.com/p/5f4d74\">▶️ See How Grep Beat Complex Embeddings</a></p>\n<h2>Do agents make traditional RAG obsolete?</h2>\n<p>Colin shared his experience building an agent for SWE-Bench Verified, a canonical AI coding evaluation where agents implement code changes based on problem descriptions. His team's agent reached the top of the leaderboard with a surprising discovery: embedding-based retrieval wasn't the bottleneck they expected.</p>\n<p>&quot;We explored adding various embedding-based retrieval tools, but found that for SweeBench tasks this was not the bottleneck - grep and find were sufficient,&quot; Colin explained. This initially surprised him, as he expected embedding models to be significantly more powerful.</p>\n<p>The evolution of code retrieval complexity has followed a clear progression:</p>\n<ul>\n<li>2023 (Completions): Simple retrieval with low latency requirements</li>\n<li>2024 (Chatbots): More complex retrieval across multiple files</li>\n<li>2025 (Agents): Highly complex retrieval across many parts of codebases</li>\n</ul>\n<p>When examining how the agent solved problems, Colin observed it would use simple tools like grep and find persistently, trying different approaches until it found what it needed. The agent's persistence effectively compensated for less sophisticated tools.</p>\n<p><em><strong>Key Takeaway:</strong></em> Agents don't necessarily make traditional RAG obsolete, but they change how we should think about retrieval systems. The persistence and course-correction capabilities of agents can sometimes overcome limitations in the underlying retrieval tools.</p>\n<p><strong>Benefits of agentic retrieval with simple tools</strong> Agentic retrieval with grep and find offers several advantages:</p>\n<ol>\n<li>Iterative retrieval becomes much simpler - instead of complex multi-step embedding processes, agents can simply run multiple searches and refine as they go</li>\n<li>Token budget management is straightforward - when the agent hits token limits, it can simply truncate old tool calls and rerun them if needed</li>\n<li>Implementation is low-effort - no need to maintain vector databases, syncing mechanisms, or other infrastructure</li>\n<li>Course correction happens naturally - if one search approach fails, the agent tries another</li>\n</ol>\n<p>However, these simple approaches have clear limitations:</p>\n<ul>\n<li>They don't scale well to large codebases</li>\n<li>They struggle with unstructured natural language content</li>\n<li>They're relatively slow and expensive compared to optimized embedding lookups</li>\n</ul>\n<p>The best approach might be combining both worlds - an agentic loop with access to high-quality embedding models as tools.</p>\n<p><strong>How to architect retrieval systems for different needs</strong> When deciding between traditional RAG, agent+grep/find, or agent+embeddings, Colin recommends considering several factors:</p>\n<ul>\n<li>Quality: How good is the final output?</li>\n<li>Latency: How quickly does the system respond?</li>\n<li>Cost: What are the computational expenses?</li>\n<li>Reliability: Does the system correct itself when it fails?</li>\n<li>Scalability: How well does it handle large indices?</li>\n<li>Maintenance effort: How much engineering work is required?</li>\n</ul>\n<p>Traditional RAG offers decent quality with excellent speed, low cost, and high scalability, but lacks course correction. Agent+grep provides excellent quality and reliability but struggles with speed, cost, and scale. Agent+embeddings combines the best of both but remains slow and expensive.</p>\n<p><em><strong>Key Takeaway:</strong></em> Don't throw away your existing retrieval systems - instead, expose them as tools to agents. This gives you the benefits of both approaches while allowing you to optimize based on your specific constraints. For implementation guidance, see the <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://jxnl.co/writing/2025/08/27/facets-context-engineering/\">Context Engineering approach to tool response design</a>, which shows how to structure tool outputs to give agents better peripheral vision of the information landscape.</p>\n<p><strong>Evaluating agentic retrieval systems</strong> Colin emphasized a &quot;vibe-first&quot; approach to evaluation:</p>\n<p>&quot;Start with 5-10 examples and do end-to-end vibe checks before moving to quantitative evaluation. With natural language systems, you can learn so much just from looking at a few examples.&quot;</p>\n<p>He noted that improving embedding models doesn't necessarily improve end-to-end performance because agents are persistent - they'll eventually find what they need even with suboptimal tools. This makes traditional embedding evaluation metrics less useful for agentic systems.</p>\n<p>For those starting from scratch, Colin recommends:</p>\n<ol>\n<li>Build the simplest possible retrieval tool</li>\n<li>Put an agent loop on top</li>\n<li>Iterate based on what causes the most pain for users</li>\n<li>Only move to quantitative evaluation once you've addressed obvious issues</li>\n</ol>\n<p>I found it refreshing that Colin focused on specific examples of queries rather than abstract discussions of model architectures. As he put it, &quot;Being a researcher is actually very similar to being a product person - you're working backwards from use cases and examples.&quot;</p>\n<p><strong>When to use embedding models vs. simple search tools</strong> While grep and find worked well for SWE-Bench's relatively small codebases, Colin identified several scenarios where embedding models become essential:</p>\n<ol>\n<li>Searching large codebases</li>\n<li>Retrieving from unstructured content like Slack messages or documentation</li>\n<li>Searching across third-party code that models haven't memorized</li>\n<li>Retrieving from non-text media like video recordings of user sessions</li>\n</ol>\n<p>&quot;If I was a human working on this use case, and I was a really persistent human that never got tired, would having this other search tool help me? If the answer is yes, then it's probably going to be useful for the agent.&quot;</p>\n<p>Colin noted that SWE-Bench is somewhat artificial - its repositories are smaller than real-world codebases, and 90% of its problems take less than an hour for a good engineer to solve. In more complex environments, embedding models become increasingly valuable.</p>\n<p><strong>Improving agentic retrieval systems</strong> To enhance agentic retrieval, Colin recommends:</p>\n<ol>\n<li>Adding re-rankers to embedding tools to improve precision and reduce token usage</li>\n<li>Training specialized embedding models for different tasks (e.g., one for code, another for Slack messages)</li>\n<li>Prompt-tuning tool schemas to guide agents toward efficient usage patterns</li>\n<li>Creating hierarchical retrieval systems that summarize files and directories</li>\n<li>Leveraging language server protocols as additional tools</li>\n</ol>\n<p>One particularly effective technique is asynchronous pre-processing: &quot;I've taken songs and used an LLM to create a dossier about each one. This simple pre-processing step took a totally non-working search system and turned it into something that works really well.&quot;</p>\n<h2>Why aren't more people training great embedding models?</h2>\n<p>When asked what question people aren't asking enough, Colin highlighted the lack of expertise in training embedding models: &quot;Very few people understand how to build and train good retrieval systems. It just confuses me why no one knows how to fine-tune really good embedding models.&quot;</p>\n<p>He attributed this partly to the specialized nature of the skill and partly to data availability. For code, there's abundant data on GitHub, but most domains lack comparable resources. Additionally, the most talented engineers often prefer working on LLMs rather than embedding models.</p>\n<p><em><strong>Key Takeaway:</strong></em> As agents become more capable, the quality of their tools becomes increasingly important. Even though agents can compensate for suboptimal tools through persistence, providing them with better retrieval mechanisms significantly improves their efficiency and capabilities.</p>\n<p><strong>Final thoughts on the future of retrieval</strong> Colin believes we're entering an era where the boundaries between traditional RAG and agentic systems are blurring. The ideal approach combines the strengths of both: the speed and efficiency of well-tuned embedding models with the persistence and course-correction of agents.</p>\n<p>As these systems evolve, we'll likely see more specialized tools emerging for different retrieval contexts, along with more sophisticated pre-processing techniques that make retrieval more effective. The key is focusing on the specific problems you're trying to solve rather than getting caught up in architectural debates.</p>\n<p>&quot;Agents are getting radically smarter, but even Einstein preferred writing on paper instead of a stone tablet,&quot; Colin noted. &quot;Yes, these agents are persistent, but you should give them whatever you can to improve the odds that they find what they're looking for.&quot;</p>\n<hr />\n<p><strong>FAQs</strong></p>\n<h2>What is agentic retrieval and how does it differ from traditional RAG?</h2>\n<p>Agentic retrieval is an approach where AI agents use tools like grep, find, or embedding models to search through code and other content. Unlike traditional RAG (Retrieval-Augmented Generation), which typically uses embedding databases and vector searches, agentic retrieval gives the agent direct control over the search process. This allows the agent to be persistent, try multiple search strategies, and course-correct when initial attempts fail. Traditional RAG is more rigid but can be faster and more efficient for certain use cases.</p>\n<h2>Do agents make traditional RAG obsolete?</h2>\n<p>No, agents don't make traditional RAG obsolete—they complement it. The best approach is often to build agentic retrieval on top of your existing retrieval system by exposing your embedding models and search capabilities as tools that an agent can use. This combines the strengths of both approaches: the persistence and flexibility of agents with the efficiency and scalability of well-tuned embedding models.</p>\n<h2>What are the benefits of using grep and find tools with agents?</h2>\n<p>Using simple tools like grep and find with agents offers several advantages:</p>\n<ul>\n<li>Iterative retrieval becomes much easier as agents can refine searches based on previous results</li>\n<li>Token budget management is simpler since old tool calls can be truncated and rerun if needed</li>\n<li>The system is easier to build and maintain without complex vector database dependencies</li>\n<li>Agents can course-correct when searches don't yield useful results by trying different approaches</li>\n</ul>\n<h2>What are the limitations of using grep and find for retrieval?</h2>\n<p>While grep and find work well for certain scenarios, they have significant limitations:</p>\n<ul>\n<li>They don't scale well to very large codebases (millions of files)</li>\n<li>They're ineffective for searching through unstructured natural language content</li>\n<li>They work best with highly structured content like code that contains distinctive keywords</li>\n<li>They can be slower than optimized embedding-based searches for large datasets</li>\n</ul>\n<h2>What's the ideal approach to retrieval for coding agents?</h2>\n<p>The best approach is often a hybrid system that combines:</p>\n<ol>\n<li>An agentic loop that gives the agent control over the search process</li>\n<li>Access to multiple search tools including grep, find, and embedding-based search</li>\n<li>The ability to choose the most appropriate tool based on the specific search task</li>\n<li>Course correction capabilities when initial searches don't yield useful results</li>\n</ol>\n<h2>How should I evaluate agentic retrieval systems?</h2>\n<p>Start with a qualitative &quot;vibe check&quot; using 5-10 examples to understand how the system performs. Observe the agent's behavior, identify patterns in successes and failures, and develop an intuition for where improvements are needed. Only after this initial assessment should you move to quantitative end-to-end evaluations or specific evaluations of individual components like embedding tools. Remember that improving a single component (like an embedding model) may not necessarily improve the end-to-end performance if the agent is already persistent enough to overcome limitations.</p>\n<h2>I already built a retrieval system with custom-trained embedding models. Should I replace it with agentic retrieval?</h2>\n<p>No, don't replace it—enhance it. Build agentic retrieval on top of your existing system by exposing your embedding models and search capabilities as tools that an agent can use. This gives you the best of both worlds: the quality and efficiency of your custom embeddings plus the persistence and flexibility of an agent.</p>\n<h2>How can I improve my agentic retrieval system?</h2>\n<p>Focus on building better tools for your agent:</p>\n<ul>\n<li>Add re-rankers to your embedding tools to improve precision and reduce token usage</li>\n<li>Train different embedding models for different specific tasks</li>\n<li>Prompt-tune your tool schemas to help the agent use them effectively</li>\n<li>Consider hierarchical retrieval approaches like creating summaries of files or directories</li>\n<li>Add specialized tools for specific retrieval tasks (like searching commit history)</li>\n</ul>\n<h2>How do memories work with agentic retrieval systems?</h2>\n<p>Memories in agentic systems can be implemented by adding tools that save and read memories. These memories can serve as a semantic cache that speeds up future searches by storing information about the codebase structure, relevant interfaces, or other insights gained during previous searches. This can significantly improve performance on similar tasks in the future.</p>\n<h2>Why did embedding models not improve performance on SWE-Bench?</h2>\n<p>For the SWE-Bench coding evaluation, embedding models didn't significantly improve performance because:</p>\n<ol>\n<li>The repositories were relatively small, making grep and find sufficient</li>\n<li>The code was highly structured with distinctive keywords that made text-based search effective</li>\n<li>The agent's persistence compensated for less sophisticated search tools</li>\n<li>The tasks were relatively simple, typically solvable by a good engineer in under an hour</li>\n</ol>\n<h2>This doesn't mean embedding models aren't valuable—they become essential for larger codebases, less structured content, or more complex retrieval tasks.</h2>\n",
      "date_modified": "2025-09-11T00:00:00+00:00",
      "date_published": "2025-09-11T00:00:00+00:00",
      "external_url": "https://jxnl.co/writing/2025/09/11/why-grep-beat-embeddings-in-our-swe-bench-agent-lessons-from-augment/",
      "id": "urn:aggr:item:303877d6f196ce754a027dfdb5ce964a207bf5f2",
      "summary": "Insights from Colin Flaherty on building autonomous coding agents and how agentic approaches reshape retrieval-augmented generation systems.",
      "tags": [
        "programming",
        "agentic systems",
        "agents",
        "autonomous coding",
        "coding agent series",
        "rag",
        "retrieval",
        "speaker series"
      ],
      "title": "Why Grep Beat Embeddings in Our SWE-Bench Agent (Lessons from Augment)",
      "url": "https://aggr.aymericbeaumet.com/items/jxnl-co/2025-09-11-why-grep-beat-embeddings-in-our-swe-bench-agent-lessons/"
    },
    {
      "content_html": "<p>I hosted Manav from Glean for a guest lecture on enterprise search and fine-tuning embedding models. This session revealed a surprisingly underutilized approach that can dramatically improve RAG system performance - building custom embedding models for each customer rather than using generic solutions.</p>\n<p><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://maven.com/p/1ea9c9\">▶️ Learn Glean's Custom Model Strategy</a></p>\n<h2>What is Glean and why does their approach to enterprise search matter?</h2>\n<p>Glean has built a comprehensive Work AI platform that unifies enterprise data across various applications (Google Drive, GitHub, Jira, Confluence) into a single system. Their flagship product, the Glean Assistant, leverages this unified data model to generate relevant answers to user questions and automate workflows.</p>\n<p>The foundation of their system is semantic search capability, which Manav emphasized is absolutely critical for enterprise AI success:</p>\n<blockquote>\n<p>&quot;Search quality matters - you can't have a good RAG system, you can't have a good overall enterprise AI product unless you have good search.&quot;</p>\n</blockquote>\n<p>This makes intuitive sense - without retrieving the right context from your enterprise data, even the best LLMs will produce hallucinations and incorrect information.</p>\n<p><strong>Key Takeaway:</strong> Search quality is the foundation of enterprise AI success. Without effective retrieval from enterprise data, even the most advanced LLMs will generate hallucinations and provide incorrect information.</p>\n<h2>What makes enterprise data uniquely challenging?</h2>\n<p>Unlike internet data, which has a significant &quot;head problem&quot; where most searches target popular websites or common information sources, enterprise data is far more heterogeneous and doesn't fit neatly into a single mold. Manav explained:</p>\n<blockquote>\n<p>&quot;Enterprise data is very different than internet data... You have your basic document data sources like Google Drive, Google Docs, Confluence, Notion... But you're also working with a bunch of different types of applications, like Slack, which is a messaging platform. You have meetings, which doesn't really meet the standard concept of what a document is. You have GitHub and GitLab... They all behave in slightly different ways.&quot;</p>\n</blockquote>\n<p>This diversity requires a robust, generalized unified data model that can handle the nuances of different data types while maintaining security and privacy. Additionally, company-specific language (project names, initiatives, internal terminology) creates another layer of complexity that generic models struggle with.</p>\n<p><strong>Key Takeaway:</strong> Enterprise search is fundamentally different from web search because of data heterogeneity and company-specific language. A unified data model that can handle diverse data types while preserving security is essential for effective enterprise AI.</p>\n<h2>Why fine-tune embedding models for each customer?</h2>\n<p>One of the most fascinating aspects of Glean's approach is that they build custom embedding models for each customer. While many companies focus on using large, general-purpose embedding models, Glean has found that smaller, fine-tuned models often perform better for specific enterprise contexts.</p>\n<h3>Glean's Custom Model Process:</h3>\n<ol>\n<li><strong>Start with a high-performance base model</strong> (typically BERT-based)</li>\n<li><strong>Perform continued pre-training</strong> on company data using masked language modeling</li>\n<li><strong>Convert the language model</strong> into an embedding model through various training techniques</li>\n<li><strong>Continuously update the model</strong> as the company evolves</li>\n</ol>\n<p>The results are impressive - after six months, they typically see a <strong>20% improvement in search performance</strong> just from learning from user feedback and adapting to company changes.</p>\n<p>Manav emphasized the power of smaller, specialized models:</p>\n<blockquote>\n<p>&quot;When you're thinking about building really performant enterprise AI... you want to also think about using smaller embedding models when you can, because small embedding models when fine-tuned to the domain and the specific task you have in hand can give you a lot better performance compared to just using large LLMs.&quot;</p>\n</blockquote>\n<p><strong>Key Takeaway:</strong> Smaller, fine-tuned embedding models often outperform large general-purpose models for enterprise contexts. Glean achieves 20% search performance improvements through continuous model adaptation to company-specific language and user feedback.</p>\n<h2>How do they generate high-quality training data?</h2>\n<p>Creating effective training data for fine-tuning embedding models is challenging, especially with enterprise privacy constraints. Glean uses several creative approaches:</p>\n<h3>Training Data Sources:</h3>\n<ul>\n<li><strong>Title-body pairs:</strong> Mapping document titles to passages from the document body</li>\n<li><strong>Anchor data:</strong> Using documents that reference other documents to create relevance pairs</li>\n<li><strong>Co-access data:</strong> Identifying documents accessed together by users in short time periods</li>\n<li><strong>Public datasets:</strong> Incorporating high-quality public datasets like MS MARCO</li>\n<li><strong>Synthetic data:</strong> Using LLMs to create question-answer pairs for documents</li>\n</ul>\n<h3>Application-Specific Intelligence</h3>\n<p>What's most impressive is their attention to application-specific nuances. For example, with Slack data, they don't just treat each message as a document. Instead, they create &quot;conversation documents&quot; from threads or messages within a short timespan, then use the first message as a title and the rest as the body.</p>\n<p>This understanding of how different applications work leads to much higher quality training data than generic approaches.</p>\n<p><strong>Key Takeaway:</strong> Generating high-quality training data requires understanding the nuances of different enterprise applications. Creative approaches like title-body pairs, anchor data, co-access signals, and synthetic data generation can provide valuable training signals even with privacy constraints.</p>\n<h2>How do they learn from user feedback?</h2>\n<p>Once users start interacting with their products, Glean incorporates those signals to further improve their models:</p>\n<h3>Search Product Feedback:</h3>\n<ul>\n<li><strong>Query-click pairs:</strong> Direct signals of relevance from user interactions</li>\n</ul>\n<h3>RAG Assistant Feedback (More Challenging):</h3>\n<p>For RAG-only settings like their Assistant product, where users don't explicitly click on documents, they face a more challenging problem:</p>\n<ul>\n<li><strong>Upvote/downvote systems</strong> (though these tend to get sparse usage)</li>\n<li><strong>Citation tracking</strong> when users click on citations to read more about a topic</li>\n<li><strong>Interaction pattern monitoring</strong> to infer relevance from various user behaviors</li>\n</ul>\n<p>Manav candidly acknowledged the difficulty:</p>\n<blockquote>\n<p>&quot;This is like a pretty hard open question&quot;</p>\n</blockquote>\n<p>Their approach of combining multiple weak signals seems pragmatic given the inherent challenge of getting explicit feedback signals for generative AI products.</p>\n<p><strong>Key Takeaway:</strong> Learning from user feedback in RAG systems is challenging, especially for generative interfaces. Combining multiple weak signals (upvotes, citation clicks, interaction patterns) provides a more robust approach than relying on any single feedback mechanism.</p>\n<h2>How do they evaluate embedding model quality?</h2>\n<p>Evaluating embedding models in enterprise settings is particularly challenging because:</p>\n<ul>\n<li><strong>Privacy constraints:</strong> You can't access customer data directly</li>\n<li><strong>Unique models:</strong> Each customer has a different model</li>\n<li><strong>Complex systems:</strong> End-to-end RAG evaluation involves many moving parts</li>\n</ul>\n<h3>Glean's &quot;Unit Test&quot; Approach</h3>\n<p>Glean's solution is to build &quot;unit tests&quot; for their models - targeted evaluations for specific behaviors they want their models to exhibit. For example, they test how well models understand paraphrases of the same query.</p>\n<p>This approach allows them to:</p>\n<ul>\n<li><strong>Set performance targets</strong> for each customer's model</li>\n<li><strong>Identify underperforming models</strong> before customers experience issues</li>\n<li><strong>Focus optimization efforts</strong> on specific areas</li>\n</ul>\n<p>Manav emphasized the importance of component-level optimization:</p>\n<blockquote>\n<p>&quot;If you want to really make good tangible progress day by day, isolating and optimizing individual components is always going to be much more scalable than trying to improve everything all together all at once.&quot;</p>\n</blockquote>\n<p><strong>Key Takeaway:</strong> Evaluating embedding models in enterprise settings requires targeted &quot;unit tests&quot; that isolate specific behaviors. This component-level approach enables scalable optimization and prevents customer-facing issues.</p>\n<h2>What role does traditional search play alongside embeddings?</h2>\n<p>Despite all the focus on embedding models, Manav emphasized that traditional search techniques remain crucial:</p>\n<blockquote>\n<p>&quot;You don't want to over-index on semanticness or LLM-based scoring as the only thing that your search system should use... you can get a lot more bang for your buck by not using any semanticness at all to answer most questions.&quot;</p>\n</blockquote>\n<h3>The 60-70% Rule</h3>\n<p>Manav estimated that for <strong>60-70% of enterprise search queries</strong>, basic lexical search with recency signals works perfectly well. Semantic search becomes more important for complex queries, particularly in agent-based systems.</p>\n<p>This aligns with practical experience - getting 80% of the way there with full-text search and then adding semantic search as the cherry on top is often the most effective approach.</p>\n<p><strong>Key Takeaway:</strong> Don't abandon traditional search techniques in pursuit of embedding-based approaches. A hybrid system that leverages both lexical and semantic search, along with signals like recency and authority, will deliver the best results for enterprise search.</p>\n<h2>How do they handle document relevance over time?</h2>\n<p>One interesting question addressed how Glean handles outdated documents that have been superseded by newer information. Their approach centers around a concept they call &quot;authoritativeness,&quot; which incorporates:</p>\n<h3>Authoritativeness Factors:</h3>\n<ul>\n<li><strong>Recency:</strong> Newer documents are generally more relevant</li>\n<li><strong>Reference patterns:</strong> Documents that continue to be linked to or accessed remain authoritative</li>\n<li><strong>User satisfaction signals:</strong> Documents that consistently satisfy user queries maintain relevance</li>\n</ul>\n<h3>Real-World Example</h3>\n<p>A document containing WiFi password information might be old but still highly relevant if people continue to reference it when answering related questions.</p>\n<p>This multi-faceted approach to document authority is more sophisticated than simply prioritizing recent content, which would miss important evergreen documents.</p>\n<p><strong>Key Takeaway:</strong> Document relevance over time requires a multi-faceted &quot;authoritativeness&quot; approach that balances recency with reference patterns and user satisfaction signals, rather than simply prioritizing the newest content.</p>\n<h2>Final thoughts on building enterprise search systems</h2>\n<p>Manav concluded with several key insights for building effective enterprise search systems:</p>\n<h3>Core Principles:</h3>\n<ul>\n<li><strong>Unified data model</strong> is critical for handling heterogeneous enterprise data</li>\n<li><strong>Company-specific language</strong> matters tremendously for search quality</li>\n<li><strong>Fine-tuned smaller models</strong> often outperform generic large models for specific tasks</li>\n<li><strong>User feedback learning</strong>, though challenging, provides invaluable signals</li>\n<li><strong>Targeted &quot;unit tests&quot;</strong> enable scalable model quality assessment</li>\n<li><strong>Traditional search techniques</strong> remain powerful and shouldn't be discarded</li>\n</ul>\n<h3>The Pragmatic Approach</h3>\n<p>Glean's approach is refreshingly pragmatic. They've learned that the path to high-quality enterprise search isn't just about using the latest, largest models, but about understanding the unique characteristics of enterprise data and building systems that address those specific challenges.</p>\n<p>The emphasis on company-specific language models is particularly noteworthy - this is an area where many companies struggle when they try to apply generic embedding models to their unique terminology and document structures.</p>\n<p><strong>Key Takeaway:</strong> Successful enterprise search requires a pragmatic approach that combines custom embedding models, unified data architecture, hybrid search techniques, and continuous learning from user feedback rather than relying solely on off-the-shelf solutions.</p>\n<h2>--8←</h2>\n<p>&quot;snippets/enrollment-button.md&quot;</p>\n<p>includes/mkdocs.md</p>\n",
      "date_modified": "2025-09-11T00:00:00+00:00",
      "date_published": "2025-09-11T00:00:00+00:00",
      "external_url": "https://jxnl.co/writing/2025/09/11/why-glean-builds-custom-embedding-models-for-every-customer/",
      "id": "urn:aggr:item:a467ab86c65c13000a52a27ee0ec5445a506ba7a",
      "summary": "How Glean achieves 20% search performance improvements through customer-specific embedding models, unified data architecture, and smart feedback loops that most enterprise AI companies are missing.",
      "tags": [
        "programming",
        "custom models",
        "embedding models",
        "enterprise search",
        "fine-tuning",
        "glean",
        "rag",
        "rag optimization",
        "speaker series"
      ],
      "title": "Why Glean Builds Custom Embedding Models for Every Customer",
      "url": "https://aggr.aymericbeaumet.com/items/jxnl-co/2025-09-11-why-glean-builds-custom-embedding-models-for-every-customer/"
    },
    {
      "content_html": "<p>I hosted a session with Walden Yan, co-founder and CPO of Cognition, to explore why multi-agent systems might not be the optimal approach for coding contexts. We discussed the theory of context engineering, the challenges of context passing between agents, and how single agents with proper context management can often outperform multi-agent setups.</p>\n<p><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://maven.com/p/446ea1\">▶️ Learn Why Devin Chose Single Agents</a></p>\n<h2>Why are multi-agent systems problematic for coding tasks?</h2>\n<p>The fundamental issue with multi-agent systems is context management. When you split a task between multiple agents, you're essentially playing a game of telephone where critical information can get lost in transmission.</p>\n<p>Consider a simple example: building a Flappy Bird clone. If you assign one agent to build the background with green pipes and hitboxes, and another agent to create the bird asset, they might develop completely incompatible components. The first agent might create something resembling Mario's environment, while the second creates a bird from an entirely different game.</p>\n<p>This miscommunication happens because each agent only understands what it was told by the main agent. While this example might seem extreme, these issues compound dramatically when dealing with complex tasks at scale:</p>\n<blockquote>\n<p>&quot;You're gonna find lots of places where miscommunications like this are just gonna get introduced in your system, especially as you as the when the user is talking with the system back and forth many times when these sub agents are getting more and more nuanced tasks, and you forget to pass some details through.&quot;</p>\n</blockquote>\n<p>The context isn't just the user's message but includes everything the agent has done - code files examined, questions asked, and answers received. These full agent traces should ideally be passed to all agents in the system.</p>\n<p><strong>Key Takeaway:</strong> Multi-agent systems often break down due to context loss between agents. What seems like a minor miscommunication in simple examples becomes a significant problem at scale with complex tasks. This connects to the broader <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://jxnl.co/writing/2025/08/28/context-engineering-index/\">Context Engineering principles</a> around managing information flow and avoiding context pollution in AI systems.</p>\n<h2>How can we improve context passing between agents?</h2>\n<p>To minimize the &quot;telephone game&quot; problem, we need to ensure sub-agents have at least the same context as the main agent. This means:</p>\n<ul>\n<li>Sub-agents should see everything the main agent saw</li>\n<li>Sub-agents should have access to the full history of actions taken by the main agent</li>\n<li>The final agent should see everything from all previous agents</li>\n</ul>\n<p>Even with this improved context passing, problems still arise when sub-agents work in parallel. They can't communicate with each other, leading to conflicts in style, APIs used, or duplicated code. When sub-agent one commits to certain decisions and sub-agent two commits to different decisions, these conflicts create integration problems.</p>\n<blockquote>\n<p>&quot;With any agentic system, lots of actions can basically carry these implicit decisions being made... anytime these decisions are made, you almost always have to actually make sure this decision is shared with everyone else, or else you might just get these conflicting decisions.&quot;</p>\n</blockquote>\n<p>A more reliable approach is a linear system where agents work sequentially. Sub-agent one completes its task, then sub-agent two works with full knowledge of what sub-agent one did, avoiding conflicts in decisions.</p>\n<p><strong>Key Takeaway:</strong> Passing full context between agents helps but doesn't solve all problems. When agents work in parallel, they make conflicting implicit decisions that create integration challenges.</p>\n<h2>What are the limitations of simple linear agent systems?</h2>\n<p>While linear agent systems solve many coordination problems, they quickly run into context window limitations. As the task progresses, the context grows larger until it exceeds what the model can effectively process.</p>\n<blockquote>\n<p>&quot;With the simple agent, you know, if you're just collecting everything that this agent's been doing, and especially for these longer tasks, you'll quickly run to context overflow issues, especially as you go down the line towards the end of your task.&quot;</p>\n</blockquote>\n<p>One solution is implementing a context compressor that intelligently selects what information to pass from one agent to the next. However, this adds complexity and requires careful engineering to ensure important details aren't lost.</p>\n<p>At Cognition, they trained a specialized model to identify and preserve critical information from earlier conversations. This approach requires significant investment but can be worthwhile for complex engineering tasks.</p>\n<blockquote>\n<p>&quot;You have to think very carefully about. Are you willing to put in the work to kind of squeeze out this extra juice from your system. In our case it makes sense, because our entire goal is to make sure that we can take on harder and harder engineering tasks.&quot;</p>\n</blockquote>\n<p><strong>Key Takeaway:</strong> Simple linear agent systems eventually hit context window limits. Context compression can help but adds complexity and requires careful engineering to avoid losing critical information.</p>\n<h2>What real-world approaches exist for managing agent context?</h2>\n<p>Several practical approaches have emerged for managing context in agent systems:</p>\n<h3>Read-only sub-agents</h3>\n<p>Both Claude Code and OpenCode use sub-agents that only read and don't make decisions. These agents perform tasks like listing project files, examining packages, or looking for imports, then report back to the main agent.</p>\n<blockquote>\n<p>&quot;The designers of these systems of these agent builders are very careful to constrain their sub agents to only doing read-only tasks. I think if you read the cloud Sys prompt. If you read open code Sysprompt, you'll probably find stuff that's like you should only use subagents for things that involve reading code.&quot;</p>\n</blockquote>\n<h3>Edit-apply models</h3>\n<p>Systems like Cursor and Windsurf use a two-stage approach where a smart coding model generates human-readable instructions for code changes, and a simpler &quot;edit-apply&quot; model implements those changes. This approach can be fragile if the instructions aren't perfectly clear.</p>\n<blockquote>\n<p>&quot;When you see complaints online about, hey? Sometimes in these code editors the edit will be like wrong and or mess something up. It is often because of this that the smart coding agent left out some important detail, or wasn't specific enough about something.&quot;</p>\n</blockquote>\n<p>Even read-only sub-agents can create problems when they return conflicting information, leaving the main agent to resolve contradictions without full context.</p>\n<p><strong>Key Takeaway:</strong> Read-only sub-agents and specialized edit-apply models can help manage context, but each approach has limitations and potential failure points.</p>\n<h2>Why should systems feel like a single agent to users?</h2>\n<p>Even if your system has complex internal components, it should present itself to users as a single, coherent entity. This creates a more intuitive user experience and avoids confusion.</p>\n<blockquote>\n<p>&quot;If you're abiding by like kind of principles of good context, engineering your system as a whole, even if it has like subparts and tasks. It kind of should feel to the user like a single agent, because the user has to feel like they're they're talking with one continuous decision maker, one continuous like consciousness.&quot;</p>\n</blockquote>\n<p>True multi-agent collaboration requires sophisticated capabilities that current models don't yet possess. Effective collaboration between humans requires mentally modeling what others know and don't know - a complex skill that LLMs haven't mastered.</p>\n<blockquote>\n<p>&quot;It actually requires being very intelligent to be good at collaborating with other people like when when I'm working with another engineer on my team, if I am mentally modeling at the same time what they know, what they don't understand and what I have to tell them.&quot;</p>\n</blockquote>\n<p>The path forward likely involves making single agents smarter and more collaborative, which will eventually enable effective multi-agent systems.</p>\n<p><strong>Key Takeaway:</strong> Systems should present as a single coherent entity to users. True multi-agent collaboration requires sophisticated capabilities that will emerge naturally as single agents become more intelligent.</p>\n<h2>What skills are current agents lacking for effective collaboration?</h2>\n<p>Current agents struggle with several key collaboration skills:</p>\n<h3>Escalation awareness</h3>\n<p>Recognizing when they've reached the limits of their decision-making authority and need human input.</p>\n<blockquote>\n<p>&quot;When can a system recognize when it reaches its own limits of the decisions it is allowed to make... How do you know, like a you should? Either way, you should notify the human that like, Hey, there's something weird that's happening here.&quot;</p>\n</blockquote>\n<h3>Confidence assessment</h3>\n<p>Accurately evaluating and communicating their confidence in proposed solutions.</p>\n<blockquote>\n<p>&quot;In Devin. One thing we do is when you collaborating with your agent, when you collaborate when you're collaborating with Devin, Devin will give you the how confident it is in its plan. And that way you, as a human, can say, like, okay, Devin is like 99% confident that this is going to work, you can roughly be hands off.&quot;</p>\n</blockquote>\n<h3>Concise communication</h3>\n<p>Providing key information without overwhelming users with unnecessary details.</p>\n<blockquote>\n<p>&quot;People hate how like, you know, Llms will just send you like a wall of text, and Markdown, a smart engineer will like manage to hone in on the key problems and the key things you have to think about with like minimal words, one sentence, 2 sentence of words.&quot;</p>\n</blockquote>\n<p>These skills are particularly important for human-agent collaboration, where the human should remain the ultimate decision-maker while the agent handles appropriate tasks autonomously.</p>\n<p><strong>Key Takeaway:</strong> Effective collaboration requires agents to know when to escalate decisions, accurately assess their confidence, and communicate concisely - skills that current systems are still developing.</p>\n<h2>How important is planning in agent systems?</h2>\n<p>Planning serves as a crucial form of context management in agent systems. By committing to a structure upfront, agents can avoid the constant recompression of information that happens in more dynamic approaches.</p>\n<blockquote>\n<p>&quot;Planning is something that we do. And the context of planning emergency came up for us is actually a form of context management. Because, again, it's about avoiding this game of telephone where you you have to constantly recompress and the information that you're you're sending to your agent.&quot;</p>\n</blockquote>\n<p>At Cognition, Devin spends significant time reading code and generating a plan at the start of any task. This planning phase is one area where sub-agent-like components might be useful, though they're integrated in a way that still feels like a single system to the user.</p>\n<p>While detailed planning (like creating extensive plan.md files) can improve results, Walden believes these capabilities will eventually be built into the systems themselves:</p>\n<blockquote>\n<p>&quot;You can be shown the plan. You can click this button to like view the full plan that the agent's working on. But the agent will work on this internally for its own sake. You know whether or not like you, tell it to or not.&quot;</p>\n</blockquote>\n<p><strong>Key Takeaway:</strong> Planning is a powerful context management technique that helps agents maintain coherence throughout complex tasks. While users can create detailed plans, these capabilities are increasingly being built into agent systems.</p>\n<h2>What frameworks and patterns show promise for agent development?</h2>\n<p>We're still in the early stages of agent development - comparable to the HTML/CSS/JavaScript era of web development before frameworks like React emerged. However, some promising patterns are emerging:</p>\n<h3>Cache-aware frameworks</h3>\n<p>Systems that optimize for caching, even before API-level caching was available.</p>\n<blockquote>\n<p>&quot;Our system has been built with caching kind of like being a very core condition. So even as you scale out and do these sub agents and things like that, having a system that still manages to like optimally cache work from before, I think is going to be important thing.&quot;</p>\n</blockquote>\n<h3>Integrated evaluation systems</h3>\n<p>Frameworks that tightly couple agent code with evaluation systems.</p>\n<blockquote>\n<p>&quot;We kind of like segment functionalities of our system into modules and these modules are tagged in such a way that not only are things that go through those modules, things that like we can track and feed into eval systems. When we change code inside these modules, we can then also back, run them against evals from before.&quot;</p>\n</blockquote>\n<h3>Universal tools over specialized integrations</h3>\n<p>Providing agents with powerful general-purpose tools like shell access rather than numerous specialized integrations.</p>\n<blockquote>\n<p>&quot;I personally like to just like, give our agent the most universal tools possible, and make sure there's very easy to use. So the shell command, like the the bash tool, for example, is so fucking powerful, it like literally, it means like anything you can do on a computer like your agent can do now.&quot;</p>\n</blockquote>\n<p>CLI tools can sometimes be more powerful than formal integrations because they can interact with the agent in sophisticated ways, almost becoming part of the agent system itself.</p>\n<p><strong>Key Takeaway:</strong> While we're still early in agent framework development, patterns around caching, evaluation integration, and universal tools are showing promise for building robust systems.</p>\n<h2>How is code review evolving with AI agents?</h2>\n<p>As AI generates more code, the review process is evolving to minimize the work required to verify changes. Walden highlighted how they're leveraging existing tools like Vercel's preview links:</p>\n<blockquote>\n<p>&quot;I tag Devin on slack. I go away. I do my thing. And 1st of all, I always tell Devin to send me a screenshot of the visual changes it makes, so I don't have to even leave slack. I will immediately see a screenshot of like, okay, is this even something reasonable that I want to review?&quot;</p>\n</blockquote>\n<p>This approach allows for quick visual verification before diving into code details. Similar infrastructure for backend systems would be valuable but hasn't been heavily invested in yet.</p>\n<p>The future of code review likely involves more automated verification and visualization tools that help humans quickly understand and validate AI-generated changes.</p>\n<p><strong>Key Takeaway:</strong> Code review is evolving to minimize human effort through screenshots, preview environments, and other tools that provide quick verification of AI-generated changes.</p>\n<h2>What exciting developments are coming for developer tools?</h2>\n<p>Walden is particularly excited about creating better synergies between local development environments and remote agents. One area with significant potential is improving code reading and understanding:</p>\n<blockquote>\n<p>&quot;Actually reading code reading code, understanding code planning how you want to make changes. Because when you're an individual developer, you're actually oftentimes you're working a massive code based searching code that isn't yours. You're trying to understand it.&quot;</p>\n</blockquote>\n<p>While current AI tools can help with implementing specific tasks, there's a cap to what they can handle without proper planning. AI has a unique advantage in reading and understanding code much faster than humans, which could transform how developers approach large codebases.</p>\n<blockquote>\n<p>&quot;The AI, even though it can like only edit code serially, it could read codes like so much faster than any of us can. And so I'm very excited about things we're going to be doing and just making that process and experience be a lot easier.&quot;</p>\n</blockquote>\n<p><strong>Key Takeaway:</strong> The next frontier for developer tools involves better code reading, understanding, and planning capabilities that leverage AI's ability to process large codebases quickly.</p>\n<h2>Final thoughts on the future of agent development</h2>\n<p>Walden believes we're just beginning to develop the concepts and principles that will become fundamental to agent development. In the future, these ideas will likely be taught in entry-level college courses.</p>\n<p>The field is evolving rapidly, with practitioners discovering new patterns and approaches through experimentation. While multi-agent systems have captured popular imagination, the immediate future likely belongs to increasingly sophisticated single agents that can handle complex tasks with proper context management.</p>\n<p>As these systems mature, the principles of context engineering will become more formalized, eventually leading to frameworks that make building robust agent systems more accessible to developers without requiring deep expertise in prompt engineering or context management.</p>\n<p><strong>Key Takeaway:</strong> We're in the early stages of developing fundamental concepts for agent systems. While multi-agent approaches capture imagination, near-term progress will come from improving single agents with better context management.</p>\n<hr />\n<h2>FAQs</h2>\n<h3>What are the main issues with multi-agent systems in coding?</h3>\n<p>Multi-agent systems often suffer from context loss and conflicting decisions. When multiple agents work in parallel, they may not share the same understanding of the original task or see each other's work, leading to inconsistencies. This creates a &quot;telephone game&quot; effect where important details get lost as information passes from one agent to another, resulting in incompatible outputs that don't work well together.</p>\n<h3>What is context engineering and why is it important?</h3>\n<p>Context engineering is the practice of carefully managing what information is available to AI agents at each step of their work. Unlike prompt engineering (which focuses on how you phrase instructions), context engineering focuses on ensuring agents have all the necessary background information and history they need to make good decisions. As models become more capable, the limiting factor is increasingly how much relevant context they have access to rather than how you phrase your prompts.</p>\n<h3>What are the two core design principles for building reliable agent systems?</h3>\n<p>The two core principles are: 1) Always share full context and agent traces between components, ensuring each agent can see what happened before, and 2) Carefully track decisions to prevent agents from making conflicting choices. These principles help maintain consistency throughout the system and prevent the &quot;telephone game&quot; problem where important details get lost between agents.</p>\n<h3>How can context overflow be managed in agent systems?</h3>\n<p>As agents work on longer tasks, they can quickly hit context window limits. One approach is using a &quot;context compressor&quot; that intelligently summarizes previous context before passing it to the next agent. This requires careful engineering to ensure important details aren't lost during compression. Another approach is designing systems that are inherently cache-aware, allowing for efficient reuse of previous computations without redundant processing.</p>\n<h3>What are &quot;read-only subagents&quot; and when are they useful?</h3>\n<p>Read-only subagents are limited to gathering information without making decisions that affect the system. They're useful for tasks like searching code, listing files, or checking imports without introducing conflicts. Both Claude and other systems constrain their subagents to these read-only tasks specifically to avoid the problems that arise when multiple agents make independent decisions in parallel.</p>\n<h3>Why might a single agent approach be more reliable than multi-agent systems?</h3>\n<p>Single agent systems avoid the problem of conflicting decisions and context loss between agents. They maintain a continuous &quot;consciousness&quot; with full access to previous context and actions. While they may be slower (lacking parallelism), they're typically more reliable because they don't suffer from coordination problems. A well-designed single agent can still use specialized components internally while presenting a unified interface to users.</p>\n<h3>How should agents handle tasks they're uncertain about?</h3>\n<p>Agents should be designed to recognize their limitations and escalate decisions to humans when appropriate. This includes providing confidence scores with their plans and outputs, so users know when to pay closer attention. Unlike how models are typically trained (to always provide an answer), agents should be willing to say &quot;I don't know&quot; or &quot;I need help&quot; rather than guessing when they lack sufficient information or expertise.</p>\n<h3>What's the &quot;edit-apply model&quot; pattern and what are its limitations?</h3>\n<p>The edit-apply model pattern uses two specialized components: a smart coding model that generates human-readable instructions for code changes, and a simpler, faster model that applies those changes to the code. This pattern can be efficient but introduces fragility because the edit-apply model lacks the full context and intelligence of the main model, making it vulnerable to misinterpreting instructions if they're not perfectly clear.</p>\n<h3>Why are current frameworks for building agents considered premature?</h3>\n<p>Current agent frameworks are considered premature because we're still discovering the fundamental principles and patterns for building reliable agent systems. Similar to web development before React, we're in an early phase where developers are still identifying best practices. Eventually, frameworks will emerge that codify these principles, but for now, it's important to focus on understanding the core concepts like context engineering.</p>\n<h3>How might agent collaboration evolve in the future?</h3>\n<p>While current multi-agent systems face significant challenges, future systems may enable more effective collaboration as models become more intelligent. The ability to model what other agents know and need is a sophisticated skill that requires advanced intelligence. As single agents become more capable, they'll naturally develop the ability to collaborate effectively with other agents, similar to how humans collaborate with teammates.</p>\n",
      "date_modified": "2025-09-11T00:00:00+00:00",
      "date_published": "2025-09-11T00:00:00+00:00",
      "external_url": "https://jxnl.co/writing/2025/09/11/why-cognition-does-not-use-multi-agent-systems/",
      "id": "urn:aggr:item:75c528a8d530bb4f0633c98b91fcdd8795dfccf9",
      "summary": "A deep dive into why multi-agent systems might not be the optimal approach for coding contexts, exploring context engineering, challenges of context passing between agents, and how single agents with proper context management can…",
      "tags": [
        "programming",
        "agent development",
        "coding",
        "coding agent series",
        "context engineering",
        "multi-agent systems",
        "rag",
        "speaker series"
      ],
      "title": "Why Cognition does not use multi-agent systems",
      "url": "https://aggr.aymericbeaumet.com/items/jxnl-co/2025-09-11-why-cognition-does-not-use-multi-agent-systems/"
    },
    {
      "content_html": "<p>I hosted a session with Ayush, an ML Engineer at LanceDB, to explore how fine-tuning re-rankers and embedding models can significantly improve retrieval performance in RAG systems. We discussed practical approaches to enhancing retrieval quality, the trade-offs involved, and when these techniques make the most business sense.</p>\n<p><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://maven.com/p/935d62\">▶️ Get the 12% Performance Boost Technique</a></p>\n<h2>What are re-rankers and why should we use them in RAG systems?</h2>\n<p>Re-rankers fit into the RAG pipeline after retrieval and before the context is provided to the LLM. When you retrieve documents from a vector database, the embedding model gets these documents based on semantic search. Re-rankers then rearrange them so that the most relevant documents to the query rank at the top.</p>\n<p>The power of re-rankers comes from their architecture. While embedding models (bi-encoders) calculate embeddings independently for documents and queries, cross-encoders process both together, allowing them to attend to each other throughout the pipeline. This cross-attention enables much better similarity assessment than independent embeddings.</p>\n<p>You might wonder why we don't just use cross-encoders for retrieval directly. The answer is computational intensity - comparing a query with each document in your database isn't feasible at scale. However, once you've narrowed down to your top 5-10 results, re-ranking becomes practical and only adds milliseconds to your pipeline.</p>\n<p><strong>Key Takeaway:</strong> Re-rankers are a &quot;low-hanging fruit&quot; for improving RAG systems because they don't disrupt your pipeline - you don't need to re-ingest data into your embedding database. They simply plug in after retrieval and before sending context to the LLM.</p>\n<h2>How do you train a re-ranker from scratch?</h2>\n<p>Training a re-ranker requires query-context pairs, similar to training embedding models. In Ayush's benchmark, he used Google's QA dataset with 3 million query-context pairs, using 2 million for training and 5,000 for evaluation.</p>\n<p>The training process involves providing an anchor (the query) along with positive examples (correct answers) and negative examples (incorrect answers). The most effective approach is mining &quot;hard negatives&quot; - wrong answers that are very close to the query in embedding space. These challenge the model to learn subtle distinctions.</p>\n<p>For the base architecture, Ayush experimented with two models: MiniLM - A small 6 million parameter model that's easy to train and test Modern BERT - A 150 million parameter model developed by AnswerAI that performs better than some billion-parameter models</p>\n<p>On top of these base models, he attached re-ranking heads (either cross-encoder or ColBERT) and trained them to function as re-rankers. The results showed significant improvements: Vector search improved by 12% at top-5 and 6% at top-10 Full-text search saw improvements up to 20% in some cases Even the small MiniLM model showed improvements over baseline performance</p>\n<p><strong>Key Takeaway:</strong> Even a small re-ranker model can provide significant retrieval improvements. If you're unsure whether re-ranking will help your specific data, try training a small model in 30-60 minutes to get a signal before investing in more sophisticated approaches.</p>\n<h2>What is the ColBERT architecture and how does it compare to cross-encoders?</h2>\n<p>ColBERT is a late interaction model that offers a middle ground between bi-encoders and cross-encoders. It independently calculates document embeddings offline (which can be stored in a vector database), but at query time, it compares token-level embeddings with the query.</p>\n<p>This approach allows for more nuanced matching than standard bi-encoders while being more computationally efficient than cross-encoders. In Ayush's benchmarks, ColBERT architectures performed well, particularly when built on top of the Modern BERT base model.</p>\n<p>The key advantage of ColBERT is that it still attends to token-level features but does so in two stages - the part that can be calculated offline is saved in a vector database, and at query time, it performs a maximum operation on token-level similarities to find the best match.</p>\n<h2>Should I fine-tune an existing re-ranker or train one from scratch?</h2>\n<p>When deciding whether to train a re-ranker from scratch or fine-tune an existing one, consider these factors:</p>\n<p>If there are no re-rankers available for your base model of choice, you'll need to train from scratch. If your dataset has become specialized enough (drifted from the original data the model was trained on), you should fine-tune to align with your distribution.</p>\n<p>In Ayush's experiments, using an off-the-shelf re-ranker (AnswerAI's ColBERT Small) immediately improved performance by about 10% over the baseline. Fine-tuning that model further improved results by an additional 2-3%.</p>\n<p>Fine-tuning has practical advantages - it converges much faster since the weights aren't random and already have context. However, be careful about catastrophic forgetting - if you fine-tune for too long on low-quality data, the model's performance can degrade from its baseline.</p>\n<p><strong>Key Takeaway:</strong> Start with an existing re-ranker if available for your use case, and fine-tune it on your domain-specific data. This approach is faster and often produces better results than training from scratch.</p>\n<h2>What are the trade-offs when implementing re-rankers?</h2>\n<p>The main trade-off with re-rankers is added latency. Ayush presented latency comparisons on an L4 GPU (similar to a T4 with larger memory): Without re-ranking: Baseline latency With ColBERT on Modern BERT: ~30ms additional latency With smaller models like MiniLM: Negligible additional latency on GPU</p>\n<p>When fetching larger result sets (overfetching documents to then re-rank them), the slowdown factor ranges from 2-3x. On CPU, the penalty is larger - up to 4-5x depending on the architecture.</p>\n<p>Given these trade-offs, who should use re-rankers? Ayush suggests that rather than asking who should use them, we should ask who shouldn't: If latency is absolutely critical (even 10ms matters), you might want to skip re-ranking In all other cases where a few hundred milliseconds doesn't matter (which is most use cases), re-rankers are worth implementing</p>\n<p><strong>Key Takeaway:</strong> Re-rankers add some latency to your pipeline, but for most applications, the performance improvement outweighs this cost. Only the most latency-sensitive applications should avoid them.</p>\n<h2>How effective is fine-tuning embedding models?</h2>\n<p>Embedding models fit into your retriever when you ingest data - they create vector representations of your documents that get stored in your database. Fine-tuning these models can also improve retrieval performance.</p>\n<p>In Ayush's experiments with the MiniLM model: Baseline performance at top-5: 48% After fine-tuning: 58% (a 10% improvement) Similar improvements at top-10</p>\n<p>However, not all embedding models should be fine-tuned. The ideal scenario for fine-tuning is when: Your domain-specific data has drifted from the original training distribution You have a sufficiently large dataset (tens of thousands of examples)</p>\n<p>If your data is just a subset of what the model was already trained on, fine-tuning might lead to overfitting and catastrophic forgetting. This was demonstrated when fine-tuning on the Stanford Question Answering Dataset (SQuAD) - only 3 of the fine-tuned models performed better than baseline because most embedding models are already trained on this common dataset.</p>\n<p><strong>Key Takeaway:</strong> Fine-tune embedding models when you have domain-specific data that differs from general web content, and when you have enough examples to properly train the model.</p>\n<h2>Can we combine re-ranking and embedding model fine-tuning?</h2>\n<p>Yes, and this approach yields the best results. In Ayush's experiments: Baseline: 48% (top-5) and 60% (top-10) Best embedding-tuned model: 62% (top-5) and 69% (top-10) Combined with best re-ranking model: 64% (top-5) and 71% (top-10)</p>\n<p>This gives you flexibility in addressing trade-offs. If latency is a concern, you might focus more on embedding model improvements. If re-ingesting data is problematic (e.g., with billions of entries), you might emphasize re-ranking instead.</p>\n<h2>How do you find and use hard negatives for training?</h2>\n<p>Hard negatives are incorrect answers that are very similar to the query in embedding space. To mine them: Take your corpus and use an independent embedding model (not the one in your RAG pipeline) Find wrong answers that are closest to your query according to the embedding model Use these as negative examples during training</p>\n<p>In real-world applications, there are other ways to find negatives: If a user deletes a citation that an LLM provided, that's a strong negative signal You can use an LLM to propose potential negatives and verify them</p>\n<p>These approaches help create training data that challenges the model to learn subtle distinctions between relevant and irrelevant content.</p>\n<h2>What are real-world examples where fine-tuning makes sense?</h2>\n<p>Fine-tuning is particularly valuable in specialized domains like legal or healthcare, where general training data from the internet doesn't contain all the necessary information or where much of the information is proprietary.</p>\n<p>Another scenario is when you understand the specific types of queries users are asking. For example, if you know users frequently ask about timing-related information, but the relevant text doesn't contain explicit time words, fine-tuning can help bridge that semantic gap.</p>\n<p>Project-specific terminology is another case - if your company uses internal codenames or jargon (like Pokemon names for projects), fine-tuning helps the model understand these connections.</p>\n<p><strong>Key Takeaway:</strong> Fine-tuning is most valuable when dealing with domain-specific knowledge, proprietary information, or company-specific terminology that general models wouldn't have encountered during training.</p>\n<h2>Should I use data augmentation or synthetic data generation?</h2>\n<p>Ayush cautions against confusing these two approaches: Data augmentation helps generalize an already good dataset with more examples to prevent overfitting Synthetic data generation creates new examples from an existing dataset</p>\n<p>The problem with synthetic data generation is that LLMs (even powerful ones like GPT-4) can hallucinate up to 70% of the time when creating synthetic query-context pairs. This means you might be adding low-quality data to your training set.</p>\n<p>In Ayush's experiments with SQuAD, synthetic data generation consistently produced worse results than baseline. Be very careful with this approach and verify the quality of synthetic data before using it.</p>\n<h2>What's the business benefit of all this fine-tuning work?</h2>\n<p>As I explained during the session, if you're not fine-tuning your embedding models or re-rankers, you're probably closer to Blockbuster than Netflix in your approach. At some point, these performance improvements directly correlate with business outcomes.</p>\n<p>For example: In product recommendations, better re-ranking might mean fewer returns or higher sales volume For content platforms like Netflix, better embeddings ensure users find relevant content without scrolling For search experiences, the difference between the 1st and 10th result is enormous for user experience</p>\n<p>Whenever you're building for humans (rather than AI), ranking quality matters tremendously. Even small improvements in retrieval performance can translate to significant business value.</p>\n<h2>What should we be thinking about next in retrieval systems?</h2>\n<p>Ayush highlighted multimodality as the next frontier. While we've made significant progress with text retrieval, humans interact with technology in multimodal ways (text, images, audio, video).</p>\n<p>The default approach for multimodal embeddings has been CLIP, with many architectures built on top of it. However, we've come a long way from the original CLIP model, and there hasn't been enough content written about these advances.</p>\n<p>Building better benchmarks for multimodal data retrieval and establishing stronger baselines for multimodal RAG systems represents a significant opportunity. While companies like Cohere now offer RAG over PDF screenshots, it's unclear how well these approaches extend to scenes in movies, audio clips, or podcasts.</p>\n<p><strong>Key Takeaway:</strong> Multimodal retrieval is likely the next wave of innovation in RAG systems, but we still lack established benchmarks and best practices in this area.</p>\n<h2>How should I approach model selection for my RAG system?</h2>\n<p>When selecting models for your RAG pipeline, it's never just about which model has the best performance. Instead, you need to consider a portfolio of factors: Maintenance costs Inference complexity Latency requirements Performance improvements</p>\n<p>You might love using a powerful model like GPT-3 as a re-ranker, but if it takes 2 minutes per API call, that's probably not practical for most applications. The best model is the one that balances these considerations for your specific use case.</p>\n<p>This is why having a range of options - from lightweight models like MiniLM to more powerful ones like Modern BERT - gives you flexibility in designing your system. Start with simpler approaches, measure their impact, and scale up complexity only when necessary.</p>\n<p><strong>Key Takeaway:</strong> Model selection should be driven by your specific constraints and requirements, not just raw performance numbers. Consider the entire system when making these decisions.</p>\n<h2>FAQs</h2>\n<h2>What are re-rankers and why should I use them?</h2>\n<p>Re-rankers are models that improve retrieval quality by reordering documents after they've been retrieved from a database. They fit into your pipeline after retrieval and before the context is provided to an LLM, helping to ensure the most relevant documents appear at the top. Re-rankers are particularly valuable because they don't disrupt your existing pipeline—you don't need to re-ingest your entire dataset, making them a low-hanging fruit for improving retrieval performance.</p>\n<h2>How do re-rankers work compared to embedding models?</h2>\n<p>While embedding models (bi-encoders) calculate embeddings for documents and queries independently, re-rankers (cross-encoders) process document-query pairs together, allowing them to calculate cross-attention between both inputs. This enables re-rankers to better understand the relevance between a query and document, resulting in more accurate rankings. However, this power comes with higher computational costs, which is why re-rankers are typically used only on a small subset of already retrieved documents.</p>\n<h2>What performance improvements can I expect from re-rankers?</h2>\n<p>Based on extensive benchmarking, re-rankers typically improve retrieval performance by 10-20% depending on the algorithm used. In the experiments presented, vector search results improved by 12% for top-5 retrieval and 6% for top-10 retrieval. Full-text search saw even more dramatic improvements, with some models showing up to 20% better performance.</p>\n<h2>What are the trade-offs when using re-rankers?</h2>\n<p>The main trade-off is latency. Re-rankers add processing time after retrieval, typically in the range of tens of milliseconds when using a GPU. On CPUs, the latency penalty is higher, potentially 3-4x the baseline retrieval time. For most applications, this additional latency is acceptable, but if your use case is extremely latency-sensitive (where even 10ms matters), you might want to consider other approaches.</p>\n<h2>What re-ranker architectures are available?</h2>\n<p>There are two main re-ranker architectures: Cross-encoders: These process the query and document together, allowing for maximum interaction but requiring more computation. ColBERT: This &quot;late interaction&quot; architecture calculates document embeddings offline and compares token-level embeddings with the query at retrieval time, offering a balance between performance and speed.</p>\n<h2>When should I train a re-ranker from scratch versus fine-tuning an existing one?</h2>\n<p>Train a re-ranker from scratch when: There are no re-rankers available for your preferred base model Your dataset has become highly specialized or has drifted significantly from general data Fine-tune an existing re-ranker when: You want faster convergence during training You have a specialized dataset but don't want to risk catastrophic forgetting Remember that fine-tuning typically converges much faster since the weights aren't random, but training for too long on low-quality data can lead to performance degradation.</p>\n<h2>How do I train a re-ranker?</h2>\n<p>To train a re-ranker, you need query-context pairs with positive (relevant) and negative (irrelevant) examples. The most effective approach is to mine &quot;hard negatives&quot;—documents that are semantically similar to the query but aren't actually relevant answers. This challenges the model to learn nuanced distinctions. Tools like sentence-transformers provide frameworks for training re-rankers with appropriate loss functions.</p>\n<h2>Should I fine-tune my embedding models as well?</h2>\n<p>Fine-tuning embedding models can also improve retrieval performance by 8-10%, but it's more disruptive to your pipeline since it requires re-embedding your entire dataset. Consider fine-tuning your embedding model when: Your data has a different distribution than what the model was originally trained on You have a sufficiently large dataset (tens of thousands of examples) Your data is domain-specific (like legal or medical content)</p>\n<h2>Can I combine re-ranking with fine-tuned embedding models?</h2>\n<p>Yes, and this approach can yield even better results. In the experiments presented, combining the best fine-tuned embedding model with the best re-ranker improved performance from a baseline of 48% to 64% for top-5 retrieval, and from 60% to 71% for top-10 retrieval. This gives you flexibility to choose the approach that best fits your latency and performance requirements.</p>\n<h2>What about data augmentation and synthetic data generation?</h2>\n<p>Be cautious with synthetic data generation. While it might seem like a solution for limited data, LLMs can hallucinate up to 70% of the time when generating synthetic query-context pairs. Data augmentation works best when you already have a good dataset and want to prevent overfitting, not as a solution for poor-quality data.</p>\n<h2>How do I evaluate if re-ranking will help my specific use case?</h2>\n<p>Start with a small experiment using a lightweight model that you can train quickly (like MiniLM). If you see improvements with this simple approach, it's a strong signal that investing in more sophisticated re-ranking will yield even better results. This allows you to validate the approach before committing significant resources.</p>\n<h2>What's the future of retrieval improvement beyond re-ranking?</h2>\n<h2>Multimodal retrieval is likely the next frontier. While much work has been done on text retrieval, there's still significant room for improvement in retrieving and ranking content across different modalities like images, audio, and video. Building better benchmarks and baselines for multimodal RAG systems represents an important area for future development.</h2>\n",
      "date_modified": "2025-09-11T00:00:00+00:00",
      "date_published": "2025-09-11T00:00:00+00:00",
      "external_url": "https://jxnl.co/writing/2025/09/11/the-12-rag-performance-boost-youre-missing-ayush-lancedb/",
      "id": "urn:aggr:item:c9ac63c12e87bfcd522bec196df8e9ebf2e9b179",
      "summary": "Practical approaches to enhancing retrieval quality through fine-tuning, re-ranking, and understanding trade-offs in RAG systems",
      "tags": [
        "programming",
        "embedding models",
        "fine-tuning",
        "lancedb",
        "rag",
        "rag performance",
        "re-rankers",
        "speaker series"
      ],
      "title": "The 12% RAG Performance Boost You're Missing (Ayush, LanceDB)",
      "url": "https://aggr.aymericbeaumet.com/items/jxnl-co/2025-09-11-the-12-rag-performance-boost-you-re-missing-ayush-lancedb/"
    },
    {
      "content_html": "<h2>Text Chunking - Anton (ChromaDB)</h2>\n<p>I hosted a special session with Anton from ChromaDB to discuss their latest technical research on text chunking for RAG applications. This session covers the fundamentals of chunking strategies, evaluation methods, and practical tips for improving retrieval performance in your AI systems.</p>\n<h2>What is chunking and why is it important for RAG systems?</h2>\n<p>Chunking is the process of splitting documents into smaller components to enable effective retrieval of relevant information. Despite what many believe, chunking remains critical even as LLM context windows grow larger.</p>\n<p>The fundamental purpose of chunking is to find the relevant text for a given query among all the divisions we've created from our documents. This becomes especially important when the information needed to answer a query spans multiple documents.</p>\n<p>There are several compelling reasons why chunking matters regardless of context window size:</p>\n<ol>\n<li>Embedding model limitations - While LLM context windows are growing, embedding models typically have fixed input sizes and will silently truncate oversized inputs</li>\n<li>Inference efficiency - You're paying per token, so retrieving only relevant information reduces costs</li>\n<li>Information accuracy - Effective chunking eliminates distractors that could confuse the model</li>\n<li>Retrieval performance - Proper chunking significantly improves your system's ability to find all relevant information</li>\n</ol>\n<p><em><strong>Key Takeaway:</strong></em> Chunking will remain important regardless of how large context windows become because it addresses fundamental challenges in retrieval efficiency, accuracy, and cost management.</p>\n<h2>What approaches exist for text chunking?</h2>\n<p>There are two broad categories of chunking approaches that are currently being used:</p>\n<p>Heuristic approaches rely on separator characters (like newlines, question marks, periods) to divide documents based on their existing structure. The most widely used implementation is the recursive character text splitter, which uses a hierarchy of splitting characters to subdivide documents into pieces not exceeding a specified maximum length.</p>\n<p>These methods generally produce good results with clean documents but become brittle when dealing with unusual formatting or special characters. They require significant preprocessing and cleaning.</p>\n<p>Semantic approaches are more experimental but promising. These use embedding or language models to identify semantic boundaries in documents - points where the topic changes. This approach avoids the brittleness of heuristics by focusing on meaning rather than characters.</p>\n<p>What's particularly interesting is that you can use the same embedding model for both chunking and retrieval, potentially finding an embedding-optimal chunking strategy. Since embeddings are relatively cheap, this approach is becoming more viable.</p>\n<p><em><strong>Key Takeaway:</strong></em> While heuristic approaches like recursive character text splitters are most common today, semantic chunking methods that identify natural topic boundaries show promise for more robust performance across diverse document types.</p>\n<h2>Does chunking strategy actually matter for performance?</h2>\n<p>According to Anton's research, chunking strategy matters tremendously. Their technical report demonstrates significant performance variations based solely on chunking approach, even when using the same embedding model and retrieval system.</p>\n<p>They discovered two fundamental rules of thumb that exist in tension with each other:</p>\n<ol>\n<li>Fill the embedding model's context window as much as possible - Strategies that produce very short chunks tend to yield noisy retrieval results</li>\n<li>Don't group unrelated information together - Embedding models struggle to summarize chunks containing contradictory or differing information</li>\n</ol>\n<p>The most important insight, however, is that you must always examine your data. Many default chunking strategies produce chunks that are far too short because the delimiter characters are in the wrong order or are the wrong characters entirely.</p>\n<p>By looking at your actual chunks, you can develop intuition about how your chunking strategy is working for your specific use case. This is critical because there's likely no universal &quot;best&quot; chunking strategy - the optimal approach depends on your data and task.</p>\n<p><em><strong>Key Takeaway:</strong></em> There's no one-size-fits-all chunking strategy. The best approach depends on your specific data and task, which is why examining your actual chunks is essential for diagnosing retrieval problems.</p>\n<h2>How should we evaluate chunking strategies?</h2>\n<p>When evaluating chunking strategies, focus on the retriever itself rather than the generative output. This differs from traditional information retrieval benchmarks in several important ways:</p>\n<p>Recall is the single most important metric. Modern models are increasingly good at ignoring irrelevant information, but they cannot complete a task if you haven't retrieved all the relevant information in the first place.</p>\n<p>You should measure recall at the passage level rather than the document level. Traditional IR benchmarks focus on whole document retrieval, but in RAG applications, we care about retrieving specific relevant passages, not just any part of a relevant document.</p>\n<p>Ranking metrics like NDCG (which consider the order of retrieved documents) are less relevant for RAG applications. As long as the information is available somewhere in the context window, the model can usually extract what it needs regardless of position.</p>\n<p>The ChromaDB team has released code for their generative benchmark, which can help evaluate chunking strategies against your specific data.</p>\n<p><em><strong>Key Takeaway:</strong></em> Focus on passage-level recall rather than document-level metrics or ranking-sensitive measures. The model can handle irrelevant information, but it can't work with information that wasn't retrieved.</p>\n<h2>What practical advice can improve our chunking implementation?</h2>\n<p>The most emphatic advice from Anton was: &quot;Always, always, always look at your data.&quot; This point was stressed repeatedly throughout the presentation.</p>\n<p>Many retrieval problems stem from poor chunking that isn't apparent until you actually examine the chunks being produced. Default settings in popular libraries often produce surprisingly poor results for specific datasets.</p>\n<p>Retrieval is not a general system - it's dependent on your specific task and data. This means your evaluation needs to be based on the types of queries you expect to see in your application, not generic benchmarks.</p>\n<p>While better tooling is being developed to help with this process, in the meantime, the best approach is to:</p>\n<ol>\n<li>Generate chunks using your chosen strategy</li>\n<li>Manually examine those chunks to ensure they make semantic sense</li>\n<li>Test with queries representative of your actual use case</li>\n<li>Measure passage-level recall to evaluate performance</li>\n</ol>\n<p>This approach acknowledges that we're in an interesting era of software development where AI application builders are being forced to learn machine learning best practices that have evolved over decades.</p>\n<p><em><strong>Key Takeaway:</strong></em> No amount of sophisticated algorithms can compensate for not understanding your data. Examining your chunks and evaluating them against representative queries is the most reliable path to improving retrieval performance.</p>\n<p><strong>Final thoughts on chunking for RAG applications</strong> The fundamental tension in chunking is between maximizing the use of the embedding model's context window and avoiding the grouping of unrelated information. Finding the right balance requires understanding your specific data and use case.</p>\n<p>While semantic chunking approaches show promise, even the most basic heuristic methods can perform well if properly configured for your data. The defaults, however, are rarely optimal.</p>\n<p>As Anton emphasized, retrieval is not a general system but a task-specific one. Your evaluation should reflect the queries your application will actually encounter rather than generic benchmarks.</p>\n<p>The ChromaDB team is developing better tooling to help with this process, but in the meantime, the most reliable approach is to manually examine your chunks and measure passage-level recall against representative queries.</p>\n<h2>By focusing on these fundamentals rather than blindly applying frameworks or following defaults, you can significantly improve the performance of your RAG applications and deliver better results to your users.</h2>\n",
      "date_modified": "2025-09-11T00:00:00+00:00",
      "date_published": "2025-09-11T00:00:00+00:00",
      "external_url": "https://jxnl.co/writing/2025/09/11/text-chunking-strategies-for-rag-applications/",
      "id": "urn:aggr:item:6ec63b1444c3f52a329b421096569ed892e37505",
      "summary": "Technical session with Anton from ChromaDB on text chunking fundamentals, evaluation methods, and practical tips for improving retrieval performance",
      "tags": [
        "programming",
        "chromadb",
        "evaluation",
        "heuristic chunking",
        "rag",
        "retrieval performance",
        "semantic chunking",
        "speaker series",
        "text chunking"
      ],
      "title": "Text Chunking Strategies for RAG Applications",
      "url": "https://aggr.aymericbeaumet.com/items/jxnl-co/2025-09-11-text-chunking-strategies-for-rag-applications/"
    },
    {
      "content_html": "<p>I hosted a session with Kelly Hong from Chroma, who presented her research on generative benchmarking for retrieval systems. She explained how to create custom evaluation sets from your own data to better test embedding models and retrieval pipelines, addressing the limitations of standard benchmarks like MTEB.</p>\n<p><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://maven.com/p/30886a\">▶️ See Chroma's Testing Framework in Action</a></p>\n<p><strong>Why public benchmarks like MTEB don't reflect real-world performance</strong></p>\n<p>When evaluating embedding models for retrieval applications, many teams rely on public benchmarks like MTEB (Massive Text Embedding Benchmark). While convenient for comparing models, these benchmarks have significant limitations:</p>\n<p>The data is too generic and doesn't match specific domain needs. Even domain-specific MTEB datasets like legal documents aren't as specific as your actual production data.</p>\n<p>The query-document pairs are artificially clean - questions are perfectly formulated and documents are perfect matches. In real applications, user queries are ambiguous, incomplete, and often phrased as statements rather than questions.</p>\n<p>There's significant risk of data contamination since embedding models may have seen these public datasets during training, making it unclear if you're testing retrieval capability or just memorization.</p>\n<p>As Kelly put it: &quot;If you have really good performance on a public benchmark for a given embedding model, that doesn't necessarily guarantee that you'll also get that good performance for your specific production pipeline.&quot;</p>\n<p><em><strong>Key Takeaway:</strong></em> Good performance on public benchmarks doesn't translate to good performance in your specific application. You need to evaluate embedding models on your own data to get meaningful results.</p>\n<p><strong>The generative benchmarking approach</strong></p>\n<p>Generative benchmarking creates custom evaluation sets directly from your data through a two-step process:</p>\n<ol>\n<li>\n<p>Chunk filtering: Using an LLM judge to identify document chunks that users would realistically query about</p>\n</li>\n<li>\n<p>Query generation: Creating realistic queries from those filtered chunks</p>\n</li>\n</ol>\n<p>This approach addresses the limitations of public benchmarks because:</p>\n<ul>\n<li>It's specific to your data</li>\n<li>It generates more realistic queries that reflect actual user behavior</li>\n<li>It tests retrieval on private data that embedding models haven't seen during training</li>\n</ul>\n<p>Kelly shared a case study with Weights &amp; Biases where they started with 13,000 document chunks and about 2,000 real user queries from production. After filtering, they generated queries that matched the style and content of real user questions.</p>\n<p>The results were revealing - the embedding model Weights &amp; Biases had been using (text-embedding-3-small) actually performed worse than alternatives they tested. Even more interesting, the rankings of embedding models on their custom benchmark contradicted the MTEB rankings.</p>\n<p><strong>Creating realistic queries that match user behavior</strong></p>\n<p>One of the most important aspects of generative benchmarking is generating queries that actually reflect how users search. Kelly demonstrated that naively generated queries (without context or examples) tend to be perfectly formed questions that make retrieval too easy.</p>\n<p>For example, a naive query generation might produce &quot;What is the purpose of artifact versioning in Weights &amp; Biases?&quot; But real users are more likely to search with something like &quot;artifact versioning not working.&quot;</p>\n<p>To generate realistic queries, Kelly recommended providing:</p>\n<ol>\n<li>\n<p>Context about your application (e.g., &quot;This is a technical support bot for Weights &amp; Biases&quot;)</p>\n</li>\n<li>\n<p>Example queries from real users (5-7 examples is usually sufficient)</p>\n</li>\n</ol>\n<p>When I work with companies building vertical AI applications, I approach this by:</p>\n<ul>\n<li>Interviewing subject matter experts about what they actually search for</li>\n<li>Having LLMs generate candidate queries through role-playing exercises</li>\n<li>Getting experts to review and validate the generated queries</li>\n</ul>\n<p>I've recently been experimenting with using OpenAI's Deep Research for this - I'll have it research a domain like &quot;search systems for lawyers dealing with employment claims&quot; and use that research as part of my prompt. This approach has been surprisingly effective for generating realistic queries.</p>\n<p><strong>The importance of human involvement in the evaluation process</strong></p>\n<p>Kelly emphasized that generative benchmarking isn't a fully automated &quot;press one button and everything's done&quot; approach. Human involvement remains critical:</p>\n<p>&quot;Human involvement is very critical. If you want really good evals, I think it applies to basically any case where you're working with AI as well. I think it's very rare that your system is going to work well with absolutely no human in the loop.&quot;</p>\n<p>For the chunk filtering step, the Chroma team:</p>\n<ol>\n<li>\n<p>Started with a small set of about 300 document chunks</p>\n</li>\n<li>\n<p>Manually labeled them as relevant or irrelevant</p>\n</li>\n<li>\n<p>Iterated on their LLM judge criteria 4-5 times to align with human judgment</p>\n</li>\n<li>\n<p>Only then scaled to the full document corpus</p>\n</li>\n</ol>\n<p>I've seen too many teams try to fully automate their evaluation process without ever looking at the actual data. I often tell clients, &quot;You can't delegate all the thinking to these LLMs.&quot; The reluctance to actually look at examples is a common problem.</p>\n<p>I find that many teams lean on LLM evals which give you this feeling of control because you're always twiddling with the generation prompt. But if your recall is very low - if you just can't find the document because of some phrasing in the text - then everything downstream is really bad. People feel empowered to adjust prompts but don't focus enough on making search better.</p>\n<p><strong>Practical applications of generative benchmarking</strong></p>\n<p>Kelly outlined several practical applications of generative benchmarking beyond just selecting the best embedding model:</p>\n<ul>\n<li>Identifying irrelevant content in your document corpus</li>\n<li>Iterating on specific components of your retrieval pipeline (reranking, chunk rewriting)</li>\n<li>Aligning your evaluation set with production query distributions</li>\n<li>Identifying knowledge gaps in your document corpus</li>\n</ul>\n<p>&quot;I think it becomes even more powerful once you actually have production traffic to work with,&quot; Kelly noted. &quot;You can cluster your production queries by topic, and you can see the distribution of topics that your users are asking about, and then you can use that distribution and align your eval set to that.&quot;</p>\n<p><strong>Balancing cost and performance in retrieval optimizations</strong></p>\n<p>When discussing contextual chunk rewriting (where you provide document context for each chunk), Kelly acknowledged its benefits but cautioned about the costs:</p>\n<p>&quot;Contextual rewriting is useful, but also very expensive. So I would be more intentional about how you use it.&quot;</p>\n<p>She suggested being selective about which chunks need context rather than rewriting everything, potentially using the chunk filtering process to identify which documents would benefit most from context.</p>\n<p>I emphasized the value of having quantifiable metrics for these decisions: &quot;This is the benefit of having an actual eval harness. The real conversation isn't 'should we use retrieval, or should we use contextual retrieval,' it is 'if we use contextual retrieval, our metrics go up 2%. Is that worth it? Probably not. But if you did contextual retrieval, and it went up 20%, then the question is, okay, well, is spending 20 cents per document worth the 20% improvement in retrieval?' That is a quantified decision that you can make.&quot;</p>\n<p>I've seen similar patterns with hybrid search. In one case, I worked with companies processing transcripts where lexical search performed nearly as well as semantic search but was 10 times faster. I told them, &quot;If I want to find meeting notes, I want that to be as fast as possible.&quot;</p>\n<p><strong>Setting up efficient evaluation infrastructure</strong></p>\n<p>When asked about hyperparameter tuning and experimentation time, Kelly shared that their process was relatively quick because they worked with small data samples first:</p>\n<p>&quot;We did around 4 or 5 iterations, but we did it across a small subset of data. So we were only working with about 300 labeled document chunks. So it was very quick iteration. We weren't doing it over our entire document corpus. So I think that probably took less than a day for me to do.&quot;</p>\n<p>I emphasized that how much time you spend on evaluation really depends on what kind of existing infrastructure you have: &quot;I've joined companies where their embedding model is like sequential call to OpenAI. And I was like, 'well, this is why it's taking you 6 days to do these experiments.'&quot;</p>\n<p>I recommend time-boxing experimentation: &quot;I mostly try to time box how much research we're going to do, and then figure out how many experiments we can run. And usually what's going to happen is as you iterate on your prompts and on your models, the marginal improvement in performance will slowly decrease.&quot;</p>\n<p>The goal isn't perfection - it's making the most of your experimental time. I personally use Modal for this - I can reembed all of Wikipedia with any arbitrary model in about 10 minutes, which makes hyperparameter sweeping much more feasible.</p>\n<p><strong>Handling metadata in retrieval evaluations</strong></p>\n<p>When asked about incorporating metadata into the evaluation process, Kelly suggested several approaches:</p>\n<p>&quot;I think maybe metadata could be useful when you're performing the retrieval task... if you have metadata like, let's say you have a query and retrieve a category of documents that have a certain metadata tag, maybe that can count as a partial retrieval success.&quot;</p>\n<p>She also noted that metadata can be valuable for pre-filtering documents before applying an LLM judge: &quot;If all this data was just scraped from the website, maybe there was a metadata tag that says 'this is from our news section.' We'd want to filter those out, because that's not really relevant to our users.&quot;</p>\n<p>I added that metadata filters can be evaluated separately from retrieval quality: &quot;If you have really simple ones like 'show me all the news articles from last week,' we can individually test our ability to select date ranges.&quot; This allows you to verify both retrieval accuracy and structured extraction tasks independently.</p>\n<p><strong>Final thoughts</strong></p>\n<p>Kelly concluded by emphasizing that generative benchmarking requires human involvement to be effective: &quot;When we talk about generative benchmarking, that also doesn't mean that this entire process is 100% automated. You do need some human in the loop if you really want a good eval.&quot;</p>\n<p>The main takeaway from the session was that good performance on public benchmarks doesn't guarantee good performance for your specific use case.</p>\n<p>As Kelly put it: &quot;I think the main takeaway from this entire talk is that if you have good performance on public benchmarks like MTEP, that doesn't necessarily guarantee that same performance for your specific use case, and generative benchmarking is a pretty good solution to that.&quot;</p>\n<p>For those interested in trying generative benchmarking, Kelly mentioned that Chroma has made their tools available, and I noted they're offering $1,000 in API credits for participants in the cohort.</p>\n<hr />\n<p>FAQs</p>\n<h2>What is generative benchmarking?</h2>\n<p>Generative benchmarking is a method to create custom evaluation sets from your own data to test AI retrieval systems. It involves generating realistic queries from your document corpus and using these query-document pairs to evaluate how well different embedding models and retrieval systems perform with your specific data. Unlike public benchmarks, this approach gives you insights directly relevant to your use case.</p>\n<h2>Why are custom benchmarks better than public benchmarks like MTEB?</h2>\n<p>Custom benchmarks address several limitations of public benchmarks like MTEB (Massive Text Embedding Benchmark). While MTEB is widely used for comparing embedding models, it uses generic data that may not reflect your specific domain, contains artificially clean query-document pairs, and may have been seen by models during training. Good performance on MTEB doesn't guarantee good performance on your specific data and use case.</p>\n<h2>How does the generative benchmarking process work?</h2>\n<p>The process involves two main steps. First, chunk filtering identifies document chunks that users would realistically query about, filtering out irrelevant content. Second, query generation creates realistic user queries from these filtered chunks. The resulting query-document pairs form your evaluation set, which you can use to test different embedding models and retrieval components.</p>\n<h2>What's involved in the chunk filtering step?</h2>\n<p>Chunk filtering uses an aligned LLM judge to identify document chunks that contain information users would actually query. This involves creating criteria for relevance, providing a small set of human-labeled examples, and iterating on the LLM judge to improve alignment with human judgment. This step helps filter out irrelevant content like news articles or marketing material that wouldn't be useful in a support context.</p>\n<h2>How do you generate realistic queries?</h2>\n<p>Query generation uses an LLM with specific context about your application and example queries. Providing this context helps the LLM focus on topics users would ask about, while example queries guide the style of generated queries. This approach creates more realistic, sometimes ambiguous queries that better reflect how users actually search, rather than perfectly formed questions that match document content exactly.</p>\n<h2>How do you evaluate retrieval performance with the generated benchmark?</h2>\n<p>Once you have your evaluation set with query-document pairs, you can test different embedding models by embedding each document chunk, storing them in a vector database, and then embedding each query to retrieve the top K document chunks. If the matching document is in the top K results, that counts as a success. This gives you metrics like recall@K and NDCG that you can compare across different models and configurations.</p>\n<h2>What insights can generative benchmarking provide?</h2>\n<p>Generative benchmarking can help you select the best embedding model for your specific data, identify irrelevant content in your document corpus, and evaluate changes to your retrieval pipeline like adding re-ranking or chunk rewriting. It can also reveal when public benchmark rankings don't align with performance on your data, as demonstrated in a case study where model rankings differed from MTEB rankings.</p>\n<h2>Do I need production data to use generative benchmarking?</h2>\n<p>No, you can use generative benchmarking even if you don't have production data yet. All you need is a document corpus to generate an evaluation set. However, if you do have production queries, you can use them to further align your generated queries to real user behavior, identify knowledge gaps in your document corpus, and make your evaluation set even more representative.</p>\n<h2>Is generative benchmarking fully automated?</h2>\n<p>No, generative benchmarking isn't 100% automated. It requires human involvement to get good results. You'll need to align your LLM judge, provide context and example queries to steer query generation, and manually review data throughout the process. The human-in-the-loop aspect is critical for creating evaluation sets that truly reflect your use case.</p>\n<h2>How can I try generative benchmarking on my own data?</h2>\n<p>You can try generative benchmarking on your own data by using Chroma's open-source tools. The full technical report is available at research.trychroma.com, and you can run the process with just a few lines of code. Chroma Cloud is also available if you want to use their hosted vector database solution.</p>\n<h2>How does contextual chunk rewriting fit into retrieval evaluation?</h2>\n<p>Contextual chunk rewriting involves adding context to document chunks to improve retrieval. While it can be effective, especially for content like tables or technical information that lacks context, it's also expensive since it requires running an LLM on every chunk. A more efficient approach might be to only rewrite chunks that need additional context, which you can identify during the filtering process. The value of this approach can be quantified through your evaluation metrics.</p>\n",
      "date_modified": "2025-09-11T00:00:00+00:00",
      "date_published": "2025-09-11T00:00:00+00:00",
      "external_url": "https://jxnl.co/writing/2025/09/11/stop-trusting-mteb-rankings-kelly-hong-chroma/",
      "id": "urn:aggr:item:d5cdad78bfceb688f282c996368a95846c1ec80e",
      "summary": "A deep dive into generative benchmarking - creating custom evaluation sets from your own data to better assess embedding model performance.",
      "tags": [
        "programming",
        "benchmarking",
        "chroma",
        "embeddings",
        "evaluation",
        "generative evals",
        "rag",
        "speaker series"
      ],
      "title": "Stop Trusting MTEB Rankings (Kelly Hong, Chroma)",
      "url": "https://aggr.aymericbeaumet.com/items/jxnl-co/2025-09-11-stop-trusting-mteb-rankings-kelly-hong-chroma/"
    },
    {
      "content_html": "<h2>What Is the RAG Master Series?</h2>\n<p>Retrieval-Augmented Generation (RAG) has become the foundation of modern AI applications that need to access and reason about external knowledge. This comprehensive series distills years of consulting experience helping companies build, improve, and scale RAG systems in production.</p>\n<p><strong>RAG systems are fundamentally different from other AI applications</strong> - they combine the complexity of information retrieval with the unpredictability of language generation. This series provides a systematic approach to mastering both aspects, from basic implementations to enterprise-grade systems serving millions of users.</p>\n<p>This guide covers everything from fundamental concepts to advanced optimization techniques, anti-patterns to avoid, and real-world case studies from successful deployments across industries.</p>\n<p>If you want hands-on help, I recommend reaching out to my friend Nila: <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://nila.is/\">nila.is</a>. Please mention you came from me.</p>\n<h2>What Makes RAG Systems Unique?</h2>\n<p>Unlike traditional AI applications, RAG systems face unique challenges:</p>\n<ul>\n<li><strong>Compound complexity</strong>: Problems can emerge in retrieval, ranking, generation, or their interactions</li>\n<li><strong>Data quality dependence</strong>: Your system is only as good as your knowledge base and retrieval pipeline</li>\n<li><strong>Dynamic failure modes</strong>: Issues often emerge only in production with real user queries</li>\n<li><strong>Multi-dimensional optimization</strong>: Balancing accuracy, latency, cost, and user experience</li>\n</ul>\n<p>This series addresses each challenge with proven methodologies developed through hundreds of implementations across industries from healthcare to finance to developer tools.</p>\n<h2>What Key Terms Should I Know?</h2>\n<ul>\n<li><strong>Retrieval-Augmented Generation (RAG)</strong>: Enhancing LLMs with external knowledge sources through retrieval</li>\n<li><strong>Vector Search/Semantic Search</strong>: Finding relevant content using embedding similarity</li>\n<li><strong>Full-text Search</strong>: Traditional keyword-based search (like BM25)</li>\n<li><strong>Hybrid Search</strong>: Combining vector and full-text search for better coverage</li>\n<li><strong>Re-ranking</strong>: Post-processing search results to improve relevance</li>\n<li><strong>Query Understanding</strong>: Extracting intent, entities, and metadata from user queries</li>\n<li><strong>Chunking</strong>: Breaking documents into retrievable pieces while preserving meaning</li>\n<li><strong>Citation/Attribution</strong>: Linking generated responses back to source documents</li>\n<li><strong>Precision/Recall</strong>: Key metrics for evaluating retrieval quality</li>\n<li><strong>Synthetic Data</strong>: Generated question-answer pairs for testing and evaluation</li>\n<li><strong>Data Flywheel</strong>: Self-reinforcing system that improves through user feedback</li>\n</ul>\n<h2>What Posts Are in This Series?</h2>\n<h3>Foundation Posts</h3>\n<h4>1. <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://jxnl.co/writing/2024/11/07/what-is-retrieval-augmented-generation/\">What is Retrieval Augmented Generation?</a></h4>\n<p><strong>Core thesis:</strong> RAG enhances LLMs by providing access to external knowledge, addressing hallucinations and factual grounding issues inherent in language models.</p>\n<p><strong>Key insight:</strong> RAG systems consist of interconnected components (knowledge base, retrieval, generation, re-ranking) that must be optimized together, not in isolation.</p>\n<p><strong>Practical takeaway:</strong> Start with understanding your use case requirements before diving into technical implementation—different applications need different RAG architectures.</p>\n<h4>2. <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://jxnl.co/writing/2024/02/28/levels-of-complexity-rag-applications/\">Levels of RAG Complexity</a></h4>\n<p><strong>Core thesis:</strong> RAG systems exist at different complexity levels, from basic chunk-and-search implementations to sophisticated multi-stage pipelines with observability and evaluation.</p>\n<p><strong>Key insight:</strong> Each complexity level introduces new capabilities but also new failure modes. Understanding these levels helps you choose the right architecture for your needs.</p>\n<p><strong>Practical takeaway:</strong> Start simple and add complexity systematically based on actual user needs rather than theoretical requirements.</p>\n<h3>Implementation and Improvement Posts</h3>\n<h4>3. <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://jxnl.co/writing/2024/05/22/systematically-improving-your-rag/\">Systematically Improving Your RAG</a></h4>\n<p><strong>Core thesis:</strong> RAG improvement requires a systematic approach focused on synthetic data, metadata utilization, hybrid search, user feedback, and continuous monitoring.</p>\n<p><strong>Key insight:</strong> Most teams spend too much time on generation quality before ensuring retrieval works correctly. Start with retrieval metrics using synthetic data.</p>\n<p><strong>Practical takeaway:</strong> Build evaluation pipelines early and use them to guide all optimization decisions. Data-driven iteration beats intuition-based improvements.</p>\n<h4>4. <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://jxnl.co/writing/2024/08/19/rag-flywheel/\">The RAG Playbook: Building a Data Flywheel</a></h4>\n<p><strong>Core thesis:</strong> Successful RAG systems create self-reinforcing improvement cycles through synthetic data generation, fast evaluations, real-world data collection, and systematic analysis.</p>\n<p><strong>Key insight:</strong> Focus on leading metrics (like number of experiments per week) rather than lagging metrics (overall system quality) to drive meaningful progress.</p>\n<p><strong>Practical takeaway:</strong> Establish baseline metrics with synthetic data before deploying to users, then build continuous improvement loops based on user interactions.</p>\n<h4>5. <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://jxnl.co/writing/2024/05/11/low-hanging-fruit-for-rag-search/\">Low-Hanging Fruit for RAG Search</a></h4>\n<p><strong>Core thesis:</strong> Most RAG systems can achieve significant improvements through simple, cost-effective optimizations that address common oversights.</p>\n<p><strong>Key insight:</strong> Basic improvements like date filters, metadata inclusion, and hybrid search often outperform complex architectural changes.</p>\n<p><strong>Practical takeaway:</strong> Audit your system for these seven common gaps before investing in sophisticated optimizations: synthetic baselines, date handling, feedback mechanisms, search metrics, full-text search, query-document alignment, and metadata utilization.</p>\n<h4>6. <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://jxnl.co/writing/2024/11/04/how-to-improve-rag-applications-6-proven-strategies/\">Six Proven Strategies for Improving RAG</a></h4>\n<p><strong>Core thesis:</strong> Systematic RAG improvement requires a portfolio of complementary strategies: data flywheels, query segmentation, specialized indices, routing, feedback collection, and response optimization.</p>\n<p><strong>Key insight:</strong> Successful RAG systems aren't built through one-time optimizations but through continuous, data-driven improvement processes.</p>\n<p><strong>Practical takeaway:</strong> Implement all six strategies as an integrated system—each reinforces the others to create compound improvements over time.</p>\n<h3>Production and Monitoring Posts</h3>\n<h4>7. <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://jxnl.co/writing/2025/05/29/systematically-improving-rag-with-raindrop-and-oleve/\">Systematically Improving RAG with Monitoring</a></h4>\n<p><strong>Core thesis:</strong> Production RAG systems require sophisticated monitoring that goes beyond traditional error tracking to identify subtle degradation patterns in AI outputs.</p>\n<p><strong>Key insight:</strong> Traditional monitoring tools don't work for AI systems because there's no explicit error when the system produces inadequate responses—monitoring must detect patterns in user behavior and system outputs.</p>\n<p><strong>Practical takeaway:</strong> Implement both implicit signals (user frustration patterns) and explicit signals (thumbs up/down) to create comprehensive monitoring dashboards that surface issues before they impact business metrics.</p>\n<h3>Anti-Patterns and Troubleshooting</h3>\n<h4>8. <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://jxnl.co/writing/2025/06/11/rag-anti-patterns-with-skylar-payne/\">RAG Anti-Patterns with Industry Expert Insights</a></h4>\n<p><strong>Core thesis:</strong> Most RAG failures stem from predictable anti-patterns across data collection, extraction, indexing, retrieval, re-ranking, and generation phases.</p>\n<p><strong>Key insight:</strong> Teams consistently make the same mistakes regardless of industry—silent data failures, naive embedding usage, poor evaluation practices, and complexity without measurement.</p>\n<p><strong>Practical takeaway:</strong> Audit your system against these common anti-patterns before building new features. Prevention through systematic data inspection beats debugging production issues.</p>\n<h3>Evaluation and Assessment</h3>\n<h4>9. <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://jxnl.co/writing/2025/05/19/there-are-only-6-rag-evals/\">The Only 6 RAG Evaluations You Need</a></h4>\n<p><strong>Core thesis:</strong> Most RAG evaluation frameworks are overcomplicated. Six core evaluations cover the essential aspects: retrieval quality, generation accuracy, relevance assessment, citation validation, latency measurement, and user satisfaction tracking.</p>\n<p><strong>Key insight:</strong> Evaluation systems should be fast enough to run continuously and comprehensive enough to catch regressions before they reach users.</p>\n<p><strong>Practical takeaway:</strong> Implement these six evaluations as automated tests that run with every change to your system, treating them like unit tests for your RAG pipeline.</p>\n<h3>Specialized Applications and Advanced Topics</h3>\n<h4>10. <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://jxnl.co/writing/2024/06/05/predictions-for-the-future-of-rag/\">RAG++: The Future Beyond Question Answering</a></h4>\n<p><strong>Core thesis:</strong> The future of RAG lies in report generation and structured analysis rather than simple question answering, enabling new business models and use cases.</p>\n<p><strong>Key insight:</strong> Reports provide more business value than isolated answers because they support decision-making, resource allocation, and standardized processes.</p>\n<p><strong>Practical takeaway:</strong> Design your RAG system architecture to support structured outputs and multi-step reasoning, not just single-turn question answering.</p>\n<h3>Enterprise and Process Posts</h3>\n<h4>11. <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://jxnl.co/writing/2025/06/09/how-to-invest-in-ai-w-mcps-and-data-analytics/\">RAG Enterprise Implementation Process</a></h4>\n<p><strong>Core thesis:</strong> Enterprise RAG deployments require different approaches than startup implementations, with emphasis on security, compliance, integration with existing systems, and change management.</p>\n<p><strong>Key insight:</strong> Technical excellence isn't sufficient for enterprise success—process, governance, and stakeholder alignment often determine project outcomes.</p>\n<p><strong>Practical takeaway:</strong> Plan for enterprise-specific requirements (security, compliance, integration) from the beginning rather than retrofitting them later.</p>\n<h2>How Should I Use This Series?</h2>\n<p><strong>If you're new to RAG:</strong></p>\n<ol>\n<li>Start with <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://jxnl.co/writing/2024/11/07/what-is-retrieval-augmented-generation/\">What is RAG?</a> for foundations</li>\n<li>Read <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://jxnl.co/writing/2024/02/28/levels-of-complexity-rag-applications/\">Levels of Complexity</a> to understand architectural options</li>\n<li>Follow the <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://jxnl.co/writing/2024/05/22/systematically-improving-your-rag/\">Systematic Improvement Guide</a> for implementation</li>\n<li>Check <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://jxnl.co/writing/2024/05/11/low-hanging-fruit-for-rag-search/\">Low-Hanging Fruit</a> for quick wins</li>\n</ol>\n<p><strong>If you're optimizing existing systems:</strong></p>\n<ol>\n<li>Review <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://jxnl.co/writing/2025/06/11/rag-anti-patterns-with-skylar-payne/\">Anti-Patterns</a> to identify issues</li>\n<li>Implement the <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://jxnl.co/writing/2024/11/04/how-to-improve-rag-applications-6-proven-strategies/\">Six Strategies</a> systematically</li>\n<li>Build monitoring with <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://jxnl.co/writing/2025/05/29/systematically-improving-rag-with-raindrop-and-oleve/\">Production Insights</a></li>\n<li>Create improvement loops using the <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://jxnl.co/writing/2024/08/19/rag-flywheel/\">RAG Flywheel</a></li>\n</ol>\n<p><strong>If you're planning enterprise deployment:</strong></p>\n<ol>\n<li>Study <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://jxnl.co/writing/2025/06/09/how-to-invest-in-ai-w-mcps-and-data-analytics/\">Enterprise Process</a> for organizational considerations</li>\n<li>Plan evaluation using <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://jxnl.co/writing/2025/05/19/there-are-only-6-rag-evals/\">The 6 Core Evaluations</a></li>\n<li>Consider advanced applications in <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://jxnl.co/writing/2024/06/05/predictions-for-the-future-of-rag/\">RAG++</a></li>\n<li>Scale with <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://jxnl.co/writing/2024/08/19/rag-flywheel/\">Data Flywheel</a> principles</li>\n</ol>\n<h2>How Does This Connect to Other Series?</h2>\n<p>This RAG series integrates closely with complementary content:</p>\n<p><strong><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://jxnl.co/writing/2025/08/28/context-engineering-index/\">Context Engineering Series</a></strong>: While RAG focuses on retrieval and generation, Context Engineering explores how to structure tool responses and information flow for agentic systems. Many Context Engineering patterns (like faceted search and structured tool outputs) directly enhance RAG implementations.</p>\n<p><strong><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://jxnl.co/writing/2025/09/11/coding-series-index/\">Coding Agents Speaker Series</a></strong>: Coding agents represent the most successful real-world application of RAG principles. The insights from companies like Cline, Devin, and Cursor show how RAG concepts apply to production systems with millions of users.</p>\n<p><strong><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://jxnl.co/writing/2024/10/15/effective-communication-in-ai-engineering-moving-beyond-vague-updates/\">AI Engineering Communication</a></strong>: RAG systems are inherently probabilistic, making effective communication about performance and improvements critical for team success.</p>\n<h2>Who Is This Series For?</h2>\n<ul>\n<li><strong>Engineering teams</strong> building or improving RAG applications</li>\n<li><strong>Product leaders</strong> evaluating RAG solutions and measuring ROI</li>\n<li><strong>AI researchers</strong> interested in practical deployment insights</li>\n<li><strong>Consultants and agencies</strong> helping clients implement RAG systems</li>\n<li><strong>Startup founders</strong> choosing between different RAG approaches</li>\n</ul>\n<p>Each post includes practical examples, implementation code, and real business metrics from companies that have successfully deployed these systems.</p>\n<h2>What's Next for RAG?</h2>\n<p>Based on trends across hundreds of implementations, RAG is evolving toward:</p>\n<p><strong>More sophisticated evaluation</strong>: Moving beyond simple accuracy metrics to business outcome measurement</p>\n<p><strong>Better integration patterns</strong>: RAG as infrastructure rather than standalone applications</p>\n<p><strong>Domain specialization</strong>: Industry-specific RAG architectures optimized for particular use cases</p>\n<p><strong>Agent integration</strong>: RAG powering more sophisticated agentic workflows rather than simple Q&amp;A</p>\n<p><strong>Compliance and governance</strong>: Enterprise-grade RAG with audit trails, explainability, and compliance features</p>\n<p>This series captures both current best practices and emerging patterns to help you build RAG systems that will remain effective as the field evolves.</p>\n<h2>How Do I Get Started?</h2>\n<p>Begin with <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://jxnl.co/writing/2024/11/07/what-is-retrieval-augmented-generation/\">What is RAG?</a> to understand the fundamentals, then choose your path through the series based on your current needs and experience level.</p>\n<p>The most successful RAG implementations aren't built through perfect initial architecture but through systematic improvement based on real user data and business metrics. This series gives you the frameworks to build that improvement process into your system from day one.</p>\n<p>Remember: the goal isn't to build the most sophisticated RAG system possible, but to build one that reliably solves real user problems while continuously improving based on feedback and data.</p>\n<h2>Want to Learn More?</h2>\n<p>I know this is a lot of material—RAG systems touch everything from data engineering to user psychology. If you want to work through this systematically, here are a couple good places to start:</p>\n<p><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://dub.link/6wk-rag-email\">Free 6-Week RAG Email Course</a></p>\n",
      "date_modified": "2025-09-11T00:00:00+00:00",
      "date_published": "2025-09-11T00:00:00+00:00",
      "external_url": "https://jxnl.co/writing/2025/09/11/rag-series-index/",
      "id": "urn:aggr:item:a321c57d398d414dbe7d78780012b0fc99391fc1",
      "summary": "Comprehensive guide to building, improving, and scaling RAG systems. From fundamentals to advanced enterprise implementations with real-world examples and proven strategies.",
      "tags": [
        "programming",
        "ai",
        "machine learning",
        "rag",
        "retrieval-augmented generation",
        "series"
      ],
      "title": "RAG Master Series: Complete Guide to Retrieval-Augmented Generation",
      "url": "https://aggr.aymericbeaumet.com/items/jxnl-co/2025-09-11-rag-master-series-complete-guide-to-retrieval-augmented/"
    },
    {
      "content_html": "<h2>Lexical Search - John Berryman</h2>\n<p>I hosted a session featuring John Berryman, who shared his expertise on lexical search and its application in RAG systems. John, who previously worked at GitHub and co-authored books on prompt engineering and information retrieval, provided valuable insights on how traditional search techniques can complement modern vector-based approaches for more effective retrieval augmented generation.</p>\n<h2>Why is lexical search still relevant in the age of semantic search?</h2>\n<p>While semantic search has become the default approach for RAG systems, John highlighted several limitations that make it problematic in certain scenarios. The primary issues include:</p>\n<ul>\n<li>Semantic search struggles with exact matches for product IDs, people's names, and specific phrases</li>\n<li>It performs poorly with niche jargon not present in the embedding training data</li>\n<li>Relevance is opaque, making debugging difficult when results aren't as expected</li>\n<li>Filtering is particularly clunky in vector-based systems</li>\n</ul>\n<p>The filtering challenge is especially significant. With semantic search, you're forced into suboptimal approaches: either search first and then filter (risking empty results if filters are strict), or filter first and then semantically re-rank (which can be computationally expensive for large datasets).</p>\n<p><em><strong>Key Takeaway:</strong></em> Semantic search excels at understanding meaning but struggles with exact matching, specialized terminology, and efficient filtering - all areas where lexical search has established strengths.</p>\n<h2>How does lexical search actually work?</h2>\n<p>John provided a concise explanation of lexical search fundamentals, breaking it down into three components:</p>\n<ol>\n<li>Indexing: Documents are processed through analysis pipelines that include character filtering (like lowercasing), tokenization (splitting text into words), stopword removal (filtering common words), and stemming (normalizing word forms). This creates an inverted index where each token points to documents containing it.</li>\n<li>Searching: The inverted index makes retrieval extremely efficient. For a search like &quot;brown AND fox,&quot; the system simply retrieves the document IDs for each term and finds their intersection. This approach handles Boolean logic naturally and can incorporate relevance scoring through methods like TF-IDF (term frequency-inverse document frequency).</li>\n<li>Results: Beyond just returning matching documents, lexical search can provide rich metadata like aggregations (facet counts showing distribution of values), snippets (relevant text excerpts), and highlights (showing where terms matched).</li>\n</ol>\n<p>The key advantage is that lexical search can process filtering and relevance scoring simultaneously, unlike the sequential approach required in semantic search.</p>\n<p><em><strong>Key Takeaway:</strong></em> Lexical search's inverted index structure enables efficient simultaneous filtering and relevance scoring, with established relevance algorithms like TF-IDF and BM25 that have been refined over decades.</p>\n<h2>How can lexical search be applied in RAG applications?</h2>\n<p>John demonstrated a practical application using the Wayfair Annotation Dataset (WANDs), containing about 43,000 e-commerce products. He showed how to:</p>\n<ol>\n<li>Structure the index with appropriate field types (product name, description, class, rating count)</li>\n<li>Build queries that combine must-match conditions with should-match boosting factors</li>\n<li>Apply filters for availability, product class, and minimum ratings</li>\n<li>Return not just results but also aggregations that provide insight into the matching set</li>\n</ol>\n<p>When integrated into a RAG application, this approach allows the LLM to:</p>\n<ul>\n<li>Search with pre-applied filters (like showing only products available in the user's state)</li>\n<li>Explore results through multiple searches to narrow options</li>\n<li>Leverage aggregation data to understand the distribution of matching products</li>\n<li>Make more informed recommendations based on both the results and metadata</li>\n</ul>\n<p>In the demonstration, a user complaint about back pain led the assistant to search for both ergonomic chairs and standing desks, then refine based on the facet data to focus on adjustable standing desks specifically.</p>\n<p><em><strong>Key Takeaway:</strong></em> Lexical search provides RAG systems with powerful filtering capabilities and metadata that allows LLMs to make more informed decisions about how to refine searches and present options to users.</p>\n<h2>What are the limitations of lexical search?</h2>\n<p>Despite its strengths, John acknowledged several significant limitations:</p>\n<ul>\n<li>Lexical search struggles with word order and context (e.g., confusing &quot;dress shoe&quot; with a shoe that is a dress)</li>\n<li>It uses a bag-of-words approach that loses semantic meaning</li>\n<li>It can't recognize synonyms or different words with the same meaning</li>\n<li>It doesn't understand negation (searching for &quot;not something&quot; still matches &quot;something&quot;)</li>\n<li>It misses contextual clues that embedding models naturally capture</li>\n</ul>\n<p>These are precisely the areas where semantic search excels, suggesting that neither approach is sufficient on its own.</p>\n<p><em><strong>Key Takeaway:</strong></em> Lexical search's limitations around understanding meaning, context, and synonyms are the exact strengths of semantic search, pointing toward hybrid approaches as the optimal solution.</p>\n<h2>What hybrid search approaches show promise?</h2>\n<p>John explored several approaches to combining the strengths of lexical and semantic search:</p>\n<ol>\n<li>Lexical search plus re-ranking: Use lexical search to filter and provide an initial ranking, then apply semantic re-ranking to the top results. This is established but complex.</li>\n<li>SPLADE (Sparse Lexical and Dense Expansion): Use language models to identify synthetic synonyms that should have been in the text but weren't, then add these to the lexical index. This expands recall but still has bag-of-words limitations.</li>\n<li>Acorn: A promising approach that traverses vector structures while simultaneously checking filter matches.</li>\n<li>Superlinked: Uses different embeddings for different data types, concatenating them into unified vectors that can be searched with nearest neighbor techniques while maintaining filtering capabilities.</li>\n</ol>\n<p>John admitted that the ideal hybrid solution remains elusive, but the industry is actively working on approaches that combine the filtering power of lexical search with the semantic understanding of vector-based methods.</p>\n<p><em><strong>Key Takeaway:</strong></em> The future likely belongs to hybrid approaches that combine lexical search's filtering capabilities with semantic search's understanding of meaning, though the ideal implementation is still evolving.</p>\n<h2>How do you optimize lexical search for specific domains?</h2>\n<p>During the Q&amp;A, John shared insights on improving search for domain-specific applications:</p>\n<ol>\n<li>Start by collecting data on what should match versus what actually matches</li>\n<li>For e-commerce, analyze click logs to understand user behavior patterns</li>\n<li>Focus on high-impact queries - often the top 10 queries account for 50% of traffic</li>\n<li>Use human curators familiar with the domain to create judgment lists</li>\n<li>Consider learn-to-rank algorithms that can automatically tune parameters based on this data</li>\n</ol>\n<p>For hybrid approaches, he suggested getting the top 100-1000 results from lexical search, then using a feature store to retrieve additional information (including embeddings) for re-ranking with models like LambdaMart or XGBoost.</p>\n<p><em><strong>Key Takeaway:</strong></em> Optimizing search requires a combination of data analysis, human expertise, and appropriate algorithms. The most effective approach often involves using lexical search for initial filtering and retrieval, then applying more sophisticated ranking methods.</p>\n<p><strong>Final thoughts on the lexical versus semantic debate</strong></p>\n<p>While John humorously admitted his bias toward lexical search (&quot;lexical search is my hammer and the world is my nail&quot;), he clearly recognized that both approaches have complementary strengths. Lexical search excels at filtering, exact matching, and providing rich metadata, while semantic search better understands meaning, context, and synonyms.</p>\n<p>The most promising direction appears to be hybrid approaches that leverage the strengths of both methods, though implementing these effectively remains challenging. As LLMs become more capable, they may also play a role in dynamically adjusting search parameters based on user needs, further blurring the line between lexical and semantic approaches.</p>\n<h2><em><strong>Key Takeaway:</strong></em> The debate isn't really about lexical versus semantic search, but rather how to effectively combine them to create retrieval systems that are both precise and understanding - offering both the filtering power of traditional search and the semantic comprehension of modern embedding-based approaches.</h2>\n",
      "date_modified": "2025-09-11T00:00:00+00:00",
      "date_published": "2025-09-11T00:00:00+00:00",
      "external_url": "https://jxnl.co/writing/2025/09/11/lexical-search-in-rag-applications/",
      "id": "urn:aggr:item:0c6740d5aaeacfcbda800b7c457e56d79831ef47",
      "summary": "Guest lecture with John Berryman on traditional search techniques, their application in RAG systems, and how lexical search complements semantic search",
      "tags": [
        "programming",
        "filtering",
        "hybrid search",
        "information retrieval",
        "lexical search",
        "rag",
        "speaker series",
        "tf-idf",
        "traditional search"
      ],
      "title": "Lexical Search in RAG Applications",
      "url": "https://aggr.aymericbeaumet.com/items/jxnl-co/2025-09-11-lexical-search-in-rag-applications/"
    },
    {
      "content_html": "<p>I hosted a special session with Eli Badgio, CTO of Extend, to discuss AI-native document processing in the cloud. Extend helps companies achieve 95%+ extraction accuracy for customers like Brex and other Fortune 500s. This session covered mapping document workflows, building task-specific evaluations, and implementing partial automation with human-in-the-loop approaches.</p>\n<p><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://maven.com/p/da0487\">▶️ Learn How Extend Hits 95%+ Accuracy</a></p>\n<h2>What are the key challenges in document automation?</h2>\n<p>Document processing is an incredibly broad field, from ingestion (RAG context for agents, data mining) to back-office processing (where documents are systems of record) to in-product experiences (traditional OCR).</p>\n<p>The most common failure case is companies trying to &quot;one-shot&quot; automation—attempting to replace an entire manual process with AI in a single step. This almost never works, even when extraction accuracy seems high in initial tests.</p>\n<blockquote>\n<p>&quot;In practice, it will never work, even if the use case is not that complex and the accuracy on the extraction is going to be close to perfect. It's extraordinarily difficult to just swap out a manual process for an automated process.&quot; — Eli Badgio</p>\n</blockquote>\n<p>This one-shot approach sets unrealistic expectations for business stakeholders and product teams. Instead, successful document automation requires a more methodical, step-by-step approach.</p>\n<p><strong>Key Takeaway:</strong> Don't try to replace entire document workflows in one go. Start with partial automation and gradually expand as you build confidence in the system and gain stakeholder buy-in.</p>\n<h2>How should companies approach understanding their document data?</h2>\n<p>The first critical step is developing a deep understanding of both your documents and the existing manual processes around them. In most companies, there's an enormous amount of tacit knowledge hidden in people's heads—whether in engineering, operations, or domain experts.</p>\n<p>One client Extend worked with took a fascinating approach to this challenge. They were trying to automate software contract ingestion but initially struggled with accuracy issues. Their solution was four-fold:</p>\n<ul>\n<li>Implemented a human-in-the-loop flow where AI-processed documents were still routed to humans for verification, allowing them to collect robust production data</li>\n<li>Brought in domain experts from within the company who had previously done this work manually</li>\n<li>Created a series of extractors that characterized normalized attributes (supplier, layout type, pricing terminology) rather than trying to extract actionable data</li>\n<li>Ran this workflow on approximately 150,000-200,000 documents (nearly a million pages) to develop an accurate representation of their historical data</li>\n</ul>\n<p>This approach allowed them to identify blind spots in their understanding—problematic suppliers, pricing terms that weren't being extracted correctly, and document types they weren't properly accounting for.</p>\n<blockquote>\n<p>&quot;Take some kind of initial first pass that is entirely focused on understanding the data, not actually trying to get the data out.&quot; — Eli Badgio</p>\n</blockquote>\n<p><strong>Key Takeaway:</strong> Invest time in understanding your documents and existing processes before attempting automation. Consider running a data characterization phase where you classify document attributes to identify patterns and edge cases.</p>\n<h2>Why are task-specific evaluations critical for document automation?</h2>\n<p>Generic benchmarks like &quot;Gemini Flash is better than GPT-X&quot; aren't particularly useful for document automation. What matters is how well a system performs on your specific documents and use cases.</p>\n<p>A healthcare vertical AI team Extend worked with discovered this when automating EHR entry of referral orders. They were dealing with 100+ page scanned PDFs containing multiple referrals mixed with patient history and provider notes.</p>\n<p>Their solution was to have their AI engineers collaborate directly with in-house nurses and medical technicians to build comprehensive, fine-grained evaluation sets across a variety of document examples. The domain experts helped optimize schemas and extraction rules to properly interpret medical jargon.</p>\n<p>This approach created a competitive advantage—proprietary evaluation data that allowed them to quickly test new models and capabilities against their specific needs. They maintained this advantage by continuing to review a percentage of documents to understand how real-world production data changes over time.</p>\n<blockquote>\n<p>&quot;You really need to invest the evals at every step of a flow when you're trying to automate an end-to-end document process.&quot; — Eli Badgio</p>\n</blockquote>\n<p><strong>Key Takeaway:</strong> Create detailed, domain-specific evaluation sets with help from subject matter experts. These evaluations should test each step of your pipeline and reflect the real-world complexity of your documents.</p>\n<h2>How should companies think about partial automation versus full automation?</h2>\n<p>Rather than aiming for 100% automation from the start, successful companies focus on what Eli called &quot;true automation rate&quot;—how high of an accuracy can you get where the end-to-end pipeline works 100% of the time, and inaccuracies are caught 100% of the time?</p>\n<p>A dental billing automation company took this approach when processing explanation of benefits (EOB) documents. They implemented:</p>\n<ul>\n<li>A multi-step process that segmented the problem, limiting what each module needed to handle</li>\n<li>Configuration workflows for different providers with known edge cases</li>\n<li>Data validations and guardrails that could declaratively identify when a document needed human review</li>\n</ul>\n<p>This approach recognized that 85% true automation (where you know with certainty which 15% need human review) is more valuable than 98% accuracy where you can't identify the problematic 2%.</p>\n<blockquote>\n<p>&quot;In most cases, for these critical systems of records ingestion use cases, that's strictly worse than if you can get to like 85% true automation rate, where the 15% of the time that's not going to properly handle the document, you can catch that declaratively.&quot; — Eli Badgio</p>\n</blockquote>\n<p><strong>Key Takeaway:</strong> Focus on &quot;automate when possible, augment otherwise&quot; rather than full automation. Design systems that can confidently identify which documents need human review rather than trying to automate everything.</p>\n<h2>How should companies redesign processes around automation?</h2>\n<p>Rather than trying to replicate existing manual processes with automation, successful implementations rethink the entire workflow with automation in mind.</p>\n<p>This might involve:</p>\n<ul>\n<li>Breaking document processing into multiple pipeline steps (parsing, classification, splitting, extraction, semantic chunking)</li>\n<li>Creating declarative validation rules to identify when automation might fail</li>\n<li>Designing human-in-the-loop workflows that efficiently handle exceptions</li>\n</ul>\n<p>When working with clients, Extend often starts by asking them to walk through their current process end-to-end. This helps identify opportunities to redesign workflows rather than simply automating existing inefficient processes.</p>\n<blockquote>\n<p>&quot;It can be very difficult to just full swap out something that's been done over many years among many different stakeholders and just make that automated as opposed to really understanding as an engineer what is the end-to-end process.&quot; — Eli Badgio</p>\n</blockquote>\n<p><strong>Key Takeaway:</strong> Don't try to replicate manual processes with automation. Instead, redesign workflows from first principles with automation in mind, focusing on validation and exception handling.</p>\n<h2>How should companies balance cost, speed, and accuracy in document processing?</h2>\n<p>Document processing involves multiple trade-offs between cost, speed, and accuracy. For back-office workflows, accuracy should generally take priority over speed.</p>\n<p>Eli recommended several approaches to manage these trade-offs:</p>\n<ul>\n<li>Use model distillation to create smaller, faster models for specific tasks like chunking</li>\n<li>Implement pre-filtering steps with lightweight classifiers to avoid running expensive operations on all documents</li>\n<li>For back-office flows, design processes that remove latency as a consideration</li>\n</ul>\n<blockquote>\n<p>&quot;If it's a back-office flow, I would encourage everyone not to think about latency as a problem. In fact, you should think about designing a process where you can effectively remove latency as being a consideration.&quot; — Eli Badgio</p>\n</blockquote>\n<p>He also noted that deterministic code-based approaches are often preferable to LLM-based solutions for certain extraction tasks:</p>\n<blockquote>\n<p>&quot;If you can solve it with code... you should do the code-based approach because then you can actually guarantee it's going to behave how you want.&quot; — Eli Badgio</p>\n</blockquote>\n<p><strong>Key Takeaway:</strong> For back-office document processing, prioritize accuracy over speed. Use lightweight models for initial classification and filtering, and consider deterministic code-based approaches where possible to reduce uncertainty.</p>\n<h2>What role should domain experts play in document automation?</h2>\n<p>One of the most overlooked aspects of document automation is the critical role of domain experts. These are the people who understand the tacit knowledge embedded in documents and processes—nurses who know how to interpret medical terminology, financial analysts who understand billing codes, or operations staff who have been processing these documents manually.</p>\n<p>Eli emphasized that involving these experts early in the process creates a significant competitive advantage:</p>\n<blockquote>\n<p>&quot;If you're an engineer and you're tasked with this kind of problem... and you don't feel like you actually understand this process, someone on your team has to. If you're doing this currently and you're in industry, someone understands this.&quot; — Eli Badgio</p>\n</blockquote>\n<p>The most successful implementations Extend has seen involve close collaboration between AI engineers and domain experts to:</p>\n<ul>\n<li>Design appropriate schemas and extraction rules</li>\n<li>Create comprehensive evaluation datasets</li>\n<li>Develop validation rules to identify edge cases</li>\n<li>Interpret domain-specific terminology and formats</li>\n</ul>\n<p><strong>Key Takeaway:</strong> Involve domain experts early in your document automation projects. Their tacit knowledge is essential for understanding document semantics, designing appropriate schemas, and creating effective evaluation datasets.</p>\n<h2>How should companies approach change management for document automation?</h2>\n<p>Implementing document automation requires careful change management. Eli recommends starting with augmentation rather than full automation:</p>\n<blockquote>\n<p>&quot;Most companies are much more willing to roll something out and deal with the change management if you focus on augmentation at start and then turn augmentation into automation.&quot; — Eli Badgio</p>\n</blockquote>\n<p>When working with clients, Extend often leads with building a human-in-the-loop system first, allowing companies to:</p>\n<ul>\n<li>Collect real-world data about their documents</li>\n<li>Build confidence in the system gradually</li>\n<li>Identify edge cases and failure modes</li>\n<li>Manage stakeholder expectations</li>\n</ul>\n<p>This approach aligns with their mantra: &quot;Automate when possible, augment otherwise.&quot;</p>\n<p><strong>Key Takeaway:</strong> Start with augmentation rather than full automation to ease change management. Build human-in-the-loop systems that gradually increase automation levels as confidence grows.</p>\n<h2>Final thoughts on document automation</h2>\n<p>Document automation is a complex field that requires deep understanding of both technical capabilities and domain-specific knowledge. The most successful implementations I've seen follow these principles:</p>\n<ul>\n<li>Understand your data and existing manual processes thoroughly</li>\n<li>Invest early in tailored evaluations specific to your domain</li>\n<li>Break processing into multiple pipeline steps</li>\n<li>Don't aim for 100% automation initially</li>\n<li>Redesign processes around automation rather than trying to replicate manual workflows</li>\n</ul>\n<p>As AI capabilities continue to advance, the companies that will benefit most are those that take a thoughtful, systematic approach to document automation rather than rushing to replace human processes overnight.</p>\n<hr />\n<h2>FAQs</h2>\n<p><strong>What is document automation and why is it important?</strong></p>\n<p>Document automation uses AI to process and extract information from documents, reducing manual effort and errors. This is particularly crucial for enterprises where manual document workflows consume significant resources. For example, a single building construction project can require processing a million documents. Effective document automation can achieve 95%+ extraction accuracy, streamlining operations across industries from finance to healthcare.</p>\n<p><strong>What are the main types of document processing use cases?</strong></p>\n<p>There are three broad categories of document processing:</p>\n<ul>\n<li>Ingestion: Including RAG context for agents and data mining</li>\n<li>Back office processing: When documents serve as systems of record (contracts, deeds, etc.)</li>\n<li>In-product experiences: Traditional OCR use cases like uploading an invoice to extract data</li>\n</ul>\n<p>The most challenging and often overlooked category is back office processing, where documents serve as critical systems of record with zero tolerance for errors.</p>\n<p><strong>What's the most common mistake when implementing document automation?</strong></p>\n<p>The most common failure is attempting to &quot;one-shot&quot; the automation—trying to replace an entire manual process with automation in a single step. This approach almost never works, even for relatively simple use cases. It sets unrealistic expectations and doesn't account for the complex change management required when transitioning from manual to automated processes.</p>\n<p><strong>How should I approach understanding my document data?</strong></p>\n<p>Take time to thoroughly understand both your documents and the existing manual processes around them. Many companies have enormous amounts of tacit knowledge hidden in people's heads—whether in engineering, operations, or domain experts. This knowledge is critical for successful automation.</p>\n<p>One effective approach is to implement a human-in-the-loop flow where AI-processed documents are still routed to humans for review. This helps collect robust production data separate from your evaluation sets. Also consider creating extractors that characterize normalized attributes of your documents (supplier types, layout patterns, terminology variations) to help cluster and understand your document landscape.</p>\n<p><strong>Why are tailored evaluations so important for document automation?</strong></p>\n<p>Generic benchmarks don't translate well to specific document automation tasks. You need evaluations tailored to your domain, data, and processes. Invest early in creating comprehensive, fine-grained evaluation sets across a variety of document examples.</p>\n<p>The best approach is to involve domain experts (like nurses for healthcare documents or billing specialists for financial documents) to help build robust evaluation sets and optimize extraction schemas. This creates a competitive advantage as you can quickly test and confidently roll out improvements when new models become available.</p>\n<p><strong>Should I aim for 100% automation from the start?</strong></p>\n<p>No. It's much more effective to start with partial automation and gradually increase automation rates. Focus on what's called the &quot;true automation rate&quot;—not just accuracy on extracted data, but how often the end-to-end pipeline works perfectly and you can catch errors 100% of the time.</p>\n<p>For example, a system with 85% true automation rate where you can reliably identify the 15% that needs human review is far better than a system with 98% accuracy but no way to identify which 2% contains errors.</p>\n<p><strong>How should I think about redesigning processes for automation?</strong></p>\n<p>Rather than trying to create a one-to-one replacement of an existing manual process, rethink the process from scratch with automation in mind. This might involve:</p>\n<ul>\n<li>Breaking complex workflows into multiple steps</li>\n<li>Creating deterministic routing based on document classifications</li>\n<li>Implementing validation checks and guardrails</li>\n<li>Designing human review processes for edge cases</li>\n</ul>\n<p>When possible, design the process around asynchronous processing rather than optimizing for latency, especially for back-office workflows where accuracy is more important than speed.</p>\n<p><strong>How can I reduce costs for document processing?</strong></p>\n<p>Several strategies can help manage costs:</p>\n<ul>\n<li>Use model distillation to create smaller, faster models for specific tasks</li>\n<li>Implement lightweight classifiers as an initial step to route documents appropriately</li>\n<li>Use pre-filtering to only apply expensive processing (like semantic chunking) to documents that require it</li>\n<li>Break processing into multiple steps with specialized models for each task</li>\n</ul>\n<p>For back-office workflows, prioritize accuracy over latency and design processes that accommodate asynchronous processing.</p>\n<p><strong>How should I involve domain experts in document automation?</strong></p>\n<p>Domain experts are crucial for successful document automation. Involve them in:</p>\n<ul>\n<li>Tailoring classifications and schema design to match industry jargon</li>\n<li>Building comprehensive evaluation sets</li>\n<li>Optimizing extraction rules to properly interpret domain-specific terminology</li>\n<li>Designing validation checks based on their knowledge of the data</li>\n</ul>\n<p>Teams that effectively incorporate domain experts gain a significant competitive advantage in document automation.</p>\n<p><strong>What's the best way to start automating a document workflow?</strong></p>\n<p>Start by identifying a single, high-priority use case rather than attempting to automate multiple workflows simultaneously. Focus on understanding the data and existing process thoroughly before designing automation.</p>\n<p>Consider these steps:</p>\n<ul>\n<li>Implement a human-in-the-loop flow to collect production data</li>\n<li>Involve domain experts to understand tacit knowledge</li>\n<li>Create robust evaluation sets for each step in the process</li>\n<li>Start with augmentation before moving to automation</li>\n<li>Design the process around automation rather than trying to replicate the manual process</li>\n</ul>\n<p>This approach delivers business value faster while building organizational comfort with automation.</p>\n",
      "date_modified": "2025-09-11T00:00:00+00:00",
      "date_published": "2025-09-11T00:00:00+00:00",
      "external_url": "https://jxnl.co/writing/2025/09/11/how-extend-achieves-95-document-automation-lessons-from-eli-badgio/",
      "id": "urn:aggr:item:c9d1761185f9fe99494901373ebdb5b6c68ac439",
      "summary": "Insights from Eli Badgio, CTO of Extend, on mapping document workflows, building task-specific evaluations, and implementing partial automation with human-in-the-loop approaches for 95%+ extraction accuracy.",
      "tags": [
        "programming",
        "document automation",
        "evaluation",
        "extend",
        "extraction",
        "human-in-the-loop",
        "rag",
        "speaker series",
        "workflow"
      ],
      "title": "How Extend Achieves 95%+ Document Automation (Lessons from Eli Badgio)",
      "url": "https://aggr.aymericbeaumet.com/items/jxnl-co/2025-09-11-how-extend-achieves-95-document-automation-lessons-from-eli/"
    },
    {
      "content_html": "<p>I hosted a session featuring Chris Lovejoy, Head of Clinical AI at Anterior, who shared valuable insights from his experience building AI agents for specialized industries. Chris brings a unique perspective as a former medical doctor who transitioned to AI, working across healthcare, education, recruiting, and retail sectors.</p>\n<p><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://maven.com/p/504453/lessons-from-building-verticalized-ai-agents?utm_medium=ll_share_link&amp;utm_source=instructor\">▶️ Lessons from building verticalized AI agents</a></p>\n<h2>Why is it so hard to successfully apply LLMs to specialized industries?</h2>\n<p>Chris identified two fundamental challenges when implementing AI in vertical industries:</p>\n<ul>\n<li>The &quot;last mile problem&quot; — taking powerful models and making them work in specific customer contexts. The challenge isn't reasoning quality anymore; today's models fail because they lack context on how workflows are performed in specific industries.</li>\n<li>Difficulty defining &quot;good&quot; and &quot;correct&quot; — in specialized fields, determining what constitutes a good output requires domain expertise that most AI engineers don't possess.</li>\n</ul>\n<p>As Chris explained, &quot;If your data looks like a legal document and you're not a lawyer, you wouldn't know whether it's right or wrong.&quot; This highlights why domain experts are essential for vertical AI applications.</p>\n<h2>How can domain experts supercharge AI development?</h2>\n<p>Chris recommends implementing a systematic review process with domain experts that creates a powerful feedback loop:</p>\n<ol>\n<li>Your production application generates AI outputs</li>\n<li>Domain experts review these outputs to provide:</li>\n<li>Performance metrics (accuracy scores)</li>\n<li>Categorized failure modes</li>\n<li>Suggested improvements</li>\n<li>Input–output pairs for training</li>\n<li>This expert‑generated data enables you to prioritize work based on which failure modes most impact your key metrics. You can create ready‑made failure mode datasets grouped by error type, allowing engineers to test improvements against specific problems.</li>\n</ol>\n<p>The most effective process is a continuous loop where:</p>\n<ol>\n<li>Production generates outputs</li>\n<li>Domain experts review and categorize issues</li>\n<li>A domain expert PM prioritizes which failure modes to fix</li>\n<li>Engineers test changes against failure mode datasets</li>\n<li>The PM approves changes to go live</li>\n</ol>\n<p><strong>Key Takeaway:</strong> Domain experts aren't just helpful — they're essential for vertical AI applications. Building systems to capture their insights creates a data‑driven improvement cycle that addresses the specific needs of specialized industries.</p>\n<h2>What's the best way to implement domain expert reviews?</h2>\n<p>While many companies start with simple tools like Google Sheets or raw traces in monitoring tools, Chris strongly recommends building a custom review dashboard. This approach gives you maximum freedom over how you present information and makes it easier to integrate review data into your production systems.</p>\n<p>A well‑designed review UI should optimize for:</p>\n<ul>\n<li>High‑quality reviews (prioritize quality over quantity)</li>\n<li>Minimizing time spent by reviewers</li>\n<li>Generating actionable data</li>\n</ul>\n<p>When building a custom UI, follow these principles:</p>\n<ul>\n<li>Clearly surface all required context (separate wider context from specific details)</li>\n<li>Optimize the review flow sequence (bake your ideal review process into the UI)</li>\n<li>Minimize friction (reduce scrolling, clicking, and manual work)</li>\n</ul>\n<p>Chris has found that spending time shadowing domain experts as they review outputs helps you understand their natural workflows, which you can then incorporate into your UI design.</p>\n<h2>Should I use prompting or fine‑tuning for vertical AI applications?</h2>\n<p>Chris believes better prompting beats fine‑tuning for improving performance in most vertical applications. This might seem counterintuitive, but he offers several reasons:</p>\n<ul>\n<li>Today's out‑of‑the‑box models already have strong domain‑specific reasoning capabilities</li>\n<li>Fine‑tuning adds complexity (managing multiple models for different tasks and customers)</li>\n<li>Fine‑tuned models can degrade as workflows evolve over time</li>\n</ul>\n<p>While fine‑tuning has benefits for speed, cost, and scalability, Chris recommends starting with prompting for performance improvements.</p>\n<p>However, he doesn't mean just tweaking prompt text — that approach can be brittle and model‑dependent. Instead, he focuses on two more powerful techniques:</p>\n<ol>\n<li>Use context augmentation — inject domain expert‑generated knowledge into your prompts at inference time. For example, in finance, you might include the specific definition of &quot;high net worth individual&quot; used by a particular bank.</li>\n<li>Leverage input–output pair examples for in‑context learning — include examples where the model initially made mistakes, along with the corrections from domain experts.</li>\n</ol>\n<p>The most powerful approach is to go beyond static prompting by implementing dynamic retrieval of this knowledge. Your domain expert UI generates knowledge that lives in a domain knowledge base, which you can then retrieve in real‑time during inference using techniques like:</p>\n<ul>\n<li>Keyword matching</li>\n<li>Semantic similarity</li>\n<li>Recency weighting</li>\n<li>Diversity optimization</li>\n</ul>\n<p><strong>Key Takeaway:</strong> Rather than fine‑tuning models, focus on dynamically augmenting prompts with domain‑specific knowledge and examples. This approach is more adaptable to changing workflows and customer needs.</p>\n<h2>How do I build and maintain customer trust?</h2>\n<p>For verticalized AI agents, Chris identifies three key factors for customer trust:</p>\n<ul>\n<li>Confidence in the AI's performance</li>\n<li>Confidence in secure data handling</li>\n<li>Protection against LLM‑specific attack vectors</li>\n</ul>\n<p>To build confidence in AI performance, implement these practices:</p>\n<ul>\n<li>Review production outputs and generate performance metrics</li>\n<li>Report these metrics to customers regularly (biweekly or monthly)</li>\n<li>Define a sampling strategy as you scale (based on uncertainty, outliers, or stratified sampling)</li>\n<li>Establish an internal response protocol for performance issues</li>\n<li>Consider using LLMs as judges to help scale monitoring</li>\n</ul>\n<p>Chris shared a case study from a healthcare application where they couldn't manually review all AI outputs — they could only review about 5%. They implemented an LLM judge system that:</p>\n<ul>\n<li>Scored the confidence of each AI output in real‑time</li>\n<li>Prioritized which cases humans should review</li>\n<li>Provided performance metrics for all outputs</li>\n<li>Created a feedback loop where human reviews improved the judge system</li>\n</ul>\n<p>For data handling, Chris recommends:</p>\n<ul>\n<li>Mapping out your data usage strategy early (during contract negotiations)</li>\n<li>Being ready to offer isolated, single‑tenant environments</li>\n<li>Investing in synthetic data generation for testing and training</li>\n</ul>\n<p>For LLM‑specific security, stay vigilant about:</p>\n<ul>\n<li>Prompt injections (using input filtering and validation)</li>\n<li>Sensitive information disclosure (through data sanitization)</li>\n<li>Data and model poisoning (tracking data origins and using version control)</li>\n</ul>\n<p><strong>Key Takeaway:</strong> Building trust requires transparent performance monitoring, secure data handling, and proactive security measures. Implement systems that give customers visibility into AI performance while protecting their sensitive information.</p>\n<h2>Which domain experts should you hire and how should you use them?</h2>\n<p>Chris strongly recommends hiring a principal domain expert who has ultimate responsibility for your AI's performance. This approach has several advantages:</p>\n<ul>\n<li>Organizational clarity (having a DRI helps you move faster)</li>\n<li>Avoiding consensus by committee</li>\n<li>Building deep intuition about your AI system's performance</li>\n</ul>\n<p>This person should be hired early and given ownership to shape your product. Beyond reviewing outputs, they can:</p>\n<ul>\n<li>Help hire and manage additional reviewers</li>\n<li>Define sampling strategies</li>\n<li>Analyze review data</li>\n<li>Monitor reviewer performance</li>\n<li>Steer product development</li>\n<li>Talk to customers</li>\n<li>Improve AI performance</li>\n</ul>\n<p>When hiring this principal domain expert, look for someone who meets a critical domain expertise threshold but also has additional skills like:</p>\n<ul>\n<li>Management/leadership experience</li>\n<li>Statistical knowledge</li>\n<li>Product development experience</li>\n<li>Technical understanding</li>\n</ul>\n<p>At Anterior, Chris played this role with his medical background, which allowed him to review outputs, hire additional reviewers from his healthcare network, define sampling strategies using his data science background, and contribute to AI implementations with his software engineering skills.</p>\n<p><strong>Key Takeaway:</strong> A principal domain expert who combines industry knowledge with leadership and technical skills can dramatically accelerate your ability to build effective vertical AI applications. This person becomes the bridge between domain expertise and technical implementation.</p>\n<h2>How do you handle citations and references in AI‑generated content?</h2>\n<p>For citation handling, Chris mentioned they've used both custom approaches and vendor solutions like Anthropic's citation API. One effective technique they've implemented is:</p>\n<ol>\n<li>Chunk content with semantic similarity‑based methods</li>\n<li>Assign unique IDs to each chunk</li>\n<li>Prompt the model to first identify relevant evidence chunks by their UIDs</li>\n<li>Have the model reason using those specific chunks</li>\n<li>Make the final decision with clear references to the source material</li>\n</ol>\n<p>This sequential approach ensures the model explicitly references the information it used, making the reasoning process more transparent and traceable.</p>\n<h2>What question are teams not asking themselves when building these systems?</h2>\n<p>When I asked Chris what question teams aren't asking themselves, he identified: &quot;What does your system look like for incorporating domain expertise?&quot;</p>\n<p>He noted many teams focus on using the latest, most sophisticated models while neglecting the process for systematically incorporating domain knowledge. The real challenge isn't just implementing the model — it's creating a flywheel that continuously improves your system based on expert feedback.</p>\n<p>Unlike traditional software where you build features and move on, AI systems require ongoing optimization to push the probability distribution in the right direction. Building this improvement flywheel is essential for successful vertical AI applications.</p>\n<p><strong>Key Takeaway:</strong> Don't treat AI applications like traditional software that you build once and maintain. Instead, design systems that continuously incorporate domain expertise to systematically improve performance over time.</p>\n<hr />\n<h2>FAQs</h2>\n<h2>What makes applying LLMs to specialized industries so challenging?</h2>\n<p>The primary challenge lies in the &quot;last mile problem&quot; — taking powerful models and making them work effectively in specific industry contexts. While today's models have strong reasoning capabilities, they often lack the contextual understanding of how specific workflows operate in specialized industries. Additionally, defining what constitutes &quot;good&quot; or &quot;correct&quot; output in these domains requires domain expertise that general AI practitioners may not possess.</p>\n<h2>How can domain experts improve AI development in vertical industries?</h2>\n<p>Domain experts are invaluable when your AI is processing specialized data that general practitioners can't properly evaluate. By implementing a structured review process with domain experts, you can gather four critical types of information:</p>\n<ul>\n<li>Performance metrics that show how well your AI is performing</li>\n<li>Specific failure modes that categorize where and how your AI makes mistakes</li>\n<li>Suggested improvements based on domain knowledge</li>\n<li>High‑quality input–output pairs for training and evaluation</li>\n</ul>\n<p>This information enables you to prioritize improvements, create targeted datasets for testing, and implement a continuous improvement cycle that addresses real‑world usage patterns.</p>\n<h2>What's the most effective way to support domain experts in reviewing AI outputs?</h2>\n<p>While you can start with basic tools like spreadsheets or third‑party evaluation platforms, building a custom review dashboard is the highest‑leverage investment you can make. A well‑designed custom UI should:</p>\n<ul>\n<li>Clearly surface all required context without overwhelming reviewers</li>\n<li>Optimize the review flow sequence to match how experts naturally evaluate information</li>\n<li>Minimize friction by reducing unnecessary scrolling and clicking</li>\n<li>Generate actionable data that can be directly integrated into your production system</li>\n</ul>\n<p>The goal is to balance high‑quality reviews with reviewer efficiency while generating data that can immediately improve your system.</p>\n<h2>Is fine‑tuning or prompting better for verticalized AI agents?</h2>\n<p>For most vertical applications, advanced prompting techniques are more effective than fine‑tuning. Today's out‑of‑the‑box models already have strong domain‑specific reasoning capabilities, and the typical issues in vertical applications relate more to context than reasoning ability. Fine‑tuning adds complexity through model management and can become outdated as workflows evolve.</p>\n<p>That said, fine‑tuning may still be valuable for optimizing speed, reducing costs, or improving scalability in certain scenarios.</p>\n<h2>What prompting techniques work best for vertical applications?</h2>\n<p>Rather than just tweaking prompt wording (which can be brittle and model‑dependent), two more robust techniques are particularly effective:</p>\n<ul>\n<li>Context augmentation: Injecting domain‑specific knowledge into the prompt context at inference time, such as customer‑specific definitions or guidelines</li>\n<li>Input–output pair examples: Using few‑shot prompting with real examples to demonstrate the desired behavior</li>\n</ul>\n<p>These approaches can be implemented dynamically, retrieving the most relevant context based on the specific query and customer needs.</p>\n<h2>How can you build and maintain customer trust in vertical AI applications?</h2>\n<p>Building trust requires addressing three key areas:</p>\n<ul>\n<li>Performance confidence: Conduct regular reviews of production outputs, report metrics to customers, implement strategic sampling as you scale, and establish clear response protocols for performance issues</li>\n<li>Data security: Map out your data usage strategy early (ideally during contracting), be prepared to offer isolated environments for security‑conscious customers, and consider investing in synthetic data generation</li>\n<li>Security against LLM‑specific threats: Implement protections against prompt injection, prevent sensitive information disclosure, and stay current with evolving best practices in LLM security</li>\n</ul>\n<p>Transparent communication about these measures helps customers understand your commitment to reliability and security.</p>\n<h2>What role should domain experts play in your organization?</h2>\n<p>Consider hiring a principal domain expert who serves as the direct responsible individual (DRI) for AI performance. This person should:</p>\n<ul>\n<li>Define what constitutes &quot;good&quot; or &quot;correct&quot; output</li>\n<li>Build intuition about how your AI system performs and how it can be improved</li>\n<li>Help hire and manage additional reviewers as you scale</li>\n<li>Define sampling strategies for efficient review</li>\n<li>Analyze review data and monitor reviewer performance</li>\n<li>Steer product development and prioritize engineering work</li>\n<li>Communicate with customers about performance and improvements</li>\n</ul>\n<p>The ideal candidate combines domain expertise with additional skills like management experience, statistical knowledge, and product development capabilities.</p>\n<h2>How can you create an effective continuous improvement cycle?</h2>\n<p>Implement a structured loop where:</p>\n<ol>\n<li>Your production application generates AI outputs</li>\n<li>Domain experts review a strategic sample of those outputs</li>\n<li>Reviews produce performance metrics and categorized failure modes</li>\n<li>Product managers prioritize which failure modes to address</li>\n<li>Engineers make changes and test against failure mode datasets</li>\n<li>Changes are validated and deployed to production</li>\n</ol>\n<p>This cycle allows you to systematically improve performance based on real‑world usage patterns and domain‑specific requirements.</p>\n<h2>How can you scale performance monitoring as your application grows?</h2>\n<p>As you scale and can no longer review all outputs, implement a strategic sampling approach based on:</p>\n<ul>\n<li>Uncertainty levels in the AI's responses</li>\n<li>Outlier detection to identify unusual cases</li>\n<li>Stratified sampling across different characteristics</li>\n</ul>\n<p>You can also implement an LLM‑as‑judge system that evaluates outputs in real‑time, providing confidence scores that help prioritize which cases need human review and potentially taking automated actions for low‑confidence outputs.</p>\n<hr />\n<h2>Connect with Chris Lovejoy</h2>\n<ul>\n<li><strong><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://forms.fillout.com/t/uYXsRp2K8Pus\">Book time here</a></strong> - Schedule a consultation with Chris</li>\n<li><strong><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://chrislovejoy.me/email\">Sign up to mailing list here</a></strong> - Get updates on vertical AI insights</li>\n</ul>\n<h2>Additional Resources</h2>\n<p><strong>Chris Lovejoy's Talks &amp; Content:</strong></p>\n<ul>\n<li><strong><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://www.youtube.com/watch?v=MRM7oA3JsFs\">How to build an LLM-Native Expert System</a></strong> - Dr Chris Lovejoy, AI Engineer World's Fair (SF, June 2025)</li>\n<li><strong><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://chrislovejoy.me/review-dashboard\">Why custom AI review dashboards transform AI products - and how to build one</a></strong> - Dr Chris Lovejoy</li>\n<li><strong><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://www.youtube.com/watch?v=cZ5ZJy19KMo\">Mission-Critical Evals at Scale</a></strong> - Dr Chris Lovejoy, AI Engineer Summit (NY, Feb 2025)</li>\n<li><strong><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://chrislovejoy.me/domain-expert-vertical-ai\">Why you need a principal domain expert for building vertical AI - and how to find them</a></strong> - Dr Chris Lovejoy</li>\n</ul>\n<p><strong>Industry Resources:</strong></p>\n<ul>\n<li><strong><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://genai.owasp.org/llm-top-10/\">2025 Top 10 Risks and Mitigations for LLM Applications &amp; Generative AI</a></strong> - OWASP</li>\n</ul>\n",
      "date_modified": "2025-09-11T00:00:00+00:00",
      "date_published": "2025-09-11T00:00:00+00:00",
      "external_url": "https://jxnl.co/writing/2025/09/11/domain-experts-the-lever-for-vertical-ai/",
      "id": "urn:aggr:item:4c4fb55b012bdb5615d52ea02e79123de59eea9c",
      "summary": "How to successfully apply LLMs in specialized industries by building domain‑expert review loops, augmenting prompts with expert knowledge, and earning customer trust.",
      "tags": [
        "programming",
        "domain experts",
        "evaluation",
        "prompting",
        "rag",
        "security",
        "speaker series",
        "trust",
        "vertical ai"
      ],
      "title": "Domain Experts: The Lever for Vertical AI",
      "url": "https://aggr.aymericbeaumet.com/items/jxnl-co/2025-09-11-domain-experts-the-lever-for-vertical-ai/"
    },
    {
      "authors": [
        {
          "name": "Jason Liu"
        }
      ],
      "content_html": "<p>This past spring, two senior engineers at different companies received the same challenge from their CEOs: &quot;We need to move faster. Use AI to get there.&quot;</p>\n<p>Both were talented, experienced engineers leading small teams. But six months later, one team had grown from 5 to 13 people and was still struggling to meet deadlines. The other stayed the same size but was shipping twice as fast.</p>\n<p><strong>What Made The Difference</strong></p>\n<p>The difference wasn't talent or effort. One engineer treats AI like a better search engine. The other treats AI like an entire engineering team working in concert.</p>\n<p><strong>The Reality Most Engineers Face</strong></p>\n<p>Your team is already swamped. They don't have time to follow every AI launch or watch dozens of tutorials. When they do try AI tools, it's between meetings or after hours when they're exhausted.</p>\n<p>The AI landscape moves faster than anyone can keep up with. The &quot;best practices&quot; you read about are often written by people who have time to experiment, not engineers who need to ship production code.</p>\n<p><strong>The Knowledge Gap That's Widening</strong></p>\n<p>The engineers winning right now learned the complete workflow—the orchestration that turns AI from a productivity hack into a fundamental shift in how software gets built.</p>\n<p>Most engineers are trying to piece together knowledge from scattered content while their actual work piles up. Companies are sending entire teams to get concentrated, practical knowledge in three days instead of months of content consumption.</p>\n<p><strong>A Tale of Two Workflows</strong></p>\n<p>Both engineers started with 90-minute product planning meetings.</p>\n<p>David took notes in Notion, then copy-pasted portions into ChatGPT: &quot;Help me plan this feature.&quot; ChatGPT responded in isolation, without any context about their actual codebase or existing architecture.</p>\n<p>Elena had Granola transcription running. Fifteen minutes after the meeting ended, she had fed the complete transcript into OpenAI Codex—directly in their codebase, and used Wispr Flow to dictate any lingering context that was missing. The agent simultaneously read their source files, understood their existing architecture, and generated:</p>\n<ul>\n<li>A comprehensive plan.md that aligned with their current system, files to edit, and tests to write, and documentation to reference and update</li>\n<li>A detailed plan.md file that considered both edge cases and existing code patterns</li>\n</ul>\n<p>By the time David was wrestling with generic ChatGPT responses with copy and paste, Elena's team had a plan that was already integrated with their actual codebase, reviewed by their team, and ready to to start agentically coding up.</p>\n<p><strong>The New Reality of Software Development</strong></p>\n<p>Right now, I am looking at metrics from companies that have figured this out. The numbers are striking:</p>\n<ul>\n<li><strong>Shopify:</strong> Managing 5,000+ repositories with 30% fewer engineers than 2 years ago</li>\n<li><strong>Coinbase:</strong> 40% of daily code is AI-generated</li>\n<li><strong>Y Combinator startups:</strong> Teams of 3 shipping what used to require teams of 15</li>\n</ul>\n<p>The difference isn't in having AI tools. Everyone has ChatGPT. Everyone can get Cursor. Everyone can open up 15 coding agents in 15 separate terminal windows and have them 'work' in parallel.</p>\n<p>But if you're an engineering leader, you've learned that AI product demos rarely translate to production reality. So why would coding agent demos be any different? Those flashy videos of parallel agents aren't how real software gets built at scale.</p>\n<p>The difference is understanding that AI has fundamentally changed the entire workflow of software development—not just the tools you use, but how you orchestrate them into actual production systems.</p>\n<p><strong>The Workflow That Changes Everything</strong></p>\n<p>The modern AI engineering workflow looks like this:</p>\n<pre><code class=\"language-text\"><span class=\"code-snippet\">Meeting → AI Transcript → PRD Generation → Ticket Creation → Plan.md → Agent Execution\n</span></code></pre>\n<p>Each step precisely feeds the next. It's not about &quot;prompt engineering.&quot; It's about exploring, planning, and executing.</p>\n<p>When Elena's team ships a feature now, they also ship:</p>\n<ul>\n<li>Comprehensive tests (because AI makes them easier to write)</li>\n<li>Load testing suites (built in an afternoon instead of never)</li>\n<li>CLI tools for operations (tiny utilities that wouldn't have been worth the time before)</li>\n<li>Documentation that's actually current (generated from the code)</li>\n</ul>\n<p><strong>Knowledge Is Power</strong></p>\n<p>You see startups with tiny teams shipping impossibly fast. You hear about companies doing more with less.</p>\n<p>It's not hype. It's knowledge—specific, practical knowledge about how to transform your engineering workflow with AI. Your engineers need to learn the complete workflow that turns AI from a toy into a transformation.</p>\n<p><strong>An Opportunity to Lead</strong></p>\n<p>Put our statements to the proof by sending your key engineers to the AI Coding Accelerator this October. For three days, they'll learn from engineers who've actually done this transformation at scale.</p>\n<ul>\n<li><strong>Day 1:</strong> The complete workflow—from meeting to production</li>\n<li><strong>Day 2:</strong> Guest Speakers from the people building these tools (Linear, Cursor, Sourcegraph, OpenAI, Cognition)</li>\n<li><strong>Day 3:</strong> An afternoon hackathon with your actual code and challenges, and a whole day of office hours and live debugging.</li>\n</ul>\n<p>The investment is $600 per engineer for our October cohort (rising to $1,200 afterward).</p>\n<p>If your engineers become even 1% more efficient, you've paid for the training in weeks. But we're not talking about 1% gains—look at the numbers: Shopify managing with 30% fewer engineers, Y Combinator startups with teams of 3 shipping what used to require teams of 15. That's like getting 5x the output without the hiring costs, onboarding time, or management overhead.</p>\n<p>What's included:</p>\n<ul>\n<li>3 days of hands-on training with the complete AI toolchain</li>\n<li>Direct access to the teams building Claude Code, Linear, Cursor, Devin, Codex, and Sourcegraph</li>\n<li>6-month Windsurf trial + 1 month Devin Team access</li>\n<li>$50/person credit from AMP</li>\n<li>Lifetime access to materials and private Discord community</li>\n<li>Most importantly: The knowledge of how to orchestrate these tools into a multiplier</li>\n</ul>\n<p><strong>A Guarantee of Value</strong></p>\n<p>If your team isn't leveraging AI workflows to ship faster by the end of the quarter, we'll refund your investment in full. We're that confident your engineers will transform how they build software.</p>\n<p><strong>About Those Two Engineers</strong></p>\n<p>David and Elena started in the same place—talented engineers asked to leverage AI. What made their paths diverge?</p>\n<p>Practical knowledge. Proven workflows. No flashy demos—just systems that ship production code.</p>\n<p><strong>The Competitive Truth</strong></p>\n<p>Six months from now, the gap between teams that understand AI workflows and teams that don't will be insurmountable.</p>\n<p>Every week your team spends copy-pasting into ChatGPT is a week your competitors are pulling further ahead. Three days of learning creates months of competitive advantage.</p>\n<p><strong>Reserve Your Spots</strong></p>\n<h3>October 15-17, 2025</h3>\n<p><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://maven.com/nila/ai-coding-accelerator?promoCode=earlybird\"><strong>Reserve Your Team's Spots →</strong></a></p>\n<p>Sincerely,</p>\n<p>Jason Liu &amp; Vignesh Mohankumar <em>Creators of the AI Coding Accelerator</em></p>\n<p>P.S. — Brian Armstrong fired engineers who wouldn't adapt to AI. That's one approach. We prefer to give engineers the knowledge they need to thrive. The choice—and the opportunity—is yours.</p>\n",
      "date_modified": "2025-09-11T00:00:00+00:00",
      "date_published": "2025-09-11T00:00:00+00:00",
      "external_url": "https://jxnl.co/writing/2025/09/11/do-your-engineers-know-how-to-leverage-ai/",
      "id": "urn:aggr:item:f1b0fec4b02bb0b0a989f39df0637f7d6a77a5de",
      "summary": "A letter to the readers of the AI Coding Accelerator.",
      "tags": [
        "programming",
        "applied ai"
      ],
      "title": "Do Your Engineers Know How to Leverage AI?",
      "url": "https://aggr.aymericbeaumet.com/items/jxnl-co/2025-09-11-do-your-engineers-know-how-to-leverage-ai/"
    },
    {
      "content_html": "<h2>What Is the Coding Agents Speaker Series?</h2>\n<p>I hosted a series of conversations with the teams behind the most successful coding agents in the industry—Cognition (Devin), Sourcegraph (Amp), Cline, and Augment. <strong>Coding agents are the most economically viable agents today</strong>—they're generating real revenue, being used daily by professional developers, and solving actual business problems at scale.</p>\n<p>This makes them incredibly important to study. While other agent applications remain largely experimental, coding agents have crossed the chasm into production use. The patterns and principles these teams discovered aren't just theoretical—they're battle-tested insights from systems processing millions of real-world tasks.</p>\n<p>This series captures those hard-won lessons, revealing what works and what doesn't when building agents that actually deliver economic value.</p>\n<p>Related Series</p>\n<p><strong><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://jxnl.co/writing/2025/08/28/context-engineering-index/\">Context Engineering Series</a></strong>: Technical implementation patterns for agentic RAG systems, including tool response design, context management, and system architecture. This Speaker Series provides strategic insights, while Context Engineering offers implementation details.</p>\n<pre><code class=\"language-text\"><span class=\"code-snippet\">**[RAG Master Series](./rag-series-index.md)**: Comprehensive guide to retrieval-augmented generation systems. Many coding agent insights (like why simple approaches beat complex ones) apply directly to RAG system design and optimization.\n</span></code></pre>\n<h2>What Are the Key Insights?</h2>\n<p>The most striking pattern across all conversations is the <strong>retreat from complexity</strong>. Every team independently discovered that simpler approaches consistently outperform elaborate architectures:</p>\n<ul>\n<li><strong>Grep beats embeddings</strong> for code search in most scenarios</li>\n<li><strong>Single agents outperform multi-agent systems</strong> due to context coherence</li>\n<li><strong>Direct exploration beats RAG</strong> for understanding codebases</li>\n<li><strong>Simple tools with agentic loops</strong> surpass complex retrieval engines</li>\n</ul>\n<p>This isn't about cutting corners—it's about letting model capabilities shine through rather than constraining them with unnecessary abstraraction layers.</p>\n<h2>What Key Terms Should I Know?</h2>\n<ul>\n<li><strong>Agentic Retrieval</strong>: Models deciding which tools to invoke to fetch context, rather than pre-computed retrieval</li>\n<li><strong>Context Coherence</strong>: Maintaining narrative integrity throughout long coding sessions</li>\n<li><strong>Tool-Calling Loops</strong>: Persistent agents that make multiple tool calls to build understanding</li>\n<li><strong>Context Pollution</strong>: Noisy outputs from tools that crowd out useful reasoning context</li>\n<li><strong>Sub-agents</strong>: Specialized agents for specific tasks (search, analysis) that preserve main agent context</li>\n<li><strong>Plan and Act Paradigm</strong>: Agents that first explore and plan, then execute changes</li>\n<li><strong>Context Engineering</strong>: Designing tool responses that teach agents how to navigate codebases</li>\n<li><strong>Narrative Integrity</strong>: Following coherent thought processes rather than jumping between disconnected snippets</li>\n<li><strong>The Bitter Lesson</strong>: As AI improves, complex application layers often become unnecessary overhead</li>\n</ul>\n<h2>How Have Coding Agents Evolved?</h2>\n<p><strong>Early Era (2023):</strong> Simple completions with basic retrieval and low latency requirements</p>\n<p><strong>Chat Era (2024):</strong> More complex retrieval across multiple files with human-in-the-loop workflows</p>\n<p><strong>Agentic Era (2025):</strong> Highly autonomous systems that explore codebases independently and make changes with minimal supervision</p>\n<p>The fundamental shift is from <strong>retrieval-first</strong> (fetch context, then reason) to <strong>exploration-first</strong> (reason about what to explore, then fetch). This inversion of control changes everything about how we build these systems.</p>\n<h2>What Posts Are in This Series?</h2>\n<h3>1. <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://jxnl.co/writing/2025/09/11/why-grep-beat-embeddings-in-our-swe-bench-agent-lessons-from-augment/\">Why Grep Beat Embeddings in Our SWE-Bench Agent (Lessons from Augment)</a></h3>\n<p><strong>Speaker:</strong> Colin Flaherty (Augment)</p>\n<p><strong>Core thesis:</strong> Agent persistence can overcome limitations in retrieval tools. Simple grep and find tools with agentic loops often outperform complex embedding-based retrieval for coding tasks.</p>\n<p><strong>Key insight:</strong> Agents don't make traditional RAG obsolete—they change how we should think about retrieval. Instead of complex monolithic engines, provide portfolios of simple, composable tools.</p>\n<p><strong>Practical takeaway:</strong> Don't throw away existing retrieval systems. Expose them as tools to agents for the best of both approaches.</p>\n<h3>2. <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://jxnl.co/writing/2025/09/11/why-cognition-does-not-use-multi-agent-systems/\">Why Cognition Does Not Use Multi-Agent Systems</a></h3>\n<p><strong>Speaker:</strong> Walden Yan (Cognition)</p>\n<p><strong>Core thesis:</strong> Multi-agent systems fail due to context loss and conflicting decisions. The &quot;telephone game&quot; effect between agents creates more problems than benefits for coding tasks.</p>\n<p><strong>Key insight:</strong> True collaboration requires mentally modeling what other agents know—a sophisticated capability current models lack. Single agents with proper context management consistently outperform multi-agent setups.</p>\n<p><strong>Practical takeaway:</strong> Systems should feel like a single coherent entity to users. Focus on making single agents smarter rather than coordinating multiple agents.</p>\n<h3>3. <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://jxnl.co/writing/2025/09/11/rethinking-rag-architecture-for-the-age-of-agents/\">Rethinking RAG Architecture for the Age of Agents</a></h3>\n<p><strong>Speaker:</strong> Beyang Liu (Sourcegraph)</p>\n<p><strong>Core thesis:</strong> The shift from RAG chat to agentic systems requires rethinking everything—context management, tool design, model selection. Many best practices from the chat era are now counterproductive.</p>\n<p><strong>Key insight:</strong> The agentic paradigm inverts context fetching control. Instead of pre-computing what context to inject, models decide which tools to invoke. This seemingly minor change has profound implications.</p>\n<p><strong>Practical takeaway:</strong> Question cargo-culted assumptions from the chat era. Focus on composable Unix-like tools rather than vertically integrated solutions.</p>\n<h3>4. <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://jxnl.co/writing/2025/09/11/why-i-stopped-using-rag-for-coding-agents-and-you-should-too/\">Why I Stopped Using RAG for Coding Agents (And You Should Too)</a></h3>\n<p><strong>Speaker:</strong> Nik Pash (Cline)</p>\n<p><strong>Core thesis:</strong> Embedding-based RAG creates unnecessary complexity and security risks for coding agents. Direct exploration approaches that mirror human engineering workflows produce better results.</p>\n<p><strong>Key insight:</strong> The &quot;bitter lesson&quot; applies to coding agents—as models improve, complex application layers become overhead. Simple approaches that let model capabilities shine through consistently win.</p>\n<p><strong>Practical takeaway:</strong> Agents should explore codebases like humans do: examining folder structures, reading files, following imports, and building coherent understanding through discovery.</p>\n<h2>What Are the Common Patterns?</h2>\n<h3>The Simplicity Advantage</h3>\n<p>Every successful coding agent team independently discovered that simpler approaches work better:</p>\n<ul>\n<li>Cline abandoned embedding-based RAG for direct exploration</li>\n<li>Augment found grep sufficient for SWE-Bench tasks</li>\n<li>Cognition chose single agents over multi-agent coordination</li>\n<li>Sourcegraph designed composable tools rather than monolithic systems</li>\n</ul>\n<h3>Context is King</h3>\n<p>All teams emphasize context management as the critical challenge:</p>\n<ul>\n<li>Maintaining narrative integrity throughout long sessions</li>\n<li>Avoiding context pollution from noisy tool outputs</li>\n<li>Using sub-agents to preserve main context window</li>\n<li>Implementing smart compaction strategies</li>\n</ul>\n<h3>Tool Design Matters</h3>\n<p>Successful agents use thoughtfully designed tool portfolios:</p>\n<ul>\n<li>Simple, composable tools over complex integrations</li>\n<li>Structured outputs that guide agent reasoning</li>\n<li>Clear separation between exploration and execution phases</li>\n<li>Unix philosophy of doing one thing well</li>\n</ul>\n<h2>How Should I Use This Series?</h2>\n<p><strong>If you're building coding agents:</strong></p>\n<ol>\n<li>Start with <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://jxnl.co/writing/2025/09/11/why-i-stopped-using-rag-for-coding-agents-and-you-should-too/\">Why I Stopped Using RAG</a> to understand why simple approaches win</li>\n<li>Read <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://jxnl.co/writing/2025/09/11/why-cognition-does-not-use-multi-agent-systems/\">Cognition's multi-agent lessons</a> to avoid coordination pitfalls</li>\n<li>Study <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://jxnl.co/writing/2025/09/11/rethinking-rag-architecture-for-the-age-of-agents/\">Sourcegraph's architecture principles</a> for design guidelines</li>\n<li>Review <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://jxnl.co/writing/2025/09/11/why-grep-beat-embeddings-in-our-swe-bench-agent-lessons-from-augment/\">Augment's retrieval insights</a> for tool selection strategy</li>\n</ol>\n<p><strong>If you're evaluating coding agents:</strong></p>\n<ul>\n<li>Focus on how agents handle context coherence in long sessions</li>\n<li>Evaluate tool simplicity and composability over feature complexity</li>\n<li>Test performance on real codebases, not synthetic benchmarks</li>\n<li>Prioritize systems that feel like coherent single entities</li>\n</ul>\n<p><strong>If you're researching agent architectures:</strong></p>\n<ul>\n<li>Study how successful teams balance autonomy with user control</li>\n<li>Investigate context management strategies beyond simple compaction</li>\n<li>Explore the economics of different tool design approaches</li>\n<li>Consider how these patterns might apply beyond coding domains</li>\n</ul>\n<h2>Who Is This Series For?</h2>\n<ul>\n<li><strong>Engineering teams</strong> building or evaluating coding agents</li>\n<li><strong>Product leaders</strong> understanding the ROI of AI-assisted development</li>\n<li><strong>AI researchers</strong> studying agent architecture and tool design</li>\n<li><strong>Developers</strong> curious about how these systems actually work in practice</li>\n</ul>\n<p>Each conversation includes specific implementation details, real-world metrics, and honest assessments of what works and what doesn't in production environments.</p>\n<h2>What's the Future of Coding Agents?</h2>\n<p>Based on these conversations, coding agents are evolving toward:</p>\n<p><strong>Simpler architectures</strong> that let model capabilities shine through rather than constraining them</p>\n<p><strong>Better context engineering</strong> that maintains coherence across long development sessions</p>\n<p><strong>Composable tool ecosystems</strong> following Unix principles of modularity and reuse</p>\n<p><strong>Tighter human-agent collaboration</strong> with appropriate escalation and confidence signaling</p>\n<p><strong>Domain expansion</strong> as patterns proven in coding extend to other structured knowledge work</p>\n<p>The teams building these systems are still discovering fundamental principles that will likely become standard practice within a few years. This series captures that learning in real-time.</p>\n<h2>How Do I Get Started?</h2>\n<p>Begin with <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://jxnl.co/writing/2025/09/11/why-i-stopped-using-rag-for-coding-agents-and-you-should-too/\">Nik Pash's perspective on RAG</a> to understand the core philosophy shift, then explore the specific architectural insights from each team based on your current implementation needs.</p>\n<p>The future of coding isn't just about AI writing code—it's about AI understanding and navigating the complex, structured information spaces that real software development requires. These conversations show us how to build that future effectively.</p>\n<h2>Want to Learn More?</h2>\n<p>I know these conversations cover a lot of ground—from architectural decisions to business strategy. If you want more material in this vein:</p>\n<p><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://dub.link/6wk-rag-email\">Free 6-Week RAG Email Course</a> <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://jxnl.co/writing/2025/09/11/rag-series-index/\">RAG resources</a></p>\n<p>If you want hands-on help, I recommend reaching out to my friend Nila: <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://nila.is/\">nila.is</a>. Please mention you came from me.</p>\n",
      "date_modified": "2025-09-11T00:00:00+00:00",
      "date_published": "2025-09-11T00:00:00+00:00",
      "external_url": "https://jxnl.co/writing/2025/09/11/coding-series-index/",
      "id": "urn:aggr:item:fb1f3ca75e65aad4cac433dc64b0612ec2125be1",
      "summary": "Deep insights from the teams behind Devin, Amp, Cline, and Augment on building effective coding agents. Learn why simple approaches are winning over complex architectures in autonomous coding systems.",
      "tags": [
        "programming",
        "coding agents",
        "context engineering",
        "multi-agent systems",
        "rag",
        "speaker series"
      ],
      "title": "Coding Agents Speaker Series: Lessons from Industry Leaders",
      "url": "https://aggr.aymericbeaumet.com/items/jxnl-co/2025-09-11-coding-agents-speaker-series-lessons-from-industry-leaders/"
    }
  ],
  "language": "en",
  "title": "Jason Liu",
  "version": "https://jsonfeed.org/version/1.1"
}