{ "@context": "https://schema.org", "@id": "https://aggr.aymericbeaumet.com/items/dbreunig-com/2026-05-10-the-cost-of-overfitting-the-harness/#webpage", "@type": [ "WebPage", "ArchiveComponent" ], "articleBody": "OpenAI [winding down fine tuning](https://x.com/bradenjhancock/status/2053309599248453999?s=20) is an interesting development and one to watch.\n\nOn one hand, model maximalists will argue the largest models keep getting better at more things, so the need to adjust the weights of them is less necessary.\n\nOn the other hand, the big labs keep pushing their models to a handful of use cases while training their harness designs into the model, rendering them less generalized. There’s an argument *this is fine*, because coding and reasoning abilities will solve most other problems.\n\nBut what we end up with are models build for their own harnesses. [Mario Zechner](https://x.com/badlogicgames/status/2052496187006054847?s=20) was wrestling with GPT in the OSS [Pi harness](https://pi.dev) this week, trying to wrangle out specific in-harness behaviors, with Claude fighting him every step of the way.\n\nIf this continues, there’s a world where 3rd party harnesses become less valuable when used with frontier lab models because the [1st party harness behavior is already *baked in*](https://www.dbreunig.com/2025/06/03/comparing-system-prompts-across-claude-versions.html). And there’s no longer a fine tuning escape hatch to generalize this behavior away.\n\nIn this world, frontier models will resemble appliances, not general platforms[^1]. With their harness trained in and no ability to adjust it? This might make application building easier for some enterprises, but the trade off is lock in. For many, improved reliability will be worth it.\n\n[^1]: I’m reminded of John Siracusa’s “[Naked Robotic Core](https://hypercritical.fireside.fm/86)” model for the iPhone, that it ideally is a common denominator device that can support many shapes of applications and interfaces.\n", "authors": [ "Drew Breunig" ], "captured_at": "2026-09-27T19:29:15.293681927+00:00", "category": "programming", "content_html": "

OpenAI winding down fine tuning is an interesting development and one to watch.

\n

On one hand, model maximalists will argue the largest models keep getting better at more things, so the need to adjust the weights of them is less necessary.

\n

On the other hand, the big labs keep pushing their models to a handful of use cases while training their harness designs into the model, rendering them less generalized. There’s an argument this is fine, because coding and reasoning abilities will solve most other problems.

\n

But what we end up with are models build for their own harnesses. Mario Zechner was wrestling with GPT in the OSS Pi harness this week, trying to wrangle out specific in-harness behaviors, with Claude fighting him every step of the way.

\n

If this continues, there’s a world where 3rd party harnesses become less valuable when used with frontier lab models because the 1st party harness behavior is already baked in. And there’s no longer a fine tuning escape hatch to generalize this behavior away.

\n

In this world, frontier models will resemble appliances, not general platforms1. With their harness trained in and no ability to adjust it? This might make application building easier for some enterprises, but the trade off is lock in. For many, improved reliability will be worth it.

\n
\n
    \n
  1. \n

    I’m reminded of John Siracusa’s “Naked Robotic Core” model for the iPhone, that it ideally is a common denominator device that can support many shapes of applications and interfaces. ↩

    \n
    2. \n
\n
\n", "content_markdown": "OpenAI [winding down fine tuning](https://x.com/bradenjhancock/status/2053309599248453999?s=20) is an interesting development and one to watch.\n\nOn one hand, model maximalists will argue the largest models keep getting better at more things, so the need to adjust the weights of them is less necessary.\n\nOn the other hand, the big labs keep pushing their models to a handful of use cases while training their harness designs into the model, rendering them less generalized. There’s an argument *this is fine*, because coding and reasoning abilities will solve most other problems.\n\nBut what we end up with are models build for their own harnesses. [Mario Zechner](https://x.com/badlogicgames/status/2052496187006054847?s=20) was wrestling with GPT in the OSS [Pi harness](https://pi.dev) this week, trying to wrangle out specific in-harness behaviors, with Claude fighting him every step of the way.\n\nIf this continues, there’s a world where 3rd party harnesses become less valuable when used with frontier lab models because the [1st party harness behavior is already *baked in*](https://www.dbreunig.com/2025/06/03/comparing-system-prompts-across-claude-versions.html). And there’s no longer a fine tuning escape hatch to generalize this behavior away.\n\nIn this world, frontier models will resemble appliances, not general platforms[^1]. With their harness trained in and no ability to adjust it? This might make application building easier for some enterprises, but the trade off is lock in. For many, improved reliability will be worth it.\n\n[^1]: I’m reminded of John Siracusa’s “[Naked Robotic Core](https://hypercritical.fireside.fm/86)” model for the iPhone, that it ideally is a common denominator device that can support many shapes of applications and interfaces.\n", "dateCreated": "2026-09-27T19:29:15.293681927+00:00", "date_created": "2026-05-10T16:26:00+00:00", "date_modified": "2026-09-15T15:44:24+00:00", "date_published": "2026-05-10T16:26:00+00:00", "description": "As labs train their own harness behavior into models and drop fine-tuning, frontier models start to look like appliances, not platforms.", "external_url": "https://www.dbreunig.com/2026/05/10/overfitting-the-harness.html", "git": { "commit": "ebb0941c5c06a2ab2377342d607535262a3e969e", "path": "items/dbreunig-com/2026/05/2026-05-10-the-cost-of-overfitting-the-harness.md", "permalink": "https://github.com/aymericbeaumet/aggr-instance/blob/ebb0941c5c06a2ab2377342d607535262a3e969e/items/dbreunig-com/2026/05/2026-05-10-the-cost-of-overfitting-the-harness.md" }, "id": "urn:aggr:item:9f959f71b9a36470722cbfbb6a40e148c6b52ece", "isBasedOn": { "@id": "https://www.dbreunig.com/2026/05/10/overfitting-the-harness.html" }, "mainEntity": { "@id": "https://www.dbreunig.com/2026/05/10/overfitting-the-harness.html", "@type": "CreativeWork", "archivedAt": { "@id": "https://aggr.aymericbeaumet.com/items/dbreunig-com/2026-05-10-the-cost-of-overfitting-the-harness/#webpage" }, "author": [ { "@type": "Person", "name": "Drew Breunig" } ], "dateModified": "2026-09-15T15:44:24+00:00", "datePublished": "2026-05-10T16:26:00+00:00", "headline": "The Cost of Overfitting the Harness", "timeRequired": "PT2M", "url": "https://www.dbreunig.com/2026/05/10/overfitting-the-harness.html", "wordCount": 247 }, "name": "Archived snapshot: The Cost of Overfitting the Harness", "preview": null, "reading_minutes": 2, "replicated_at": null, "source": "dbreunig-com", "source_name": "Drew Breunig", "summary": "As labs train their own harness behavior into models and drop fine-tuning, frontier models start to look like appliances, not platforms.", "tags": [], "temporalCoverage": "2026-09-27T19:29:15.293681927+00:00", "title": "The Cost of Overfitting the Harness", "url": "https://aggr.aymericbeaumet.com/items/dbreunig-com/2026-05-10-the-cost-of-overfitting-the-harness/", "word_count": 247 }