I released ttok 0.4, ran uv tool upgrade ttok, piped a file into the new version... and realized that it was defaulting to the GPT-4 tokenizer when it should very clearly default to GPT-5/GPT-6 instead!
I figured switching the default was a reasonable excuse to finally ship a 1.0.
\nOpenAI haven't actually confirmed that GPT-6 uses the same tokenizer as the GPT-5 family yet - there's an angry issue about it - but I found this commit by William Liu which reports on an experiment he ran confirming that the tokenizers are likely the same:
\n\n\n", "content_markdown": "I released [ttok 0.4](https://simonwillison.net/2026/Oct/8/ttok/), ran `uv tool upgrade ttok`, piped a file into the new version... and realized that it was defaulting to the GPT-4 tokenizer when it should very clearly default to GPT-5/GPT-6 instead!\n\nI figured switching the default was a reasonable excuse to finally ship a 1.0.\n\nOpenAI haven't actually confirmed that GPT-6 uses the same tokenizer as the GPT-5 family yet - there's an [angry issue about it](https://github.com/openai/tiktoken/issues/608) - but I found [this commit](https://github.com/williamliu-ai/token-count-compare/commit/8d8a2178538bb37beb548fc37378135e4ff4b0c8) by William Liu which reports on an experiment he ran confirming that the tokenizers are likely the same:\n\n> All seven GPT models (5.5, 5.6 Sol/Terra/Luna, 6 Astra/Sol/Luna) report 44,794 tokens and match each other on every one of the 31 fixtures. GPT-6 introduces no input-count change on this corpus.\n", "dateCreated": "2026-10-09T05:58:27.034165253+00:00", "date_created": "2026-10-09T00:34:43+00:00", "date_modified": "2026-10-09T00:34:43+00:00", "date_published": "2026-10-09T00:34:43+00:00", "description": "Release: ttok 1.0 I released ttok 0.4, ran uv tool upgrade ttok, piped a file into the new version... and realized that it was defaulting to the GPT-4 tokenizer when it should very clearly default to GPT-5/GPT-6 instead! I figured…", "external_url": "https://simonwillison.net/2026/Oct/9/ttok/", "git": { "commit": "ebb0941c5c06a2ab2377342d607535262a3e969e", "path": "items/simonwillison-net/2026/10/2026-10-09-ttok-1-0.md", "permalink": "https://github.com/aymericbeaumet/aggr-instance/blob/ebb0941c5c06a2ab2377342d607535262a3e969e/items/simonwillison-net/2026/10/2026-10-09-ttok-1-0.md" }, "id": "urn:aggr:item:acfcd0e6c458fe319266d1bf39147e00bb6d2b75", "isBasedOn": { "@id": "https://simonwillison.net/2026/Oct/9/ttok/" }, "mainEntity": { "@id": "https://simonwillison.net/2026/Oct/9/ttok/", "@type": "CreativeWork", "archivedAt": { "@id": "https://aggr.aymericbeaumet.com/items/simonwillison-net/2026-10-09-ttok-1-0/#webpage" }, "dateModified": "2026-10-09T00:34:43+00:00", "datePublished": "2026-10-09T00:34:43+00:00", "headline": "ttok 1.0", "timeRequired": "PT1M", "url": "https://simonwillison.net/2026/Oct/9/ttok/", "wordCount": 136 }, "name": "Archived snapshot: ttok 1.0", "preview": null, "reading_minutes": 1, "replicated_at": null, "source": "simonwillison-net", "source_name": "Simon Willison's Weblog", "summary": "Release: ttok 1.0 I released ttok 0.4, ran uv tool upgrade ttok, piped a file into the new version... and realized that it was defaulting to the GPT-4 tokenizer when it should very clearly default to GPT-5/GPT-6 instead! I figured switching the default was a reasonable excuse to finally ship a 1.0. OpenAI haven't actually confirmed that GPT-6 uses the same tokenizer as the GPT-5 family yet - there's an angry issue about it - but I found this commit by William Liu which reports on an experiment he ran confirming that the tokenizers are likely the same: All seven GPT models (5.5, 5.6 Sol/Terra/Luna, 6 Astra/Sol/Luna) report 44,794 tokens and match each other on every one of the 31 fixtures. GPT-6 introduces no input-count change on this corpus. Tags: projects, ai, openai, generative-ai, llms, tokenization", "tags": [ "ai", "generative-ai", "llms", "openai", "projects", "tokenization" ], "temporalCoverage": "2026-10-09T05:58:27.034165253+00:00", "title": "ttok 1.0", "url": "https://aggr.aymericbeaumet.com/items/simonwillison-net/2026-10-09-ttok-1-0/", "word_count": 136 }All seven GPT models (5.5, 5.6 Sol/Terra/Luna, 6 Astra/Sol/Luna) report 44,794 tokens and match each other on every one of the 31 fixtures. GPT-6 introduces no input-count change on this corpus.
\n