Keyboard shortcuts

Scrolling

dorCtrl+d
Scroll down
uorCtrl+u
Scroll up
Ctrl+e
One line down
Ctrl+y
One line up
gtheng
Top of article
G
Bottom of article

Articles

j
Older article
k
Newer article
O
Open original
H
Open hackernews
R
Open reddit

Lists

jor↓
Next item
kor↑
Previous item
gtheng
First item
G
Last item
oorEnter
Open selected item

Go to

gthenforgtheni
Feed
gthen1–9
Open feed entry
gthenb
Browse
gthenp
Preferences

Search and help

⌘+kCtrl+kor/
Global search
?
Show this help
Esc
Close this help

ttok 1.0

simonwillison.net original

I released ttok 0.4, ran uv tool upgrade ttok, piped a file into the new version... and realized that it was defaulting to the GPT-4 tokenizer when it should very clearly default to GPT-5/GPT-6 instead!

I figured switching the default was a reasonable excuse to finally ship a 1.0.

OpenAI haven't actually confirmed that GPT-6 uses the same tokenizer as the GPT-5 family yet - there's an angry issue about it - but I found this commit by William Liu which reports on an experiment he ran confirming that the tokenizers are likely the same:

All seven GPT models (5.5, 5.6 Sol/Terra/Luna, 6 Astra/Sol/Luna) report 44,794 tokens and match each other on every one of the 31 fixtures. GPT-6 introduces no input-count change on this corpus.