ttok 1.0
Simon Willison's Weblog (LLM / AI engineering) · Oct 9, 2026
Claim: ttok 1.0 ships with GPT-5/GPT-6 as default tokenizer; experiments show GPT-6 variants match GPT-5 family tokenization
Evidence: William Liu experiment across seven GPT models (5.5, 5.6 Sol/Terra/Luna, 6 Astra/Sol/Luna) showing identical token counts (44,794) on 31 fixtures; commit referenced but not linked; author notes OpenAI has not officially confirmed tokenizer parity
Rationale
ttok is a tokenizer utility that Timothy's monorepo may use for token budgeting, making it applicable to his stack. However, evidence is anecdotal (one experiment from a third party, no official OpenAI confirmation, no reproducible test suite linked). Durability is low because tokenizer defaults are volatile across model releases—the 1.0 bump is reactive, not structural. The release is maintenance-driven, not a capability uplift. No real hype. Source baseline held at 2 (plausible indie project).