InjazAI

Simon Willison released ttok 1.0 after changing its default tokenizer from GPT-4 to the GPT-5/GPT-6 family, based on an experiment suggesting GPT-6 shares the same tokenizer.

1 min read

ttok 1.0 Switches Default Tokenizer to GPT-5/GPT-6 Family

What's new in ttok 1.0?

FAQ

What is ttok?

An open-source tool by Simon Willison that counts tokens in text before sending it to language models, helping estimate cost and context limits.

Why did the default tokenizer change in 1.0?

The previous version defaulted to the GPT-4 tokenizer, while an experiment suggests the GPT-5/GPT-6 family shares the same tokenizer, so the default now matches newer models.

Has OpenAI confirmed GPT-6 uses the GPT-5 tokenizer?

No, OpenAI has not officially confirmed it yet, but the experiment cited in the source suggests they match based on identical results across 31 fixtures.

Should MENA teams update now?

If teams rely on ttok for cost estimates, updating to 1.0 gives figures closer to modern models, while always verifying against OpenAI's official documentation.

Source: Simon Willison (LLM & tools)

AI-assisted summary of the source above. High-confidence stories publish automatically; others wait for review. Facts that are not in the source are not added.