Simon Willison released ttok 1.0, updating the default tokenizer from GPT-4 to the GPT-5/GPT-6 family. The change was prompted by an experiment by William Liu suggesting GPT-5 and GPT-6 models share the same tokenizer, though OpenAI has not officially confirmed this.
SynthePulse Insight · AI deep reading
Version 1 · 1 source
Simon Willison released ttok 1.0 on October 9, 2026, switching the default tokenizer from GPT-4 to GPT-5/GPT-6. This decision rests on an experiment by William Liu, not on official confirmation from OpenAI—whether GPT-6 shares the same tokenizer as the GPT-5 series remains an open question to this day.
Simon Willison released ttok 1.0 on October 9, 2026, a tool for counting tokens and truncating text. The core change in this release is not a feature expansion but a switch of the default tokenizer: from GPT-4 to GPT-5/GPT-6. Willison himself recounts that after releasing ttok 0.4 and upgrading via uv tool upgrade ttok, he discovered the new version defaulted to the GPT-4 tokenizer, whereas he believed it should default to the GPT-5/GPT-6 tokenizer.
This judgment does not come from official OpenAI documentation. Willison explicitly states that OpenAI has not yet confirmed whether GPT-6 uses the same tokenizer as the GPT-5 series, and that there is an issue expressing dissatisfaction about this. In other words, ttok 1.0's default rests on a premise that has not received first-party confirmation.
Willison cites a commit by William Liu, which reports an experiment concluding that these tokenizers are very likely the same. The specific result of the experiment is: seven GPT models—GPT-5.5, GPT-5.6 Sol/Terra/Luna, GPT-6 Astra/Sol/Luna—all reported 44,794 tokens and were consistent with one another across all 31 test samples. The experiment also states that GPT-6 introduced no change in input counts on this corpus.
This set of data constitutes the direct basis for ttok 1.0's default switch. But its level of evidence needs to be noted: this is William Liu's experimental conclusion, not an official OpenAI statement. Whether the 31 test samples covered by the experiment are representative is not explained in the source.
From this experiment it can be inferred that if the GPT-6 and GPT-5 series tokenizers are indeed the same, then ttok's default switch will not change token counts on these models. The confidence of this inference is limited by the experiment itself: it verifies "very likely the same," not "confirmed the same."
The reverse risk is equally clear: if the GPT-6 tokenizer is in fact different, ttok 1.0's default setting could lead to token count discrepancies. For users who rely on ttok for token counting and truncation, this means the accuracy of the default on GPT-6 depends on a premise that OpenAI has not yet confirmed.
The case of ttok 1.0 demonstrates a common engineering decision pattern: when official documentation is absent, developers set tool defaults based on reproducible third-party experiments. Willison regarded switching the default as a reasonable occasion for releasing 1.0, and this choice is itself pragmatic.
But defaults have an amplifying effect. Once ttok 1.0 is widely used, its default tokenizer choice will silently affect the token count results of a large number of downstream tasks. Until OpenAI confirms the GPT-6 tokenizer, this default always carries an uncertainty that has not been eliminated.
All facts in this article come from Simon Willison's release notes of October 9, 2026, and the William Liu experiment he cites. The release date of ttok 1.0, the default tokenizer switch, OpenAI's lack of confirmation of the GPT-6 tokenizer, the seven models reporting 44,794 tokens, and the consistency across 31 test samples are all source statements; among them, "the tokenizers are very likely the same" is William Liu's experimental conclusion, not official OpenAI confirmation. Inferences about the consequences of the switch are explicitly labeled as inferences.
ttok 1.0's default tokenizer switch is an engineering decision driven by a third-party experiment and not yet confirmed by OpenAI. It holds on the corpus covered by the experiment, but its accuracy on GPT-6 still depends on an open question: whether GPT-6 truly shares the same tokenizer as the GPT-5 series.
Primary report
Primary source