Why does Dutch cost more tokens?

Tokenizers learn their vocabulary mostly from English text. Dutch words get split into pieces more often, and every piece is a token.

Were Claude and Gemini measured?

Not yet. Their tokenizers aren't public; counting only works through their API. We'll add that measurement.

Can I use these numbers?

Yes, with attribution (guide.wyst.nl). The data is in data/tokenindex.yaml and the script in scripts/tokenindex.py.

Token index: how much extra does Dutch cost?

Our measurement: the same text costs 24% to 60% more tokens in Dutch than in English, depending on the tokenizer. With open method and data.

In short

The same text costs 24% more tokens in Dutch than in English with the tokenizer of current OpenAI models, and up to 58-60% more with Qwen3 and the old GPT-4 tokenizer. Budget about 1.45 tokens per Dutch word with OpenAI. Measured on 2,009 parallel sentences (FLORES-200), November 2026.

Fact-checked on 2 November 2026

Dutch vs. English: extra tokens for the same text

OpenAI o200k_baseGPT-4o tot en met GPT-6
+24%
Mistral TekkenMistral Small 3.1 en verwante Mistral-modellen
+41%
DeepSeek-V3DeepSeek-V3 en R1
+49%
Qwen3Qwen3-modellen (Alibaba, open gewichten)
+58%
OpenAI cl100k_baseGPT-4 en GPT-3.5 (2023)
+60%

Tokenizer	Tokens per Dutch word	Dutch	German	French	Spanish
OpenAI o200k_base	1.45	1.24×	1.31×	1.36×	1.32×
Mistral Tekken	1.70	1.41×	1.30×	1.31×	1.32×
DeepSeek-V3	1.74	1.49×	1.51×	1.54×	1.50×
Qwen3	1.89	1.58×	1.55×	1.57×	1.52×
OpenAI cl100k_base	1.88	1.60×	1.58×	1.60×	1.55×

Corpus: FLORES-200 (dev + devtest) (2,009 sentences, CC BY-SA 4.0). Measured on 2026-11-02.

What this means for you

Cost estimates: with OpenAI, budget 1.45 tokens per Dutch word. On formal text (legal texts) the gap grows to 37%.
Open models: Mistral (+41%) and DeepSeek (+49%) are less efficient for Dutch than OpenAI. Mistral is European, but not more efficient with Dutch.
Context window: more tokens also means less Dutch text fits in the same window.

Method

Text: FLORES-200, 2,009 sentences from Wikipedia articles, translated into each language by professional translators. Same content, so a fair comparison.
Counting: each sentence tokenized separately without special tokens, summed per language. OpenAI via tiktoken, the open models via their official tokenizer.json.
Check: on the Universal Declaration of Human Rights (formal text), o200k came out at +37% and 1.46 tokens per word.
Limitation: encyclopedic text. Chat messages and code may differ. Claude and Gemini are still missing.