Findings · measured 27 Oct 2026

How wrong is “four characters per token”? It depends what you write.

The 4-characters rule underestimates Tamil by 5.35× on cl100k_base (0.74 characters per token). For English news prose on o200k_base it overestimates: 0.84× (4.68 characters per token).

01 · One language, two tokenizers

Tamil on cl100k_base vs o200k_base.

cl100k_base · GPT-4, GPT-3.5

5.35×

0.74 chars/token · 3,352 real tokens vs estimate 627

o200k_base · GPT-4o, GPT-5, gpt-oss (Groq)

1.41×

2.8 chars/token · 884 real tokens vs estimate 627

Same 15 sentences, 2,478 characters. The newer vocabulary needs 3.79× fewer tokens for them.

02 · All text, all tokenizers

Characters per token, by what you write.

o200k_basecl100k_baseLlama 3.1Gemma 3┆ the 4.0 rule

Text012345 chars/tokeno200k_base · cl100k_base · Llama 3.1 · Gemma 3

English

4.68 · 4.62 · 4.62 · 4.60

Spanish

4.42 · 3.73 · 3.73 · 4.44

French

4.16 · 3.54 · 3.54 · 3.92

German

4.27 · 3.58 · 3.58 · 4.22

Russian

3.52 · 2.05 · 3.03 · 3.65

Arabic

3.23 · 1.45 · 2.61 · 2.94

Hindi

3.06 · 1.03 · 1.89 · 3.57

Tamil

2.80 · 0.74 · 0.74 · 3.60

Chinese

1.25 · 0.83 · 1.21 · 1.40

Japanese

1.25 · 0.93 · 1.40 · 1.70

Code

4.05 · 4.07 · 4.07 · 3.36

Markdown

4.60 · 4.58 · 4.58 · 4.07

JSON

3.10 · 3.11 · 3.11 · 2.55

Emoji

2.36 · 2.02 · 2.06 · 2.85

Real tokens per estimated token (above 1× the estimate is too low)

Text	Samples	Estimate	o200k_base	cl100k_base	Llama 3.1	Gemma 3
English	15	557	0.84× 470	0.85× 476	0.85× 476	0.86× 478
Spanish	15	641	0.90× 575	1.06× 682	1.06× 682	0.89× 573
French	15	646	0.96× 617	1.12× 725	1.12× 724	1.01× 654
German	15	657	0.93× 610	1.11× 728	1.11× 727	0.94× 618
Russian	15	607	1.13× 683	1.93× 1,170	1.31× 793	1.08× 658
Arabic	15	490	1.22× 598	2.72× 1,333	1.51× 740	1.34× 658
Hindi	15	564	1.30× 732	3.87× 2,184	2.10× 1,186	1.11× 628
Tamil	15	627	1.41× 884	5.35× 3,352	5.35× 3,352	1.10× 688
Chinese	15	186	3.09× 575	4.67× 869	3.19× 594	2.76× 514
Japanese	15	241	3.12× 752	4.22× 1,018	2.80× 675	2.30× 555
Code	15	3,053	0.99× 3,011	0.98× 2,999	0.98× 2,999	1.19× 3,630
Markdown	11	1,893	0.87× 1,642	0.87× 1,651	0.87× 1,651	0.98× 1,854
JSON	12	1,358	1.28× 1,744	1.28× 1,741	1.28× 1,741	1.56× 2,118
Emoji	12	189	1.66× 314	1.95× 368	1.90× 360	1.38× 260

03 · Real usage, 50 requests

The tokens you never typed.

Short requests to free-tier models. Overhead is the reported input tokens minus the local tokenizer's count of the prompt and system text: tokens the provider adds that nobody typed.

Model	Local tokenizer	Requests	Overhead, no system	Overhead, with system	Median estimate error
openai/gpt-oss-20b	o200k_base	25	+71 (71–71, n=12)	+74 (74–74, n=13)	-76.7%
gemini-3.1-flash-lite	gemma3	25	+1 (1–1, n=12)	+1 (1–1, n=13)	-38.5%

100%

Held-out reconciliation. Overhead was measured on a fixed random half (25 requests, seed 20261027). On the other 25, the named reasons explain 100% of the input-token gap: 0 of 1,067 tokens unexplained.

Gemini's tokenizer isn't public; Gemma 3, its closest open relative, stands in for the count.

04 · Method, sources, licences

How it was measured.

estimate = ceil(Unicode characters / 4); tokens = local tokenizer count without special tokens; factor = real tokens / estimated tokens; errorPct = (estimate - real) / real * 100

200 samples, corpus built 26 Oct 2026. Every number on this page is read from results/study.json; the corpus with each sample's source and licence is study/corpus.json in the repo.

Sources & licences

NTREX-128 newstest2019-ref.eng-US.txt (CC BY-SA 4.0)
NTREX-128 newstest2019-ref.spa.txt (CC BY-SA 4.0)
NTREX-128 newstest2019-ref.fra.txt (CC BY-SA 4.0)
NTREX-128 newstest2019-ref.deu.txt (CC BY-SA 4.0)
NTREX-128 newstest2019-ref.rus.txt (CC BY-SA 4.0)
NTREX-128 newstest2019-ref.arb.txt (CC BY-SA 4.0)
NTREX-128 newstest2019-ref.hin.txt (CC BY-SA 4.0)
NTREX-128 newstest2019-ref.tam.txt (CC BY-SA 4.0)
NTREX-128 newstest2019-ref.zho-CN.txt (CC BY-SA 4.0)
NTREX-128 newstest2019-ref.jpn.txt (CC BY-SA 4.0)
Rust and Python sample code (MIT)
Overrun TypeScript source (MIT (this repo))
technical documentation (MIT)
Overrun product spec (MIT (this repo))
synthetic, written for this study (MIT (this repo))

Parallel prose: NTREX-128, CC BY-SA 4.0. Rust and Python sample code and technical documentation: MIT. Synthetic JSON and emoji posts: written for this study, MIT.

MIT License text

MIT License

Copyright (c) [COPYRIGHT HOLDER]

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

Overrun itself is MIT licensed; see the LICENSE file in its repository.