# Tokenizer cost: English vs Hindi vs Hinglish vs dev-roman

Run at 2026-10-12T16:53:56+00:00 · Python 3.13.16 · tiktoken 0.14.0, tokenizers 0.23.3, transformers 5.19.0, huggingface_hub 2.2.0, datasets 5.1.0, sentencepiece 0.2.2, protobuf 7.36.1

No special tokens. Each sentence encoded separately, counts summed. Text NFC-normalised.


## Ratio vs English: school (12 sentences)

Tokens for the same content divided by English tokens (1.00 = same cost as English).

| Tokenizer | en tokens | hi | hinglish | hi_mixed |
|---|---:|---:|---:|---:|
| tiktoken/o200k_base | 199 | 1.61 | 1.48 | 1.25 |
| tiktoken/cl100k_base | 202 | 4.91 | 1.62 | 3.25 |
| Qwen/Qwen2.5-0.5B-Instruct | 204 | 4.55 | 1.61 | 3.05 |
| Qwen/Qwen3-0.6B | 204 | 4.55 | 1.61 | 3.05 |
| mistralai/Mistral-7B-v0.1 | 215 | 4.85 | 1.68 | 3.34 |
| sarvamai/sarvam-1 | 211 | 1.30 | 1.92 | 1.18 |
| ai4bharat/IndicBARTSS | 257 | 1.13 | 1.74 | 1.15 |
| bert-base-multilingual-cased | 211 | 1.95 | 1.50 | 1.39 |

## Full results

| Tokenizer | Dataset | Lang | Sents | Tokens | Chars | Words | Tok/char | Tok/word | Ratio vs en |
|---|---|---|---:|---:|---:|---:|---:|---:|---:|
| tiktoken/o200k_base | school | en | 12 | 199 | 972 | 172 | 0.205 | 1.157 | 1.00 |
| tiktoken/o200k_base | school | hi | 12 | 321 | 990 | 201 | 0.324 | 1.597 | 1.61 |
| tiktoken/o200k_base | school | hinglish | 12 | 295 | 1134 | 199 | 0.260 | 1.482 | 1.48 |
| tiktoken/o200k_base | school | hi_mixed | 12 | 249 | 1028 | 199 | 0.242 | 1.251 | 1.25 |
| tiktoken/cl100k_base | school | en | 12 | 202 | 972 | 172 | 0.208 | 1.174 | 1.00 |
| tiktoken/cl100k_base | school | hi | 12 | 991 | 990 | 201 | 1.001 | 4.930 | 4.91 |
| tiktoken/cl100k_base | school | hinglish | 12 | 327 | 1134 | 199 | 0.288 | 1.643 | 1.62 |
| tiktoken/cl100k_base | school | hi_mixed | 12 | 657 | 1028 | 199 | 0.639 | 3.301 | 3.25 |
| Qwen/Qwen2.5-0.5B-Instruct | school | en | 12 | 204 | 972 | 172 | 0.210 | 1.186 | 1.00 |
| Qwen/Qwen2.5-0.5B-Instruct | school | hi | 12 | 929 | 990 | 201 | 0.938 | 4.622 | 4.55 |
| Qwen/Qwen2.5-0.5B-Instruct | school | hinglish | 12 | 329 | 1134 | 199 | 0.290 | 1.653 | 1.61 |
| Qwen/Qwen2.5-0.5B-Instruct | school | hi_mixed | 12 | 623 | 1028 | 199 | 0.606 | 3.131 | 3.05 |
| Qwen/Qwen3-0.6B | school | en | 12 | 204 | 972 | 172 | 0.210 | 1.186 | 1.00 |
| Qwen/Qwen3-0.6B | school | hi | 12 | 929 | 990 | 201 | 0.938 | 4.622 | 4.55 |
| Qwen/Qwen3-0.6B | school | hinglish | 12 | 329 | 1134 | 199 | 0.290 | 1.653 | 1.61 |
| Qwen/Qwen3-0.6B | school | hi_mixed | 12 | 623 | 1028 | 199 | 0.606 | 3.131 | 3.05 |
| mistralai/Mistral-7B-v0.1 | school | en | 12 | 215 | 972 | 172 | 0.221 | 1.250 | 1.00 |
| mistralai/Mistral-7B-v0.1 | school | hi | 12 | 1042 | 990 | 201 | 1.052 | 5.184 | 4.85 |
| mistralai/Mistral-7B-v0.1 | school | hinglish | 12 | 361 | 1134 | 199 | 0.318 | 1.814 | 1.68 |
| mistralai/Mistral-7B-v0.1 | school | hi_mixed | 12 | 719 | 1028 | 199 | 0.699 | 3.613 | 3.34 |
| sarvamai/sarvam-1 | school | en | 12 | 211 | 972 | 172 | 0.217 | 1.227 | 1.00 |
| sarvamai/sarvam-1 | school | hi | 12 | 274 | 990 | 201 | 0.277 | 1.363 | 1.30 |
| sarvamai/sarvam-1 | school | hinglish | 12 | 406 | 1134 | 199 | 0.358 | 2.040 | 1.92 |
| sarvamai/sarvam-1 | school | hi_mixed | 12 | 248 | 1028 | 199 | 0.241 | 1.246 | 1.18 |
| ai4bharat/IndicBARTSS | school | en | 12 | 257 | 972 | 172 | 0.264 | 1.494 | 1.00 |
| ai4bharat/IndicBARTSS | school | hi | 12 | 291 | 990 | 201 | 0.294 | 1.448 | 1.13 |
| ai4bharat/IndicBARTSS | school | hinglish | 12 | 446 | 1134 | 199 | 0.393 | 2.241 | 1.74 |
| ai4bharat/IndicBARTSS | school | hi_mixed | 12 | 296 | 1028 | 199 | 0.288 | 1.487 | 1.15 |
| bert-base-multilingual-cased | school | en | 12 | 211 | 972 | 172 | 0.217 | 1.227 | 1.00 |
| bert-base-multilingual-cased | school | hi | 12 | 411 | 990 | 201 | 0.415 | 2.045 | 1.95 |
| bert-base-multilingual-cased | school | hinglish | 12 | 317 | 1134 | 199 | 0.280 | 1.593 | 1.50 |
| bert-base-multilingual-cased | school | hi_mixed | 12 | 293 | 1028 | 199 | 0.285 | 1.472 | 1.39 |

## Tokenizer versions

- tiktoken 0.14.0 o200k_base (n_vocab=200019)
- tiktoken 0.14.0 cl100k_base (n_vocab=100277)
- Qwen/Qwen2.5-0.5B-Instruct@7ae557604adf (Qwen2Tokenizer, vocab=151665)
- Qwen/Qwen3-0.6B@c1899de289a0 (Qwen2Tokenizer, vocab=151669)
- mistralai/Mistral-7B-v0.1@27d67f1b5f57 (TokenizersBackend, vocab=32000)
- sarvamai/sarvam-1@e9607337286d (LlamaTokenizer, vocab=68096)
- ai4bharat/IndicBARTSS@4b2669d25bc2 (AlbertTokenizer, vocab=64015)
- bert-base-multilingual-cased@3f076fdb1ab6 (BertTokenizer, vocab=119547)

## Skipped

| Item | Kind | Reason |
|---|---|---|
| openlanguagedata/flores_plus | dataset | gated repo on the HF Hub; no HF access token in this environment. GatedRepoError: 401 Client Error. (Request ID: Root=1-6acd109c-2ba4570a57b625fa3992f44a;fc8b26b9-c2b4-4c12-af0e-46a0a9fc966c) |
| ai4bharat/IN22-Gen | dataset | gated repo on the HF Hub; no HF access token in this environment. GatedRepoError: 401 Client Error. (Request ID: Root=1-6acd109c-70cfda3b4a248c3870abb059;42775a68-ff76-49da-9297-5ad1a2f8724f) |
| google/gemma-3-270m | tokenizer | gated repo on the HF Hub; no HF access token in this environment. GatedRepoError: 401 Client Error. (Request ID: Root=1-6acd10a0-689f3be5188dd42e48e2850e;b7ff130e-c2e2-48b3-ab59-28dab6aef449) |
| meta-llama/Llama-3.2-1B | tokenizer | gated repo on the HF Hub; no HF access token in this environment. GatedRepoError: 401 Client Error. (Request ID: Root=1-6acd10a1-7673471a1baee5d16cdc9181;b035914f-8461-48a8-9a4f-b353b88f838f) |