IndiaSome popular languages need up to 5 times more AI tokens than others.
More tokens mean higher bills, slower answers and less room for documents and conversation history.
Tokenizers favour English, so Burmese, Khmer and Lao fare worst, and across 200 languages one model’s gap grew 17 times, researchers found.
Script matters, not only rarity: Latin-script Asturian, with few speakers, fares well, the Coalition for Language Equity in AI Research found.
How each outlet framed it
- Rest of World
- frames the AI cost gap as a hidden 'tokenizer tax', measuring 1.5–5x token cost gaps across the top 20 languages
Sources: Rest of World