Papers with MUTANT-Indic
MUTANT: A Recipe for Multilingual Tokenizer Design (2026.acl-long)
Copied to clipboard
| Challenge: | Subword tokenization schemes such as Byte Pair Encoding (BPE) are widely adopted, but their effectiveness in multilingual settings remains understudied. |
| Approach: | They propose a multilingual tokenizer that produces linguistically coherent tokens for multilingual LLMs. |
| Outcome: | The proposed tokenizer improves fertility score by 39.5% over LLaMA4 and 18% over Sutra. |