Papers by Kaushik Gopalan
Comparative Analysis of the Intrinsic Metrics for Tokenizers and their effect on Downstream Tasks for Hindi and Marathi (2026.acl-long)
Copied to clipboard
| Challenge: | Various studies have shown that the performance of language models is poor in non-English or non-European languages. |
| Approach: | They propose a grapheme cluster tokenizer which shows better performance than other popular tokenizers. |
| Outcome: | The proposed tokenizers show better or competitiveness on question-answering tasks . the proposed tokenization model is highly correlated to the performance of other tokenizer models . |