Papers by Nancy Hamdan
R-BPE: Improving BPE-Tokenizers with Token Reuse (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large pretrained language models prioritize high-resource languages in their vocabularies, leaving others with poor coverage. |
| Approach: | They propose a framework that reuses existing tokenizers and creates ID-based maps to resolve the new tokens of the chosen language. |
| Outcome: | The proposed framework reduces subword fertility by 24.4% on Arabic models and preserves performance on EnglishMMLU. |