Papers by Tomohiro Sawada
Train It and Forget It: Merge Lists are Unnecessary for BPE Inference in Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing work shows that byte-pair encoding (BPE) tokenization uses a learned merge list to iteratively combine subword units into tokens during inference time. |
| Approach: | They propose to use a standard byte-pair encoding algorithm to pair a learned token vocabulary with a detailed merge list to compress text. |
| Outcome: | The proposed algorithms differ from the encoding process during training and show that the targetted deviation from merge lists exhibits significant degradation in language model performance. |