Papers by DongHyeok Lee
Morpheme Matters: Morpheme-Based Subword Tokenization for Korean Language Models (2026.eacl-short)
Copied to clipboard
| Challenge: | Existing tokenizers rely on frequency-based segmentation to represent words . this often leads to inefficient token representations and oversegmentation . |
| Approach: | They propose a tokenization method that emphasizes the importance of Korean morphological structures in eojeol. |
| Outcome: | The proposed method outperforms existing tokenizers on Korean benchmark tasks and produces significantly fewer tokens per input sequence. |