Papers by Bar Gazit
Splintering Nonconcatenative Languages for Better Tokenization (2025.findings-acl)
Copied to clipboard
| Challenge: | Subword tokenizers are not effective in nonconcatenative languages such as Hebrew and Arabic, where morphology is encoded in root-template patterns, or Malay and Georgian, where split affixes are common. |
| Approach: | They propose a pre-processing step which rearranges text into a linear form that better represents nonconcatenative morphologies. |
| Outcome: | The proposed method is based on measures evaluating token vocabularies in Hebrew, Arabic, and Malay, and on BERT-architecture models trained for Hebrew. |