Length-aware Byte Pair Encoding for Mitigating Over-segmentation in Korean Machine Translation (2024.findings-acl)
Copied to clipboard
Jungseob Lee, Hyeonseok Moon, Seungjun Lee, Chanjun Park, Sugyeong Eo, Hyunwoong Ko, Jaehyung Seo, Seungyoon Lee, Heuiseok Lim
| Challenge: | Byte Pair Encoding (BPE) is an effective approach in machine translation across several languages, but it is prone to over-segmentation in Korean, an agglutinative and morphologically rich language. |
| Approach: | They propose a new method that incorporates long words into the Korean vocabulary by strategically preserving morphological information and reducing semantic confusion. |
| Outcome: | The proposed method outperforms BPE and surpasses state-of-the-art morpheme-aware tokenization methods. |
Similar Papers
Jamo Pair Encoding: Subcharacter Representation-based Extreme Korean Vocabulary Compression for Efficient Subword Tokenization (2020.lrec-1)
Copied to clipboard
| Challenge: | a vocabulary size problem is unsolved in multilingual pre-training tasks. |
| Approach: | They propose two algorithms that increase the elasticity of budget for building the vocabulary in Byte-Pair Encoding inspired tokenizers, and reduce the cost of supporting Korean in a multilingual model. |
| Outcome: | The proposed method reduces the cost of supporting Korean in a multilingual model. |
Investigating the Effectiveness of BPE: The Power of Shorter Sequences (D19-1)
Copied to clipboard
| Challenge: | Byte-Pair Encoding (BPE) is an unsupervised sub-word tokenization technique, but its reasons for its effectiveness are not well understood. |
| Approach: | They link BPE to the broader family of dictionary-based compression algorithms and compare it with other members of this family. |
| Outcome: | The proposed method is compared with dictionary-based compression algorithms and improves on a fixed vocabulary size budget. |
Parity-Aware Byte-Pair Encoding: Improving Cross-lingual Fairness in Tokenization (2026.acl-long)
Copied to clipboard
Negar Foroutan, Clara Meister, Debjit Paul, Joel Niklaus, Sina Ahmadi, Antoine Bosselut, Rico Sennrich
| Challenge: | Tokenization is the first step of most NLP pipelines. |
| Approach: | They propose a parity-aware byte pair encoder that maximizes the compression gain of the currently worst-compressed language for cross-lingual parity. |
| Outcome: | a new algorithm reduces tokenization inequality by 89% compared to classical BPE . the proposed algorithm is based on a fair-max rule that maximizes the compression gain of the currently worst-compressed language . |
BPE-knockout: Pruning Pre-existing BPE Tokenisers with Backwards-compatible Morphological Semi-supervision (2024.naacl-long)
Copied to clipboard
| Challenge: | Subword tokenisation is a mainstay in natural language processing (NLP) it allows the representation of an infinite space of text with a finite set of units. |
| Approach: | They propose to use byte-pair encoding to represent an infinite space of text with a finite set of units by removing subwords from the BPE vocabulary without impeding further use of merges that relied on them. |
| Outcome: | The proposed method improves token-based tasks in English, Dutch and German while saving training costs. |
An Empirical Study of Tokenization Strategies for Various Korean NLP Tasks (2020.aacl-main)
Copied to clipboard
| Challenge: | Traditionally, tokenization is the very first step in most text processing works. |
| Approach: | They propose to use morphological segmentation followed by BPE for Korean NLP tasks . they empirically examine what is the best tokenization strategy for Korean to/from English . |
| Outcome: | The proposed approach is best for Korean to/from English machine translation and natural language understanding tasks. |
MorphBPE: Morphology-Aware Tokenization for Efficient LLM Training (2026.findings-acl)
Copied to clipboard
| Challenge: | Tokenization is a key design choice in modern NLP systems and a critical bottleneck for multilingual Large Language Models. |
| Approach: | They propose a tokenization extension that constrains merge operations to respect morpheme boundaries while preserving inference. |
| Outcome: | The proposed tokenization improves morphological coherence and language model cross-entropy in four languages. |
AdaptBPE: From General Purpose to Specialized Tokenizers (2026.eacl-long)
Copied to clipboard
| Challenge: | Subword tokenization methods impact performance and efficiency of large language models . generic tokens can incur inefficiencies when applying the model to specific domains or languages . |
| Approach: | They propose a subword tokenization technique that selectively replaces low-utility tokens with more relevant ones based on their frequency in an adaptation corpus. |
| Outcome: | The proposed method compresses test corpora more effectively than baselines using the same vocabulary size. |
Exploring morphology-aware tokenization: A case study on Spanish language modeling (2025.emnlp-main)
Copied to clipboard
| Challenge: | a recent study shows that subword tokenization improves performance of neural language models. |
| Approach: | They propose a linguistically grounded approach to train a tokenizer on morphologically segmented data. |
| Outcome: | The proposed tokenizer improves on a Spanish language model with morphological information. |
BPE-Dropout: Simple and Effective Subword Regularization (2020.acl-main)
Copied to clipboard
| Challenge: | Subword segmentation is widely used to address the open vocabulary problem in machine translation. |
| Approach: | They propose a method that stochastically corrupts the segmentation procedure of BPE and produces multiple segmentations within the same fixed BPE framework. |
| Outcome: | The proposed method produces multiple segmentations within the same fixed BPE framework. |
Adaptive BPE Tokenization for Enhanced Vocabulary Adaptation in Finetuning Pretrained Language Models (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Current vocabulary adaptation approaches append the target domainspecific vocabulary (V DOMAIN) at the end of the PLM vocabulary. |
| Approach: | They propose a vocabulary adaptation scheme that appends a target domain-specific vocabulary (V DOMAIN) at the end of the PLM vocabulary. |
| Outcome: | The proposed approach improves by 3.57% (in terms of accuracy) and 1.87% (royal-L) over various classification and summarization tasks. |