| Challenge: | Function units are hard to map across languages, while being the most frequent tokens. |
| Approach: | They analyze subword tokens in terms of their productivity and try to find thresholds that best distinguish function from content tokens. |
| Outcome: | The proposed method can be used to identify functional lexical units in low-resource languages with minimal annotated data. |
Similar Papers
An Empirical Study of Tokenization Strategies for Various Korean NLP Tasks (2020.aacl-main)
Copied to clipboard
| Challenge: | Traditionally, tokenization is the very first step in most text processing works. |
| Approach: | They propose to use morphological segmentation followed by BPE for Korean NLP tasks . they empirically examine what is the best tokenization strategy for Korean to/from English . |
| Outcome: | The proposed approach is best for Korean to/from English machine translation and natural language understanding tasks. |
Where are we Still Split on Tokenization? (2024.findings-eacl)
Copied to clipboard
| Challenge: | Identifying tokens is a crucial first step for many tasks in Natural Language Processing (NLP) gold tokenization is often assumed, but some work on token-level tasks is more challenging. |
| Approach: | They propose an efficient method for tokenization with subword-based language models and evaluate it on 122 languages in 20 scripts. |
| Outcome: | The proposed method performs on par with the state-of-the-art on 122 languages in 20 scripts. |
AdaptBPE: From General Purpose to Specialized Tokenizers (2026.eacl-long)
Copied to clipboard
| Challenge: | Subword tokenization methods impact performance and efficiency of large language models . generic tokens can incur inefficiencies when applying the model to specific domains or languages . |
| Approach: | They propose a subword tokenization technique that selectively replaces low-utility tokens with more relevant ones based on their frequency in an adaptation corpus. |
| Outcome: | The proposed method compresses test corpora more effectively than baselines using the same vocabulary size. |
How Important Is Tokenization in French Medical Masked Language Models? (2024.lrec-main)
Copied to clipboard
| Challenge: | Word tokenization into subword units has become the prevailing standard in the field of natural language processing (NLP) over recent years . the precise factors contributing to its success remain unclear . |
| Approach: | They propose a tokenization strategy that integrates morpheme-enriched word segmentation into existing tokenization methods. |
| Outcome: | The proposed tokenization strategy outperforms character and word tokenization but the precise factors contributing to its success remain unclear. |
Adaptive BPE Tokenization for Enhanced Vocabulary Adaptation in Finetuning Pretrained Language Models (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Current vocabulary adaptation approaches append the target domainspecific vocabulary (V DOMAIN) at the end of the PLM vocabulary. |
| Approach: | They propose a vocabulary adaptation scheme that appends a target domain-specific vocabulary (V DOMAIN) at the end of the PLM vocabulary. |
| Outcome: | The proposed approach improves by 3.57% (in terms of accuracy) and 1.87% (royal-L) over various classification and summarization tasks. |
Tokenization with Factorized Subword Encoding (2023.findings-acl)
Copied to clipboard
| Challenge: | Subword tokenization methods are often used to project subwords onto triplets . a typical tokenizer consists of 10 000s of subword mapped onto a single index . |
| Approach: | They propose a subword tokenization method that factorizes subwords onto triplets using a VQ-VAE model. |
| Outcome: | The proposed tokenization method is more appropriate and robust for morphological tasks than the commonly used byte-pair encoding (BPE) tokenization algorithm. |
Multilingual Tokenization through the Lens of Indian Languages: Challenges and Insights (2026.findings-acl)
Copied to clipboard
Maharaj Brahma, N J Karthika, Rajat Verma, Nagasai Saketh Naidu, Rohit Saluja, Maunendra Sankar Desarkar, Ganesh Ramakrishnan
| Challenge: | Existing tokenizers are often skewed towards high-resource languages limiting their effectiveness for linguistically diverse and morphologically rich languages. |
| Approach: | They evaluate multilingual tokenization across 17 Indic languages spanning 11 scripts and two language families. |
| Outcome: | The proposed method improves tokenization quality and vocabulary size in 17 languages . poor tokenization can lead to increase in sequence lengths, fragment meaningful units, weaken model's ability to capture linguistic structure and semantics. |
From Where Words Come: Efficient Regularization of Code Tokenizers Through Source Attribution (2026.acl-long)
Copied to clipboard
| Challenge: | Currently, subword tokenization is the most common approach for vocabulary building in large models. |
| Approach: | They propose to regularize training and minimize overfitting by using source-attributed BPE . they find that undertrained tokens are prone to producing unused, unusable tokens . |
| Outcome: | The proposed techniques reduce the number of under-trained tokens while maintaining the same inference procedure as with regular BPE. |
Lexically Grounded Subword Segmentation (2024.emnlp-main)
Copied to clipboard
| Challenge: | Statistical word segmentation algorithms have remained a thorn in the side of many researchers. |
| Approach: | They propose to use unsupervised morphological analysis with Morfessor as pre-tokenization and an algebraic method for obtaining subword embeddings grounded in a word embeddable space. |
| Outcome: | The proposed methods improve morphological plausibility and Rényi efficiency on part-of-speech tagging and machine translation tasks. |
Tokenization Impacts Multilingual Language Modeling: Assessing Vocabulary Allocation and Overlap Across Languages (2023.findings-acl)
Copied to clipboard
| Challenge: | Multilingual language models perform surprisingly well in a variety of NLP tasks for diverse languages. |
| Approach: | They propose to evaluate the quality of lexical representation and vocabulary overlap observed in sub-word tokenizers. |
| Outcome: | The proposed criteria show that the overlap of vocabulary across languages can be detrimental to certain downstream tasks. |