Papers with tokenizer
Unknown Script: Impact of Script on Cross-Lingual Transfer (2024.naacl-srw)
Copied to clipboard
| Challenge: | Existing models for high-resource languages are not available for all languages, and the vast majority of the world's languages are excluded from these models. |
| Approach: | They propose to use pre-trained models to analyze the effect of the target language and its script on cross-lingual transfer. |
| Outcome: | The proposed model is based on six models pre-trained on NER and POS tasks in the original script and romanized version. |
Trainable, Multiword-aware Linguistic Tokenization Using Modern Neural Networks (2026.eacl-srw)
Copied to clipboard
| Challenge: | Tokenization is a fundamental task in natural language processing that forms the first step of many pipelines. |
| Approach: | They propose to use a standard tokenizer trained without MWE-awareness as a baseline and a character-level SRN+CRF model to train token-level models. |
| Outcome: | The proposed tokenizers are based on a character-level and token-level sequence labeling problem and are consistent with the proposed pipelines. |
Joint Optimization of Tokenization and Downstream Model (2021.findings-acl)
Copied to clipboard
| Challenge: | Existing studies have reported that an appropriate tokenization depends on each downstream task. |
| Approach: | They propose a method to find an appropriate tokenization to a downstream task by optimizing a tokenizer and a model. |
| Outcome: | The proposed method improves on text classification and machine translation tasks. |
TransMI: A Framework to Create Strong Baselines from Multilingual Pretrained Language Models for Transliterated Data (2025.coling-main)
Copied to clipboard
| Challenge: | Existing mPLMs that handle non-transliterated data are not sufficient to train crosslingual models. |
| Approach: | They propose a framework to transliterate related languages into a common script by exploiting existing mPLMs and their tokenizer without any training. |
| Outcome: | The proposed framework can create strong baselines for data that is transliterated into a common script by exploiting an existing mPLM and its tokenizer without any training. |
Tokenization Is More Than Compression (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing tokenization approaches like Byte-Pair Encoding (BPE) have been suggested that their effectiveness stems from their ability to condense text into a relatively small number of tokens. |
| Approach: | They propose a tokenizer that segments a document’s text into the minimum number of tokens for a given vocabulary and propose fewer tokens to improve downstream performance. |
| Outcome: | The proposed tokenizers can initialize vocabulary construction and pre-tokenization, and the results show that fewer tokens lead to better performance. |
An Embarrassingly Simple Method to Mitigate Undesirable Properties of Pretrained Language Model Tokenizers (2022.acl-short)
Copied to clipboard
| Challenge: | a standard tokenizer does not cover all characters of a word but preserves key aspects of its morphological structure . a novel method to improve tokenization of pretrained language models is proposed . |
| Approach: | They propose a method to improve the tokenization of pretrained language models . they use the vocabulary of a standard tokenizer but preserves morphological structure . |
| Outcome: | The proposed method improves tokenization of pretrained language models on morphological gold segmentations and text classification tasks. |
Incorporating Context into Subword Vocabularies (2023.eacl-main)
Copied to clipboard
| Challenge: | Current tokenizers are trained on word frequency statistics over a corpus without considering information about co-occurrence or context. |
| Approach: | They propose a tokenizer that bakes in contextualized signal at the vocabulary creation phase to tailor subwords for their downstream use. |
| Outcome: | The proposed tokenizer is able to keep token contexts cohesive while not incurring a large price in terms of encoding efficiency or domain robustness. |
SAC: Neural Speech Codec with Semantic-Acoustic Dual-Stream Quantization (2026.acl-long)
Copied to clipboard
Wenxi Chen, Ruiqi Yan, Yushen Chen, Zhikang Niu, Ziyang Ma, Xiquan Li, Yuzhe Liang, null Wenhanlin, Shunshun Yin, Ming Tao, Xinsheng Wang, Xie Chen
| Challenge: | Existing speech codecs struggle to balance high-quality reconstruction with semantically rich representations, limiting their effectiveness in both generative and understanding tasks. |
| Approach: | They propose a neural speech codec with semantic-acoustic dual-stream quantization that disentangles semantic and acousian modeling into two dedicated streams. |
| Outcome: | The proposed codec outperforms state-of-the-art speech tokenizers in auto-propagating text-to-speech models. |
One Tokenizer To Rule Them All: Emergent Language Plasticity via Multilingual Tokenizers (2026.acl-long)
Copied to clipboard
Diana Abagyan, Alejandro R. Salamanca, Andres Felipe Cruz-Salinas, Kris Cao, Hangyu Lin, Acyr Locatelli, Marzieh Fadaee, Ahmet Üstün, Sara Hooker
| Challenge: | Existing approaches to train multilingual large language models for many languages at once are limited due to limited model capacity, scarce high-quality data, and compute constraints. |
| Approach: | They propose to use a universal tokenizer to improve language plasticity and adaptability to new languages by up to 20%. |
| Outcome: | The proposed tokenizer improves language plasticity and improves plasticity towards languages that are completely unseen in the tokenizer and pretraining, by up to 5% win rate gain. |
T-VEC: A Telecom-Specific Vectorization Model with Enhanced Semantic Understanding via Deep Triplet Loss Fine-Tuning (2025.emnlp-industry)
Copied to clipboard
| Challenge: | Generic embedding models struggle to represent telecom-specific semantics . specialized terminology and ambiguous terms often limit their utility in retrieval and downstream tasks. |
| Approach: | They propose a domain-adapted embedding model fine-tuned from a gte-Qwen2-1.5B-instruct backbone. |
| Outcome: | The proposed model outperforms MPNet, BGE, Jina and E5 on a custom benchmark . it is open source and has a triplet loss objective . |
Multilingual Word Segmentation: Training Many Language-Specific Tokenizers Smoothly Thanks to the Universal Dependencies Corpus (L18-1)
Copied to clipboard
| Challenge: | Towards language scalability, major progress has been achieved in multilingual language technology in recent years. |
| Approach: | They propose a tokenizer that can be trained from any Universal Dependencies corpus dataset . they argue that tokenization should be seen as a supervised task and scalability requires a software engineering process across languages. |
| Outcome: | The proposed tokenizer can be trained from any dataset in the corpus UD2 . the proposed software tool relies on elephant to perform the training . |
MANTa: Efficient Gradient-Based Tokenization for End-to-End Robust Language Modeling (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Subword tokenization algorithms have been an essential component of language modeling but their static nature results in important flaws that degrade the models’ downstream performance and robustness. |
| Approach: | They propose a module for Adaptive Neural TokenizAtion that is differentiable and trained end-to-end with the language model. |
| Outcome: | The proposed tokenizer improves robustness to character perturbations and out-of-domain data. |
E-magyar – A Digital Language Processing System (L18-1)
Copied to clipboard
Tamás Váradi, Eszter Simon, Bálint Sass, Iván Mittelholcz, Attila Novák, Balázs Indig, Richárd Farkas, Veronika Vincze
| Challenge: | e-magyar is a free, open, modular text processing pipeline for Hungarian . existing tools were overhauled to operate in the pipeline with a uniform encoding and run in the same Java platform. |
| Approach: | e-magyar is a free, open, modular text processing pipeline for Hungarian . it was created by a collaborative effort by the language technology community . the system is aimed at a broad range of users, from language developers to researchers . |
| Outcome: | The proposed tool is open source and available for download on the HFST framework. |
WECHSEL: Effective initialization of subword embeddings for cross-lingual transfer of monolingual language models (2022.naacl-main)
Copied to clipboard
| Challenge: | Existing methods to train large pretrained language models require more computational resources and are expensive to train in other languages. |
| Approach: | They propose a method to transfer pretrained language models to new languages using subword-based tokenization and embeddings. |
| Outcome: | The proposed method outperforms existing methods on low-resource languages and makes training large models more accessible and less damaging to the environment. |
Sudachi: a Japanese Tokenizer for Business (L18-1)
Copied to clipboard
| Challenge: | Lack of token unit compatibility is one of the critical problems of Japanese language resources. |
| Approach: | They develop a Japanese tokenizer called Sudachi and its accompanying dictionary . they use multi-granular output and normalization of notation variations to improve tokenization . |
| Outcome: | The proposed tokenizer and dictionary improve tokenization in Japanese for business use. |
Data Collection Pipeline for Low-Resource Languages: A Case Study on Constructing a Tetun Text Corpus (2024.lrec-main)
Copied to clipboard
| Challenge: | Labadain Crawler is a data collection pipeline designed to automate and optimize the process of constructing textual corpora from the web, with a specific target to low-resource languages. |
| Approach: | They propose a data collection pipeline built on top of Nutch, an open-source web crawler and data extraction framework, and a tokenizer and identifier for Tetun. |
| Outcome: | The proposed pipeline is based on Nutch, an open-source web crawler and data extraction framework, and is tested with Tetun, one of Timor-Leste’s official languages. |
Ted-Tok: Maintaining an Evolving Vocabulary for Lifelong Learning (2026.acl-long)
Copied to clipboard
| Challenge: | a static tokenizer fragments newly emerging lexical items as language evolves . as language grows, a dynamic tokenizer reduces compression efficiency and performance . |
| Approach: | They propose a Temporal Drift Tokenizer that maintains an evolving vocabulary that adapts to emerging linguistic patterns over time. |
| Outcome: | The proposed tokenizer maintains an evolving vocabulary that adapts to emerging linguistic patterns over time. |
Multi-Modal Multi-Granularity Tokenizer for Chu Bamboo Slips (2025.coling-main)
Copied to clipboard
| Challenge: | Using a multi-modal multi-granularity tokenizer, we analyze ancient Chinese scripts . a large proportion of the characters in ancient Chinese are rare or undeciphered . |
| Approach: | They propose a multi-modal multi-granularity tokenizer specifically designed for ancient Chinese scripts. |
| Outcome: | The proposed tokenizer improves on the part-of-speech tagging task on the Chu bamboo slip script. |
SPEAK: Spiking Neurons as an Entropy-Aware Tokenizer for Large Language Models (2026.acl-long)
Copied to clipboard
| Challenge: | Existing tokenizers fail to explicitly leverage historical tokenization results . large language models (LLMs) have demonstrated remarkable effectiveness across NLP tasks . |
| Approach: | They propose a tokenizer that integrates spiking neurons to explicitly leverage historical tokenization results. |
| Outcome: | The proposed tokenizer leverages historical tokenization results, but does not selectively leverage history based on contextual relevance. |
Developing a Spell and Grammar Checker for Icelandic using an Error Corpus (2022.lrec-1)
Copied to clipboard
| Challenge: | lack of datasets for spelling and grammar correction in Icelandic has caused a dearth of spell and grammar checking systems for the low-resource language. |
| Approach: | They propose to use an error corpus to guide the development of a spell and grammar checking tool for Icelandic, using tokenizer, tagger, and parser. |
| Outcome: | The proposed system is based on an error corpus at all stages of development and is reproducible for other low- or medium-resource languages and especially well-suited to morphologically rich languages. |
Romanization-based Large-scale Adaptation of Multilingual Language Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Large multilingual pretrained language models are limited by their vocabulary size and parameter budget. |
| Approach: | They explore the potential of leveraging transliteration on a massive scale to improve performance for multilingual pretrained language models. |
| Outcome: | The proposed transliteration tool outperforms other methods on low-resource languages. |
PToco: Prefix-based Token-level Collaboration Enhances Reasoning for Multi-LLMs (2025.coling-main)
Copied to clipboard
| Challenge: | Existing approaches to collaboration between multiple Large Language Models (LLMs) rely on highly capable models with strong self-reflection abilities or are limited to models sharing the same tokenizer. |
| Approach: | They propose a mechanism that enables collaboration among less capable LLMs independent of tokenizer differences. |
| Outcome: | The proposed mechanism improves performance over individual models and generalizes well across different quantities and sizes of participating models. |
Tokenization is Sensitive to Language Variation (2025.findings-acl)
Copied to clipboard
| Challenge: | Variation in language is often linked to regional, social, and contextual factors. |
| Approach: | They propose a method to estimate tokenizer impact on downstream LLM performance . they pre-train BERT models with the popular Byte-Pair Encoding algorithm . |
| Outcome: | The proposed model improves on Rényi efficiency and other metrics on language variation. |
MoVoC: Morphology-Aware Subword Construction for Ge’ez Script Languages (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Subword-based tokenization methods fail to preserve morphological boundaries, a limitation especially pronounced in low-resource, morphology complex languages such as those written in the Ge‘ez script. |
| Approach: | They propose a tokenizer that integrates supervised morphological analysis into the subword vocabulary and propose morpheme-based tokenization with Byte Pair Encoding (BPE) tokens. |
| Outcome: | The proposed tokenizer preserves morphological integrity while maintaining lexical meaning. |
Tokenization Consistency Matters for Generative Models on Extractive NLP Tasks (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Pretrained sequence-to-sequence (seq2sequ) models have been widely used to solve extractive tasks, where parts of the input are extracted to form the desired output. |
| Approach: | They propose a simple fix to tokenization inconsistency that damages extractive nature of generative models by causing performance drop and hallucination. |
| Outcome: | The proposed model performs better in both in-domain and out-of-domain datasets with a notable average of +1.7 F1 gain when a BART model is trained on SQuAD and evaluated on 8 QA datasets. |
Unsupervised Discrete Representations of American Sign Language (2024.emnlp-main)
Copied to clipboard
| Challenge: | Modern NLP models use discrete tokens to represent continuous signals, such as videos, audio, or gestures . modalities that are continuous are difficult to use with discrete models, such a LLM . |
| Approach: | They propose a method that discretizes sequences of fingerspelling signs into tokens . they also propose 'loss function' to improve interpretability of the tokens. |
| Outcome: | The proposed method improves the performance of the tokenizer on downstream tasks. |
Splintering Nonconcatenative Languages for Better Tokenization (2025.findings-acl)
Copied to clipboard
| Challenge: | Subword tokenizers are not effective in nonconcatenative languages such as Hebrew and Arabic, where morphology is encoded in root-template patterns, or Malay and Georgian, where split affixes are common. |
| Approach: | They propose a pre-processing step which rearranges text into a linear form that better represents nonconcatenative morphologies. |
| Outcome: | The proposed method is based on measures evaluating token vocabularies in Hebrew, Arabic, and Malay, and on BERT-architecture models trained for Hebrew. |
Dictionaries to the Rescue: Cross-Lingual Vocabulary Transfer for Low-Resource Languages Using Bilingual Dictionaries (2025.findings-acl)
Copied to clipboard
Haruki Sakajo, Yusuke Ide, Justin Vasselli, Yusuke Sakai, Yingtao Tian, Hidetaka Kamigaito, Taro Watanabe
| Challenge: | Existing approaches to cross-lingual vocabulary transfer face challenges when dealing with low-resource languages. |
| Approach: | They propose a dictionary-based crosslingual vocabulary transfer method that leverages bilingual dictionaries, which are available for many languages thanks to descriptive linguists. |
| Outcome: | The proposed method outperforms existing methods for low-resource languages. |
Two Counterexamples to Tokenization and the Noiseless Channel (2024.lrec-main)
Copied to clipboard
| Challenge: | Nevertheless, Rényi efficiency is not perfect and the metric is difficult to evaluate because training multiple tokenizers can be prohibitively expensive and takes days or weeks. |
| Approach: | They propose to use Rényi efficiency as an intrinsic mechanism to evaluate a tokenizer for NLP tasks without the expensive step of training multiple models with different tokenizers. |
| Outcome: | The proposed metric is better correlated to downstream model performance than a percentile frequency metric. |
Beyond Text Compression: Evaluating Tokenizers Across Scales (2025.acl-long)
Copied to clipboard
| Challenge: | Language models rely on tokenizers to convert text into machine-interpretable tokens, which shape the statistical patterns that language models learn to estimate. |
| Approach: | They propose to use Zipf's law to measure tokenizer performance by combining several metrics to capture multiple aspects of tokenizer behavior. |
| Outcome: | The proposed metrics correlate more strongly with downstream performance than text compression when modeling unseen languages. |
Exploring morphology-aware tokenization: A case study on Spanish language modeling (2025.emnlp-main)
Copied to clipboard
| Challenge: | a recent study shows that subword tokenization improves performance of neural language models. |
| Approach: | They propose a linguistically grounded approach to train a tokenizer on morphologically segmented data. |
| Outcome: | The proposed tokenizer improves on a Spanish language model with morphological information. |
From Where Words Come: Efficient Regularization of Code Tokenizers Through Source Attribution (2026.acl-long)
Copied to clipboard
| Challenge: | Currently, subword tokenization is the most common approach for vocabulary building in large models. |
| Approach: | They propose to regularize training and minimize overfitting by using source-attributed BPE . they find that undertrained tokens are prone to producing unused, unusable tokens . |
| Outcome: | The proposed techniques reduce the number of under-trained tokens while maintaining the same inference procedure as with regular BPE. |
MUTANT: A Recipe for Multilingual Tokenizer Design (2026.acl-long)
Copied to clipboard
| Challenge: | Subword tokenization schemes such as Byte Pair Encoding (BPE) are widely adopted, but their effectiveness in multilingual settings remains understudied. |
| Approach: | They propose a multilingual tokenizer that produces linguistically coherent tokens for multilingual LLMs. |
| Outcome: | The proposed tokenizer improves fertility score by 39.5% over LLaMA4 and 18% over Sutra. |