ReBPE: Iteratively Improving the Internal Structure of a Structured Tokeniser by Mining its Internal Structure (2026.findings-eacl)
Copied to clipboard
| Challenge: | Recent work explores pruning merges from BPE subword tokenisers using corpus data as a signal for which merges to prune. |
| Approach: | They propose a pruning algorithm that inspects the effects left by pruning . they propose reification of the tokenisers and a new pruning algorithm . |
| Outcome: | The proposed algorithm outperforms the original BPE-knockout algorithm on alignment in all 14 languages tested by over 11% F1 on average. |
Similar Papers
BPE-knockout: Pruning Pre-existing BPE Tokenisers with Backwards-compatible Morphological Semi-supervision (2024.naacl-long)
Copied to clipboard
| Challenge: | Subword tokenisation is a mainstay in natural language processing (NLP) it allows the representation of an infinite space of text with a finite set of units. |
| Approach: | They propose to use byte-pair encoding to represent an infinite space of text with a finite set of units by removing subwords from the BPE vocabulary without impeding further use of merges that relied on them. |
| Outcome: | The proposed method improves token-based tasks in English, Dutch and German while saving training costs. |
MorphBPE: Morphology-Aware Tokenization for Efficient LLM Training (2026.findings-acl)
Copied to clipboard
| Challenge: | Tokenization is a key design choice in modern NLP systems and a critical bottleneck for multilingual Large Language Models. |
| Approach: | They propose a tokenization extension that constrains merge operations to respect morpheme boundaries while preserving inference. |
| Outcome: | The proposed tokenization improves morphological coherence and language model cross-entropy in four languages. |
BPE Gets Picky: Efficient Vocabulary Refinement During Tokenizer Training (2024.emnlp-main)
Copied to clipboard
| Challenge: | Tokenization is a relatively understudied area, but it can greatly impact model performance and efficiency. |
| Approach: | They propose a modified BPE tokenizer that removes merges that leave intermediate "junk" tokens from the vocabulary. |
| Outcome: | The proposed method improves vocabulary efficiency, eliminates under-trained tokens, and does not compromise text compression. |
AdaptBPE: From General Purpose to Specialized Tokenizers (2026.eacl-long)
Copied to clipboard
| Challenge: | Subword tokenization methods impact performance and efficiency of large language models . generic tokens can incur inefficiencies when applying the model to specific domains or languages . |
| Approach: | They propose a subword tokenization technique that selectively replaces low-utility tokens with more relevant ones based on their frequency in an adaptation corpus. |
| Outcome: | The proposed method compresses test corpora more effectively than baselines using the same vocabulary size. |
Adaptive BPE Tokenization for Enhanced Vocabulary Adaptation in Finetuning Pretrained Language Models (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Current vocabulary adaptation approaches append the target domainspecific vocabulary (V DOMAIN) at the end of the PLM vocabulary. |
| Approach: | They propose a vocabulary adaptation scheme that appends a target domain-specific vocabulary (V DOMAIN) at the end of the PLM vocabulary. |
| Outcome: | The proposed approach improves by 3.57% (in terms of accuracy) and 1.87% (royal-L) over various classification and summarization tasks. |
Teaching Old Tokenizers New Words: Efficient Tokenizer Adaptation for Pretrained Models (2026.findings-eacl)
Copied to clipboard
| Challenge: | Extending existing vocabulary is a widely used step in adapting pre-trained language models to new domains or languages. |
| Approach: | They propose to extend a pre-trained tokenizer by continuing the BPE merge learning process on new data. |
| Outcome: | The proposed method improves tokenization efficiency and improves model utilization. |
Exploring morphology-aware tokenization: A case study on Spanish language modeling (2025.emnlp-main)
Copied to clipboard
| Challenge: | a recent study shows that subword tokenization improves performance of neural language models. |
| Approach: | They propose a linguistically grounded approach to train a tokenizer on morphologically segmented data. |
| Outcome: | The proposed tokenizer improves on a Spanish language model with morphological information. |
From Where Words Come: Efficient Regularization of Code Tokenizers Through Source Attribution (2026.acl-long)
Copied to clipboard
| Challenge: | Currently, subword tokenization is the most common approach for vocabulary building in large models. |
| Approach: | They propose to regularize training and minimize overfitting by using source-attributed BPE . they find that undertrained tokens are prone to producing unused, unusable tokens . |
| Outcome: | The proposed techniques reduce the number of under-trained tokens while maintaining the same inference procedure as with regular BPE. |
R-BPE: Improving BPE-Tokenizers with Token Reuse (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large pretrained language models prioritize high-resource languages in their vocabularies, leaving others with poor coverage. |
| Approach: | They propose a framework that reuses existing tokenizers and creates ID-based maps to resolve the new tokens of the chosen language. |
| Outcome: | The proposed framework reduces subword fertility by 24.4% on Arabic models and preserves performance on EnglishMMLU. |
Tokenization Falling Short: On Subword Robustness in Large Language Models (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Language models typically tokenize raw text into sequences of subword identifiers from a predefined vocabulary. |
| Approach: | They propose to tokenize raw text into sequences of subword identifiers from a predefined vocabulary . they also investigate the challenges and their impact on large language models . |
| Outcome: | The proposed model can mitigate tokenization issues, but still suffer from typos and other variations. |