FLEXITOKENS: Flexible Tokenization for Evolving Language Models (2026.findings-acl)
Copied to clipboard
| Challenge: | Widely used subword tokenizers overfragment sequences in unseen domains, languages, and scripts . inefficient tokenizer models can cause overfragments in out-of-distribution domains if not trained properly . |
| Approach: | They propose a byte-level LM with learnable tokenizers to make tokenization adaptive . they propose 'flexitoken' which enables significantly greater flexibility during adaptation . |
| Outcome: | The proposed method significantly reduces token overfragmentation and improves on multilingual benchmarks and domains. |
Similar Papers
Learn Your Tokens: Word-Pooled Tokenization for Language Modeling (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Language models typically tokenize text into subwords, using a deterministic, hand-engineered heuristic of combining characters into longer surface-level strings such as ‘ing’ or whole words. |
| Approach: | They propose a 'learn your tokens' scheme which pooles bytes/characters into word representations and decodes individual characters/bytes per word in parallel. |
| Outcome: | The proposed tokenizer outperforms subword models and byte/character models over the word boundary and outperformed on rare words by a factor of 30! |
Retrofitting Large Language Models with Dynamic Tokenization (2025.acl-long)
Copied to clipboard
| Challenge: | Current language models use a static tokenizer, which results in degraded efficiency and language capabilities. |
| Approach: | They propose a way to dynamically decide on token boundaries based on input text . they merge frequent subword sequences in a batch and apply a hypernetwork to compute token embeddings . |
| Outcome: | The proposed method reduces token sequence lengths by >20% across 14 languages while promoting fairness across languages. |
Teaching Old Tokenizers New Words: Efficient Tokenizer Adaptation for Pretrained Models (2026.findings-eacl)
Copied to clipboard
| Challenge: | Extending existing vocabulary is a widely used step in adapting pre-trained language models to new domains or languages. |
| Approach: | They propose to extend a pre-trained tokenizer by continuing the BPE merge learning process on new data. |
| Outcome: | The proposed method improves tokenization efficiency and improves model utilization. |
AdaptBPE: From General Purpose to Specialized Tokenizers (2026.eacl-long)
Copied to clipboard
| Challenge: | Subword tokenization methods impact performance and efficiency of large language models . generic tokens can incur inefficiencies when applying the model to specific domains or languages . |
| Approach: | They propose a subword tokenization technique that selectively replaces low-utility tokens with more relevant ones based on their frequency in an adaptation corpus. |
| Outcome: | The proposed method compresses test corpora more effectively than baselines using the same vocabulary size. |
Tokenization Falling Short: On Subword Robustness in Large Language Models (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Language models typically tokenize raw text into sequences of subword identifiers from a predefined vocabulary. |
| Approach: | They propose to tokenize raw text into sequences of subword identifiers from a predefined vocabulary . they also investigate the challenges and their impact on large language models . |
| Outcome: | The proposed model can mitigate tokenization issues, but still suffer from typos and other variations. |
Token-Level Self-Evolution Training for Sequence-to-Sequence Learning (2023.acl-short)
Copied to clipboard
| Challenge: | Adaptive training approaches do not consider the variation of learning difficulty in different training steps, making the learning deterministic and sub-optimal. |
| Approach: | They propose a dynamic token-level self-evolution training method that reweighs the training losses of different target tokens based on priors. |
| Outcome: | Empirically, the proposed method yields significant improvements on three translation tasks. |
BPE-knockout: Pruning Pre-existing BPE Tokenisers with Backwards-compatible Morphological Semi-supervision (2024.naacl-long)
Copied to clipboard
| Challenge: | Subword tokenisation is a mainstay in natural language processing (NLP) it allows the representation of an infinite space of text with a finite set of units. |
| Approach: | They propose to use byte-pair encoding to represent an infinite space of text with a finite set of units by removing subwords from the BPE vocabulary without impeding further use of merges that relied on them. |
| Outcome: | The proposed method improves token-based tasks in English, Dutch and German while saving training costs. |
One Tokenizer To Rule Them All: Emergent Language Plasticity via Multilingual Tokenizers (2026.acl-long)
Copied to clipboard
Diana Abagyan, Alejandro R. Salamanca, Andres Felipe Cruz-Salinas, Kris Cao, Hangyu Lin, Acyr Locatelli, Marzieh Fadaee, Ahmet Üstün, Sara Hooker
| Challenge: | Existing approaches to train multilingual large language models for many languages at once are limited due to limited model capacity, scarce high-quality data, and compute constraints. |
| Approach: | They propose to use a universal tokenizer to improve language plasticity and adaptability to new languages by up to 20%. |
| Outcome: | The proposed tokenizer improves language plasticity and improves plasticity towards languages that are completely unseen in the tokenizer and pretraining, by up to 5% win rate gain. |
MANTa: Efficient Gradient-Based Tokenization for End-to-End Robust Language Modeling (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Subword tokenization algorithms have been an essential component of language modeling but their static nature results in important flaws that degrade the models’ downstream performance and robustness. |
| Approach: | They propose a module for Adaptive Neural TokenizAtion that is differentiable and trained end-to-end with the language model. |
| Outcome: | The proposed tokenizer improves robustness to character perturbations and out-of-domain data. |
An Embarrassingly Simple Method to Mitigate Undesirable Properties of Pretrained Language Model Tokenizers (2022.acl-short)
Copied to clipboard
| Challenge: | a standard tokenizer does not cover all characters of a word but preserves key aspects of its morphological structure . a novel method to improve tokenization of pretrained language models is proposed . |
| Approach: | They propose a method to improve the tokenization of pretrained language models . they use the vocabulary of a standard tokenizer but preserves morphological structure . |
| Outcome: | The proposed method improves tokenization of pretrained language models on morphological gold segmentations and text classification tasks. |