| Challenge: | Subword regularization reduces the dependency on exact tokenizations, augments training corpus, and exposes model to unique contexts during training. |
| Approach: | They propose an algorithm to uniformly sample subword tokenizations to replace stochastic variants that are biased towards a small set of tokenization per word. |
| Outcome: | The proposed algorithm reduces the dependency on exact tokenizations and augments the training corpus. |
Similar Papers
BPE-Dropout: Simple and Effective Subword Regularization (2020.acl-main)
Copied to clipboard
| Challenge: | Subword segmentation is widely used to address the open vocabulary problem in machine translation. |
| Approach: | They propose a method that stochastically corrupts the segmentation procedure of BPE and produces multiple segmentations within the same fixed BPE framework. |
| Outcome: | The proposed method produces multiple segmentations within the same fixed BPE framework. |
GRaMPa: Subword Regularisation by Skewing Uniform Segmentation Distributions with an Efficient Path-counting Markov Model (2025.acl-long)
Copied to clipboard
| Challenge: | Subword regularisations are known to be stochastic, but only a handful of possible segmentations are sampled. |
| Approach: | They propose to randomise word segmentations from a subword tokeniser instead of randomising them by weighting paths in an unweighted segmentation graph. |
| Outcome: | The proposed method outperforms existing methods on token-level tasks with spelling errors. |
Tokenization Falling Short: On Subword Robustness in Large Language Models (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Language models typically tokenize raw text into sequences of subword identifiers from a predefined vocabulary. |
| Approach: | They propose to tokenize raw text into sequences of subword identifiers from a predefined vocabulary . they also investigate the challenges and their impact on large language models . |
| Outcome: | The proposed model can mitigate tokenization issues, but still suffer from typos and other variations. |
From Where Words Come: Efficient Regularization of Code Tokenizers Through Source Attribution (2026.acl-long)
Copied to clipboard
| Challenge: | Currently, subword tokenization is the most common approach for vocabulary building in large models. |
| Approach: | They propose to regularize training and minimize overfitting by using source-attributed BPE . they find that undertrained tokens are prone to producing unused, unusable tokens . |
| Outcome: | The proposed techniques reduce the number of under-trained tokens while maintaining the same inference procedure as with regular BPE. |
Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates (P18-1)
Copied to clipboard
| Challenge: | Subword units are an effective way to alleviate the open vocabulary problems in neural machine translation. |
| Approach: | They propose a method to regularize subword segmentations probabilistically by sampling subwords . they also propose 'unigram' language model to be used for better subword sampling . |
| Outcome: | The proposed method improves on low resource and out-of-domain settings with multiple corpora. |
Causal Estimation of Tokenisation Bias (2025.acl-long)
Copied to clipboard
| Challenge: | Modern language models define probabilities over character-strings, but in practice, it does . Ideally, the choice of the tokeniser should not affect the probability assigned to the underlying character- string. |
| Approach: | They quantify a type of tokenisation bias by framing it as a causal effect and estimating it using the regression discontinuity design. |
| Outcome: | The proposed model can estimate tokenisation bias by comparing subwords around arbitrary cutoff points. |
Learn Your Tokens: Word-Pooled Tokenization for Language Modeling (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Language models typically tokenize text into subwords, using a deterministic, hand-engineered heuristic of combining characters into longer surface-level strings such as ‘ing’ or whole words. |
| Approach: | They propose a 'learn your tokens' scheme which pooles bytes/characters into word representations and decodes individual characters/bytes per word in parallel. |
| Outcome: | The proposed tokenizer outperforms subword models and byte/character models over the word boundary and outperformed on rare words by a factor of 30! |
AdaptBPE: From General Purpose to Specialized Tokenizers (2026.eacl-long)
Copied to clipboard
| Challenge: | Subword tokenization methods impact performance and efficiency of large language models . generic tokens can incur inefficiencies when applying the model to specific domains or languages . |
| Approach: | They propose a subword tokenization technique that selectively replaces low-utility tokens with more relevant ones based on their frequency in an adaptation corpus. |
| Outcome: | The proposed method compresses test corpora more effectively than baselines using the same vocabulary size. |
Tokenization and the Noiseless Channel (2023.acl-long)
Copied to clipboard
| Challenge: | Subword tokenization is a key part of most NLP pipelines, but little is known about why some combinations lead to improved downstream model performance. |
| Approach: | They propose that good tokenizers lead to efficient channel usage . they propose that an optimal encoding assigns extremely long codes to low-frequency subwords . |
| Outcome: | The proposed tokenizers have a very strong correlation with BLEU in machine translation . the proposed function can be used to improve model performance in the downstream task . |
How Important Is Tokenization in French Medical Masked Language Models? (2024.lrec-main)
Copied to clipboard
| Challenge: | Word tokenization into subword units has become the prevailing standard in the field of natural language processing (NLP) over recent years . the precise factors contributing to its success remain unclear . |
| Approach: | They propose a tokenization strategy that integrates morpheme-enriched word segmentation into existing tokenization methods. |
| Outcome: | The proposed tokenization strategy outperforms character and word tokenization but the precise factors contributing to its success remain unclear. |