| Challenge: | Existing evaluations of subword tokenization focus on engineering criteria such as compression rate . a recent study evaluated subwords for their cognitive plausibility in languages with limited vocabulary size . |
| Approach: | They propose a new evaluation paradigm that focuses on the cognitive plausibility of subword tokenization. |
| Outcome: | The proposed tokenization algorithm yields less cognitively plausible tokenization behavior and worse coverage of derivational morphemes than previous evaluations. |
Similar Papers
Where are we Still Split on Tokenization? (2024.findings-eacl)
Copied to clipboard
| Challenge: | Identifying tokens is a crucial first step for many tasks in Natural Language Processing (NLP) gold tokenization is often assumed, but some work on token-level tasks is more challenging. |
| Approach: | They propose an efficient method for tokenization with subword-based language models and evaluate it on 122 languages in 20 scripts. |
| Outcome: | The proposed method performs on par with the state-of-the-art on 122 languages in 20 scripts. |
SubTokenTest: A Practical Benchmark for Real-World Sub-token Understanding (2026.acl-long)
Copied to clipboard
| Challenge: | Existing benchmarks have highlighted character-level tasks as lacking practical relevance . many real-world applications rely heavily on precise sub-token understanding . |
| Approach: | They propose a benchmark that assesses sub-token understanding through practical tasks . they examine the impact of test-time scaling on sub-word reasoning . |
| Outcome: | The proposed benchmark assesses sub-token understanding through practical tasks . it includes ten tasks across four domains and isolates tokenization-related failures . |
Words, Subwords, and Morphemes: What Really Matters in the Surprisal-Reading Time Relationship? (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies using LLMs on psycholinguistic data have gone unverified . a growing body of research is using word-level prediction as a computational proxy . |
| Approach: | They compare morphological, morphologic, and BPE tokenization estimates with reading time data. |
| Outcome: | The proposed method could be used to evaluate morphological prediction. |
Revisiting subword tokenization: A case study on affixal negation in large language models (2024.naacl-long)
Copied to clipboard
| Challenge: | Negation is central to language understanding but is not properly captured by modern NLP methods. |
| Approach: | They propose to use subword tokenization methods to detect negation in large language models . they find that models can reliably recognize negation, despite mismatches in tokenization accuracy . |
| Outcome: | The proposed models can detect negation in English using subword tokenization methods despite some mismatches in tokenization accuracy and negation detection performance. |
Which Pieces Does Unigram Tokenization Really Need? (2026.findings-acl)
Copied to clipboard
| Challenge: | Despite its theoretical elegance, its implementation in practice is complex, limiting its adoption to SentencePiece. |
| Approach: | They propose a Unigram-based probabilistic alternative to the greedy heuristics of Byte-Pair Encoding that is based on C++. |
| Outcome: | The proposed algorithm is remarkably robust to hyperparameter choices and can be simplified to reduce computational costs. |
Tokenization Falling Short: On Subword Robustness in Large Language Models (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Language models typically tokenize raw text into sequences of subword identifiers from a predefined vocabulary. |
| Approach: | They propose to tokenize raw text into sequences of subword identifiers from a predefined vocabulary . they also investigate the challenges and their impact on large language models . |
| Outcome: | The proposed model can mitigate tokenization issues, but still suffer from typos and other variations. |
Stop Taking Tokenizers for Granted: They Are Core Design Decisions in Large Language Models (2026.eacl-long)
Copied to clipboard
| Challenge: | Subword tokenization approaches misalign with linguistic structure and waste capacity across languages and domains. |
| Approach: | They argue for a context-aware framework that integrates tokenizer and model co-design . they argue that tokenization should be treated as a core design problem, not an afterthought . |
| Outcome: | The proposed framework integrates tokenizer and model co-design, guided by linguistic, domain, and deployment considerations. |
Beyond Text Compression: Evaluating Tokenizers Across Scales (2025.acl-long)
Copied to clipboard
| Challenge: | Language models rely on tokenizers to convert text into machine-interpretable tokens, which shape the statistical patterns that language models learn to estimate. |
| Approach: | They propose to use Zipf's law to measure tokenizer performance by combining several metrics to capture multiple aspects of tokenizer behavior. |
| Outcome: | The proposed metrics correlate more strongly with downstream performance than text compression when modeling unseen languages. |
Evaluating Morphological Plausibility of Subword Tokenization via Statistical Alignment with Morpho-Syntactic Features (2026.findings-eacl)
Copied to clipboard
| Challenge: | Existing metric for subword tokenization evaluation for morphological plausibility requires unavailable or inconsistent gold segmentation data. |
| Approach: | They propose a morpho-syntactic feature-based metric for subword tokenization evaluation. |
| Outcome: | The proposed metric correlates well with traditional morpheme boundary recall while being more broadly applicable across languages with different morphological systems. |
Learn Your Tokens: Word-Pooled Tokenization for Language Modeling (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Language models typically tokenize text into subwords, using a deterministic, hand-engineered heuristic of combining characters into longer surface-level strings such as ‘ing’ or whole words. |
| Approach: | They propose a 'learn your tokens' scheme which pooles bytes/characters into word representations and decodes individual characters/bytes per word in parallel. |
| Outcome: | The proposed tokenizer outperforms subword models and byte/character models over the word boundary and outperformed on rare words by a factor of 30! |