SubTokenTest: A Practical Benchmark for Real-World Sub-token Understanding (2026.acl-long)
Copied to clipboard
| Challenge: | Existing benchmarks have highlighted character-level tasks as lacking practical relevance . many real-world applications rely heavily on precise sub-token understanding . |
| Approach: | They propose a benchmark that assesses sub-token understanding through practical tasks . they examine the impact of test-time scaling on sub-word reasoning . |
| Outcome: | The proposed benchmark assesses sub-token understanding through practical tasks . it includes ten tasks across four domains and isolates tokenization-related failures . |
Similar Papers
CUTE: Measuring LLMs’ Understanding of Their Tokens (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) perform well on a wide variety of tasks, authors say . they lack direct access to characters, which can be difficult to generalize to new languages . |
| Approach: | They propose a benchmark to test the orthographic knowledge of Large Language Models . they find that most LLMs seem to know the spelling of their tokens - yet fail to manipulate text . |
| Outcome: | The proposed benchmark tests the orthographic knowledge of large language models . it finds that most LLMs seem to know the spelling of their tokens, but fail to manipulate text . |
Where are we Still Split on Tokenization? (2024.findings-eacl)
Copied to clipboard
| Challenge: | Identifying tokens is a crucial first step for many tasks in Natural Language Processing (NLP) gold tokenization is often assumed, but some work on token-level tasks is more challenging. |
| Approach: | They propose an efficient method for tokenization with subword-based language models and evaluate it on 122 languages in 20 scripts. |
| Outcome: | The proposed method performs on par with the state-of-the-art on 122 languages in 20 scripts. |
Learn Your Tokens: Word-Pooled Tokenization for Language Modeling (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Language models typically tokenize text into subwords, using a deterministic, hand-engineered heuristic of combining characters into longer surface-level strings such as ‘ing’ or whole words. |
| Approach: | They propose a 'learn your tokens' scheme which pooles bytes/characters into word representations and decodes individual characters/bytes per word in parallel. |
| Outcome: | The proposed tokenizer outperforms subword models and byte/character models over the word boundary and outperformed on rare words by a factor of 30! |
Tokenization Falling Short: On Subword Robustness in Large Language Models (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Language models typically tokenize raw text into sequences of subword identifiers from a predefined vocabulary. |
| Approach: | They propose to tokenize raw text into sequences of subword identifiers from a predefined vocabulary . they also investigate the challenges and their impact on large language models . |
| Outcome: | The proposed model can mitigate tokenization issues, but still suffer from typos and other variations. |
EXECUTE: A Multilingual Benchmark for LLM Token Understanding (2025.findings-acl)
Copied to clipboard
| Challenge: | EXECUTE is an expandable X(Cross)-Lingual Extension of CUTE that can be expanded to any language. |
| Approach: | They extend the CUTE benchmark to more languages with diverse scripts and writing systems, introducing EXECUTE. |
| Outcome: | The extended framework allows expansion to any language. |
Analyzing Cognitive Plausibility of Subword Tokenization (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing evaluations of subword tokenization focus on engineering criteria such as compression rate . a recent study evaluated subwords for their cognitive plausibility in languages with limited vocabulary size . |
| Approach: | They propose a new evaluation paradigm that focuses on the cognitive plausibility of subword tokenization. |
| Outcome: | The proposed tokenization algorithm yields less cognitively plausible tokenization behavior and worse coverage of derivational morphemes than previous evaluations. |
Spelling-out is not Straightforward: LLMs’ Capability of Tokenization from Token to Characters (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models (LLMs) can spell out tokens character by character with high accuracy, yet struggle with more complex character-level tasks. |
| Approach: | They examine how large language models internally represent character-level information during the spelling-out process. |
| Outcome: | The embedding layer does not fully encode character-level information, especially beyond the first character. |
Revisiting subword tokenization: A case study on affixal negation in large language models (2024.naacl-long)
Copied to clipboard
| Challenge: | Negation is central to language understanding but is not properly captured by modern NLP methods. |
| Approach: | They propose to use subword tokenization methods to detect negation in large language models . they find that models can reliably recognize negation, despite mismatches in tokenization accuracy . |
| Outcome: | The proposed models can detect negation in English using subword tokenization methods despite some mismatches in tokenization accuracy and negation detection performance. |
The Strawberry Problem: Emergence of Character-level Understanding in Tokenized Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models fail at simple character-level tasks due to low mutual information, study finds . authors propose a lightweight architectural modification that improves character- level reasoning . |
| Approach: | They propose a lightweight architectural modification that improves character-level reasoning while preserving the inductive advantages of subword models. |
| Outcome: | The proposed model improves character-level reasoning while preserving the advantages of subword models. |
What do tokens know about their characters and how do they know it? (2022.naacl-main)
Copied to clipboard
| Challenge: | Pre-trained language models that use subword tokenization schemes can succeed at a variety of language tasks that require character-level information. |
| Approach: | They propose to use word tokenization schemes to probe what word pieces encode . they show that larger models can encode character-level information . |
| Outcome: | The proposed models can encode character-level information and perform better on non-Latin alphabets. |