The Strawberry Problem: Emergence of Character-level Understanding in Tokenized Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models fail at simple character-level tasks due to low mutual information, study finds . authors propose a lightweight architectural modification that improves character- level reasoning . |
| Approach: | They propose a lightweight architectural modification that improves character-level reasoning while preserving the inductive advantages of subword models. |
| Outcome: | The proposed model improves character-level reasoning while preserving the advantages of subword models. |
Similar Papers
What do tokens know about their characters and how do they know it? (2022.naacl-main)
Copied to clipboard
| Challenge: | Pre-trained language models that use subword tokenization schemes can succeed at a variety of language tasks that require character-level information. |
| Approach: | They propose to use word tokenization schemes to probe what word pieces encode . they show that larger models can encode character-level information . |
| Outcome: | The proposed models can encode character-level information and perform better on non-Latin alphabets. |
How Do Language Models Acquire Character-Level Information? (2026.eacl-long)
Copied to clipboard
| Challenge: | Language models (LMs) implicitly encode character-level information, despite not being explicitly provided during training. |
| Approach: | They analyze how language models acquire character-level knowledge by comparing them to standard settings. |
| Outcome: | The results show that LMs do not treat words as opaque tokens, but instead treat them as tokens. |
How Tokenization Limits Phonological Knowledge Representation in Language Models and How to Improve Them (2026.acl-long)
Copied to clipboard
| Challenge: | Tokenization is the first step in every language model (LM), yet it never takes the sounds of words into account. |
| Approach: | They propose a lightweight IPA-based fine-tuning method that infuses phonological awareness into LMs. |
| Outcome: | The proposed method improves phonological awareness across three phonology-related tasks while preserving math and general reasoning ability. |
Learn Your Tokens: Word-Pooled Tokenization for Language Modeling (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Language models typically tokenize text into subwords, using a deterministic, hand-engineered heuristic of combining characters into longer surface-level strings such as ‘ing’ or whole words. |
| Approach: | They propose a 'learn your tokens' scheme which pooles bytes/characters into word representations and decodes individual characters/bytes per word in parallel. |
| Outcome: | The proposed tokenizer outperforms subword models and byte/character models over the word boundary and outperformed on rare words by a factor of 30! |
Tokenization Falling Short: On Subword Robustness in Large Language Models (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Language models typically tokenize raw text into sequences of subword identifiers from a predefined vocabulary. |
| Approach: | They propose to tokenize raw text into sequences of subword identifiers from a predefined vocabulary . they also investigate the challenges and their impact on large language models . |
| Outcome: | The proposed model can mitigate tokenization issues, but still suffer from typos and other variations. |
Spelling-out is not Straightforward: LLMs’ Capability of Tokenization from Token to Characters (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models (LLMs) can spell out tokens character by character with high accuracy, yet struggle with more complex character-level tasks. |
| Approach: | They examine how large language models internally represent character-level information during the spelling-out process. |
| Outcome: | The embedding layer does not fully encode character-level information, especially beyond the first character. |
Token Knowledge: A New Perspective For Knowledge in Large Language Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Predicting the presence and absence of certain knowledge in large language models could aid hallucination avoidance. |
| Approach: | They propose a token knowledge dataset construction method and use the intermediate states during inference to train probes. |
| Outcome: | The proposed method increases the model's latent potential by 60% to 90% with strong out-of-distribution generalization by training on just a few dozen prompts. |
Small Language Models Also Work With Small Vocabularies: Probing the Linguistic Abilities of Grapheme- and Phoneme-Based Baby Llamas (2025.coling-main)
Copied to clipboard
| Challenge: | Existing studies on LMs have focused on linguistic generalizations and representations from developmentally plausible data. |
| Approach: | They propose to use phoneme- and grapheme-based language models to learn linguistic units at and below the word level. |
| Outcome: | The proposed models can achieve strong performance on syntactic and novel benchmarks and match grapheme-based models in standard tasks and novel evaluations. |
SubTokenTest: A Practical Benchmark for Real-World Sub-token Understanding (2026.acl-long)
Copied to clipboard
| Challenge: | Existing benchmarks have highlighted character-level tasks as lacking practical relevance . many real-world applications rely heavily on precise sub-token understanding . |
| Approach: | They propose a benchmark that assesses sub-token understanding through practical tasks . they examine the impact of test-time scaling on sub-word reasoning . |
| Outcome: | The proposed benchmark assesses sub-token understanding through practical tasks . it includes ten tasks across four domains and isolates tokenization-related failures . |
From Characters to Words: Hierarchical Pre-trained Language Model for Open-vocabulary Language Understanding (2023.acl-long)
Copied to clipboard
| Challenge: | Current models for natural language understanding require a preprocessing step to convert raw text into discrete tokens. |
| Approach: | They propose a hierarchical open-vocabulary language model that adopts a shallow Transformer architecture to learn word representations from their characters and a deep inter-word Transformer module that contextualizes each word representation by attending to the entire word sequence. |
| Outcome: | The proposed model outperforms baselines on various downstream tasks and is robust to textual corruption and domain shift. |