Unpacking Tokenization: Evaluating Text Compression and its Correlation with Model Performance (2024.findings-acl)
Copied to clipboard
| Challenge: | Despite being the cornerstone of BPE, the importance of compression in the tokenization process is still unclear. |
| Approach: | They argue for the theoretical importance of compression in the tokenization process . they also demonstrate the empirical importance of compressing tokenizers for downstream success of pre-trained language models. |
| Outcome: | The proposed method can be viewed as 0-gram language modeling where equal probability is assigned to all tokens. |
Similar Papers
Tokenization Is More Than Compression (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing tokenization approaches like Byte-Pair Encoding (BPE) have been suggested that their effectiveness stems from their ability to condense text into a relatively small number of tokens. |
| Approach: | They propose a tokenizer that segments a document’s text into the minimum number of tokens for a given vocabulary and propose fewer tokens to improve downstream performance. |
| Outcome: | The proposed tokenizers can initialize vocabulary construction and pre-tokenization, and the results show that fewer tokens lead to better performance. |
Beyond Text Compression: Evaluating Tokenizers Across Scales (2025.acl-long)
Copied to clipboard
| Challenge: | Language models rely on tokenizers to convert text into machine-interpretable tokens, which shape the statistical patterns that language models learn to estimate. |
| Approach: | They propose to use Zipf's law to measure tokenizer performance by combining several metrics to capture multiple aspects of tokenizer behavior. |
| Outcome: | The proposed metrics correlate more strongly with downstream performance than text compression when modeling unseen languages. |
Investigating the Effectiveness of BPE: The Power of Shorter Sequences (D19-1)
Copied to clipboard
| Challenge: | Byte-Pair Encoding (BPE) is an unsupervised sub-word tokenization technique, but its reasons for its effectiveness are not well understood. |
| Approach: | They link BPE to the broader family of dictionary-based compression algorithms and compare it with other members of this family. |
| Outcome: | The proposed method is compared with dictionary-based compression algorithms and improves on a fixed vocabulary size budget. |
Stop Taking Tokenizers for Granted: They Are Core Design Decisions in Large Language Models (2026.eacl-long)
Copied to clipboard
| Challenge: | Subword tokenization approaches misalign with linguistic structure and waste capacity across languages and domains. |
| Approach: | They argue for a context-aware framework that integrates tokenizer and model co-design . they argue that tokenization should be treated as a core design problem, not an afterthought . |
| Outcome: | The proposed framework integrates tokenizer and model co-design, guided by linguistic, domain, and deployment considerations. |
Parity-Aware Byte-Pair Encoding: Improving Cross-lingual Fairness in Tokenization (2026.acl-long)
Copied to clipboard
Negar Foroutan, Clara Meister, Debjit Paul, Joel Niklaus, Sina Ahmadi, Antoine Bosselut, Rico Sennrich
| Challenge: | Tokenization is the first step of most NLP pipelines. |
| Approach: | They propose a parity-aware byte pair encoder that maximizes the compression gain of the currently worst-compressed language for cross-lingual parity. |
| Outcome: | a new algorithm reduces tokenization inequality by 89% compared to classical BPE . the proposed algorithm is based on a fair-max rule that maximizes the compression gain of the currently worst-compressed language . |
How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Models (2021.acl-long)
Copied to clipboard
| Challenge: | Using pretraining data, we find that a designated monolingual tokenizer plays an equally important role in the downstream performance of the model. |
| Approach: | They propose to compare pretrained multilingual models with their monolingual counterparts on a set of five diverse monolingual downstream tasks. |
| Outcome: | The proposed models offer previously unmatched performance in all NLP tasks. |
Tokenizer Choice For LLM Training: Negligible or Crucial? (2024.findings-naacl)
Copied to clipboard
Mehdi Ali, Michael Fromm, Klaudia Thellmann, Richard Rutmann, Max Lübbering, Johannes Leveling, Katrin Klug, Jan Ebert, Niclas Doll, Jasper Buschhoff, Charvi Jain, Alexander Weber, Lena Jurkschat, Hammam Abdelwahab, Chelsea John, Pedro Ortiz Suarez, Malte Ostendorff, Samuel Weinbach, Rafet Sifa, Stefan Kesselheim, Nicolas Flores-Herr
| Challenge: | Recent success of large language models has been driven by curating the training dataset composition, scaling of model architectures and advancements in pretraining objectives, leaving tokenizer influence as a blind spot. |
| Approach: | They conduct a comprehensive study on the influence of tokenizer choice on LLM downstream performance by training 24 mono- and multilingual LLMs at a 2.6B parameter scale. |
| Outcome: | The proposed model can significantly impact the model's downstream performance and training costs. |
Tokenization is Sensitive to Language Variation (2025.findings-acl)
Copied to clipboard
| Challenge: | Variation in language is often linked to regional, social, and contextual factors. |
| Approach: | They propose a method to estimate tokenizer impact on downstream LLM performance . they pre-train BERT models with the popular Byte-Pair Encoding algorithm . |
| Outcome: | The proposed model improves on Rényi efficiency and other metrics on language variation. |
BPE Gets Picky: Efficient Vocabulary Refinement During Tokenizer Training (2024.emnlp-main)
Copied to clipboard
| Challenge: | Tokenization is a relatively understudied area, but it can greatly impact model performance and efficiency. |
| Approach: | They propose a modified BPE tokenizer that removes merges that leave intermediate "junk" tokens from the vocabulary. |
| Outcome: | The proposed method improves vocabulary efficiency, eliminates under-trained tokens, and does not compromise text compression. |
Byte Pair Encoding is Suboptimal for Language Model Pretraining (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Subword tokenization is a popular language model that can be used to segment text. |
| Approach: | They analyze differences between byte-pair encoding (BPE) and unigram LM tokenization methods to find subword units that align more closely with morphology. |
| Outcome: | The proposed method recovers subword units that align more closely with morphology and avoids problems stemming from BPE’s greedy construction procedure. |