TokCollate: A Comprehensive Tool for Tokenizer Evaluation and Visualization across Languages (2026.acl-demo)
Copied to clipboard
| Challenge: | Tokenization quality varies significantly across languages, leading to disparities in LLM performance and cost for speakers of less-resourced languages. |
| Approach: | They propose a Python-based evaluation framework and a JavaScript visualization interface that evaluates tokenizers in a variety of languages. |
| Outcome: | TokCollate is an evaluation framework for tokenizers with a JavaScript visualization interface. |
Similar Papers
TokLens: A Multilingual Lens on Tokenizer Quality for LLMs (2026.acl-srw)
Copied to clipboard
| Challenge: | TokLens is an open-source toolkit for evaluating tokenizer quality across languages . authors evaluated 24 tokenizers from major LLM families across 15 typologically diverse languages - a gap that is stark in Japanese . |
| Approach: | They evaluate 24 tokenizers from major LLM families across 15 typologically diverse languages and correlate these metrics with downstream performance. |
| Outcome: | The proposed tokenizers produce 56x more tokens per word in Japanese than in English . the newer tokenizer Qwen2.5 and Gemma-2 reduce this gap to under 4x . |
Beyond Text Compression: Evaluating Tokenizers Across Scales (2025.acl-long)
Copied to clipboard
| Challenge: | Language models rely on tokenizers to convert text into machine-interpretable tokens, which shape the statistical patterns that language models learn to estimate. |
| Approach: | They propose to use Zipf's law to measure tokenizer performance by combining several metrics to capture multiple aspects of tokenizer behavior. |
| Outcome: | The proposed metrics correlate more strongly with downstream performance than text compression when modeling unseen languages. |
TFDP: Token-Efficient Disparity Audits for Autoregressive LLMs via Single-Token Masked Evaluation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for auditing autoregressive Large Language Models for disparities are limited and expensive. |
| Approach: | They propose a method to detect disparities in autoregressive Large Language Models by token querying . they propose 'token-focused disparity probing' to measure disparities between sentence pairs . |
| Outcome: | The proposed method detects disparities with 42 times fewer output tokens than previous methods. |
Beyond Literal Token Overlap: Token Alignability for Multilinguality (2025.naacl-short)
Copied to clipboard
| Challenge: | Existing studies have shown that token overlap is a strong predictor of multilinguality and cross-lingual knowledge transfer between languages with different scripts. |
| Approach: | They propose a subword token alignability metric to understand the impact and quality of multilingual tokenisation. |
| Outcome: | The proposed metric predicts multilinguality much better when scripts are disparate and the overlap of literal tokens is low. |
Where are we Still Split on Tokenization? (2024.findings-eacl)
Copied to clipboard
| Challenge: | Identifying tokens is a crucial first step for many tasks in Natural Language Processing (NLP) gold tokenization is often assumed, but some work on token-level tasks is more challenging. |
| Approach: | They propose an efficient method for tokenization with subword-based language models and evaluate it on 122 languages in 20 scripts. |
| Outcome: | The proposed method performs on par with the state-of-the-art on 122 languages in 20 scripts. |
Tokenization Impacts Multilingual Language Modeling: Assessing Vocabulary Allocation and Overlap Across Languages (2023.findings-acl)
Copied to clipboard
| Challenge: | Multilingual language models perform surprisingly well in a variety of NLP tasks for diverse languages. |
| Approach: | They propose to evaluate the quality of lexical representation and vocabulary overlap observed in sub-word tokenizers. |
| Outcome: | The proposed criteria show that the overlap of vocabulary across languages can be detrimental to certain downstream tasks. |
Learn Your Tokens: Word-Pooled Tokenization for Language Modeling (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Language models typically tokenize text into subwords, using a deterministic, hand-engineered heuristic of combining characters into longer surface-level strings such as ‘ing’ or whole words. |
| Approach: | They propose a 'learn your tokens' scheme which pooles bytes/characters into word representations and decodes individual characters/bytes per word in parallel. |
| Outcome: | The proposed tokenizer outperforms subword models and byte/character models over the word boundary and outperformed on rare words by a factor of 30! |
How do different tokenizers perform on downstream tasks in scriptio continua languages?: A case study in Japanese (2023.acl-srw)
Copied to clipboard
| Challenge: | Existing studies on scriptio continua languages lack comprehensiveness of tokenizers . authors use Byte-Pair-Encoding or Unigram instead of WordPiece for subword tokenizer . |
| Approach: | They investigate the effect of tokenizers on the downstream performance of pretrained language models in scriptio continua languages where no explicit spaces exist between words. |
| Outcome: | The proposed tokenizers perform better on a wide range of tasks compared with other tokenizer methods . the results show that each task has an optimal morphological analyzer . |
How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Models (2021.acl-long)
Copied to clipboard
| Challenge: | Using pretraining data, we find that a designated monolingual tokenizer plays an equally important role in the downstream performance of the model. |
| Approach: | They propose to compare pretrained multilingual models with their monolingual counterparts on a set of five diverse monolingual downstream tasks. |
| Outcome: | The proposed models offer previously unmatched performance in all NLP tasks. |
MorphBPE: Morphology-Aware Tokenization for Efficient LLM Training (2026.findings-acl)
Copied to clipboard
| Challenge: | Tokenization is a key design choice in modern NLP systems and a critical bottleneck for multilingual Large Language Models. |
| Approach: | They propose a tokenization extension that constrains merge operations to respect morpheme boundaries while preserving inference. |
| Outcome: | The proposed tokenization improves morphological coherence and language model cross-entropy in four languages. |