| Challenge: | Lack of token unit compatibility is one of the critical problems of Japanese language resources. |
| Approach: | They develop a Japanese tokenizer called Sudachi and its accompanying dictionary . they use multi-granular output and normalization of notation variations to improve tokenization . |
| Outcome: | The proposed tokenizer and dictionary improve tokenization in Japanese for business use. |
Similar Papers
User-Generated Text Corpus for Evaluating Japanese Morphological Analysis and Lexical Normalization (2021.naacl-main)
Copied to clipboard
| Challenge: | Morphological analysis (MA) and lexical normalization (LN) are important tasks for Japanese user-generated text. |
| Approach: | They construct a publicly available Japanese UGT corpus annotated with morphological and normalization information. |
| Outcome: | The proposed corpus shows low performance for non-general words and non-standard forms . morphological analysis is an important task in Japanese user-generated text . |
How do different tokenizers perform on downstream tasks in scriptio continua languages?: A case study in Japanese (2023.acl-srw)
Copied to clipboard
| Challenge: | Existing studies on scriptio continua languages lack comprehensiveness of tokenizers . authors use Byte-Pair-Encoding or Unigram instead of WordPiece for subword tokenizer . |
| Approach: | They investigate the effect of tokenizers on the downstream performance of pretrained language models in scriptio continua languages where no explicit spaces exist between words. |
| Outcome: | The proposed tokenizers perform better on a wide range of tasks compared with other tokenizer methods . the results show that each task has an optimal morphological analyzer . |
Multilingual Tokenization through the Lens of Indian Languages: Challenges and Insights (2026.findings-acl)
Copied to clipboard
Maharaj Brahma, N J Karthika, Rajat Verma, Nagasai Saketh Naidu, Rohit Saluja, Maunendra Sankar Desarkar, Ganesh Ramakrishnan
| Challenge: | Existing tokenizers are often skewed towards high-resource languages limiting their effectiveness for linguistically diverse and morphologically rich languages. |
| Approach: | They evaluate multilingual tokenization across 17 Indic languages spanning 11 scripts and two language families. |
| Outcome: | The proposed method improves tokenization quality and vocabulary size in 17 languages . poor tokenization can lead to increase in sequence lengths, fragment meaningful units, weaken model's ability to capture linguistic structure and semantics. |
TokCollate: A Comprehensive Tool for Tokenizer Evaluation and Visualization across Languages (2026.acl-demo)
Copied to clipboard
| Challenge: | Tokenization quality varies significantly across languages, leading to disparities in LLM performance and cost for speakers of less-resourced languages. |
| Approach: | They propose a Python-based evaluation framework and a JavaScript visualization interface that evaluates tokenizers in a variety of languages. |
| Outcome: | TokCollate is an evaluation framework for tokenizers with a JavaScript visualization interface. |
An Empirical Study of Tokenization Strategies for Various Korean NLP Tasks (2020.aacl-main)
Copied to clipboard
| Challenge: | Traditionally, tokenization is the very first step in most text processing works. |
| Approach: | They propose to use morphological segmentation followed by BPE for Korean NLP tasks . they empirically examine what is the best tokenization strategy for Korean to/from English . |
| Outcome: | The proposed approach is best for Korean to/from English machine translation and natural language understanding tasks. |
Exploring morphology-aware tokenization: A case study on Spanish language modeling (2025.emnlp-main)
Copied to clipboard
| Challenge: | a recent study shows that subword tokenization improves performance of neural language models. |
| Approach: | They propose a linguistically grounded approach to train a tokenizer on morphologically segmented data. |
| Outcome: | The proposed tokenizer improves on a Spanish language model with morphological information. |
NLPre: A Revised Approach towards Language-centric Benchmarking of Natural Language Preprocessing Systems (2024.lrec-main)
Copied to clipboard
| Challenge: | GLUE benchmarking system enables ongoing evaluation of multiple NLPre tools while credibly tracking their performance. |
| Approach: | They propose a language-centric benchmarking system that enables ongoing evaluation of multiple NLPre tools while credibly tracking their performance. |
| Outcome: | The proposed system is configured for Polish and integrated with the thoroughly assembled NLPre-PL benchmark. |
MUTANT: A Recipe for Multilingual Tokenizer Design (2026.acl-long)
Copied to clipboard
| Challenge: | Subword tokenization schemes such as Byte Pair Encoding (BPE) are widely adopted, but their effectiveness in multilingual settings remains understudied. |
| Approach: | They propose a multilingual tokenizer that produces linguistically coherent tokens for multilingual LLMs. |
| Outcome: | The proposed tokenizer improves fertility score by 39.5% over LLaMA4 and 18% over Sutra. |
From Where Words Come: Efficient Regularization of Code Tokenizers Through Source Attribution (2026.acl-long)
Copied to clipboard
| Challenge: | Currently, subword tokenization is the most common approach for vocabulary building in large models. |
| Approach: | They propose to regularize training and minimize overfitting by using source-attributed BPE . they find that undertrained tokens are prone to producing unused, unusable tokens . |
| Outcome: | The proposed techniques reduce the number of under-trained tokens while maintaining the same inference procedure as with regular BPE. |
Phonemes to the Rescue: Multilingual Tokenization Based on International Phonetic Alphabet (2026.acl-long)
Copied to clipboard
| Challenge: | Widely-used subword tokenization approaches favor high-resource languages and tokenizer-free methods yield longer sequences for scripts with a higher bytes-per-character ratio. |
| Approach: | They propose to use the International Phonetic Alphabet (IPA) as a language-agnostic input representation for multilingual tokenizers. |
| Outcome: | The proposed model improves tokenization quality and generalizes more effectively to unseen languages and scripts. |