Papers with tokenization
Dive into Deep Learning for Natural Language Processing (D19-2)
Copied to clipboard
| Challenge: | GluonNLP is a powerful new toolkit that automates the most laborious aspects of deep learning for NLP. |
| Approach: | This hands-on tutorial demonstrates how to scale unsupervised pre-training techniques with Apache MXNet and GluonNLP. |
| Outcome: | This hands-on tutorial examines the challenges of scaling these models and algorithms effectively with Apache MXNet and GluonNLP. |
Where are we Still Split on Tokenization? (2024.findings-eacl)
Copied to clipboard
| Challenge: | Identifying tokens is a crucial first step for many tasks in Natural Language Processing (NLP) gold tokenization is often assumed, but some work on token-level tasks is more challenging. |
| Approach: | They propose an efficient method for tokenization with subword-based language models and evaluate it on 122 languages in 20 scripts. |
| Outcome: | The proposed method performs on par with the state-of-the-art on 122 languages in 20 scripts. |
Trankit: A Light-Weight Transformer-based Toolkit for Multilingual Natural Language Processing (2021.eacl-demos)
Copied to clipboard
| Challenge: | Trankit is a lightweight, pre-trained toolkit for multilingual natural language processing. |
| Approach: | They propose a transformer-based toolkit for multilingual natural language processing that trains pipelines over 100 languages and 90 pretrained pipelines for 56 languages. |
| Outcome: | The proposed tool outperforms existing pipelines over sentence segmentation, part-of-speech tagging, morphological feature tabbing, and dependency parsing while maintaining competitive performance over tokenization, multi-word token expansion, and lemmatization over 90 Universal Dependencies treebanks. |
DadmaTools: Natural Language Processing Toolkit for Persian Language (2022.naacl-demo)
Copied to clipboard
| Challenge: | Existing tools for Persian language processing are based on conventional non-neural models and do not take full advantage of the latest developments. |
| Approach: | They propose to use a Python neural pipeline for Persian text processing tasks . they use 'parsBERT' to fine-tune the Python pipeline using the PerDT dataset . |
| Outcome: | The proposed toolkit can achieve state-of-the-art performance on multiple NLP tasks. |
Stanza: A Python Natural Language Processing Toolkit for Many Human Languages (2020.acl-demos)
Copied to clipboard
| Challenge: | Existing tools that support only a few major languages are under-optimized for accuracy due to a focus on efficiency or use of less powerful models. |
| Approach: | They introduce a Python natural language processing toolkit that supports 66 languages . they train Stanza on 112 datasets and show it generalizes well on all languages compared to other tools . |
| Outcome: | The proposed toolkit performs well on 112 datasets and is compatible with the popular Java CoreNLP software. |
An Empirical Study of Tokenization Strategies for Various Korean NLP Tasks (2020.aacl-main)
Copied to clipboard
| Challenge: | Traditionally, tokenization is the very first step in most text processing works. |
| Approach: | They propose to use morphological segmentation followed by BPE for Korean NLP tasks . they empirically examine what is the best tokenization strategy for Korean to/from English . |
| Outcome: | The proposed approach is best for Korean to/from English machine translation and natural language understanding tasks. |
Do Diacritics Matter? Evaluating the Impact of Arabic Diacritics on Tokenization and LLM Benchmarks (2026.findings-eacl)
Copied to clipboard
| Challenge: | Diacritics can significantly influence language processing tasks in Arabic . their presence can increase subword fragmentation during tokenization, reducing performance . |
| Approach: | They analyze the impact of diacritics on tokenization and benchmark task performance across major Large Language Models. |
| Outcome: | The proposed model is robust to diacritics, but full diacritization leads to token fragmentation and degraded performance. |
Tokenization Is More Than Compression (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing tokenization approaches like Byte-Pair Encoding (BPE) have been suggested that their effectiveness stems from their ability to condense text into a relatively small number of tokens. |
| Approach: | They propose a tokenizer that segments a document’s text into the minimum number of tokens for a given vocabulary and propose fewer tokens to improve downstream performance. |
| Outcome: | The proposed tokenizers can initialize vocabulary construction and pre-tokenization, and the results show that fewer tokens lead to better performance. |
An Embarrassingly Simple Method to Mitigate Undesirable Properties of Pretrained Language Model Tokenizers (2022.acl-short)
Copied to clipboard
| Challenge: | a standard tokenizer does not cover all characters of a word but preserves key aspects of its morphological structure . a novel method to improve tokenization of pretrained language models is proposed . |
| Approach: | They propose a method to improve the tokenization of pretrained language models . they use the vocabulary of a standard tokenizer but preserves morphological structure . |
| Outcome: | The proposed method improves tokenization of pretrained language models on morphological gold segmentations and text classification tasks. |
BanSuite: A Unified Toolkit and Software Platform for Low-Resource NLP in Bangla (2026.eacl-demo)
Copied to clipboard
Md. Abu Sayed, Faisal Ahamed Khan, Jannatul Ferdous Tuli, Nabeel Mohammed, Mohammad Ruhul Amin, Mohammad Mamun Or Rashid
| Challenge: | Existing efforts to improve Bangla's NLP performance have focused on isolated tasks such as Part-of-Speech tagging and Named Entity Recognition (NER) but comprehensive, integrated systems for core NLP tasks such Shallow Parsing and Dependency Parser are largely absent. |
| Approach: | They propose to integrate a large-scale, manually annotated Bangla Treebank with high-quality pretrained models for POS tagging, NER, shallow parsing, and dependency parse. |
| Outcome: | The proposed system achieves strong in-domain baseline performance while maintaining high efficiency in resource usage. |
Multi-word Tokenization for Sequence Compression (2023.emnlp-industry)
Copied to clipboard
| Challenge: | Large Language Models have proven successful at modelling tasks, but they are expensive and slow to scale. |
| Approach: | They propose a Multi-Word Tokenizer that represents frequent multi-word expressions as single tokens. |
| Outcome: | The proposed tokenizer is more robust across shorter sequence lengths, allowing for major speedups via early sequence truncation. |
Assessing Emoji Use in Modern Text Processing Tools (2021.acl-long)
Copied to clipboard
| Challenge: | Emojis are textual elements that are encoded as characters but rendered as small digital images or icons that can be used to express an idea or emotion. |
| Approach: | They propose to use a set of popular NLP tools to assess the support of emojis in tweets. |
| Outcome: | The proposed methods show that many systems still have notable shortcomings when operating on text containing emojis. |
An Untold Story of Preprocessing Task Evaluation: An Alignment-based Joint Evaluation Approach (2024.lrec-main)
Copied to clipboard
Eunkyul Leah Jo, Angela Yoonseo Park, Grace Tianjiao Zhang, Izia Xiaoxiao Wang, Junrui Wang, MingJia Mao, Jungyeul Park
| Challenge: | a preprocessing task such as tokenization and sentence boundary detection (SBD) has been considered as a solution to many NLP challenges . however, the low error rates of current methods are mainly specific to certain tasks and rule-based tokenization can be difficult to use across different systems. |
| Approach: | They propose an evaluation algorithm that combines both tokenization and SBD results to improve evaluation reliability. |
| Outcome: | The proposed evaluation algorithm improves the reliability of evaluations by reevaluating the counts of true positive cases for F1 measures in both preprocessing tasks jointly. |
Optimizing Word Segmentation for Downstream Task (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods to optimize tokenizations for downstream tasks are not suitable for traditional NLP. |
| Approach: | They propose a method to explore a tokenization appropriate for a downstream task . they train a model to assign a high probability to such appropriate tokenization based on the downstream task loss . |
| Outcome: | The proposed method improves sentiment analysis and textual entailment tasks . it is also integrated into state-of-the-art contextualized embeddings and reports a positive effect . |
TArC: Tunisian Arabish Corpus, First complete release (2022.lrec-1)
Copied to clipboard
| Challenge: | a project focused on Tunisian Arabic encoded in Arabizi is a hybrid approach to linguistics and linguistic research . Arabic dialects are notoriously under-resourced linguistic systems . |
| Approach: | They propose to use Arabic script as a linguistic corpus and a neural network architecture to annotate the latter with various levels of linguistic information. |
| Outcome: | The proposed approach is hybrid and combines linguistic and linguistic tools . the proposed approach produces in cascade different levels of annotation . |
LLM The Genius Paradox: A Linguistic and Math Expert’s Struggle with Simple Word-based Counting Problems (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing conjectures about the reason for deficiency of LLMs in simple word-based counting problems are invalid. |
| Approach: | They propose to evaluate model transferability from specialized LLMs to simple counting tasks by comparing their results to popular conjectures . |
| Outcome: | The proposed model evaluations show that engaging reasoning is the most robust and efficient way to help LLMs better perceive tasks with more accurate responses. |
Parsivar: A Language Processing Toolkit for Persian (L18-1)
Copied to clipboard
| Challenge: | a preprocessing step is required to convert text into a standard format for NLP tasks. |
| Approach: | They propose a Persian preprocessing toolkit that performs various kinds of activities . they use a plagiarism detection system to exploit the proposed toolkit . |
| Outcome: | The proposed tool outperforms available Persian preprocessing tools by about 8 percent in terms of F1 . the proposed toolkit performs normalization, space correction, tokenization, stemming, parts of speech tagging and shallow parsing tasks. |
Unsupervised Tokenization Learning (2022.emnlp-main)
Copied to clipboard
| Challenge: | Unsupervised language learning has attracted great attention in recent years . Glushchenko et al. (2015) suggested using "deep patterns" with hierarchical "symbolic" grammatical pattern structures learned from texts as a way to model grammars and domain ontologies for natural languages. |
| Approach: | They propose to use a "transition freedom" metric to measure unsupervised tokenization . they find that different languages require different offshoots of that metric for tokenization. |
| Outcome: | The proposed method provides better tokenization quality than or comparable to lexicon-based ones, depending on the language. |
Evaluating the Elementary Multilingual Capabilities of Large Language Models with MultiQ (2024.findings-acl)
Copied to clipboard
| Challenge: | a global majority of non-English speakers are underrepresented by large language models . however, most open LLMs are limited in their language coverage . |
| Approach: | They propose a silver standard benchmark for basic open-ended question answering with 27.4k test questions across a typologically diverse set of 137 languages. |
| Outcome: | The proposed model can answer questions in 27.4k questions across 137 languages. |
Analyzing Cognitive Plausibility of Subword Tokenization (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing evaluations of subword tokenization focus on engineering criteria such as compression rate . a recent study evaluated subwords for their cognitive plausibility in languages with limited vocabulary size . |
| Approach: | They propose a new evaluation paradigm that focuses on the cognitive plausibility of subword tokenization. |
| Outcome: | The proposed tokenization algorithm yields less cognitively plausible tokenization behavior and worse coverage of derivational morphemes than previous evaluations. |
TriEmbed: Bridge the Gap between Text and Token Indices with Embedding Reparameterization (2025.findings-acl)
Copied to clipboard
| Challenge: | a current paradigm of language modeling discards linguistic relations between tokens during tokenization, creating a fundamental gap . empirical results show that TriEmbed provides more linguistically informative token embeddings . |
| Approach: | They propose a reparameterization method that incorporates morphological relationships . they propose to organize the vocabulary into a Trie structure to reparametrize embeddings . |
| Outcome: | Empirical results show that TriEmbed outperforms existing token embeddings while offering more linguistically informative token embeds. |
How Do Language Models Acquire Character-Level Information? (2026.eacl-long)
Copied to clipboard
| Challenge: | Language models (LMs) implicitly encode character-level information, despite not being explicitly provided during training. |
| Approach: | They analyze how language models acquire character-level knowledge by comparing them to standard settings. |
| Outcome: | The results show that LMs do not treat words as opaque tokens, but instead treat them as tokens. |
Local Structure Matters Most: Perturbation Study in NLU (2022.findings-acl)
Copied to clipboard
| Challenge: | Recent research shows that neural models are insensitive to word-order perturbations, but other studies suggest that models learn some abstract notion of syntax. |
| Approach: | They develop order-altering perturbations on the order of words, subwords, and characters to analyze their effect on neural models’ performance on language understanding tasks. |
| Outcome: | The proposed models are insensitive to word-order perturbations while the local ordering remains relatively unperturbed. |
Byte Pair Encoding is Suboptimal for Language Model Pretraining (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Subword tokenization is a popular language model that can be used to segment text. |
| Approach: | They analyze differences between byte-pair encoding (BPE) and unigram LM tokenization methods to find subword units that align more closely with morphology. |
| Outcome: | The proposed method recovers subword units that align more closely with morphology and avoids problems stemming from BPE’s greedy construction procedure. |
Lexically Grounded Subword Segmentation (2024.emnlp-main)
Copied to clipboard
| Challenge: | Statistical word segmentation algorithms have remained a thorn in the side of many researchers. |
| Approach: | They propose to use unsupervised morphological analysis with Morfessor as pre-tokenization and an algebraic method for obtaining subword embeddings grounded in a word embeddable space. |
| Outcome: | The proposed methods improve morphological plausibility and Rényi efficiency on part-of-speech tagging and machine translation tasks. |
MaxMatch-Dropout: Subword Regularization for WordPiece (2022.coling-1)
Copied to clipboard
| Challenge: | Existing subword regularization methods are specialized to a particular tokenizer type. |
| Approach: | They propose a subword regularization method for WordPiece that uses a maximum matching algorithm for tokenization. |
| Outcome: | The proposed method improves the performance of text classification and machine translation tasks as well as other subword regularization methods. |
Incorporating Domain Knowledge into Materials Tokenization (2025.acl-long)
Copied to clipboard
| Challenge: | Recent advances in language models have expanded their applications in materials science, but they often produce excessive fragmentation and semantic loss. |
| Approach: | They propose a frequency-centric tokenization approach that integrates material knowledge into tokenization. |
| Outcome: | The proposed tokenization approach outperforms existing tokenization methods and achieves an average performance gain of 4% and 2% in the generation and classification tasks. |
OpenKorPOS: Democratizing Korean Tokenization with Voting-Based Open Corpus Annotation (2022.lrec-1)
Copied to clipboard
| Challenge: | Korean uses spaces at larger-than-word boundaries, unlike other East-Asian languages. |
| Approach: | They propose to use Korean morphological analyzers to provide a sequence of morpheme-level tokens, losing information in the tokenization process. |
| Outcome: | The proposed scheme improves existing tagging scheme and makes it friendlier to generative tasks. |
A Morphologically Annotated Corpus of Emirati Arabic (L18-1)
Copied to clipboard
| Challenge: | Emirati Arabic corpus is first large-scale morphologically manually annotated corpus . resources for dialectal Arabic NLP tasks are still lacking compared to those for modern standard Arabic (MSA). |
| Approach: | They propose to annotate a large-scale corpus of Emirati Arabic using a morphologically manually annotated corpus from eight Gumar novels . they discuss the guidelines for each part of the annotation components, and the annotation interface they use. |
| Outcome: | The annotated corpus includes about 200,000 words from eight Gumar novels in the Emirati Arabic variety. |
Modeling Overregularization in Children with Small Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing research has analyzed regularization in language acquisition only by modeling word inflection directly, which is unnatural in light of human language acquisition. |
| Approach: | They hypothesize that language models that imitate errors children make during language acquisition have a learning process more similar to humans. |
| Outcome: | The proposed model shows child-like U-shaped learning curves clearly for certain verbs, but the preferences for types of overgeneralization did not fully match the observations in children. |
Improbable Bigrams Expose Vulnerabilities of Incomplete Tokens in Byte-Level Tokenizers (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies have revealed that tokenizers can be exploited to elicit unwanted behavior. |
| Approach: | They propose to exploit incomplete tokens with stray bytes to exploit their dependency . they propose to use improbable bigrams to exploit the dependency of their adjacent tokens . |
| Outcome: | The proposed tokenizers can be exploited to elicit unwanted behavior in language models. |
Token Alignment via Character Matching for Subword Completion (2024.findings-acl)
Copied to clipboard
Ben Athiwaratkun, Shiqi Wang, Mingyue Shang, Yuchen Tian, Zijian Wang, Sujan Kumar Gonugondla, Sanjay Krishna Gouda, Robert Kwiatkowski, Ramesh Nallapati, Parminder Bhatia, Bing Xiang
| Challenge: | Generative models struggle with prompts corresponding to partial tokens due to tokenization, where partial token is out-of-distribution during inference. |
| Approach: | They propose a method to alleviate tokenization artifact on text completion by backtracking to the last complete tokens and aligning subsequent generations to match with the prompt. |
| Outcome: | The proposed method shows that it improves on partial token scenarios with only a minor time increase. |
Vulnerability of LLMs to Vertically Aligned Text Manipulations (2025.acl-long)
Copied to clipboard
| Challenge: | Recent research shows that vertical text input significantly degrades the accuracy of large language models (LLMs) in text classification tasks. |
| Approach: | They investigate the impact of vertical text input on the performance of LLMs . they find that chain of thought reasoning does not help LLM recognize vertical input . |
| Outcome: | The proposed model can significantly mislead models, posing a risk of bypassing detection in real-world scenarios involving harmful or sensitive information. |
On the Proper Treatment of Tokenization in Psycholinguistics (2024.emnlp-main)
Copied to clipboard
| Challenge: | Language models are used in computational psycholinguistics to test theories that relate the surprisal of a region of interest to its cognitive cost experienced by readers. |
| Approach: | They propose to marginalize token-level language models into character-level ones before they are used in psycholinguistic studies. |
| Outcome: | The proposed model over token strings is better than character-level model, the authors show . the proposed model marginalizes token-level models into character-based models before they are used in psycholinguistic studies. |
Unsupervised Morphological Tree Tokenizer (2025.findings-acl)
Copied to clipboard
| Challenge: | Conventional statistical tokenizers often disrupt constituent boundaries within words, thereby corrupting semantic information. |
| Approach: | They propose a method that uses morphological structure guidance to induce character-level structures of words by training a deep model. |
| Outcome: | Empirical results show that the proposed method retains complete morphemes and outperforms existing methods on morphological segmentation and language modeling tasks. |
LoPT: Lossless Parallel Tokenization Acceleration for Long Context Inference of Large Language Model (2026.acl-long)
Copied to clipboard
| Challenge: | Existing parallel tokenization methods suffer from inconsistent results due to boundary artifacts that occur after merging. |
| Approach: | They propose a Lossless Parallel Tokenization framework that ensures output identical to standard sequential tokenization. |
| Outcome: | The proposed method achieves significant speedup while guaranteeing lossless tokenization. |
ZAEBUC-Spoken: A Multilingual Multidialectal Arabic-English Speech Corpus (2024.lrec-main)
Copied to clipboard
| Challenge: | a corpus of multilingual Arabic-English speech is presented in a new paper . a major bottleneck is the lack of data needed for training NLP models . |
| Approach: | They propose a multilingual multidialectal Arabic-English speech corpus with a set of guidelines for automatic speech recognition. |
| Outcome: | The proposed corpus includes two languages with Arabic and English spoken in multiple variants and Arabic and Arabic with various accents. |
Vocab Diet: Reshaping the Vocabulary of LLMs via Vector Arithmetic (2026.findings-acl)
Copied to clipboard
| Challenge: | Large language models often encode word-form variation as linear directions in the embedding space. |
| Approach: | They propose a compact reshaping of large language models' vocabulary by using shared vectors instead of unique tokens. |
| Outcome: | The proposed approach frees 10-40% of vocabulary slots to be reallocated where tokenization is inefficient. |
TokDrift: When LLM Speaks in Subwords but Code Speaks in Grammar (2026.acl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) for code rely on subword tokenizers learned from mixed natural language text and programming language code but driven by statistics rather than grammar. |
| Approach: | They propose a framework that applies semantic-preserving rewrite rules to create code variants differing only in tokenization. |
| Outcome: | The proposed framework can create code variants differing only in tokenization . the findings highlight the need for grammar-aware tokenization for future code LLMs. |