Papers with pre-tokenization
Tokenization Is More Than Compression (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing tokenization approaches like Byte-Pair Encoding (BPE) have been suggested that their effectiveness stems from their ability to condense text into a relatively small number of tokens. |
| Approach: | They propose a tokenizer that segments a document’s text into the minimum number of tokens for a given vocabulary and propose fewer tokens to improve downstream performance. |
| Outcome: | The proposed tokenizers can initialize vocabulary construction and pre-tokenization, and the results show that fewer tokens lead to better performance. |
Fast WordPiece Tokenization (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for tokenization of text are not efficient, but they are based on Aho-Corasick's algorithm. |
| Approach: | They propose an efficient algorithm for WordPiece tokenization using a longest-match-first strategy . they propose an algorithm whose tokenization complexity is strictly O(n) |
| Outcome: | The proposed method is 8.2x faster than HuggingFace Tokenizers and 5.1x faster on average for general text tokenization. |
Pre-tokenization of Multi-word Expressions in Cross-lingual Word Embeddings (2020.emnlp-main)
Copied to clipboard
| Challenge: | Multi-Word Expressions (MWEs) are common in every language, but they are not translated by cross-lingual word embeddings. |
| Approach: | They propose a method for word translation of Multi-Word Expressions (MWEs) they compile lists of MWEs in each language and tokenize them as single tokens before training word embeddings. |
| Outcome: | The proposed method can translate multi-word expressions to and from English in 10 languages. |
Egalitarian Language Representation in Language Models: It All Begins with Tokenizers (2025.coling-main)
Copied to clipboard
| Challenge: | Tokenizers influence how language is represented in large language models . pre-tokenization choices can be problematic for some languages . |
| Approach: | They propose a tokenization algorithm that incorporates graphemes to improve tokenization . they validate this algorithm with Tamil, Sinhala, and Hindi scripts . |
| Outcome: | The proposed method outperforms tokenizers on Tamil, Sinhala, and Hindi scripts. |
Lexically Grounded Subword Segmentation (2024.emnlp-main)
Copied to clipboard
| Challenge: | Statistical word segmentation algorithms have remained a thorn in the side of many researchers. |
| Approach: | They propose to use unsupervised morphological analysis with Morfessor as pre-tokenization and an algebraic method for obtaining subword embeddings grounded in a word embeddable space. |
| Outcome: | The proposed methods improve morphological plausibility and Rényi efficiency on part-of-speech tagging and machine translation tasks. |
The Devil Is in the Word Alignment Details: On Translation-Based Cross-Lingual Transfer for Token Classification Tasks (2025.findings-acl)
Copied to clipboard
| Challenge: | Translation-based strategies for cross-lingual transfer XLT include label projection . word aligners (WAs) are commonly used for label projection, but low-level design decisions for using them have not been investigated . |
| Approach: | They revisit word aligners (WAs) for label projection and propose a new projection strategy that outperforms WAs. |
| Outcome: | The proposed projection strategy outperforms marker-based methods in token classification tasks. |