Tokenization Matters: Navigating Data-Scarce Tokenization for Gender Inclusive Language Technologies (2024.findings-naacl)
Copied to clipboard
Anaelia Ovalle, Ninareh Mehrabi, Palash Goyal, Jwala Dhamala, Kai-Wei Chang, Richard Zemel, Aram Galstyan, Yuval Pinter, Rahul Gupta
| Challenge: | a recent study documented the harmful limitations of gender binary-centric large language models . data scarcity is a known culprit, but the precise mechanisms through which scarcity affects this behavior remain underexplored. |
| Approach: | They propose to use BPE tokenization to enforce consistent tokenization across gendered pronouns to improve neopronoun proficiency. |
| Outcome: | The proposed methods outperform finetuning with standard BPE, and improve neopronoun proficiency. |
Similar Papers
MISGENDERED: Limits of Large Language Models in Understanding Pronouns (2023.acl-long)
Copied to clipboard
| Challenge: | excluding non-binary gender identities can perpetuate harm against non-bisexual individuals through exclusion and marginalization. |
| Approach: | They propose a framework for evaluating large language models’ ability to correctly use preferred pronouns. |
| Outcome: | The proposed framework evaluates language models' ability to correctly use preferred pronouns in English. |
Parity-Aware Byte-Pair Encoding: Improving Cross-lingual Fairness in Tokenization (2026.acl-long)
Copied to clipboard
Negar Foroutan, Clara Meister, Debjit Paul, Joel Niklaus, Sina Ahmadi, Antoine Bosselut, Rico Sennrich
| Challenge: | Tokenization is the first step of most NLP pipelines. |
| Approach: | They propose a parity-aware byte pair encoder that maximizes the compression gain of the currently worst-compressed language for cross-lingual parity. |
| Outcome: | a new algorithm reduces tokenization inequality by 89% compared to classical BPE . the proposed algorithm is based on a fair-max rule that maximizes the compression gain of the currently worst-compressed language . |
Tokenization Falling Short: On Subword Robustness in Large Language Models (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Language models typically tokenize raw text into sequences of subword identifiers from a predefined vocabulary. |
| Approach: | They propose to tokenize raw text into sequences of subword identifiers from a predefined vocabulary . they also investigate the challenges and their impact on large language models . |
| Outcome: | The proposed model can mitigate tokenization issues, but still suffer from typos and other variations. |
Gender Identity in Pretrained Language Models: An Inclusive Approach to Data Creation and Probing (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Pretrained language models encode binary gender information of text authors, raising the risk of skewed representations and downstream harms. |
| Approach: | They use a corpus of YouTube transcripts from transgender, cisgender and non-binary speakers to examine whether pretrained language models encode binary gender information. |
| Outcome: | The proposed model encodes gender information for all gender identities but to different extents. |
Stop Taking Tokenizers for Granted: They Are Core Design Decisions in Large Language Models (2026.eacl-long)
Copied to clipboard
| Challenge: | Subword tokenization approaches misalign with linguistic structure and waste capacity across languages and domains. |
| Approach: | They argue for a context-aware framework that integrates tokenizer and model co-design . they argue that tokenization should be treated as a core design problem, not an afterthought . |
| Outcome: | The proposed framework integrates tokenizer and model co-design, guided by linguistic, domain, and deployment considerations. |
Welcome to the Modern World of Pronouns: Identity-Inclusive Natural Language Processing beyond Gender (2022.coling-1)
Copied to clipboard
| Challenge: | Current modeling of 3rd person pronouns ignores neopronoun phenomena like naive pronounes, which are not (yet) widely established. |
| Approach: | They propose to validate existing and novel approaches for modeling 3rd person pronouns in language technology and validate them through a survey. |
| Outcome: | The proposed model excludes non-binary users, while ignoring gender-specific phenomena. |
Egalitarian Language Representation in Language Models: It All Begins with Tokenizers (2025.coling-main)
Copied to clipboard
| Challenge: | Tokenizers influence how language is represented in large language models . pre-tokenization choices can be problematic for some languages . |
| Approach: | They propose a tokenization algorithm that incorporates graphemes to improve tokenization . they validate this algorithm with Tamil, Sinhala, and Hindi scripts . |
| Outcome: | The proposed method outperforms tokenizers on Tamil, Sinhala, and Hindi scripts. |
Tokenizer-Aware Cross-Lingual Adaptation of Decoder-Only LLMs through Embedding Relearning and Swapping (2026.eacl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have been primarily focused on English, leaving the multilingual ability unexplored. |
| Approach: | They propose a technique that creates new tokenizers and tunes embeddings on fixed model weights for target language adaptation. |
| Outcome: | The proposed method is light-weight and performant but has limitations for older models and high resource languages. |
Tokenizer Choice For LLM Training: Negligible or Crucial? (2024.findings-naacl)
Copied to clipboard
Mehdi Ali, Michael Fromm, Klaudia Thellmann, Richard Rutmann, Max Lübbering, Johannes Leveling, Katrin Klug, Jan Ebert, Niclas Doll, Jasper Buschhoff, Charvi Jain, Alexander Weber, Lena Jurkschat, Hammam Abdelwahab, Chelsea John, Pedro Ortiz Suarez, Malte Ostendorff, Samuel Weinbach, Rafet Sifa, Stefan Kesselheim, Nicolas Flores-Herr
| Challenge: | Recent success of large language models has been driven by curating the training dataset composition, scaling of model architectures and advancements in pretraining objectives, leaving tokenizer influence as a blind spot. |
| Approach: | They conduct a comprehensive study on the influence of tokenizer choice on LLM downstream performance by training 24 mono- and multilingual LLMs at a 2.6B parameter scale. |
| Outcome: | The proposed model can significantly impact the model's downstream performance and training costs. |
AdaptBPE: From General Purpose to Specialized Tokenizers (2026.eacl-long)
Copied to clipboard
| Challenge: | Subword tokenization methods impact performance and efficiency of large language models . generic tokens can incur inefficiencies when applying the model to specific domains or languages . |
| Approach: | They propose a subword tokenization technique that selectively replaces low-utility tokens with more relevant ones based on their frequency in an adaptation corpus. |
| Outcome: | The proposed method compresses test corpora more effectively than baselines using the same vocabulary size. |