WECHSEL: Effective initialization of subword embeddings for cross-lingual transfer of monolingual language models (2022.naacl-main)
Copied to clipboard
| Challenge: | Existing methods to train large pretrained language models require more computational resources and are expensive to train in other languages. |
| Approach: | They propose a method to transfer pretrained language models to new languages using subword-based tokenization and embeddings. |
| Outcome: | The proposed method outperforms existing methods on low-resource languages and makes training large models more accessible and less damaging to the environment. |
Similar Papers
UNKs Everywhere: Adapting Multilingual Language Models to New Scripts (2021.emnlp-main)
Copied to clipboard
| Challenge: | Massively multilingual language models offer state-of-the-art cross-lingual transfer performance on a range of NLP tasks, but there is a profound performance gap between resource-rich and resource-poor target languages. |
| Approach: | They propose a series of data-efficient methods that enable quick and effective adaptation of pretrained multilingual models to low-resource languages and unseen scripts. |
| Outcome: | The proposed methods improve learning of the new dedicated embedding matrix in the target language and for low-resource languages written in unseen scripts. |
A Multi-dimensional Evaluation of Tokenizer-free Multilingual Pretrained Models (2023.findings-eacl)
Copied to clipboard
| Challenge: | Recent work on tokenizer-free models shows promising results in cross-lingual transfer . previous work focused on reporting accuracy on a limited set of tasks and data settings . |
| Approach: | They compare tokenizer-free and subword-based models using various dimensions . they find subword models are still the most practical choice in many settings . |
| Outcome: | The proposed model improves cross-lingual transfer and reduces engineering overhead. |
HYPEROFA: Expanding LLM Vocabulary to New Languages via Hypernetwork-Based Embedding Initialization (2025.acl-srw)
Copied to clipboard
| Challenge: | Pre-trained language models exhibit suboptimal performance on mid- and low-resource languages due to limited exposure to these languages during pre-training. |
| Approach: | They propose a similarity-based subword embedding initialization heuristic that introduces new tokens specific to target languages, initializes their embedders, and applies continual pre-training on target-language data. |
| Outcome: | The proposed method outperforms random initialization baseline and matches or exceeds OFA in both continual pre-training convergence and downstream task performance. |
TransMI: A Framework to Create Strong Baselines from Multilingual Pretrained Language Models for Transliterated Data (2025.coling-main)
Copied to clipboard
| Challenge: | Existing mPLMs that handle non-transliterated data are not sufficient to train crosslingual models. |
| Approach: | They propose a framework to transliterate related languages into a common script by exploiting existing mPLMs and their tokenizer without any training. |
| Outcome: | The proposed framework can create strong baselines for data that is transliterated into a common script by exploiting an existing mPLM and its tokenizer without any training. |
OFA: A Framework of Initializing Unseen Subword Embeddings for Efficient Large-scale Multilingual Continued Pretraining (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing methods to pretrain multilingual models are limited by the number of embedding parameters and the complexity of the model. |
| Approach: | They propose a framework that initializes the embeddings of unseen subwords and can adapt a model to multiple languages. |
| Outcome: | The proposed framework can adapt a pre-trained model to multiple languages efficiently and effectively. |
One Tokenizer To Rule Them All: Emergent Language Plasticity via Multilingual Tokenizers (2026.acl-long)
Copied to clipboard
Diana Abagyan, Alejandro R. Salamanca, Andres Felipe Cruz-Salinas, Kris Cao, Hangyu Lin, Acyr Locatelli, Marzieh Fadaee, Ahmet Üstün, Sara Hooker
| Challenge: | Existing approaches to train multilingual large language models for many languages at once are limited due to limited model capacity, scarce high-quality data, and compute constraints. |
| Approach: | They propose to use a universal tokenizer to improve language plasticity and adaptability to new languages by up to 20%. |
| Outcome: | The proposed tokenizer improves language plasticity and improves plasticity towards languages that are completely unseen in the tokenizer and pretraining, by up to 5% win rate gain. |
Tokenizer-Aware Cross-Lingual Adaptation of Decoder-Only LLMs through Embedding Relearning and Swapping (2026.eacl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have been primarily focused on English, leaving the multilingual ability unexplored. |
| Approach: | They propose a technique that creates new tokenizers and tunes embeddings on fixed model weights for target language adaptation. |
| Outcome: | The proposed method is light-weight and performant but has limitations for older models and high resource languages. |
Adapting Word Embeddings to New Languages with Morphological and Phonological Subword Representations (D18-1)
Copied to clipboard
| Challenge: | Existing approaches to generalization to resource-rich languages are difficult . a recent study shows that word representations can be useful in low resource languages . |
| Approach: | They propose two approaches for improving generalization to low-resource languages by adapting continuous word representations using linguistically motivated subword units. |
| Outcome: | The proposed method improves generalization to low resource languages . it requires neither parallel corpora nor bilingual dictionaries and requires no parallel training . |
What is the best recipe for character-level encoder-only modelling? (2023.acl-long)
Copied to clipboard
| Challenge: | aims to benchmark recent progress in language understanding models that output contextualised representations at the character level. |
| Approach: | They aim to find the best way to build and train character-level BERT-like models by comparing architectural innovations with pretraining objectives. |
| Outcome: | The proposed model outperforms a token-based model on a set of evaluation tasks with a fixed training procedure. |
MonoByte: A Pool of Monolingual Byte-level Language Models (2022.coling-1)
Copied to clipboard
| Challenge: | Existing studies have shown that multilingual models can achieve zero-shot cross-lingual performance on various NLP tasks, but due to the cost of pretraining, they often use public models with limited budgets. |
| Approach: | They propose to use tokenized models to test cross-lingual ability in multilingual and monolingual corpora. |
| Outcome: | The results show that models pretrained on multilingual and even monolingual corpora perform better than models pre-trained on SOTA models. |