False Friends Are Not Foes: Investigating Vocabulary Overlap in Multilingual Language Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Prior work has shown that token overlap facilitates cross-lingual transfer or introduces interference between languages? |
| Approach: | They devised a controlled experiment where they train bilingual autoregressive models on multiple language pairs under systematically varied vocabulary overlap settings. |
| Outcome: | The proposed model outperforms models with disjointed vocabularies on XNLI and XQuAD and shows that token overlap is beneficial for multilingual tokenizers. |
Similar Papers
Tokenization Impacts Multilingual Language Modeling: Assessing Vocabulary Allocation and Overlap Across Languages (2023.findings-acl)
Copied to clipboard
| Challenge: | Multilingual language models perform surprisingly well in a variety of NLP tasks for diverse languages. |
| Approach: | They propose to evaluate the quality of lexical representation and vocabulary overlap observed in sub-word tokenizers. |
| Outcome: | The proposed criteria show that the overlap of vocabulary across languages can be detrimental to certain downstream tasks. |
Beyond Literal Token Overlap: Token Alignability for Multilinguality (2025.naacl-short)
Copied to clipboard
| Challenge: | Existing studies have shown that token overlap is a strong predictor of multilinguality and cross-lingual knowledge transfer between languages with different scripts. |
| Approach: | They propose a subword token alignability metric to understand the impact and quality of multilingual tokenisation. |
| Outcome: | The proposed metric predicts multilinguality much better when scripts are disparate and the overlap of literal tokens is low. |
How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Models (2021.acl-long)
Copied to clipboard
| Challenge: | Using pretraining data, we find that a designated monolingual tokenizer plays an equally important role in the downstream performance of the model. |
| Approach: | They propose to compare pretrained multilingual models with their monolingual counterparts on a set of five diverse monolingual downstream tasks. |
| Outcome: | The proposed models offer previously unmatched performance in all NLP tasks. |
XLM-V: Overcoming the Vocabulary Bottleneck in Multilingual Masked Language Models (2023.emnlp-main)
Copied to clipboard
Davis Liang, Hila Gonen, Yuning Mao, Rui Hou, Naman Goyal, Marjan Ghazvininejad, Luke Zettlemoyer, Madian Khabsa
| Challenge: | Large multilingual models rely on a single vocabulary shared across 100+ languages . this vocabulary bottleneck limits the representational capabilities of multilingual model XLM-R . |
| Approach: | They propose a new approach for scaling to large multilingual vocabularies by de-emphasizing token sharing between languages with little lexical overlap and assigning vocabulary capacity to achieve sufficient coverage for each individual language. |
| Outcome: | The proposed model outperforms XLM-R on all language tasks and is particularly effective on low-resource tasks. |
Overlap-based Vocabulary Generation Improves Cross-lingual Transfer Among Related Languages (2022.acl-long)
Copied to clipboard
| Challenge: | Pre-trained multilingual models have shown great potential for zero-shot cross-lingual transfer to low web-resource languages (LRLs). |
| Approach: | They propose a vocabulary generation algorithm which enhances lexical overlap across related languages by generating a token that increases the representation of LRLs. |
| Outcome: | The proposed approach improves cross-lingual transfer accuracy without reducing HRL representation and accuracy. |
Beyond Shared Vocabulary: Increasing Representational Word Similarities across Languages for Multilingual Machine Translation (2023.emnlp-main)
Copied to clipboard
| Challenge: | Using a shared vocabulary is common practice in multilingual machine translation . however, when words overlap is small, e.g., using different writing systems, knowledge transfer is inhibited . |
| Approach: | They propose a re-parameterized method for building word embeddings using word equivalence classes and graph networks to fuse word embeds across languages. |
| Outcome: | The proposed method achieves evident BLEU improvements on high- and low-resource MNMT scenarios. |
Emerging Cross-lingual Structure in Pretrained Language Models (2020.acl-main)
Copied to clipboard
| Challenge: | Recent work has shown that multilingual pretraining works, but is unable to measure these effects. |
| Approach: | They propose to use multilingual masked language modeling to train a model on concatenated text from multiple languages to find universal latent symmetries in embedding spaces. |
| Outcome: | The proposed models can be trained on concatenated text from multiple languages without shared vocabulary or domain similarity. |
Take a Closer Look at Multilinguality! Improve Multilingual Pre-Training Using Monolingual Corpora Only (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent studies have demonstrated remarkable cross-lingual capability of pre-trained language models . however, semantic alignments may be the reason behind such capability but remain under-explored. |
| Approach: | They propose token-level and semantic-level code-switched masked language modeling to improve cross-lingual interactions over mono-mPLMs without parallel sentences. |
| Outcome: | The proposed method outperforms mono-mPLMs on natural language understanding and unsupervised machine translation tasks. |
False Friends or Cognates? A Cross-lingual Semantic Ambiguity Evaluation for Galician, Portuguese and Spanish (2026.acl-long)
Copied to clipboard
| Challenge: | Closely related languages exhibit a high degree of lexical and orthographic similarity, which can facilitate cross-lingual understanding but also give rise to systematic semantic ambiguity. |
| Approach: | They introduce six cross-lingual datasets that are manually or semi-automatically generated and are able to identify and process false friends among these languages. |
| Outcome: | The proposed models can identify and process false friends among Galician, Portuguese, and Spanish. |
Understanding Cross-Lingual Alignment—A Survey (2024.findings-acl)
Copied to clipboard
| Challenge: | Cross-lingual alignment is the meaningful similarity of representations across languages in multilingual language models. |
| Approach: | They propose a taxonomy of methods to improve cross-lingual alignment . they argue that an effective trade-off between language-neutral and language-specific information is key . |
| Outcome: | The proposed methods can be applied to encoder models and encoder-decoder-only models . they show that language-neutral and language-specific information is key . |