Improving Multilingual Models with Language-Clustered Vocabularies (2020.emnlp-main)
Copied to clipboard
| Challenge: | State-of-the-art multilingual models depend on vocabularies that cover all languages . but the methods for generating those vocalaries are not ideal for massively multilingual applications. |
| Approach: | They propose a procedure for multilingual vocabulary generation that combines separately trained vocabularies of several automatically derived language clusters. |
| Outcome: | The proposed procedure shows improvements across languages on multilingual benchmark tasks . the proposed procedure reduces out-of-vocabulary rate by a factor of 8 . |
Similar Papers
XLM-V: Overcoming the Vocabulary Bottleneck in Multilingual Masked Language Models (2023.emnlp-main)
Copied to clipboard
Davis Liang, Hila Gonen, Yuning Mao, Rui Hou, Naman Goyal, Marjan Ghazvininejad, Luke Zettlemoyer, Madian Khabsa
| Challenge: | Large multilingual models rely on a single vocabulary shared across 100+ languages . this vocabulary bottleneck limits the representational capabilities of multilingual model XLM-R . |
| Approach: | They propose a new approach for scaling to large multilingual vocabularies by de-emphasizing token sharing between languages with little lexical overlap and assigning vocabulary capacity to achieve sufficient coverage for each individual language. |
| Outcome: | The proposed model outperforms XLM-R on all language tasks and is particularly effective on low-resource tasks. |
Language Clustering for Multilingual Named Entity Recognition (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Recent work in multilingual natural language processing has shown progress on tasks such as natural language inference and joint multilingual translation. |
| Approach: | They propose a technique that groups similar languages together by embeddings from a pre-trained masked language model and automatically discovering language clusters in this embeddable space. |
| Outcome: | The proposed technique outperforms baselines on 15 languages in the WikiAnn dataset showing meaningful multilingual transfer for low-resource languages (Swahili and Yoruba). |
Multilingual Generation in Abstractive Summarization: A Comparative Study (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing models for multilingual generation lack thorough analysis due to extensive linguistic diversity. |
| Approach: | They propose to classify multilingual generation methodologies into three categories based on their underlying modeling principles . they introduce an automatic metric to mitigate spurious correlations associated with language mixing . |
| Outcome: | The proposed model improves in high-resource, low-resourced, and zero-shot scenarios. |
A Closer Look at Clustering Bilingual Comparable Corpora (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing methods for clustering comparable corpora are not suitable for bilingual corpors. |
| Approach: | They propose new clustering models fully adapted to comparable corpora based on a deep variant of Kmeans . they illustrate their behavior on bilingual collections created from Wikipedia . |
| Outcome: | The proposed models show that they can cluster comparable corpora on bilingual collections . the proposed models are based on a state-of-the-art deep variant of Kmeans . |
A Recipe of Parallel Corpora Exploitation for Multilingual Large Language Models (2025.findings-naacl)
Copied to clipboard
| Challenge: | Recent studies have highlighted the potential of exploiting parallel corpora to enhance multilingual large language models. |
| Approach: | They investigate the impact of parallel corpora quality and quantity, training objectives, and model size on performance of multilingual large language models enhanced with parallel corporeal. |
| Outcome: | The proposed approach improves performance in bilingual and general-purpose tasks. |
An Empirical Study on Cross-lingual Vocabulary Adaptation for Efficient Language Model Inference (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Cross-lingual vocabulary adaptation (CVA) methods have been proposed for adapting models to a target language . but effectiveness of these methods on increasing inference efficiency of generative large language models has not been explored. |
| Approach: | They propose to use cross-lingual vocabulary adaptation methods to adapt models to a target language to improve downstream performance. |
| Outcome: | The proposed methods significantly speed up models in four languages and four natural language understanding tasks. |
How Vocabulary Sharing Facilitates Multilingualism in LLaMA? (2024.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) show strong performance on English tasks, but their performance in other languages is limited. |
| Approach: | They conducted an exhaustive analysis of the multilingual capability of LLMs by examining the performance gap before and after embedding fine-tuning across 101 languages. |
| Outcome: | The proposed model improves on the attributes of four quadrants in the model and provides actionable and efficient guidelines for tuning these languages. |
KIT-Multi: A Translation-Oriented Multilingual Embedding Corpus (L18-1)
Copied to clipboard
| Challenge: | Cross-lingual word embeddings are representations of words across languages in a shared continuous vector space. |
| Approach: | They propose a multilingual word embedding corpus which is acquired by neural machine translation and is based on monolingual data. |
| Outcome: | The proposed method is competitive with existing methods but on the cross-lingual document classification task, it obtains the best figures. |
Vocabulary-informed Language Encoding (2022.coling-1)
Copied to clipboard
| Challenge: | A Multilingual model relies on language encodings to identify input languages . a method to compute a vocabulary-informed language coding can improve multilingual models . |
| Approach: | They propose a method to compute a vocabulary-informed language encoding as the language representation for a required language. |
| Outcome: | The proposed method improves performance on unsupervised translation and cross-lingual embedding. |
Towards Continual Learning for Multilingual Machine Translation via Vocabulary Substitution (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing approaches to multilingual machine translation rely on training models on monolingual data for all languages in a multitask setup. |
| Approach: | They propose a vocabulary adaptation scheme to extend the language capacity of multilingual machine translation models by combining monolingual data with a dictionary. |
| Outcome: | The proposed model improves on existing models by preserving the original model and allowing for competitive performance even with only monolingual data. |