Knowledge Enhanced Pre-training for Cross-lingual Dense Retrieval (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing mPLMs neglect the importance of knowledge in cross-lingual dense retrieval. |
| Approach: | They propose a novel mPLM that leverages knowledge to learn language-agnostic semantic representations from a multilingual knowledge base and an annotation of Wiki. |
| Outcome: | The proposed model achieves strong multilingual and cross-lingual retrieval performance with significant improvements over existing mPLMs. |
Similar Papers
CLEAR: Cross-Lingual Enhancement in Retrieval via Reverse-training (2026.acl-long)
Copied to clipboard
| Challenge: | Existing multilingual embedding models often struggle to capture cross-lingual alignment during training. |
| Approach: | They propose a novel loss function that leverages an English passage as a bridge to strengthen alignments between target language and English. |
| Outcome: | The proposed model improves retrieval performance across cross-lingual scenarios while minimizing performance degradation in English. |
Breaking the Script Barrier in Multilingual Pre-Trained Language Models with Transliteration-Based Post-Training Alignment (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Recent mPLMs have shown impressive performance on crosslingual transfer tasks . however, the performance is often hindered when a lowresource target language is written in a different script than the high-resource source language. |
| Approach: | They propose a transliteration-based method to improve cross-lingual alignment between languages using diverse scripts. |
| Outcome: | The proposed method outperforms the original model on Englishcentric transfer tasks up to 50%. |
Cross-lingual Transfer of Monolingual Models (2022.lrec-1)
Copied to clipboard
| Challenge: | Existing studies on cross-lingual learning using multilingual models cast doubt on shared vocabulary and joint pre-training . et al. (2005) show that model knowledge learned in the source language enhances the learning of the target language independently of language proximity. |
| Approach: | They propose a method for transferring monolingual models to other languages through continuous pre-training and investigate their results in English. |
| Outcome: | The proposed method outperforms a model trained from scratch in the GLUE benchmark for English . it shows that model knowledge from the source language enhances the learning of syntactic and semantic knowledge in english. |
Enhancing Multilingual Language Model with Massive Multilingual Knowledge Triples (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for language model pretraining use limited knowledge graph data for knowledge-intensive tasks. |
| Approach: | They propose to make better use of multilingual annotations and language agnostic properties of KG triples for pretraining LMs. |
| Outcome: | The proposed models show significant performance improvements on a wide range of knowledge-intensive cross-lingual tasks. |
Enhancing LLM Language Adaption through Cross-lingual In-Context Pre-training (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for enhancing cross-lingual transfer are limited by parallel resources and lack linguistic and domain coverage. |
| Approach: | They propose a cross-lingual in-context pre-training approach that leverages semantically related bilingual Wikipedia documents to enhance cross-linguistic transfer. |
| Outcome: | The proposed approach improves multilingual performance on three models across six target languages. |
Unsupervised Multilingual Dense Retrieval via Generative Pseudo Labeling (2024.findings-eacl)
Copied to clipboard
| Challenge: | Existing sparse retrieval methods often yield inferior performance in multilingual retrieval, requiring a large amount of paired data, which is costly. |
| Approach: | They propose an Unsupervised Multilingual dense Retriever trained without paired data which iteratively improves performance of multilingual retrievers. |
| Outcome: | The proposed framework outperforms supervised baselines on two benchmark datasets and shows that iterative training improves the performance. |
Take a Closer Look at Multilinguality! Improve Multilingual Pre-Training Using Monolingual Corpora Only (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent studies have demonstrated remarkable cross-lingual capability of pre-trained language models . however, semantic alignments may be the reason behind such capability but remain under-explored. |
| Approach: | They propose token-level and semantic-level code-switched masked language modeling to improve cross-lingual interactions over mono-mPLMs without parallel sentences. |
| Outcome: | The proposed method outperforms mono-mPLMs on natural language understanding and unsupervised machine translation tasks. |
Leveraging LLMs for Synthesizing Training Data Across Many Languages in Multilingual Dense Retrieval (2024.naacl-long)
Copied to clipboard
| Challenge: | et al., 2020: performance of dense retrieval models in multilingual retrieval is limited due to uneven and scarce training data available across multiple languages. |
| Approach: | They propose a synthetic retrieval training dataset containing 33 languages for fine-tuning multilingual retrievers without human supervision. |
| Outcome: | The proposed model outperforms human-supervised retrieval models on three retrieval benchmarks. |
mPMR: A Multilingual Pre-trained Machine Reader at Scale (2023.acl-short)
Copied to clipboard
| Challenge: | Existing mPLMs only transfer NLU capability from source to target languages . mPMR allows direct inheritance of multilingual NLU capabilities to downstream tasks . |
| Approach: | They propose a method to guide multilingual pre-trained language models to perform natural language understanding in multiple languages. |
| Outcome: | mPMR enables multilingual pre-trained language models to perform natural language understanding (NLU) in multiple languages. |
Cross-View Language Modeling: Towards Unified Cross-Lingual Cross-Modal Pre-training (2023.acl-long)
Copied to clipboard
| Challenge: | Empirical results show that CCLM significantly outperforms the prior state-of-the-art with an average absolute improvement of over 10%. |
| Approach: | They introduce a pre-training framework that unifies cross-lingual and cross-modal pre-trained models with shared architectures and objectives. |
| Outcome: | The proposed framework outperforms the state-of-the-art in two multi-lingual datasets and two multilingual image-text retrieval datasets. |