Enhancing Cross-lingual Sentence Embedding for Low-resource Languages with Word Alignment (2024.findings-naacl)
Copied to clipboard
| Challenge: | Current approaches to obtain cross-lingual sentence embeddings rely on pre-trained language models that implicitly align the contextual representations of similar units of sentences in different languages. |
| Approach: | They propose a framework that explicitly aligns words between English and eight low-resource languages by using off-the-shelf word alignment models. |
| Outcome: | The proposed framework improves on the bitext retrieval task and in high-resource languages. |
Similar Papers
Cross-Align: Modeling Deep Cross-lingual Interactions for Word Alignment (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing word alignment models capture few interactions between input sentence pairs, which severely degrades the word alignment quality. |
| Approach: | They propose to model deep interactions between input and target sentences using a two-stage training framework to train the model. |
| Outcome: | The proposed model achieves the state-of-the-art (SOTA) performance on four out of five language pairs. |
Massively Multilingual Document Alignment with Cross-lingual Sentence-Mover’s Distance (2020.aacl-main)
Copied to clipboard
| Challenge: | Document alignment aims to identify pairs of documents in two distinct languages that are of comparable content or translations of each other. |
| Approach: | They propose an unsupervised scoring function that leverages cross-lingual sentence embeddings to compute the semantic distance between documents in different languages. |
| Outcome: | The proposed scoring function outperforms baseline methods on high-resource language pairs, 15% on mid-resourced language pairs and 22% on low-resourcing language pairs. |
Aligning Cross-lingual Sentence Representations with Dual Momentum Contrast (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing work uses sentences within the same batch as negatives, which suffers from easy negatives. |
| Approach: | They propose to align sentence representations from different languages into a unified embedding space . they adapt MoCo to further improve the quality of alignment . |
| Outcome: | The proposed model achieves state-of-the-art on several tasks. |
ACROSS: An Alignment-based Framework for Low-Resource Many-to-One Cross-Lingual Summarization (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing studies ignore data imbalance in multilingual settings and do not utilize monolingual data. |
| Approach: | They propose a cross-lingual summarization model that aligns cross-linguistic data with high-resource monolingual data via contrastive and consistency loss. |
| Outcome: | The proposed model outperforms baseline models and consistently dominates on 45 language pairs. |
Improving Low-Resource Cross-lingual Document Retrieval by Reranking with Deep Bilingual Representations (P19-1)
Copied to clipboard
Rui Zhang, Caitlin Westerfield, Sungrok Shim, Garrett Bingham, Alexander Fabbri, William Hu, Neha Verma, Dragomir Radev
| Challenge: | Experimental results show that our model outperforms competitive translation-based baselines on cross-lingual relevance ranking tasks. |
| Approach: | They propose to match queries and documents in both source and target languages with deep bilingual query-document representations. |
| Outcome: | The proposed model outperforms translation-based baselines on English-Swahili, English-Tagalog, and English-Somali cross-lingual retrieval tasks. |
A Massively Multilingual Analysis of Cross-linguality in Shared Embedding Space (2021.emnlp-main)
Copied to clipboard
| Challenge: | Cross-lingual language models house representations for many different languages in the same space. |
| Approach: | They investigate linguistic and non-linguistic factors affecting sentence-level alignment in cross-lingual pretrained language models for 101 languages and 5,050 language pairs. |
| Outcome: | The results show that word order agreement and agreement in morphological complexity are strongest predictors of cross-linguality. |
Improving Low-Resource Languages in Pre-Trained Multilingual Language Models (2022.emnlp-main)
Copied to clipboard
| Challenge: | Pre-trained multilingual language models are the foundation of many NLP approaches, but are often not well-supported by these models due to small available monolingual corpora. |
| Approach: | They propose an unsupervised approach to improve cross-lingual representations of low-resource languages by bootstrapping word translation pairs from monolingual corpora and using them to improve language alignment. |
| Outcome: | The proposed approach improves cross-lingual representations on low-resource languages using word retrieval and zero-shot named entity recognition. |
Meeting the Needs of Low-Resource Languages: The Value of Automatic Alignments via Pretrained Models (2023.eacl-main)
Copied to clipboard
Abteen Ebrahimi, Arya D. McCarthy, Arturo Oncevay, John E. Ortega, Luis Chiruzzo, Gustavo Giménez-Lugo, Rolando Coto-Solano, Katharina Kann
| Challenge: | Large multilingual models have inspired a new class of word alignment methods, which work well for pretraining languages. |
| Approach: | They propose to use transformer-based word alignment methods to extract alignments from massive pretrained models. |
| Outcome: | The proposed methods outperform traditional methods for languages unseen to pretraining models, and are competitive with each other. |
Sub-Word Alignment is Still Useful: A Vest-Pocket Method for Enhancing Low-Resource Machine Translation (2022.acl-short)
Copied to clipboard
| Challenge: | Low-resource machine translation (MT) is challenging due to the scarcity of parallel data and lack of bilingual dictionaries. |
| Approach: | They propose to leverage embedding duplication between aligned sub-words to extend the Parent-Child transfer learning method to improve low-resource machine translation. |
| Outcome: | The proposed method achieves BLEU scores of 22.5, 28.0 and 18.1 respectively. |
Understanding Cross-Lingual Alignment—A Survey (2024.findings-acl)
Copied to clipboard
| Challenge: | Cross-lingual alignment is the meaningful similarity of representations across languages in multilingual language models. |
| Approach: | They propose a taxonomy of methods to improve cross-lingual alignment . they argue that an effective trade-off between language-neutral and language-specific information is key . |
| Outcome: | The proposed methods can be applied to encoder models and encoder-decoder-only models . they show that language-neutral and language-specific information is key . |