| Challenge: | Existing language models (LMs) provide powerful representations for internal text structure, but there are important applications for multi-text tasks. |
| Approach: | They propose a pretraining approach that incorporates two key ideas into the masked language modeling objective. |
| Outcome: | The proposed model improves over existing models and sets of long-range transformers and can be easily applied to multiple multi-text tasks. |
Similar Papers
Cross-View Language Modeling: Towards Unified Cross-Lingual Cross-Modal Pre-training (2023.acl-long)
Copied to clipboard
| Challenge: | Empirical results show that CCLM significantly outperforms the prior state-of-the-art with an average absolute improvement of over 10%. |
| Approach: | They introduce a pre-training framework that unifies cross-lingual and cross-modal pre-trained models with shared architectures and objectives. |
| Outcome: | The proposed framework outperforms the state-of-the-art in two multi-lingual datasets and two multilingual image-text retrieval datasets. |
Emerging Cross-lingual Structure in Pretrained Language Models (2020.acl-main)
Copied to clipboard
| Challenge: | Recent work has shown that multilingual pretraining works, but is unable to measure these effects. |
| Approach: | They propose to use multilingual masked language modeling to train a model on concatenated text from multiple languages to find universal latent symmetries in embedding spaces. |
| Outcome: | The proposed models can be trained on concatenated text from multiple languages without shared vocabulary or domain similarity. |
LinkBERT: Pretraining Language Models with Document Links (2022.acl-long)
Copied to clipboard
| Challenge: | Existing language model pretraining methods do not capture dependencies or knowledge that span across documents. |
| Approach: | They propose a language model pretraining method that leverages links between documents . they use masked language modeling and document relation prediction to model LMs . |
| Outcome: | The proposed method outperforms existing methods on downstream tasks across two domains. |
Cross-lingual Visual Pre-training for Multimodal Machine Translation (2021.eacl-main)
Copied to clipboard
Ozan Caglayan, Menekse Kuyu, Mustafa Sercan Amac, Pranava Madhyastha, Erkut Erdem, Aykut Erdem, Lucia Specia
| Challenge: | Pre-trained language models have been shown to improve performance in many natural language tasks. |
| Approach: | They propose to combine cross-lingual and visual pre-training to learn visually-grounded cross-linguistic representations using masked region classification and three-way parallel vision & language corpora. |
| Outcome: | The proposed models obtain state-of-the-art performance when fine-tuned for multimodal machine translation. |
Combining Deep Generative Models and Multi-lingual Pretraining for Semi-supervised Document Classification (2021.eacl-main)
Copied to clipboard
| Challenge: | Semi-supervised learning and multilingual pretraining have been shown to be effective for task-specific labelled data shortages. |
| Approach: | They propose to combine semi-supervised deep generative models and multi-lingual pretraining to form a pipeline for document classification task. |
| Outcome: | The proposed method outperforms state-of-the-art models in low-resource settings across several languages and outperformed existing models in English. |
On-the-fly Cross-lingual Masking for Multilingual Pre-training (2023.acl-long)
Copied to clipboard
| Challenge: | In multilingual pre-training, multilingual models only learn cross-linguality implicitly from isomorphic spaces formed by overlapping different language spaces due to the lack of explicit cross-linguistic forward pass. |
| Approach: | They propose a dynamic token-wise masking scheme for multilingual pre-training that uses a special token [C]x to replace a random token in the input sentence. |
| Outcome: | The proposed model improves the performance of UNMT models on De, Ro, Ne En. |
Improving Pretrained Cross-Lingual Language Models via Self-Labeled Word Alignment (2021.acl-long)
Copied to clipboard
| Challenge: | Experimental results show that denoising word alignment improves cross-lingual transferability . most applications and resources are still English-centric, making non-English users hard to access. |
| Approach: | They propose to denoise word alignment as a cross-lingual pre-training task . they first self-label word alignments for parallel sentences and then mask tokens . |
| Outcome: | The proposed model improves cross-lingual transferability on token-level tasks, especially on question answering, and structured prediction. |
Explicit Cross-lingual Pre-training for Unsupervised Machine Translation (D19-1)
Copied to clipboard
| Challenge: | Existing approaches to build initial unsupervised machine translation models with cross-lingual n-gram embeddings are inexplicit and limited. |
| Approach: | They propose a cross-lingual pre-training method that incorporates cross-linguistic training signals into pre-trained models by randomly choosing source n-grams in the input text stream. |
| Outcome: | The proposed method significantly improves the performance of unsupervised machine translation. |
InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training (2021.naacl-main)
Copied to clipboard
Zewen Chi, Li Dong, Furu Wei, Nan Yang, Saksham Singhal, Wenhui Wang, Xia Song, Xian-Ling Mao, Heyan Huang, Ming Zhou
| Challenge: | Existing methods for learning cross-lingual representations are lacking in the field of NLP. |
| Approach: | They propose a framework that formulates cross-lingual language model pre-training as maximizing mutual information between multilingual-multi-granularity texts. |
| Outcome: | The proposed approach improves cross-lingual transferability on benchmarks. |
Universal Sentence Representation Learning with Conditional Masked Language Model (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to learn sentence representations on unlabeled corpora are difficult and expensive to obtain, making it hard to cover many domains and languages. |
| Approach: | They propose a method to train sentence representations on large unlabeled corpora by conditioning on the encoded vectors of adjacent sentences. |
| Outcome: | The proposed method outperforms existing models on SentEval and can be extended to a broad range of languages and domains. |