Multilingual Multimodal Pre-training for Zero-Shot Cross-Lingual Transfer of Vision-Language Models (2021.naacl-main)
Copied to clipboard
| Challenge: | a new study examines zero-shot cross-lingual transfer of vision-language models . we study multilingual text-to-video search in non-English languages without annotations . |
| Approach: | They propose a Transformer-based model that learns contextual multilingual multimodal embeddings . they propose 'zero-shot cross-lingual transfer' to improve multilingual search . |
| Outcome: | The proposed model outperforms baselines on multilingual text-to-video search and multilingual image search on VTT and VATEX. |
Similar Papers
A Simple and Effective Method to Improve Zero-Shot Cross-Lingual Transfer Learning (2022.coling-1)
Copied to clipboard
Kunbo Ding, Weijie Liu, Yuejian Fang, Weiquan Mao, Zhe Zhao, Tao Zhu, Haoyan Liu, Rong Tian, Yiren Chen
| Challenge: | Existing zero-shot cross-lingual transfer methods rely on parallel corpora or bilingual dictionaries . however, its effect is limited by the gap between embedding clusters of different languages . |
| Approach: | They propose Embedding-Push, Attention-Pull, and Robust targets to transfer English embeddings to virtual multilingual embedders without semantic loss. |
| Outcome: | Experimental results show that the proposed method outperforms existing methods on cross-lingual tasks and can achieve a better multilingual alignment. |
Multi Task Learning For Zero Shot Performance Prediction of Multilingual Models (2022.acl-long)
Copied to clipboard
| Challenge: | Massively Multilingual Transformer based Language Models have been shown to be effective on zero-shot transfer across languages, though performance varies from language to language depending on pivot language(s) used for fine-tuning. |
| Approach: | They propose to combine multi-task learning problems with multi-lingual Transformers to model zero-shot transfer across languages. |
| Outcome: | The proposed model can predict zero-shot transfer across languages with a multi-task learning problem with pretraining data in very few languages. |
Cross-lingual Cross-modal Pretraining for Multimodal Retrieval (2021.naacl-main)
Copied to clipboard
| Challenge: | Recent pretrained vision-language models have achieved impressive performance on cross-modal retrieval tasks in English. |
| Approach: | They propose a new approach to learn cross-lingual cross-modal representations for matching images and captions in multiple languages using an annotated corpus. |
| Outcome: | The proposed model achieves impressive performance on two multimodal multilingual image caption benchmarks: Multi30k with German captions and MSCOCO with Japanese captions. |
Few-shot Learning with Multilingual Generative Language Models (2022.emnlp-main)
Copied to clipboard
Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu Wang, Shuohui Chen, Daniel Simig, Myle Ott, Naman Goyal, Shruti Bhosale, Jingfei Du, Ramakanth Pasunuru, Sam Shleifer, Punit Singh Koura, Vishrav Chaudhary, Brian O’Horo, Jeff Wang, Luke Zettlemoyer, Zornitsa Kozareva, Mona Diab, Veselin Stoyanov, Xian Li
| Challenge: | Large-scale generative language models such as GPT-3 are competitive few-shot learners. |
| Approach: | They train multilingual generative language models on a corpus covering a diverse set of languages and study their few- and zero-shot learning capabilities. |
| Outcome: | The proposed model outperforms GPT-3 on 171 out of 182 directions with 32 training examples and surpasses the official supervised baseline in 45 directions. |
Cross-lingual Visual Pre-training for Multimodal Machine Translation (2021.eacl-main)
Copied to clipboard
Ozan Caglayan, Menekse Kuyu, Mustafa Sercan Amac, Pranava Madhyastha, Erkut Erdem, Aykut Erdem, Lucia Specia
| Challenge: | Pre-trained language models have been shown to improve performance in many natural language tasks. |
| Approach: | They propose to combine cross-lingual and visual pre-training to learn visually-grounded cross-linguistic representations using masked region classification and three-way parallel vision & language corpora. |
| Outcome: | The proposed models obtain state-of-the-art performance when fine-tuned for multimodal machine translation. |
Can Monolingual Pretrained Models Help Cross-Lingual Classification? (2020.aacl-main)
Copied to clipboard
| Challenge: | Multilingual pretrained language models have shown impressive results for cross-lingual transfer, but due to the constant model capacity, multilingual pre-training usually lags behind the monolingual competitors. |
| Approach: | They propose to transfer the knowledge from monolingual pretrained models to multilingual ones to improve zero-shot cross-lingual classification by using machine translation systems. |
| Outcome: | The proposed methods outperform vanilla multilingual fine-tuning on two cross-lingual classification benchmarks. |
Zero-shot Reading Comprehension by Cross-lingual Transfer Learning with Multi-lingual Language Representation Model (D19-1)
Copied to clipboard
| Challenge: | Existing studies on RC datasets in English have limited results due to lack of training data. |
| Approach: | They systematically explore zero-shot cross-lingual transfer learning on reading comprehension tasks with pre-trained language representation model. |
| Outcome: | The proposed model performs well on reading comprehension tasks on pre-trained language representation models. |
Unifying Cross-Lingual and Cross-Modal Modeling Towards Weakly Supervised Multilingual Vision-Language Pre-training (2023.acl-long)
Copied to clipboard
| Challenge: | Existing studies address the problem of translating English data into other languages, but they are limited in form and scale. |
| Approach: | They propose a framework to unify cross-lingual and cross-modal pre-training by using English data. |
| Outcome: | The proposed framework unifies cross-lingual and cross-modal pre-training on different data. |
Towards Making the Most of Cross-Lingual Transfer for Zero-Shot Neural Machine Translation (2022.acl-long)
Copied to clipboard
| Challenge: | Existing unsupervised neural machine translation systems can degrade when labeled data is limited. |
| Approach: | They propose a multilingual pretraining and multilingual fine-tuning for facilitating cross-lingual transfer in zero-shot translation using a parallel dataset. |
| Outcome: | The proposed model outperforms state-of-the-art models on many-to-English translation by over 7.2 and 5.0 BLEU. |
How to Adapt Pre-trained Vision-and-Language Models to a Text-only Input? (2022.coling-1)
Copied to clipboard
| Challenge: | Current language models have been criticised for learning language from text alone without connection between words and their meaning. |
| Approach: | They propose to train models on more sources than text to provide the lacking connection between words and their meanings. |
| Outcome: | The proposed model adaptation methods perform differently for different models and unimodal model counterparts perform on par with the VL models regardless of adaptation. |