CTAL: Pre-training Cross-modal Transformer for Audio-and-Language Representations (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing audio-language task-specific predictive approaches focus on building complicated late-fusion mechanisms. |
| Approach: | They propose a cross-modal transformer for audio-and-language that learns inter-modal connections between audio and language through two proxy tasks on a large amount of audio- and-language pairs. |
| Outcome: | The proposed model improves on multiple audio-and-language tasks and can be used in fine-tuning phase. |
Similar Papers
Vision-and-Language or Vision-for-Language? On Cross-Modal Influence in Multimodal Transformers (2021.emnlp-main)
Copied to clipboard
| Challenge: | Pretrained vision-and-language BERTs aim to learn representations that combine information from both modalities. |
| Approach: | They propose a diagnostic method based on cross-modal input ablation to assess the extent to which pretrained models integrate cross-module information. |
| Outcome: | The proposed method evaluates the model's performance on the other modality based on inputs from one or both modality. |
SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processing (2022.acl-long)
Copied to clipboard
Junyi Ao, Rui Wang, Long Zhou, Chengyi Wang, Shuo Ren, Yu Wu, Shujie Liu, Tom Ko, Qing Li, Yu Zhang, Zhihua Wei, Yao Qian, Jinyu Li, Furu Wei
| Challenge: | Existing work shows that pre-trained models can improve in various natural language processing tasks. |
| Approach: | They propose a unified-modal encoder-decoder framework that pre-trains speech-text representations using large-scale unlabeled speech and text data. |
| Outcome: | The proposed framework is superior to existing models on speech-to-text processing tasks. |
Self-supervised Cross-modal Pretraining for Speech Emotion Recognition and Sentiment Analysis (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to multimodal speech emotion recognition and sentiment analysis have not improved results due to their relatively simple fusion mechanisms and lack of proper cross-modal pretraining. |
| Approach: | They propose a deep-fused audio-text bi-modal transformer with carefully designed cross-modal fusion mechanism and stage-wise cross-mod pretraining scheme to facilitate cross-modulation. |
| Outcome: | The proposed method exceeds benchmarks on public IEMOCAP emotion and CMU-MOSEI sentiment datasets by a large margin. |
Cross-View Language Modeling: Towards Unified Cross-Lingual Cross-Modal Pre-training (2023.acl-long)
Copied to clipboard
| Challenge: | Empirical results show that CCLM significantly outperforms the prior state-of-the-art with an average absolute improvement of over 10%. |
| Approach: | They introduce a pre-training framework that unifies cross-lingual and cross-modal pre-trained models with shared architectures and objectives. |
| Outcome: | The proposed framework outperforms the state-of-the-art in two multi-lingual datasets and two multilingual image-text retrieval datasets. |
CLASP: Cross-modal Alignment Using Pre-trained Unimodal Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent advances in speech-text pretraining rely on parallel speech- text data . however, data accessibility is a challenge due to the limited data available. |
| Approach: | They propose a framework for jointly performing speech and text processing without parallel corpora during pre-training but only downstream. |
| Outcome: | The proposed framework extracts distinct representations for speech and text, aligning them effectively in a newly defined space using a multi-level contrastive learning mechanism. |
Phone-ing it in: Towards Flexible Multi-Modal Language Model Training by Phonetic Representations of Data (2022.acl-long)
Copied to clipboard
| Challenge: | Pre-trained language models are increasingly applied in ways that are agnostic to targeted downstream tasks. |
| Approach: | They propose a multi-modal approach to train language models using whatever text and/or audio data might be available in a language. |
| Outcome: | The proposed approach improves on pre-trained models on Swahili and Kinyarwanda data, with an improvement of up to 6% over models that are trained from scratch. |
RC3: Regularized Contrastive Cross-lingual Cross-modal Pre-training (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing V&L pre-training methods rely on strictly-aligned multilingual image-text pairs generated from English-centric datasets. |
| Approach: | They propose a regularized cross-lingual visual contrastive learning objective that constrains representation proximity of weakly-aligned multilingual image-text pairs. |
| Outcome: | The proposed model outperforms competing models with weak zero-shot capability on 5 multi-modal tasks across 6 languages. |
Unified Speech-Text Pre-training for Speech Translation and Recognition (2022.acl-long)
Copied to clipboard
Yun Tang, Hongyu Gong, Ning Dong, Changhan Wang, Wei-Ning Hsu, Jiatao Gu, Alexei Baevski, Xian Li, Abdelrahman Mohamed, Michael Auli, Juan Pino
| Challenge: | Existing methods to pre-train speech and text use unlabeled data to learn universal feature representations. |
| Approach: | They propose a method to jointly pre-train speech and text in an encoder-decoder modeling framework for speech translation and recognition. |
| Outcome: | The proposed method achieves between 1.7 and 2.3 BLEU improvement above the state of the art on the MuST-C speech translation dataset and comparable WERs to wav2vec 2.0 on the Librispeech speech recognition task. |
UniXcoder: Unified Cross-Modal Pre-training for Code Representation (2022.acl-long)
Copied to clipboard
| Challenge: | Pre-trained models for programming languages have demonstrated great success on code intelligence . however, such pre-tried models are sub-optimal for auto-regressive tasks . |
| Approach: | They propose a unified cross-modal pre-trained model for programming language that leverages cross-module contents like AST and code comment to enhance code representation. |
| Outcome: | The proposed model achieves state-of-the-art on most code-related tasks and compares with existing models on zero-shot code-to-code search. |
Cross-lingual Visual Pre-training for Multimodal Machine Translation (2021.eacl-main)
Copied to clipboard
Ozan Caglayan, Menekse Kuyu, Mustafa Sercan Amac, Pranava Madhyastha, Erkut Erdem, Aykut Erdem, Lucia Specia
| Challenge: | Pre-trained language models have been shown to improve performance in many natural language tasks. |
| Approach: | They propose to combine cross-lingual and visual pre-training to learn visually-grounded cross-linguistic representations using masked region classification and three-way parallel vision & language corpora. |
| Outcome: | The proposed models obtain state-of-the-art performance when fine-tuned for multimodal machine translation. |