VLM: Task-agnostic Video-Language Model Pre-training for Video Understanding (2021.findings-acl)
Copied to clipboard
Hu Xu, Gargi Ghosh, Po-Yao Huang, Prahal Arora, Masoumeh Aminzadeh, Christoph Feichtenhofer, Florian Metze, Luke Zettlemoyer
| Challenge: | Existing methods for multimodal video understanding are task-specific, limiting their use for retrieval-style end tasks. |
| Approach: | They propose a task-agnostic multimodal pre-training approach that can accept video or text input, or both, for a variety of end tasks. |
| Outcome: | The proposed approach outperforms existing methods on a wider range of tasks while maintaining separability. |
Similar Papers
End-to-End Unsupervised Vision-and-Language Pre-training with Referring Expression Matching (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing unsupervised vision-and-language pre-training methods take pre-extracted region-based visual features from external object detectors, which limits flexibility and reduces computational efficiency. |
| Approach: | They propose an unsupervised vision-and-language pre-training task that predicts which patches contain an object referred to in natural language from the encoded visual features. |
| Outcome: | The proposed approach outperforms existing methods and obtains state-of-the-art results on four vision-and-language tasks. |
Fusion or Defusion? Flexible Vision-and-Language Pre-Training (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to vision-and-language pretraining (VLP) lack effectiveness and efficiency in downstream multimodal tasks. |
| Approach: | They propose a flexible vision-and-language pre-training model by incorporating cross-modal fusions into a dual-encoder architecture and a cross-module knowledge transfer strategy to guide the training process. |
| Outcome: | The proposed model is well-equipped with effectiveness and efficiency compared with other strong VLP models. |
Cross-lingual Visual Pre-training for Multimodal Machine Translation (2021.eacl-main)
Copied to clipboard
Ozan Caglayan, Menekse Kuyu, Mustafa Sercan Amac, Pranava Madhyastha, Erkut Erdem, Aykut Erdem, Lucia Specia
| Challenge: | Pre-trained language models have been shown to improve performance in many natural language tasks. |
| Approach: | They propose to combine cross-lingual and visual pre-training to learn visually-grounded cross-linguistic representations using masked region classification and three-way parallel vision & language corpora. |
| Outcome: | The proposed models obtain state-of-the-art performance when fine-tuned for multimodal machine translation. |
How to Adapt Pre-trained Vision-and-Language Models to a Text-only Input? (2022.coling-1)
Copied to clipboard
| Challenge: | Current language models have been criticised for learning language from text alone without connection between words and their meaning. |
| Approach: | They propose to train models on more sources than text to provide the lacking connection between words and their meanings. |
| Outcome: | The proposed model adaptation methods perform differently for different models and unimodal model counterparts perform on par with the VL models regardless of adaptation. |
CLASP: Cross-modal Alignment Using Pre-trained Unimodal Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent advances in speech-text pretraining rely on parallel speech- text data . however, data accessibility is a challenge due to the limited data available. |
| Approach: | They propose a framework for jointly performing speech and text processing without parallel corpora during pre-training but only downstream. |
| Outcome: | The proposed framework extracts distinct representations for speech and text, aligning them effectively in a newly defined space using a multi-level contrastive learning mechanism. |
Multimodal Unlearning Across Vision, Language, Video, and Audio: Survey of Methods, Datasets, and Benchmarks (2026.findings-acl)
Copied to clipboard
| Challenge: | Survey aims to identify challenges of multimodal unlearning for vision, language, audio and video . retraining after deletion requests or policy updates is often impractical, survey finds . |
| Approach: | They propose to enable selective removal across modalities while retaining overall utility. |
| Outcome: | This study compares models with existing models to identify weaknesses and improves performance. |
Grafting Pre-trained Models for Multimodal Headline Generation (2022.emnlp-industry)
Copied to clipboard
| Challenge: | Existing approaches to generate video headlines with pre-trained language models are labor intensive and impractical. |
| Approach: | They propose to graft the encoder from the pre-trained video-language model on the generative pre-trainer model and propose a consensus fusion mechanism for the integration of different components. |
| Outcome: | The proposed model achieves strong results on a brand-new dataset collected from real-world applications. |
CDLM: Cross-Document Language Modeling (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Existing language models (LMs) provide powerful representations for internal text structure, but there are important applications for multi-text tasks. |
| Approach: | They propose a pretraining approach that incorporates two key ideas into the masked language modeling objective. |
| Outcome: | The proposed model improves over existing models and sets of long-range transformers and can be easily applied to multiple multi-text tasks. |
VIMI: Grounding Video Generation through Multi-modal Instruction (2024.emnlp-main)
Copied to clipboard
Yuwei Fang, Willi Menapace, Aliaksandr Siarohin, Tsai-Shien Chen, Kuan-Chieh Wang, Ivan Skorokhodov, Graham Neubig, Sergey Tulyakov
| Challenge: | Existing text-to-video diffusion models rely on text-only encoders for their pretraining, restricting their versatility and application in multimodal integration. |
| Approach: | They propose a multimodal conditional video generation framework for pretraining on augmented text prompts and then utilize a two-stage training strategy to enable diverse video generation tasks within a model. |
| Outcome: | The proposed model can synthesize consistent and temporally coherent videos with large motion while retaining the semantic control. |
Vision-Language Pre-Training for Multimodal Aspect-Based Sentiment Analysis (2022.acl-long)
Copied to clipboard
| Challenge: | Existing approaches to multimodal Aspect-Based Sentiment Analysis (MABSA) ignore crossmodalalignment and use pre-trained visual and textual models. |
| Approach: | They propose a multimodal multimodal encoder-decoder framework for MABSA that uses a unified multimodal decoder architecture for all the pretrainingand downstream tasks. |
| Outcome: | The proposed framework outperforms state-of-the-art approaches on three MABSA subtasks. |