Cesar Ilharco, Afsaneh Shirazi, Arjun Gopalan, Arsha Nagrani, Blaz Bratanic, Chris Bregler, Christina Funk, Felipe Ferreira, Gabriel Barcik, Gabriel Ilharco, Georg Osang, Jannis Bulian, Jared Frank, Lucas Smaira, Qin Cao, Ricardo Marino, Roma Patel, Thomas Leung, Vaiva Imbrasaite
| Challenge: | This tutorial introduces the multimodal entailment task for detecting semantic alignments . the task requires fine-grained understanding of visual and linguistic semantics questions . |
| Approach: | This tutorial introduces the multimodal entailment task to machine learning . it introduces a dataset for recognizing multimodal alignments . |
| Outcome: | This tutorial introduces the multimodal entailment task . it can be useful for detecting semantic alignments when a single modality alone is not enough . |
Similar Papers
Tutorial on Multimodal Machine Learning (2022.naacl-tutorials)
Copied to clipboard
| Challenge: | Multimodal machine learning is a challenging but crucial area with numerous applications in multimedia, affective computing, robotics, finance, HCI, and healthcare. |
| Approach: | This tutorial will describe an updated taxonomy on multimodal machine learning synthesizing its core technical challenges and major directions for future research. |
| Outcome: | The proposed taxonomy synthesizes the core technical challenges and major directions for future research. |
Multimodality for NLP-Centered Applications: Resources, Advances and Frontiers (2022.lrec-1)
Copied to clipboard
| Challenge: | resurgence of multimodal datasets has attracted significant research interest, but there is no comprehensive survey for this task. |
| Approach: | They present a survey of a multimodal dataset with different modalities according to the applications. |
| Outcome: | The proposed datasets are available online and discuss the new frontier and motivate future researches. |
Multimodal Logical Inference System for Visual-Textual Entailment (P19-2)
Copied to clipboard
| Challenge: | Recent studies of multimodal inference provide challenging tasks such as visual question answering and visual reasoning. |
| Approach: | They propose an unsupervised multimodal logical inference system that can prove entailment relations between texts and images by combing semantic parsing and theorem proving. |
| Outcome: | The proposed system can handle semantically complex sentences for visual-textual inference. |
Exploring and Evaluating Multimodal Knowledge Reasoning Consistency of Multimodal Large Language Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | MLLMs have achieved significant breakthroughs in understanding across text and vision, but current models still face inconsistencies in reasoning outcomes. |
| Approach: | They propose to evaluate multimodal large language models using a multimodal knowledge reasoning dataset to examine the extent of consistency degradation. |
| Outcome: | The proposed evaluation tasks show that MLLMs are inefficient at integrating knowledge across modalities . |
Aligning Images and Text with Semantic Role Labels for Fine-Grained Cross-Modal Understanding (2022.lrec-1)
Copied to clipboard
| Challenge: | Currently, image retrieval systems can retrieve relevant results for diverse inputs, but they do not provide a way to intentionally inject variety into the search results. |
| Approach: | They propose a multimodal dataset that combines semantic annotations with image bounding boxes. |
| Outcome: | The proposed system improves image retrieval performance and flexibility. |
Multi-Head Attention with Diversity for Learning Grounded Multilingual Multimodal Representations (D19-1)
Copied to clipboard
| Challenge: | Recent studies have advanced learning VSE under the monolingual setup. |
| Approach: | They propose a model with diverse multi-head attention to learn grounded multilingual multimodal representations by leveraging visual object detection. |
| Outcome: | The proposed model performs well in German-Image and English-Image matching tasks and in the Semantic Textual Similarity task with English descriptions of visual content. |
MM-AVS: A Full-Scale Dataset for Multi-modal Summarization (2021.naacl-main)
Copied to clipboard
| Challenge: | Multimodal summarization materials lacking a holistic organization by integrating resources from various modalities. |
| Approach: | They propose a multimodal article and video summarization dataset that integrates resources from different modalities. |
| Outcome: | The proposed dataset validates the important assistance role of external information for multimodal summarization. |
Retrieving Multimodal Information for Augmented Generation: A Survey (2023.findings-emnlp)
Copied to clipboard
Ruochen Zhao, Hailin Chen, Weishi Wang, Fangkai Jiao, Xuan Long Do, Chengwei Qin, Bosheng Ding, Xiaobao Guo, Minzhi Li, Xingxuan Li, Shafiq Joty
| Challenge: | Large Language Models (LLMs) are increasingly using multimodality to augment their generation ability, but there is no unified perception of at which stage and how to incorporate different modalities. |
| Approach: | They propose to use multimodality to augment Large Language Models (LLMs) this will provide scholars with a deeper understanding of the methods' applications and encourage them to adapt existing techniques to the fast-growing field of LLMs. |
| Outcome: | The proposed methods improve factuality, reasoning, interpretability, and robustness of the generated content. |
Rethinking Multimodal Entity and Relation Extraction from a Translation Point of View (2023.acl-long)
Copied to clipboard
| Challenge: | Special attention is paid to the cross-modal misalignment in text-image datasets which may mislead the learning. |
| Approach: | They propose a multimodal back-translation method which uses diffusion-based generative models for pseudo-paralleled pairs and a divergence estimator to construct a high-resource corpora as a bridge for low-ressource learners. |
| Outcome: | The proposed method outperforms 14 state-of-the-art methods in both entity and relation extraction tasks. |
Multimodal Large Language Models for Human-AI Interaction: Foundations, Agents, and Inclusive Applications (2026.eacl-tutorials)
Copied to clipboard
| Challenge: | This tutorial presents foundations, agentic capabilities, and inclusive applications of multimodal large language models. |
| Approach: | This tutorial presents foundations, agentic capabilities, and inclusive applications of multimodal large language models. |
| Outcome: | This tutorial covers foundations, agentic capabilities, and inclusive applications of multimodal large language models. |