MIRTT: Learning Multimodal Interaction Representations from Trilinear Transformers for Visual Question Answering (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Existing bilinear methods focus on inter-modality information between images and questions . existing models focus on the interaction between images, questions, and images . |
| Approach: | They propose a trilinear interaction framework that incorporates attention mechanisms for capturing inter-modality and intra-modal relationships. |
| Outcome: | The proposed model outperforms bilinear models on the Visual7W Telling task and VQA-1.0 Multiple Choice task and outperformed baselines on the VQA, TDIUC and GQA datasets. |
Similar Papers
Open-Ended Visual Question Answering by Multi-Modal Domain Adaptation (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to visual question answering (VQA) are not suitable for real-world applications. |
| Approach: | They propose a supervised multi-modal domain adaptation method for visual question answering in images that exploits supervised domain adaptation. |
| Outcome: | The proposed method outperforms state-of-the-art methods on the benchmark VQA 2.0 and VizWiz datasets. |
Multimodal Graph Transformer for Multimodal Question Answering (2023.eacl-main)
Copied to clipboard
| Challenge: | a myriad of complex tasks require both prior knowledge and reasoning intelligence. |
| Approach: | They propose a plug-and-play quasi-attention mechanism to integrate multimodal graph information to vanilla self-attention as effective prior. |
| Outcome: | The proposed model is able to perform reasoning across multiple modalities. |
Incorporating Probing Signals into Multimodal Machine Translation via Visual Question-Answering Pairs (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies show that multimodal machine translation systems exhibit decreased sensitivity to visual information when text inputs are complete. |
| Approach: | They propose to generate parallel VQA style pairs from source text to foster more robust cross-modal interaction. |
| Outcome: | The proposed approach generates parallel VQA style pairs from the source text, fostering more robust cross-modal interaction. |
MMCoQA: Conversational Question Answering over Text, Tables, and Images (2022.acl-long)
Copied to clipboard
| Challenge: | Existing conversational QA systems only use a single knowledge source, e.g., paragraphs or knowledge graph, and assume it contains enough evidence to extract answers to users' questions. |
| Approach: | They propose a task to answer users' questions with multimodal knowledge sources via multi-turn conversations using a multimodal dataset. |
| Outcome: | The proposed task brings a series of research challenges, including but not limited to priority, consistency, and complementarity of multimodal knowledge. |
MIRe: Enhancing Multimodal Queries Representation via Fusion-Free Modality Interaction for Multimodal Retrieval (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing methods focus on textual queries that include visual information, but lack the ability to address multimodal queries that encompass both textual and visual information. |
| Approach: | They propose a retrieval framework that achieves modality interaction without fusing textual features during the alignment. |
| Outcome: | The proposed method achieves modality interaction without fusing textual features during the alignment. |
Delving Deeper into Cross-lingual Visual Question Answering (2023.findings-eacl)
Copied to clipboard
| Challenge: | Existing studies on cross-lingual VQA have reported poor zero-shot transfer performance of current multilingual multimodal Transformers . lack of multilingual resources has hindered development and evaluation of VQA methods beyond the English language . |
| Approach: | They analyze cross-lingual VQA across different question types of varying complexity . they show that simple modifications to the standard training setup can substantially reduce the transfer gap to monolingual English performance. |
| Outcome: | The proposed model significantly reduces the transfer gap to monolingual English performance . the proposed model also improves on question types and languages . |
xGQA: Cross-Lingual Visual Question Answering (2022.findings-acl)
Copied to clipboard
Jonas Pfeiffer, Gregor Geigle, Aishwarya Kamath, Jan-Martin Steitz, Stefan Roth, Ivan Vulić, Iryna Gurevych
| Challenge: | a lack of multilingual multimodal datasets has hindered multimodal vision and language modeling efforts. |
| Approach: | They propose a multilingual evaluation benchmark for the visual question answering task . they extend the established English GQA dataset to 7 typologically diverse languages . |
| Outcome: | The proposed methods outperform current state-of-the-art models in zero-shot cross-lingual settings, but the accuracy remains low across languages. |
MMFT-BERT: Multimodal Fusion Transformer with BERT Encodings for Visual Question Answering (2020.findings-emnlp)
Copied to clipboard
| Challenge: | MMFT-BERT is a multimodal fusion transformer that decomposes input modalities into different BERT instances with similar architectures, but variable weights. |
| Approach: | They propose a multimodal fusion transformer with BERT encodings to solve Visual Question Answering (VQA) . |
| Outcome: | The proposed method achieves SOTA results on the TVQA dataset and TVQA-Visual, an isolated diagnostic subset of TVQA, which strictly requires the knowledge of visual (V) modality based on a human annotator’s judgment. |
Can We Learn Question, Answer, and Distractors All from an Image? A New Task for Multiple-choice Visual Question Answering (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing studies focus on generating QADs from image and question, but a novel task is needed to generate meaningful questions, correct answers, and challenging distractors. |
| Approach: | They propose a task to generate QADs from images and encode images together . they use contrastive learning to ensure consistency of QAD generated and tested . |
| Outcome: | Empirical evaluations on the benchmark dataset validate the performance of the proposed task. |
Seeing Beyond: Enhancing Visual Question Answering with Multi-Modal Retrieval (2025.coling-industry)
Copied to clipboard
| Challenge: | Multi-modal Large language models still suffer from model hallucination and lack of specific knowledge when answering challenging questions. |
| Approach: | They propose to use a multi-modal retrieval augmented generation method to integrate knowledge from all modalities into a model to enable alignment between query and knowledge. |
| Outcome: | The proposed method achieves significant performance improvement on the VQA dataset. |