| Challenge: | Existing models for visual dialog infer the answer through multiple reasoning steps. |
| Approach: | They propose a model for visual dialog that uses multi-step reasoning to answer questions about an image. |
| Outcome: | The proposed model achieves a new state-of-the-art of 64.47% on the VisDial v1.0 dataset . |
Similar Papers
Dual Attention Networks for Visual Reference Resolution in Visual Dialog (D19-1)
Copied to clipboard
| Challenge: | Visual dialog (VisDial) requires a dialog agent to answer a series of questions grounded in an image. |
| Approach: | They propose dual attention networks (DAN) for visual reference resolution in VisDial. |
| Outcome: | The proposed model outperforms the previous state-of-the-art model on VisDial datasets. |
Dialog Generation Using Multi-Turn Reasoning Neural Networks (N18-1)
Copied to clipboard
| Challenge: | Existing methods for dialog generation are limited and short at generalization. |
| Approach: | They propose a generalizable dialog generation approach that adapts multi-turn reasoning to generate responses by taking current conversation session context as a document and current query as 'question' they separate the single memory used for document comprehension into different groups for speaker-specific topic and opinion embedding. |
| Outcome: | Experiments on Japanese 10-sentence (5-round) conversation modeling show that multi-turn reasoning can produce more diverse and acceptable responses than state-of-the-art single-turn and non-reasoning baselines. |
Tell-and-Answer: Towards Explainable Visual Question Answering using Attributes and Captions (D18-1)
Copied to clipboard
| Challenge: | Existing approaches to visual question answering represent images using pre-trained CNNs . but they rarely provide any insight, apart from the answer, into the VQA process . |
| Approach: | They propose to break up the end-to-end VQA into two steps: explaining and reasoning . they first extract attributes and generate descriptions as explanations for an image . a reasoning module utilizes these explanations in place of the image to infer an answer . |
| Outcome: | The proposed system achieves comparable performance with baselines, but with added benefits of explanability and the ability to improve with higher quality explanations. |
II-MMR: Identifying and Improving Multi-modal Multi-hop Reasoning in Visual Question Answering (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing studies have focused on assessing the model’s overall accuracy without evaluating it on different reasoning cases. |
| Approach: | They propose a novel idea to identify and improve multi-modal multi-hop reasoning in VQA by using two new language prompts to find a reasoning path to reach its answer. |
| Outcome: | The proposed model improves multi-modal multi-hop reasoning in visual question answering (VQA) it finds that the proposed model is easy to answer, simply demanding “single-hop” reasoning, whereas only a few questions require “multi-hop.” |
Causal Reasoning through Two Cognition Layers for Improving Generalization in Visual Question Answering (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing attempts to generalize VQA focus on unimodal aspects, overlooking enhancements in multimodal aspects. |
| Approach: | They propose to decompose the responsibility of each stage into distinct experts and a cognition-enabled component (CC) they prioritize answer predictions governed by pathways involving both CCs while disregarding answers produced by either CC. |
| Outcome: | The proposed model improves multimodal predictions by emphasizing causal reasoning factors. |
Neuro-Symbolic Visual Dialog (2022.coling-1)
Copied to clipboard
| Challenge: | Existing methods for visual dialog require large amounts of training data, which is prohibitive for most settings. |
| Approach: | They propose a method that integrates deep learning and symbolic program execution for multi-round visual reasoning. |
| Outcome: | The proposed model outperforms existing methods on long-distance co-reference resolution and vanishing question-answering performance. |
Check It Again:Progressive Visual Question Answering via Visual Entailment (2021.acl-long)
Copied to clipboard
| Challenge: | Existing approaches to Visual Question Answering (VQA) only address superficial correlations between image and answer. |
| Approach: | They propose a select-and-rerank progressive framework based on Visual Entailment to address this problem. |
| Outcome: | The proposed framework improves on the Visual Question Answering (VQA) task with 7.55% accuracy. |
Co-VQA : Answering by Interactive Sub Question Sequence (2022.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to Visual Question Answering (VQA) answer questions directly, but people usually decompose a complex question into a sequence of simple sub questions. |
| Approach: | They propose a conversation-based VQA framework that decomposes questions into sub questions and answers them one-by-one. |
| Outcome: | The proposed framework achieves state-of-the-art on VQA 2.0 and VQA-CP v2 datasets. |
CLEVR-Dialog: A Diagnostic Dataset for Multi-Round Reasoning in Visual Dialog (N19-1)
Copied to clipboard
| Challenge: | Visual Dialog is a multimodal task of answering a sequence of questions grounded in an image. |
| Approach: | They construct a dialog grammar that is grounded in the scene graphs of the images from the CLEVR dataset and use it to benchmark performance of standard visual dialog models. |
| Outcome: | The proposed model is based on a large diagnostic dataset for studying multi-round reasoning in visual dialog. |
MAGIC-VQA: Multimodal And Grounded Inference with Commonsense Knowledge for Visual Question Answering (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing Large Vision-Language Models (LVLMs) lack integrated commonsense knowledge . lack of integrated common knowledge limits their robustness and accuracy in VQA . |
| Approach: | They propose a framework to enhance multimodal inference by integrating commonsense reasoning. |
| Outcome: | MAGIC-VQA improves comprehensive benchmark datasets, surpassing existing models in tasks requiring advanced commonsense reasoning. |