CLEVR-Dialog: A Diagnostic Dataset for Multi-Round Reasoning in Visual Dialog (N19-1)
Copied to clipboard
| Challenge: | Visual Dialog is a multimodal task of answering a sequence of questions grounded in an image. |
| Approach: | They construct a dialog grammar that is grounded in the scene graphs of the images from the CLEVR dataset and use it to benchmark performance of standard visual dialog models. |
| Outcome: | The proposed model is based on a large diagnostic dataset for studying multi-round reasoning in visual dialog. |
Similar Papers
Neuro-Symbolic Visual Dialog (2022.coling-1)
Copied to clipboard
| Challenge: | Existing methods for visual dialog require large amounts of training data, which is prohibitive for most settings. |
| Approach: | They propose a method that integrates deep learning and symbolic program execution for multi-round visual reasoning. |
| Outcome: | The proposed model outperforms existing methods on long-distance co-reference resolution and vanishing question-answering performance. |
CLEVR_HYP: A Challenge Dataset and Baselines for Visual Question Answering with Hypothetical Actions over Images (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing research on visual question answering is limited to information explicitly present in an image or a video. |
| Approach: | They propose a vision-language question answering task based on a CLEVR dataset . they modify existing methods and propose baseline solvers for this task . |
| Outcome: | The proposed model motivates the development of better vision-language models . it provides insights about the capability of diverse architectures to perform joint reasoning over image-text modality. |
DVD: A Diagnostic Dataset for Multi-step Reasoning in Video Grounded Dialogue (2021.acl-long)
Copied to clipboard
| Challenge: | Existing benchmarks do not have enough annotations to analyze video-grounded dialogue systems and understand their capabilities and limitations in isolation. |
| Approach: | They present a Diagnostic Dataset for Video-grounded dialogue with minimal biases and detailed annotations for the different types of reasoning over the spatio-temporal space of video. |
| Outcome: | The proposed system is based on 11k CATER synthetic videos and contains 10 instances of 10-round dialogues for each video. |
QLEVR: A Diagnostic Dataset for Quantificational Language and Elementary Visual Reasoning (2022.findings-naacl)
Copied to clipboard
| Challenge: | Synthetic datasets have been used to test visual question-answering datasets for reasoning abilities. |
| Approach: | They propose a visual question-answering dataset that is minimally biased and diagnostic . they propose to use the dataset to test visual reasoning abilities . |
| Outcome: | The proposed dataset is compared with existing models and shows it is far superior to existing models. |
The World in My Mind: Visual Dialog with Adversarial Multi-modal Feature Encoding (N19-1)
Copied to clipboard
| Challenge: | Visual Dialog is a multi-modal task that requires a model to participate in a dialog grounded on an image and generate correct, human-like responses. |
| Approach: | They propose a framework for effective and robust auxiliary training of visual dialog systems using multi-modal encoding. |
| Outcome: | The proposed framework outperforms supervised learning baselines and fine-tuning methods on most metrics of VisDial v0.5/v0.9 generative tasks. |
Multi-step Reasoning via Recurrent Dual Attention for Visual Dialog (P19-1)
Copied to clipboard
| Challenge: | Existing models for visual dialog infer the answer through multiple reasoning steps. |
| Approach: | They propose a model for visual dialog that uses multi-step reasoning to answer questions about an image. |
| Outcome: | The proposed model achieves a new state-of-the-art of 64.47% on the VisDial v1.0 dataset . |
MMDialog: A Large-scale Multi-turn Dialogue Dataset Towards Multi-modal Open-domain Conversation (2023.acl-long)
Copied to clipboard
| Challenge: | MMDialog is a dataset of 1.08 million real-world dialogues with 1.53 million unique images across 4,184 topics. |
| Approach: | They propose to use a curated set of 1.08 million dialogues with 1.53 million unique images to generalize the open domain. |
| Outcome: | The proposed system can predict responses to multi-modal content with state-of-the-art techniques and measure their performance. |
The PhotoBook Dataset: Building Common Ground through Visually-Grounded Dialogue (P19-1)
Copied to clipboard
| Challenge: | Using the PhotoBook dataset, we investigate shared dialogue history accumulating during conversation . human interlocutors are known to collaboratively establish a shared repository of mutual information during a conversation - this common ground is then used to optimise understanding and communication efficiency. |
| Approach: | They propose a data-collection task formulated as a collaborative game prompting two online participants to refer to images utilising both their visual context and previously established referring expressions. |
| Outcome: | The proposed model takes into account shared information accumulated in a reference chain and is important to resolve later descriptions. |
History for Visual Dialog: Do we really need it? (2020.acl-main)
Copied to clipboard
| Challenge: | Recent studies have shown that dialog-based interaction grounded in visual information is not as effective as previous VQA tasks because of its dialog history. |
| Approach: | They propose a visual dialogue subset which explicitly encodes dialog history and a NDCG benchmark of 63%. |
| Outcome: | The proposed subset (VisdialConv) of the VisdialVal set achieves state-of-the-art performance on 72 % of the data. |
DialogUSR: Complex Dialogue Utterance Splitting and Reformulation for Multiple Intent Detection (2022.findings-emnlp)
Copied to clipboard
Haoran Meng, Zheng Xin, Tianyu Liu, Zizhen Wang, He Feng, Binghuai Lin, Xuemin Zhao, Yunbo Cao, Zhifang Sui
| Challenge: | DialogUSR is a plug-in and domain-agnostic module that empowers multi-intent detection for chatbots . a single user query triggers inquiries on highspeed train ticket price and weather of destination. |
| Approach: | They propose a dialog utterance splitting and reformulation task that splits multi-intent user query into multiple single-intention sub-queries and recovers all coreferred and omitted information in the sub-questions. |
| Outcome: | The proposed model can be used to split multi-intent user queries into multiple sub-queries . it can be trained in two stages and perform in-depth analyses on the proposed models . |