Papers by Dhruv Batra
SOrT-ing VQA Models : Contrastive Gradient Learning for Improved Consistency (2021.naacl-main)
Copied to clipboard
| Challenge: | Current visual question answering models are inconsistent in their understanding of the world . they answer difficult reasoning questions correctly but get associated sub-questions wrong . |
| Approach: | They propose a gradient-based interpretability approach to determine the questions most strongly correlated with the reasoning question on an image and a contrastive gradient learning based approach called Sub-question Oriented Tuning (SOrT). |
| Outcome: | The proposed approach improves model consistency by up to 6.5% points over existing approaches while improving visual grounding and robustness to rephrasings of questions. |
CoDraw: Collaborative Drawing as a Testbed for Grounded Goal-driven Communication (P19-1)
Copied to clipboard
Jin-Hwa Kim, Nikita Kitaev, Xinlei Chen, Marcus Rohrbach, Byoung-Tak Zhang, Yuandong Tian, Dhruv Batra, Devi Parikh
| Challenge: | a goal-driven collaborative drawing task combines language, perception, and actions in a partially observable environment . et al., 1990: 138K messages exchanged between human players. |
| Approach: | They propose a goal-driven collaborative task that combines language, perception, and action . they collect a clip art dataset and use it to build an image-drawing game between two agents . |
| Outcome: | The proposed task integrates language, perception, and action in a virtual world . it is based on a dataset of 10K dialogs and 138K messages exchanged between humans . |
CLEVR-Dialog: A Diagnostic Dataset for Multi-Round Reasoning in Visual Dialog (N19-1)
Copied to clipboard
| Challenge: | Visual Dialog is a multimodal task of answering a sequence of questions grounded in an image. |
| Approach: | They construct a dialog grammar that is grounded in the scene graphs of the images from the CLEVR dataset and use it to benchmark performance of standard visual dialog models. |
| Outcome: | The proposed model is based on a large diagnostic dataset for studying multi-round reasoning in visual dialog. |
Where Are You? Localization from Embodied Dialog (2020.emnlp-main)
Copied to clipboard
| Challenge: | Observer and Locator perform a cooperative localization task in a 3D environment. |
| Approach: | They propose a dataset of 6k dialogs in which two humans complete a cooperative localization task. |
| Outcome: | The proposed model achieves 32.7% success at identifying the Observer’s location within 3m in unseen buildings, vs. 70.4% for human Locators. |
Improving Generative Visual Dialog by Answering Diverse Questions (D19-1)
Copied to clipboard
| Challenge: | Prior work on training generative Visual Dialog models with reinforcement learning has shown that this improvement saturates and starts degrading after a few rounds of interaction, and does not lead to a better Visual Dialog model. |
| Approach: | They propose a Q-Bot-A-Bot image-guessing game that allows Q-BOT to ask diverse questions, thus reducing repetitions and enabling A-BOTT to explore a larger state space during RL. |
| Outcome: | The proposed approach improves Q-Bot-A-Bot image-guessing performance but degrades after a few rounds of interaction and does not lead to a better Visual Dialog model. |