Papers by Sanjay Subramanian
AllenNLP Interpret: A Framework for Explaining Predictions of NLP Models (D19-3)
Copied to clipboard
| Challenge: | Existing interpretation codebases make it difficult to apply these methods to new models and tasks. |
| Approach: | They propose a framework for interpreting NLP models that provides explanations for specific models. |
| Outcome: | The proposed framework provides interpretation primitives for any AllenNLP model and task, a suite of built-in interpretation methods, and a library of front-end visualization components. |
From Wrong To Right: A Recursive Approach Towards Vision-Language Explanation (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for generating insightful explanations with limited annotations are limited. |
| Approach: | They propose a method that iteratively computes visual features, an answer, and an explanation to improve the explanation quality step by step until the answer converges. |
| Outcome: | The proposed method outperforms previous methods while utilizing 5% of the human-annotated explanations across 10 metrics, showing up to 4.2 and 1.3 increases in BLEU-1 score on the VCR and VQA-X datasets. |
Modular Visual Question Answering via Code Generation (2023.acl-short)
Copied to clipboard
Sanjay Subramanian, Medhini Narasimhan, Kushal Khangaonkar, Kevin Yang, Arsha Nagrani, Cordelia Schmid, Andy Zeng, Trevor Darrell, Dan Klein
| Challenge: | a framework for visual question answering is based on modular code generation . the scope of reasoning needed for visual questions is vast, and requires many skills . |
| Approach: | They propose a framework that formulates visual question answering as modular code generation. |
| Outcome: | The proposed framework improves accuracy on COVR and GQA datasets by 3% and 2% compared to the few-shot baseline that does not employ code generation. |
ReCLIP: A Strong Zero-Shot Baseline for Referring Expression Comprehension (2022.acl-long)
Copied to clipboard
| Challenge: | Visual referring expression comprehension (ReC) models can be trained for a domain, but it remains unclear if they can be applied in a zero-shot manner to more complex tasks like ReC. |
| Approach: | They propose a method that repurposes CLIP, a state-of-the-art large-scale model, for training a referring expression comprehension model for a new visual domain. |
| Outcome: | The proposed model reduces the gap between zero-shot baselines from prior work and supervised models by as much as 29% on RefCOCOg, and on ReFGTA (video game imagery), and its relative improvement over supervised ReC models is 8%. |
MedICaT: A Dataset of Medical Images, Captions, and Textual References (2020.findings-emnlp)
Copied to clipboard
Sanjay Subramanian, Lucy Lu Wang, Ben Bogin, Sachin Mehta, Madeleine van Zuylen, Sravanthi Parasa, Sameer Singh, Matt Gardner, Hannaneh Hajishirzi
| Challenge: | Existing largescale datasets explicitly exclude compound figures . existing systems lack this ability to identify relevant subfigures . |
| Approach: | They propose a dataset of medical images in context that allows figure-to-text alignment . they use captions, inline references and manually annotated subfigures for compound figures . |
| Outcome: | The proposed dataset demonstrates the utility of inline references in image-text matching. |
Using Language Models to Disambiguate Lexical Choices in Translation (2024.emnlp-main)
Copied to clipboard
| Challenge: | In translation, a concept represented by a single word can have multiple variations in a target language. |
| Approach: | They evaluate language models that can be used to generate English rules for lexical selection . they find weaker models with high-quality lexicals improve accuracy . |
| Outcome: | The proposed model outperforms existing models on the lexical selection task in English and with native speakers. |
Evaluating Models’ Local Decision Boundaries via Contrast Sets (2020.findings-emnlp)
Copied to clipboard
Matt Gardner, Yoav Artzi, Victoria Basmov, Jonathan Berant, Ben Bogin, Sihao Chen, Pradeep Dasigi, Dheeru Dua, Yanai Elazar, Ananth Gottumukkala, Nitish Gupta, Hannaneh Hajishirzi, Gabriel Ilharco, Daniel Khashabi, Kevin Lin, Jiangming Liu, Nelson F. Liu, Phoebe Mulcaire, Qiang Ning, Sameer Singh, Noah A. Smith, Sanjay Subramanian, Reut Tsarfaty, Eric Wallace, Ally Zhang, Ben Zhou
| Challenge: | Standard test sets for supervised learning evaluate in-distribution generalization but are misleading when a dataset has systematic gaps. |
| Approach: | They propose a more rigorous annotation paradigm for NLP that helps to close systematic gaps in the test data. |
| Outcome: | The proposed model performs significantly lower on contrast sets than on the original test sets—up to 25% in some cases. |
TraveLER: A Modular Multi-LMM Agent Framework for Video Question-Answering (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods that can find relevant information, extract it, and answer video questions in a single pass are not able to adapt if insufficient or incorrect information is collected. |
| Approach: | They propose a modular multi-LMM agent framework that can find relevant information, extract it, and answer the question simultaneously. |
| Outcome: | The proposed model improves performance on several VideoQA benchmarks without fine-tuning on specific datasets. |
Latent Compositional Representations Improve Systematic Generalization in Grounded Question Answering (2021.tacl-1)
Copied to clipboard
| Challenge: | Neural networks fail to generalize to out-of-distribution examples that contain new compositions. |
| Approach: | They propose a model that computes a representation and denotation for all question spans in a bottom-up, compositional manner using a CKY-style parser. |
| Outcome: | The proposed model achieves 96.1% accuracy on a challenging dataset compared to baseline models on . previous models failed to generalize to out-of-distribution examples . |
Obtaining Faithful Interpretations from Compositional Neural Networks (2020.acl-main)
Copied to clipboard
Sanjay Subramanian, Ben Bogin, Nitish Gupta, Tomer Wolfson, Sameer Singh, Jonathan Berant, Matt Gardner
| Challenge: | Neural module networks (NMNs) are a popular approach for modeling compositionality but prior work implicitly assumed that the structure of the network modules provides a faithful explanation of the model’s reasoning. |
| Approach: | They propose to use auxiliary supervision to train a model with a structured model that can understand the reasoning process and make better choices for module architecture. |
| Outcome: | The proposed models on two datasets show that the proposed models do not provide a faithful explanation of model behaviour. |
An Improved Neural Baseline for Temporal Relation Extraction (D19-1)
Copied to clipboard
| Challenge: | Existing datasets are small and/or have low inter-annotator agreements. |
| Approach: | They propose a new neural system that achieves 10% absolute accuracy improvement over the previous best system. |
| Outcome: | The proposed system achieves 10% absolute improvement over the previous best system on two benchmark datasets. |