Papers by Dheeru Dua
To Adapt or to Annotate: Challenges and Interventions for Domain Adaptation in Open-Domain Question Answering (2023.acl-long)
Copied to clipboard
| Challenge: | Recent advances in open-domain question answering have demonstrated impressive accuracy on general-purpose domains like Wikipedia. |
| Approach: | They propose a more realistic end-to-end domain shift evaluation setting covering five diverse domains to assess model adaption. |
| Outcome: | The proposed model improves by 24 points when adapted to unsupervised datasets. |
Benefits of Intermediate Annotations in Reading Comprehension (2020.acl-main)
Copied to clipboard
| Challenge: | a lack of intermediate supervision makes learning difficult for complex compositional reading comprehension datasets . lack of supervision makes the learning difficult, leading to the lack of correct latent reasoning steps. |
| Approach: | They propose to collect intermediate reasoning supervision along with the final answer during data collection . they find that this helps combat some of the biases introduced during the data collection process . |
| Outcome: | The use of intermediate reasoning supervision improves model performance for complex compositional datasets. |
Comprehensive Multi-Dataset Evaluation of Reading Comprehension (D19-58)
Copied to clipboard
| Challenge: | Recent research aims to facilitate training and evaluation on several reading comprehension datasets at the same time. |
| Approach: | They propose an evaluation server that reports performance on seven diverse reading comprehension datasets and includes synthetic augmentations to test models' ability to handle out-of-domain questions. |
| Outcome: | The evaluation server performs on seven reading comprehension datasets, and collects and includes synthetic augmentations for these datasets to test models' ability to handle out-of-domain questions. |
Dynamic Sampling Strategies for Multi-Task Reading Comprehension (2020.acl-main)
Copied to clipboard
| Challenge: | Prior work focused on model architecture or generalization to held out datasets and largely passed over the particulars of the multi-task learning set up. |
| Approach: | They propose a dynamic sampling strategy that selects instances proportional to the model's current performance on a dataset relative to its single task performance. |
| Outcome: | The proposed model outperforms the best model on ORB, a recent multitask reading comprehension benchmark. |
LOFT: Scalable and More Realistic Long-Context Evaluation (2025.findings-naacl)
Copied to clipboard
Jinhyuk Lee, Anthony Chen, Zhuyun Dai, Dheeru Dua, Devendra Singh Sachan, Michael Boratko, Yi Luan, Séb Arnold, Vincent Perot, Siddharth Dalmia, Hexiang Hu, Xudong Lin, Panupong Pasupat, Aida Amini, Jeremy R. Cole, Sebastian Riedel, Iftekhar Naim, Ming-Wei Chang, Kelvin Guu
| Challenge: | Long-context language models (LCLMs) can be used to perform tasks traditionally reliant on external tools like retrieval systems or databases. |
| Approach: | They propose a benchmark to evaluate LCLMs' performance on in-context retrieval and reasoning tasks using a set of tokens. |
| Outcome: | The proposed model outperforms state-of-the-art retrieval and RAG systems on in-context retrieval tasks while still requiring prompting strategies. |
Easy, Reproducible and Quality-Controlled Data Collection with CROWDAQ (2020.emnlp-demos)
Copied to clipboard
Qiang Ning, Hao Wu, Pradeep Dasigi, Dheeru Dua, Matt Gardner, Robert L. Logan IV, Ana Marasović, Zhen Nie
| Challenge: | Efficient data collection is important for advancing research and building time-sensitive applications. |
| Approach: | They propose an open-source platform that standardizes the data collection pipeline . it includes customizable user interface components, automated annotator qualification, and saved pipelines . |
| Outcome: | The proposed platform simplifies data annotation significantly on diverse datasets . it can be used by researchers and engineers to improve reproducibility and minimize overhead . |
Learning with Instance Bundles for Reading Comprehension (2021.emnlp-main)
Copied to clipboard
| Challenge: | a study shows that training reading comprehension models assumes that the training instances are independent and identically distributed . however, this assumption can cause the learner to ignore distinguishing cues between related or minimally different questions . |
| Approach: | They propose to normalize question-answer scores across neighborhoods of closely contrasting questions and/or answers by adding a cross entropy loss term to the supervision signal. |
| Outcome: | The proposed methods show up to 9% absolute gains in accuracy on two datasets. |
Evaluating Models’ Local Decision Boundaries via Contrast Sets (2020.findings-emnlp)
Copied to clipboard
Matt Gardner, Yoav Artzi, Victoria Basmov, Jonathan Berant, Ben Bogin, Sihao Chen, Pradeep Dasigi, Dheeru Dua, Yanai Elazar, Ananth Gottumukkala, Nitish Gupta, Hannaneh Hajishirzi, Gabriel Ilharco, Daniel Khashabi, Kevin Lin, Jiangming Liu, Nelson F. Liu, Phoebe Mulcaire, Qiang Ning, Sameer Singh, Noah A. Smith, Sanjay Subramanian, Reut Tsarfaty, Eric Wallace, Ally Zhang, Ben Zhou
| Challenge: | Standard test sets for supervised learning evaluate in-distribution generalization but are misleading when a dataset has systematic gaps. |
| Approach: | They propose a more rigorous annotation paradigm for NLP that helps to close systematic gaps in the test data. |
| Outcome: | The proposed model performs significantly lower on contrast sets than on the original test sets—up to 25% in some cases. |
DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs (N19-1)
Copied to clipboard
| Challenge: | a large body of work has highlighted the brittleness of reading comprehension systems . a crowdsourced, adversarially-created, 55k-question benchmark requires a more comprehensive understanding of paragraphs . |
| Approach: | They propose a reading comprehension benchmark that requires Discrete Reasoning over the content of paragraphs. |
| Outcome: | The proposed benchmarks show that the best systems only achieve 38.4% F1 on the generalized accuracy metric, while human performance is 96%. |
PoMo: Generating Entity-Specific Post-Modifiers in Context (N19-1)
Copied to clipboard
Jun Seok Kang, Robert Logan, Zewei Chu, Yang Chen, Dheeru Dua, Kevin Gimpel, Sameer Singh, Niranjan Balasubramanian
| Challenge: | Using crowdsourcing, we show that contextual relevance is necessary for accurate post-modifier generation. |
| Approach: | They introduce entity post-modifier generation as an instance of a collaborative writing task . they build a post- modifier dataset from news articles that provides contextually relevant information about the target entity. |
| Outcome: | The proposed system can generate a post-modifier phrase that provides contextually relevant information about the target entity. |
Generative Context Pair Selection for Multi-hop Question Answering (2021.emnlp-main)
Copied to clipboard
Dheeru Dua, Cicero Nogueira dos Santos, Patrick Ng, Ben Athiwaratkun, Bing Xiang, Matt Gardner, Sameer Singh
| Challenge: | Recent studies have shown that discriminative training results in models that exploit these underlying biases to achieve a better held-out performance, without learning the right way to reason. |
| Approach: | They propose a generative context selection model for multi-hop QA that reasons about how the given question could have been generated given a context pair and not just independent contexts. |
| Outcome: | The proposed model outperforms the state-of-the-art model on hotpotQA while being comparable to the state of the art answering performance on adversarial held-out set. |
Tricks for Training Sparse Translation Models (2022.naacl-main)
Copied to clipboard
| Challenge: | Multitask learning with an unbalanced data distribution skews model learning towards high resource tasks. |
| Approach: | They propose to use a temperature heating mechanism and dense pre-training to mitigate this by training models with a fixed model capacity. |
| Outcome: | The proposed techniques improve performance on two multilingual translation benchmarks compared to BASELayers and Dense scaling baselines and in combination, more than 2x model convergence speed. |
Successive Prompting for Decomposing Complex Questions (2022.emnlp-main)
Copied to clipboard
| Challenge: | Recent work leverages the capabilities of large language models (LMs) to perform complex question answering in a few-shot setting by generating latent reasoning steps before answering the question. |
| Approach: | They propose a method to iteratively break down a complex task into a simple task, solve it, and repeat the process until we get the final solution. |
| Outcome: | The proposed model achieves an improvement in F1 of 5% when compared with a state-of-the-art model with synthetic augmentations and few-shot version of the DROP dataset. |