Papers by Bhuwan Dhingra

Neural Models for Reasoning over Multiple Mentions Using Coreference (N18-2)

Challenge:	Existing models implicitly recover the original text, but it is unclear when they rely on context and when they implicitly do so.
Approach:	They propose to use a dictionary to recover adversarial words by using a phonetic, typo, and visual attack to study word recovery performance.
Outcome:	The proposed model outperforms open-source models on hateful, offensive, and toxic classification tasks.

Copied to clipboard

Bhuwan Dhingra, Qiao Jin, Zhilin Yang, William Cohen, Ruslan Salakhutdinov

Information Extraction & Text Mining

Challenge:	Existing Recurrent Neural Network (RNN) layers are biased towards short-term dependencies and hence not suited to such tasks.
Approach:	They propose a recurrent layer which is instead biased towards coreferent dependencies and uses coreference annotations extracted from an external system to connect entity mentions belonging to the same cluster.
Outcome:	The proposed layer improves performance on Wikihop, LAMBADA and the bAbi AI datasets with large gains when training data is scarce.

Tailoring Vaccine Messaging with Common-Ground Opinions (2024.findings-naacl)

Copied to clipboard

Rickard Stureborg, Sanxing Chen, Roy Xie, Aayushi Patel, Christopher Li, Chloe Zhu, Tingnan Hu, Jun Yang, Bhuwan Dhingra

Challenge:	Vaccine interventions aim to answer concerns expressed about vaccination.
Approach:	They propose a dataset to evaluate how well responses are tailored to a common-ground opinion . they find that GPT-4-Turbo performs significantly better than others .
Outcome:	The proposed dataset outperforms fine tuned LLMs on the task of tailoring vaccine responses to common-ground opinions.

Handling Divergent Reference Texts when Evaluating Table-to-Text Generation (P19-1)

Copied to clipboard

Bhuwan Dhingra, Manaal Faruqui, Ankur Parikh, Ming-Wei Chang, Dipanjan Das, William Cohen

Structured Data in NLP

Challenge:	Existing text generation metrics rely on reference texts, such as BLEU and ROUGE, but they are too expensive to apply repeatedly.
Approach:	They propose a metric which aligns n-grams from the generated texts to the semi-structured data before computing their precision and recall.
Outcome:	The proposed metric correlates with human judgments better than existing text generation metrics while being easier to use.

Salient Span Masking for Temporal Understanding (2023.eacl-main)

Copied to clipboard

Jeremy R. Cole, Aditi Chaudhary, Bhuwan Dhingra, Partha Talukdar

DiffQG: Generating Questions to Summarize Factual Changes (2023.eacl-main)

Challenge:	Salient Span Masking (SSM) has shown to be effective for closed-book question answering . authors of this study found that SSM alone improves performance on temporal tasks .
Approach:	They introduce Temporal Span Masking (TSM) to improve performance on temporal tasks . they find that SSM alone improves the downstream performance by +5.8 points .
Outcome:	The proposed approach improves performance on three temporal tasks by +5.8 points . the additional targeted spans achieved by adding the TSM task are the best .

Copied to clipboard

Jeremy R. Cole, Palak Jain, Julian Martin Eisenschlos, Michael J.Q. Zhang, Eunsol Choi, Bhuwan Dhingra

Text Generation

Challenge:	Existing methods to identify factual changes between paired documents are limited . specialized entailment-like resources and models have been applied to fact verification .
Approach:	They propose to represent factual changes between paired documents as question-answer pairs . they propose to generate a discriminating question given an answer span such that the question is answerable by one passage but not the other .
Outcome:	The proposed model can flexibly and concisely capture the updated contents of paired documents.

Text Generation with Exemplar-based Adaptive Decoding (N19-1)

Copied to clipboard

Hao Peng, Ankur Parikh, Manaal Faruqui, Bhuwan Dhingra, Dipanjan Das

Text Generation

Challenge:	Empirical results show that the proposed model achieves strong performance and outperforms comparable baselines.
Approach:	They propose a conditioned text generation model that uses a template-based approach to generate content from input text.
Outcome:	The proposed model outperforms baselines on abstractive text summarization and data-to-text generation.

Atomic Self-Consistency for Better Long Form Generations (2024.emnlp-main)

Copied to clipboard

Raghuveer Thirukovalluru, Yukun Huang, Bhuwan Dhingra

Challenge:	Recent work has aimed to improve LLM generations by filtering out hallucinations, thereby improving the accuracy of the information in responses.
Approach:	They propose a technique that improves the recall of relevant information in an LLM.
Outcome:	The proposed technique improves the recall of relevant information in an LLM.

Simple and Effective Semi-Supervised Question Answering (N18-2)

Copied to clipboard

Bhuwan Dhingra, Danish Danish, Dheeraj Rajagopal

Differentiable Open-Ended Commonsense Reasoning (2021.naacl-main)

Challenge:	Existing deep learning systems for extractive Question Answering are limited and expensive to construct.
Approach:	They propose a semi-supervised QA system where end user specifies a set of documents and only a few labelled examples.
Outcome:	The proposed system achieves 50% F1 score on SQuAD and TriviaQA with very little labeled data.

Copied to clipboard

Bill Yuchen Lin, Haitian Sun, Bhuwan Dhingra, Manzil Zaheer, Xiang Ren, William Cohen

Reasoning

Challenge:	Existing commonsense reasoning models work by scoring a question-candidate pair, but new approaches are needed to answer multiple-choice questions.
Approach:	They propose to use a corpus of commonsense facts to answer a commonsensical question without any pre-defined choices as a resource.
Outcome:	The proposed model outperforms baseline methods by a large margin in the open-ended commonsense reasoning task.

Fuzzy Speculative Decoding for a Tunable Accuracy-Runtime Tradeoff (2025.findings-acl)

Copied to clipboard

Maximilian Holsman, Yukun Huang, Bhuwan Dhingra

Challenge:	Speculative Decoding (SD) enforces strict distributional equivalence to the target model when accepting candidate tokens.
Approach:	They propose a decoding algorithm that generalizes SD by accepting candidate tokens based on the divergences between the target and draft model distributions.
Outcome:	Using Fuzzy Speculative Decoding (FSD) we show that the proposed method can achieve significant runtime improvements of over 5 tokens per second faster than SD at only an approximate 2% reduction in benchmark accuracy.

Time-Aware Language Models as Temporal Knowledge Bases (2022.tacl-1)

Copied to clipboard

Bhuwan Dhingra, Jeremy R. Cole, Julian Martin Eisenschlos, Daniel Gillick, Jacob Eisenstein, William W. Cohen

SumCSE: Summary as a transformation for Contrastive Learning (2024.findings-naacl)

Challenge:	Existing language models are trained on snapshots of data collected at a specific moment in time.
Approach:	They propose a diagnostic dataset aimed at probing LMs for factual knowledge that changes over time.
Outcome:	The proposed method improves memorization of seen facts and calibration on unseen facts from future time periods.

Copied to clipboard

Raghuveer Thirukovalluru, Xiaolan Wang, Jun Chen, Shuyang Li, Jie Lei, Rong Jin, Bhuwan Dhingra

GenEOL: Harnessing the Generative Power of LLMs for Training-Free Sentence Embeddings (2025.findings-naacl)

Challenge:	Sentence embedding models are typically trained using contrastive learning (CL) using human annotations directly or by repurposing other annotated datasets.
Approach:	They propose to use generative language models to generate CL data using annotated data.
Outcome:	The proposed method outperforms the previous best unsupervised method by 1.8 points and SimCSE, a strong supervised baseline by 0.3 points on the semantic text similarity (STS) benchmark.

Copied to clipboard

Raghuveer Thirukovalluru, Bhuwan Dhingra

Weakly- and Semi-supervised Evidence Extraction (2020.findings-emnlp)

Challenge:	Training-free embedding methods focus on optimizing embeddable prompts . previous methods have overlooked the benefits of utilizing generative abilities of LLMs - GenEOL .
Approach:	They propose a method that leverages pretrained large language models to embed text . they propose generating diverse transformations of a sentence that preserve its meaning .
Outcome:	The proposed method outperforms existing training-free embedding methods by 2.85 points on the sentence semantic text similarity (STS) benchmark.

Copied to clipboard

Danish Pruthi, Bhuwan Dhingra, Graham Neubig, Zachary C. Lipton

Combating Adversarial Misspellings with Robust Word Recognition (P19-1)

Challenge:	Existing methods to combine evidence annotations with document labels are limited to a minority of training examples.
Approach:	They propose to combine evidence annotations with abundant document labels for evidence extraction task.
Outcome:	The proposed method outperforms baselines on two classification tasks with evidence annotations.

Copied to clipboard

Danish Pruthi, Bhuwan Dhingra, Zachary C. Lipton

Over-Searching in Search-Augmented Large Language Models (2026.eacl-long)

Challenge:	a sub-field of word recognition models is emerging to combat adversarial spelling mistakes . imperceptible attacks can cause models to misclassify examples, but training robust models remains a challenge .
Approach:	They propose to place a word recognition model in front of a downstream classifier to combat adversarial spelling mistakes.
Outcome:	The proposed model outperforms adversarial training and off-the-shelf spell checkers in a word recognition task.

Copied to clipboard

Roy Xie, Deepak Gopinath, David Qiu, Dong Lin, Haitian Sun, Saloni Potdar, Bhuwan Dhingra

ToTTo: A Controlled Table-To-Text Generation Dataset (2020.emnlp-main)

Challenge:	Search-augmented large language models (LLMs) excel at knowledge-intensive tasks by integrating external retrieval.
Approach:	They conduct a systematic evaluation of over-searching across multiple dimensions including query types, model categories, retrieval conditions, and multi-turn conversations.
Outcome:	The proposed model improves answer accuracy on answerable queries but harms abstention on unanswerable ones .

Copied to clipboard

Ankur Parikh, Xuezhi Wang, Sebastian Gehrmann, Manaal Faruqui, Bhuwan Dhingra, Diyi Yang, Dipanjan Das

Structured Data in NLP

Challenge:	Existing methods for data-to-text generation often hallucinate phrases not supported by the Wikipedia table.
Approach:	They propose a controlled task where annotators directly revise existing Wikipedia sentences to generate a one-sentence description.
Outcome:	The proposed task produces a one-sentence description from a Wikipedia table and highlighted cells.

ReCaLL: Membership Inference via Relative Conditional Log-Likelihoods (2024.emnlp-main)

Copied to clipboard

Roy Xie, Junlin Wang, Ruomin Huang, Minxing Zhang, Rong Ge, Jian Pei, Neil Gong, Bhuwan Dhingra

Fool Me Twice: Entailment from Wikipedia Gamification (2021.naacl-main)

Challenge:	ReCaLL (Relative Conditional Log-Likelihood) is a membership inference attack that can detect LLMs’ pretraining data by leveraging their conditional language modeling capabilities.
Approach:	They propose a membership inference attack to detect LLMs’ pretraining data by leveraging their conditional language modeling capabilities.
Outcome:	The proposed model achieves state-of-the-art performance on the WikiMIA dataset, even with random and synthetic prefixes, and can be further improved using an ensemble approach.

Copied to clipboard

Julian Eisenschlos, Bhuwan Dhingra, Jannis Bulian, Benjamin Börschinger, Jordan Boyd-Graber

Reasoning

Challenge:	a new dataset of entailment pairs is released to challenge the goal of examining arbitrary statements.
Approach:	They propose a multi-player game that challenges players to solve entailment pairs . the game is open source and encourages adversarial examples .
Outcome:	The proposed game lowers the number of examples that can be solved using "shortcuts" the game is open source and the code is available for free.

Learning to Deceive with Attention-Based Explanations (2020.acl-main)

Copied to clipboard

Danish Pruthi, Mansi Gupta, Bhuwan Dhingra, Graham Neubig, Zachary C. Lipton

Real-time Factuality Assessment from Adversarial Feedback (2025.acl-long)

Challenge:	Attention mechanisms are ubiquitous components in neural network architectures and are often claimed to confer interpretability.
Approach:	They propose a method for training models to produce deceptive attention masks by combining weights assigned to designated impermissible tokens with a weighted sum.
Outcome:	The proposed method reduces the weight assigned to designated impermissible tokens while still using them across multiple models and tasks.

Copied to clipboard

Sanxing Chen, Yukun Huang, Bhuwan Dhingra

MatViX: Multimodal Information Extraction from Visually Rich Articles (2025.naacl-long)

Challenge:	Existing evaluations for assessing the factuality of news from conventional sources, such as claims on fact-checking websites, result in high accuracies over time for LLM-based detectors.
Approach:	They propose a pipeline that leverages natural language feedback from a RAG-based detector to iteratively modify real-time news into deceptive variants that challenge LLMs.
Outcome:	The proposed pipeline reduces the binary classification ROC-AUC by 17.5 percent for a strong RAG-based GPT-4o detector.

Copied to clipboard

Ghazal Khalighinejad, Sharon Scott, Ollie Liu, Kelly L. Anderson, Rickard Stureborg, Aman Tyagi, Bhuwan Dhingra

Multimodality

Challenge:	Existing methods for multimodal information extraction are limited due to the multimodal nature of scientific articles and complex interconnections between data points.
Approach:	They propose a benchmark to extract structured information from scientific articles . they use curated JSON files extracted from text, tables, and figures .
Outcome:	The proposed benchmark is based on 324 full-length research articles and 1,688 complex structured JSON files curated by experts in polymer nanocomposites and biodegradation.

Calibrating Long-form Generations From Large Language Models (2024.findings-emnlp)

Copied to clipboard

Yukun Huang, Yixin Liu, Raghuveer Thirukovalluru, Arman Cohan, Bhuwan Dhingra

Learning the Legibility of Visual Text Perturbations (2023.eacl-main)

Challenge:	Conventional calibration methods treat answer correctness as binary and do not work for long-form generation where an answer can be partially correct.
Approach:	They propose a framework where correctness of LLMs' responses and associated confidence levels are treated as distributions across a range of scores.
Outcome:	The proposed framework treats the correctness of the LLMs’ responses and their associated confidence levels as distributions across a range of scores.

Copied to clipboard

Dev Seth, Rickard Stureborg, Danish Pruthi, Bhuwan Dhingra

Selectively Answering Ambiguous Questions (2023.emnlp-main)

Challenge:	Existing adversarial attacks in NLP perturb text to produce visually similar strings ('ergo', 'rgo') which are legible to humans but degrade model performance.
Approach:	They use a human-annotated dataset comprising the legibility of visually perturbed text to build models that predict the legible inputs and rank them based on their legibility.
Outcome:	The proposed models achieve an F score of 0.91 and an accuracy of 0.86 in predicting which of two perturbations is more legible.

Copied to clipboard

Jeremy Cole, Michael Zhang, Daniel Gillick, Julian Eisenschlos, Bhuwan Dhingra, Jacob Eisenstein

Question Answering

Challenge:	Prior work has focused on the case in which the question is clear and the answer is unambiguous but possibly unknown.
Approach:	They propose to use a sampled set of questions to calibrate answers to ambiguous questions with varying model scales.
Outcome:	The results show that sampling-based confidence scores help calibrate answers to relatively unambiguous questions, with more dramatic improvements on ambiguous ones.

DeepFact: Co-Evolving Benchmarks and Agents for Deep Research Factuality (2026.acl-long)

Copied to clipboard

Yukun Huang, Leonardo F. R. Ribeiro, Momchil Hardalov, Bhuwan Dhingra, Markus Dreyer, Venkatesh Saligrama

Challenge:	Existing fact-checkers usually target general-domain atomic claims . citation-grounded fact- checking ignores claims without explicit citations .
Approach:	They propose to use a benchmark to test whether claim-level factuality is transferable . they instantiate Audit-then-Score as a versioned DRR factualism benchmark .
Outcome:	The proposed benchmark outperforms the best prior deep-research and traditional fact-checkers by 14.3 and 24.9 points.

Extracting Polymer Nanocomposite Samples from Full-Length Documents (2024.findings-acl)

Copied to clipboard

Ghazal Khalighinejad, Defne Circi, L. Brinson, Bhuwan Dhingra

Challenge:	Using large language models (LLMs) to extract sample lists of polymer nanocomposites (PNCs) from full-length materials science research papers is challenging due to the complexity of the data.
Approach:	They propose a benchmark and evaluation technique for extracting sample lists of polymer nanocomposites from full-length materials science research papers.
Outcome:	The proposed method improves the performance of LLMs and incorporates self-consistency to improve the performance.

Raccoon: Prompt Extraction Benchmark of LLM-Integrated Applications (2024.findings-acl)

Copied to clipboard

Junlin Wang, Tianyi Yang, Roy Xie, Bhuwan Dhingra

Evaluating Explanations: How Much Do Explanations from the Teacher Aid Students? (2022.tacl-1)

Challenge:	Recent advances in Large Language Models (LLMs) have markedly shifted the landscape of AI, enabling these models to tackle complex, real-world tasks through natural language instructions.
Approach:	They propose a benchmark which evaluates a model's susceptibility to prompt extraction attacks by employing a dual approach to evaluate the effectiveness of existing defenses and the resilience of the models.
Outcome:	The proposed benchmark assesses models under both defenseless and defended scenarios, employing a dual approach to evaluate the effectiveness of existing defenses and the resilience of the models.

Copied to clipboard

Danish Pruthi, Rachit Bansal, Bhuwan Dhingra, Livio Baldini Soares, Michael Collins, Zachary C. Lipton, Graham Neubig, William W. Cohen

Explainability & Interpretability in NLP

Challenge:	Existing methods to explain predictions by highlighting salient features are often unstated.
Approach:	They propose a framework to quantify the value of explanations via the accuracy gains that they confer on a student model trained to simulate a teacher model.
Outcome:	The proposed framework allows principled, automatic, model-agnostic evaluation of attributions.

Sequence Reducible Holdout Loss for Language Model Pretraining (2024.lrec-main)

Copied to clipboard

Raghuveer Thirukovalluru, Nicholas Monath, Bhuwan Dhingra, Sam Wiseman

Open Domain Question Answering Using Early Fusion of Knowledge Bases and Text (D18-1)

Challenge:	Data selection techniques have shown empirical benefits in reducing the number of gradient steps to train neural models.
Approach:	They propose to modify an existing data selection technique to adapt it to the sequence losses typical in language modeling.
Outcome:	The proposed technique reduces the number of steps required to train neural models by 4.3% and improves generalization ability on out of domain datasets.

Copied to clipboard

Haitian Sun, Bhuwan Dhingra, Manzil Zaheer, Kathryn Mazaitis, Ruslan Salakhutdinov, William Cohen

Natural Language Interfaces

Challenge:	Specialized neural models have been developed for extracting answers from text alone or Knowledge Bases (KBs) alone.
Approach:	They propose a novel model for extracting answers from a question-specific subgraph containing text and KB entities and relations.
Outcome:	The proposed model outperforms existing methods in a combination of a KB and entity-linked text in QA over a large text corpus.

Adversarial Math Word Problem Generation (2024.findings-emnlp)

Copied to clipboard

Roy Xie, Chengxuan Huang, Junlin Wang, Bhuwan Dhingra

ASQA: Factoid Questions Meet Long-Form Answers (2022.emnlp-main)

Challenge:	Large language models (LLMs) have revolutionized the educational landscape due to the great improvements in their natural language generation and problem-solving capabilities.
Approach:	They propose a cost-effective approach to attack large language models using abstract syntax trees to generate adversarial examples that preserve the structure and difficulty of the original questions aimed for assessment.
Outcome:	The proposed approach significantly degrades students' math problem-solving ability on open- and closed-source LLMs.

Copied to clipboard

Ivan Stelmakh, Yi Luan, Bhuwan Dhingra, Ming-Wei Chang

Question Answering

Challenge:	Recent progress on factoid question answering (QA) does not easily transfer to the task of long-form QA where the goal is to generate detailed explanations.
Approach:	They propose a task that focuses on ambiguous factoid questions which have different correct answers depending on interpretation.
Outcome:	The proposed metric is reliable and demonstrates agreement between this metric and human judgments, and reveals a considerable gap between human performance and strong baselines.

Evaluating Morphological Compositional Generalization in Large Language Models (2025.naacl-long)

Copied to clipboard

Mete Ismayilzada, Defne Circi, Jonne Sälevä, Hale Sirin, Abdullatif Köksal, Bhuwan Dhingra, Antoine Bosselut, Duygu Ataman, Lonneke Van Der Plas