Papers by Sameer Singh

MisgenderMender: A Community-Informed Approach to Interventions for Misgendering (2024.naacl-long)

Challenge:	Existing methods to embed knowledge bases into large pre-training models do not contain any explicit grounding to real world entities and are difficult to recover factual knowledge.
Approach:	They propose a method to embed multiple knowledge bases (KBs) into large pretrained models with a Knowledge Attention and Recontextualization mechanism.
Outcome:	The proposed model improves perplexity, ability to recall facts and word sense disambiguation.

Copied to clipboard

Tamanna Hossain, Sunipa Dev, Sameer Singh

Ethical NLP

Challenge:	Misgendering is the act of incorrectly addressing someone’s gender and is pervasive in everyday use platforms and technologies.
Approach:	They propose a task and evaluation dataset to assess the effectiveness of automated misgendering interventions for text-based misgending in the US.
Outcome:	The proposed dataset includes 3790 instances of social media content and LLM-generations about non-cisgender public figures, annotated for the presence of misgendering, with additional annotations for correcting misgending in LLM generated text.

Gradient-based Analysis of NLP Models is Manipulable (2020.findings-emnlp)

Copied to clipboard

Junlin Wang, Jens Tuyls, Eric Wallace, Sameer Singh

Structurally Diverse Sampling for Sample-Efficient Training and Comprehensive Evaluation (2022.findings-emnlp)

Challenge:	Recent work has shown that explanation techniques can be unstable and can be manipulated to hide the actual reasoning behind the predictions of NLP models.
Approach:	They propose to merge a BERT-based sentiment classifier with a Facade Model that overwhelms the gradients without affecting the predictions.
Outcome:	The proposed model overwhelms the gradients without affecting the predictions on a variety of NLP tasks, such as sentiment analysis, NLI, and QA.

Copied to clipboard

Shivanshu Gupta, Sameer Singh, Matt Gardner

Universal Adversarial Triggers for Attacking and Analyzing NLP (D19-1)

Challenge:	Existing approaches to generalize compositionally are inadequate, but there is no evidence for this.
Approach:	They propose a model-agnostic algorithm for subsampling instances with diverse structures from a labeled instance pool with structured outputs.
Outcome:	The proposed algorithm leads to comparable or better generalization than prior algorithms in 9 out of 10 dataset-split type pairs.

Copied to clipboard

Eric Wallace, Shi Feng, Nikhil Kandpal, Matt Gardner, Sameer Singh

Interpretation of Natural Language Rules in Conversational Machine Reading (D18-1)

Challenge:	Using adversarial triggers, a model can produce a specific prediction . adversarial attacks are useful for evaluation and interpretation .
Approach:	They propose a gradient-guided search over tokens that finds short adversarial triggers that successfully trigger the target prediction.
Outcome:	The proposed algorithm finds short trigger sequences that successfully trigger the target prediction.

Copied to clipboard

Marzieh Saeidi, Max Bartolo, Patrick Lewis, Sameer Singh, Tim Rocktäschel, Mike Sheldon, Guillaume Bouchard, Sebastian Riedel

Natural Language Interfaces

Challenge:	Existing work on question answering problems requires the reading of text because it contains a recipe to derive an answer together with the reader’s background knowledge.
Approach:	They formalise a task and develop a crowd-sourcing strategy to collect 37k task instances based on real-world rules and crowd-generated questions and scenarios.
Outcome:	The proposed task is based on 37k task instances based in real-world rules and crowd-generated questions and scenarios.

Paired Examples as Indirect Supervision in Latent Decision Models (2021.emnlp-main)

Copied to clipboard

Nitish Gupta, Sameer Singh, Matt Gardner, Dan Roth

Challenge:	a new method to learn compositional structured models is needed . end-task supervision provides only a weak indirect signal on values the latent decisions should take.
Approach:	They propose a way to leverage paired examples that provide stronger cues for learning latent decisions . they use a DROP dataset to acquire paired questions that provide strong cue signals .
Outcome:	The proposed approach improves compositional question answering on a DROP dataset.

To Adapt or to Annotate: Challenges and Interventions for Domain Adaptation in Open-Domain Question Answering (2023.acl-long)

Copied to clipboard

Dheeru Dua, Emma Strubell, Sameer Singh, Pat Verga

Low-Resource NLP

Challenge:	Recent advances in open-domain question answering have demonstrated impressive accuracy on general-purpose domains like Wikipedia.
Approach:	They propose a more realistic end-to-end domain shift evaluation setting covering five diverse domains to assess model adaption.
Outcome:	The proposed model improves by 24 points when adapted to unsupervised datasets.

Investigating Robustness and Interpretability of Link Prediction via Adversarial Modifications (N19-1)

Copied to clipboard

Pouya Pezeshkpour, Yifan Tian, Sameer Singh

Interpreting Predictions of NLP Models (2020.emnlp-tutorials)

Challenge:	Existing approaches focus on improving accuracy and overlook other aspects such as robustness and interpretability.
Approach:	They propose adversarial modifications for link prediction models that identify influential facts and evaluate their sensitivity to addition of fake facts.
Outcome:	The proposed model evaluates the robustness of the model to the addition of fake facts and the interpretability of the models.

Copied to clipboard

Eric Wallace, Matt Gardner, Sameer Singh

Benefits of Intermediate Annotations in Reading Comprehension (2020.acl-main)

Challenge:	This tutorial will provide a background on interpretation techniques for neural NLP models.
Approach:	This tutorial will provide a background on interpretation techniques for NLP models . it will examine saliency maps, input perturbations, adversarial attacks and influence functions .
Outcome:	This tutorial will provide a background on interpretation techniques . examples-specific interpretations include saliency maps, input perturbations, adversarial attacks, influence functions .

Copied to clipboard

Dheeru Dua, Sameer Singh, Matt Gardner

Evaluating Question Answering Evaluation (D19-58)

Challenge:	a lack of intermediate supervision makes learning difficult for complex compositional reading comprehension datasets . lack of supervision makes the learning difficult, leading to the lack of correct latent reasoning steps.
Approach:	They propose to collect intermediate reasoning supervision along with the final answer during data collection . they find that this helps combat some of the biases introduced during the data collection process .
Outcome:	The use of intermediate reasoning supervision improves model performance for complex compositional datasets.

Copied to clipboard

Anthony Chen, Gabriel Stanovsky, Sameer Singh, Matt Gardner

Question Answering

Challenge:	Existing n-gram based QA metrics have a number of drawbacks and are not suitable for all extractive tasks.
Approach:	They propose to use BERTScore to evaluate translation for question answering (QA) they also explore whether existing n-gram based metrics are suitable for generative QA .
Outcome:	The proposed BERTScore metric fails to provide stronger correlation with human judgements .

On Importance Sampling-Based Evaluation of Latent Language Models (2020.acl-main)

Copied to clipboard

Robert L. Logan IV, Matt Gardner, Sameer Singh

Do NLP Models Know Numbers? Probing Numeracy in Embeddings (D19-1)

Challenge:	Existing approaches to evaluate language models using latent structures are intractable as they require marginalizing over the latent space.
Approach:	They propose to use importance sampling to evaluate latent language models . they elucidate subtle differences in how importance sampling is applied .
Outcome:	The proposed model performs better on tasks requiring structure and interpretability.

Copied to clipboard

Eric Wallace, Yizhong Wang, Sujian Li, Sameer Singh, Matt Gardner

Enforcing Consistency in Weakly Supervised Semantic Parsing (2021.acl-short)

Challenge:	Existing models cannot capture numeracy, but they can be useful for complex reasoning tasks.
Approach:	They investigate numerical reasoning capabilities of a question-answering model . they probe token embedding methods on synthetic list maximum, number decoding, and addition tasks.
Outcome:	The proposed model excels on questions that require numerical reasoning, i.e., it already captures numeracy.

Copied to clipboard

Nitish Gupta, Sameer Singh, Matt Gardner

EchoPrompt: Instructing the Model to Rephrase Queries for Improved In-context Learning (2024.naacl-short)

Challenge:	Existing methods for training semantic parsers from only (utterance, denotation) supervision are challenging.
Approach:	They propose to use consistency between output programs for related inputs to reduce the impact of spurious programs.
Outcome:	The proposed formalisms improve model performance even without consistency-based training.

Copied to clipboard

Raja Sekhar Reddy Mekala, Yasaman Razeghi, Sameer Singh

ReCLIP: A Strong Zero-Shot Baseline for Referring Expression Comprehension (2022.acl-long)

Challenge:	Language models are adopting inference-time prompting techniques such as zero-shot and few-shot prompting.
Approach:	They propose a prompting technique that prompts the model to rephrase its queries before answering them.
Outcome:	The proposed prompt improves zero-shot-CoT performance of code-davinci-002 by 5% . the proposed prompt is tailored for four scenarios in both zero- and few-shot settings .

Copied to clipboard

Sanjay Subramanian, William Merrill, Trevor Darrell, Matt Gardner, Sameer Singh, Anna Rohrbach

Comprehensive Multi-Dataset Evaluation of Reading Comprehension (D19-58)

Challenge:	Visual referring expression comprehension (ReC) models can be trained for a domain, but it remains unclear if they can be applied in a zero-shot manner to more complex tasks like ReC.
Approach:	They propose a method that repurposes CLIP, a state-of-the-art large-scale model, for training a referring expression comprehension model for a new visual domain.
Outcome:	The proposed model reduces the gap between zero-shot baselines from prior work and supervised models by as much as 29% on RefCOCOg, and on ReFGTA (video game imagery), and its relative improvement over supervised ReC models is 8%.

Copied to clipboard

Dheeru Dua, Ananth Gottumukkala, Alon Talmor, Sameer Singh, Matt Gardner

Dynamic Sampling Strategies for Multi-Task Reading Comprehension (2020.acl-main)

Challenge:	Recent research aims to facilitate training and evaluation on several reading comprehension datasets at the same time.
Approach:	They propose an evaluation server that reports performance on seven diverse reading comprehension datasets and includes synthetic augmentations to test models' ability to handle out-of-domain questions.
Outcome:	The evaluation server performs on seven reading comprehension datasets, and collects and includes synthetic augmentations for these datasets to test models' ability to handle out-of-domain questions.

Copied to clipboard

Ananth Gottumukkala, Dheeru Dua, Sameer Singh, Matt Gardner

Combining Feature and Instance Attribution to Detect Artifacts (2022.findings-acl)

Challenge:	Prior work focused on model architecture or generalization to held out datasets and largely passed over the particulars of the multi-task learning set up.
Approach:	They propose a dynamic sampling strategy that selects instances proportional to the model's current performance on a dataset relative to its single task performance.
Outcome:	The proposed model outperforms the best model on ORB, a recent multitask reading comprehension benchmark.

Copied to clipboard

Pouya Pezeshkpour, Sarthak Jain, Sameer Singh, Byron Wallace

Impact of Pretraining Term Frequencies on Few-Shot Numerical Reasoning (2022.findings-emnlp)

Challenge:	In this paper, we evaluate use of different attribution methods for aiding identification of training data artifacts.
Approach:	They propose hybrid methods that combine saliency maps and instance attribution methods to aid in identifying training data artifacts.
Outcome:	The proposed methods can be used to efficiently uncover artifacts in training data when a challenging validation set is available.

Copied to clipboard

Yasaman Razeghi, Robert L Logan IV, Matt Gardner, Sameer Singh

Concealed Data Poisoning Attacks on NLP Models (2021.naacl-main)

Challenge:	Pretrained language models have demonstrated ability to perform numerical reasoning by extrapolating from a few examples in few-shot settings.
Approach:	They investigate how well pretrained language models reason with terms less frequent in pretraining data.
Outcome:	The models are more accurate on instances whose terms are more prevalent, in some cases above 70% more accurate than the bottom 10%.

Copied to clipboard

Eric Wallace, Tony Zhao, Shi Feng, Sameer Singh

MedICaT: A Dataset of Medical Images, Captions, and Textual References (2020.findings-emnlp)

Challenge:	In contrast, adversarial attacks can cause model errors by modifying inputs, such as the universal triggers attack.
Approach:	They propose a data poisoning attack that allows an adversary to control model predictions whenever a desired trigger phrase is present in the input.
Outcome:	The proposed attack can cause model errors by modifying inputs, but it can also cause extra human annotation.

Copied to clipboard

Sanjay Subramanian, Lucy Lu Wang, Ben Bogin, Sachin Mehta, Madeleine van Zuylen, Sravanthi Parasa, Sameer Singh, Matt Gardner, Hannaneh Hajishirzi

Multimodality

Challenge:	Existing largescale datasets explicitly exclude compound figures . existing systems lack this ability to identify relevant subfigures .
Approach:	They propose a dataset of medical images in context that allows figure-to-text alignment . they use captions, inline references and manually annotated subfigures for compound figures .
Outcome:	The proposed dataset demonstrates the utility of inline references in image-text matching.

Compositional Questions Do Not Necessitate Multi-hop Reasoning (P19-1)

Copied to clipboard

Sewon Min, Eric Wallace, Sameer Singh, Matt Gardner, Hannaneh Hajishirzi, Luke Zettlemoyer

Coverage-based Example Selection for In-Context Learning (2023.findings-emnlp)

Challenge:	a single-hop reasoning model can solve much more of the dataset than previously thought.
Approach:	They propose a single-hop BERT-based RC model that achieves 67 F1 . they propose an evaluation setting where humans are not shown all paragraphs .
Outcome:	The proposed model achieves 67 F1—comparable to state-of-the-art multi-hop models.

Copied to clipboard

Shivanshu Gupta, Matt Gardner, Sameer Singh

Nudging: Inference-time Alignment of LLMs via Guided Decoding (2025.acl-long)

Challenge:	In-context learning (ICL) is a training-free paradigm of fewshot inference that can generalize to novel tasks by conditioning on a few task examples.
Approach:	They show that BERTScore-Recall (BSR) selects better examples that demonstrate more of the salient aspects of the test input.
Outcome:	The proposed model outperforms methods that leverage task or LLM-specific training on compositional tasks.

Copied to clipboard

Yu Fei, Yasaman Razeghi, Sameer Singh

Challenge:	Large language models (LLMs) require alignment to effectively and safely follow user instructions.
Approach:	They propose a simple, training-free algorithm that aligns any base model at inference time using a small aligned model.
Outcome:	The proposed algorithm outperforms large aligned models on open-instruction tasks without training.

Plot Twist: Multimodal Models Don’t Comprehend Simple Chart Details (2024.findings-emnlp)

Copied to clipboard

Yasaman Razeghi, Ishita Dasgupta, Fangyu Liu, Vinay Ramasesh, Sameer Singh

Multimodality

Challenge:	Recent advances in multimodal models show remarkable performance in real-world benchmarks for chart and figure understanding like ChartQA that involve interpreting trends, comparing data points, and extracting insights from visuals.
Approach:	They propose to ask models basic questions about axes ranges and values to examine their visual understanding abilities in the context of charts.
Outcome:	The models perform well on complex tasks, but lack basic capabilities on basic questions.

Prompt Waywardness: The Curious Case of Discretized Interpretation of Continuous Prompts (2022.naacl-main)

Copied to clipboard

Daniel Khashabi, Xinxi Lyu, Sewon Min, Lianhui Qin, Kyle Richardson, Sean Welleck, Hannaneh Hajishirzi, Tushar Khot, Ashish Sabharwal, Sameer Singh, Yejin Choi

Entity-Based Knowledge Conflicts in Question Answering (2021.emnlp-main)

Challenge:	Recent work shows the surprising power of continuous prompts to language models for controlled generation and solving a wide range of tasks.
Approach:	They propose to extract a discrete (textual) interpretation of continuous prompts faithful to the problem they solve.
Outcome:	The proposed model can find prompts that solve a task while being projected to an arbitrary text with a smaller drop in accuracy.

Copied to clipboard

Shayne Longpre, Kartik Perisetla, Anthony Chen, Nikhil Ramesh, Chris DuBois, Sameer Singh

Measuring and Improving Attentiveness to Partial Inputs with Counterfactuals (2024.findings-emnlp)

Challenge:	Knowledge-dependent tasks typically use two sources of knowledge: parametric, learned at training time, and contextual, given as a passage at inference time.
Approach:	They propose a method to mitigate over-reliance on parametric knowledge, which minimizes hallucination, and improves out-of-distribution generalization by 4% - 7%.
Outcome:	The proposed method minimizes hallucination and improves generalization to evolving information by 4% - 7%.

Copied to clipboard

Yanai Elazar, Bhargavi Paranjape, Hao Peng, Sarah Wiegreffe, Khyathi Chandu, Vivek Srikumar, Sameer Singh, Noah Smith

Challenge:	Existing studies have found that datasets with paired inputs are prone to spurious correlations, resulting in models trained only on those outperform chance.
Approach:	They propose a counterfactual attentiveness test to measure reliance on spurious correlations by replacing part of the input with its counterpart from a different example.
Outcome:	The proposed method improves models' attentiveness on ten datasets spanning four tasks: natural language inference, reading comprehension, paraphrase detection, and visual & language reasoning.

Selective Perception: Learning Concise State Descriptions for Language Model Actors (2024.naacl-short)

Copied to clipboard

Kolby Nottingham, Yasaman Razeghi, Kyungmin Kim, Jb Lanier, Pierre Baldi, Roy Fox, Sameer Singh

MISGENDERED: Limits of Large Language Models in Understanding Pronouns (2023.acl-long)

Challenge:	Recent large language models support longer contexts, but requiring them to process redundant or irrelevant data increases inference time and cost.
Approach:	They propose a method that leverages a small finetuned LM to sample the minimal set of input features that maximizes the performance of a downstream LM.
Outcome:	The proposed method reduces the length of LM actor input by 87% and 99% while improving task success rates by 158% and 54% on NetHack and robot planning.

Copied to clipboard

Tamanna Hossain, Sunipa Dev, Sameer Singh

Evaluating Entity Disambiguation and the Role of Popularity in Retrieval-Based NLP (2021.acl-long)

Challenge:	excluding non-binary gender identities can perpetuate harm against non-bisexual individuals through exclusion and marginalization.
Approach:	They propose a framework for evaluating large language models’ ability to correctly use preferred pronouns.
Outcome:	The proposed framework evaluates language models' ability to correctly use preferred pronouns in English.

Copied to clipboard

Anthony Chen, Pallavi Gudipati, Shayne Longpre, Xiao Ling, Sameer Singh

Information Retrieval

Challenge:	Existing studies show that retrievers underperform on rarer entities that share a name . open-domain tasks require a knowledge source to perform reasoning and produce an answer .
Approach:	They propose an evaluation benchmark for retrieving entities that share a name . they define Ambiguous Entity Retrieval sets as a collection of entities that have a common name - and query about those entities.
Outcome:	The proposed sets underperform on rarer entities that share a name . the retrievers exhibit popularity bias, and are twice as likely to retrieve erroneous documents .

Semantically Equivalent Adversarial Rules for Debugging NLP models (P18-1)

Copied to clipboard

Marco Tulio Ribeiro, Sameer Singh, Carlos Guestrin

TurtleBench: A Visual Programming Benchmark in Turtle Geometry (2025.naacl-long)

Challenge:	Complex machine learning models are often brittle, making different predictions for input instances that are extremely similar semantically.
Approach:	They propose to generalize semantically equivalent adversarial rules that induce adversaries on many instances to detect brittle models.
Outcome:	The proposed rules generate high-quality local adversaries for more instances than humans and induce four times as many mistakes as human experts.

Copied to clipboard

Sina Rismanchian, Yasaman Razeghi, Sameer Singh, Shayan Doroudi

Multimodality

Challenge:	Large Multimodal Models (LMMs) are capable of reasoning about geometric patterns, but they are still a challenge to evaluate.
Approach:	They propose a benchmark to evaluate LMMs’ ability to interpret geometric patterns and generate precise code outputs.
Outcome:	The benchmark compared LMMs with turtle geometry and GPT-4V, and found that few-shot prompting improved their performance (2%).

MOCHA: A Dataset for Training and Evaluating Generative Reading Comprehension Metrics (2020.emnlp-main)

Copied to clipboard

Anthony Chen, Gabriel Stanovsky, Sameer Singh, Matt Gardner

Are Red Roses Red? Evaluating Consistency of Question-Answering Models (P19-1)

Challenge:	Existing reading comprehension metrics rely on token overlap and are agnostic to the nuances of reading comprehension.
Approach:	They propose a benchmark for training and evaluating generative reading comprehension metrics: MOdeling Correctness with Human Annotations.
Outcome:	The proposed benchmark outperforms baseline metrics by 10 to 36 absolute Pearson points on held-out annotations.

Copied to clipboard

Marco Tulio Ribeiro, Carlos Guestrin, Sameer Singh

Question Answering

Challenge:	Existing question-answering systems are limited in their ability to test reasoning and comprehension.
Approach:	They propose a method to automatically extract implications from QA datasets to evaluate models' consistency . they use a heuristic to generate such questions and retrain models with implication-augmented data .
Outcome:	The proposed method shows that generated implications are well formed and valid . retraining with implication-augmented data improves consistency on both synthetic and human-generated implications.

FRUIT: Faithfully Reflecting Updated Information in Text (2022.naacl-main)

Copied to clipboard

Robert Iv, Alexandre Passos, Sameer Singh, Ming-Wei Chang

Challenge:	Existing work on generating edits grounded in external knowledge has focused on correcting grammar and reducing repetitive typing.
Approach:	They propose a novel task where the goal is to update an existing article given new evidence by using a dataset of 170K distantly supervised data produced from Wikipedia snapshots.
Outcome:	The proposed model can update Wikipedia articles faithfully with new capabilities and opens doors to many new applications.

Learning with Instance Bundles for Reading Comprehension (2021.emnlp-main)

Copied to clipboard

Dheeru Dua, Pradeep Dasigi, Sameer Singh, Matt Gardner

GenderQuant: Quantifying Mention-Level Genderedness (N19-1)

Challenge:	a study shows that training reading comprehension models assumes that the training instances are independent and identically distributed . however, this assumption can cause the learner to ignore distinguishing cues between related or minimally different questions .
Approach:	They propose to normalize question-answer scores across neighborhoods of closely contrasting questions and/or answers by adding a cross entropy loss term to the supervision signal.
Outcome:	The proposed methods show up to 9% absolute gains in accuracy on two datasets.

Copied to clipboard

null Ananya, Nitya Parthasarthi, Sameer Singh

Ethical NLP

Challenge:	Existing approaches to detect gendered language require considerable annotation efforts for each language, domain, and author, and often require handcrafted lexicons and features.
Approach:	They use existing NLP pipelines to automatically annotate gender of mentions in the text and train a supervised classifier to predict the gender of any mention from its context and evaluate it on unseen text.
Outcome:	The proposed method can detect gendered language on movie summaries, movie reviews, news articles, and fiction novels.

COMPACT: Building Compliance Paralegals via Clause Graph Reasoning over Contracts (2026.eacl-long)

Copied to clipboard

Ayush Singh, Dishank Aggarwal, Pranav Bhagat, Ainulla Khan, Sameer Malik, Amar Prakash Azad

Structured Data in NLP

Challenge:	Existing legal NLP benchmarks focus on single-clause tasks, such as ContractNLI and CUAD.
Approach:	They propose a framework that models cross-clause dependencies through structured clause graphs by extracting deontic-temporal entities from clauses and constructs typed relationship graphs capturing definitional dependencies, exception hierarchies, and temporal sequences.
Outcome:	The proposed framework extracts deontic-temporal entities from clauses and constructs typed relationship graphs capturing definitional dependencies, exception hierarchies, and temporal sequences.

Evaluating Models’ Local Decision Boundaries via Contrast Sets (2020.findings-emnlp)

Copied to clipboard

Matt Gardner, Yoav Artzi, Victoria Basmov, Jonathan Berant, Ben Bogin, Sihao Chen, Pradeep Dasigi, Dheeru Dua, Yanai Elazar, Ananth Gottumukkala, Nitish Gupta, Hannaneh Hajishirzi, Gabriel Ilharco, Daniel Khashabi, Kevin Lin, Jiangming Liu, Nelson F. Liu, Phoebe Mulcaire, Qiang Ning, Sameer Singh, Noah A. Smith, Sanjay Subramanian, Reut Tsarfaty, Eric Wallace, Ally Zhang, Ben Zhou

Challenge:	Standard test sets for supervised learning evaluate in-distribution generalization but are misleading when a dataset has systematic gaps.
Approach:	They propose a more rigorous annotation paradigm for NLP that helps to close systematic gaps in the test data.
Outcome:	The proposed model performs significantly lower on contrast sets than on the original test sets—up to 25% in some cases.

Snoopy: An Online Interface for Exploring the Effect of Pretraining Term Frequencies on Few-Shot LM Performance (2022.emnlp-demos)

Copied to clipboard

Yasaman Razeghi, Raja Sekhar Reddy Mekala, Robert L Logan Iv, Matt Gardner, Sameer Singh

Competency Problems: On Finding and Removing Artifacts in Language Data (2021.emnlp-main)

Challenge:	Snoopy allows researchers to analyze the impact of the overlap between pretraining corpus and test data on model performance statistics.
Approach:	They propose to align terms in test instances with their frequency in the Pile to explore correlations between model accuracy and model size and number.
Outcome:	Snoopy allows researchers to analyze term frequency statistics in large language models on NLP benchmarks.

Copied to clipboard

Matt Gardner, William Merrill, Jesse Dodge, Matthew Peters, Alexis Ross, Sameer Singh, Noah A. Smith

Zero- and Few-Shot NLP with Pretrained Language Models (2022.acl-tutorials)

Challenge:	Recent work in NLP has documented dataset artifacts, bias, and spurious correlations . how to tell which features have spurious instead of legitimate correlations is typically left unspecified .
Approach:	They propose a class of competency problems to formalize this notion into a classification . they show that realistic datasets will increasingly deviate from competency problems .
Outcome:	The proposed model can be used to show that models are inappropriately affected by these less extreme biases.

Copied to clipboard

Iz Beltagy, Arman Cohan, Robert Logan IV, Sewon Min, Sameer Singh

DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs (N19-1)

Challenge:	a tutorial aims to introduce NLP researchers to the latest techniques for learning from little-to-no data . aims at bringing interested researchers up to speed about the latest and ongoing techniques .
Approach:	They aim to introduce techniques for learning from little-to-no data using pretrained language models.
Outcome:	This tutorial aims to bring interested NLP researchers up to speed about recent techniques . it will cover methods from manual engineering, better inference algorithms to better tuning methods .

Copied to clipboard

Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, Matt Gardner

Robustness and Adversarial Examples in Natural Language Processing (2021.emnlp-tutorials)

Challenge:	a large body of work has highlighted the brittleness of reading comprehension systems . a crowdsourced, adversarially-created, 55k-question benchmark requires a more comprehensive understanding of paragraphs .
Approach:	They propose a reading comprehension benchmark that requires Discrete Reasoning over the content of paragraphs.
Outcome:	The proposed benchmarks show that the best systems only achieve 38.4% F1 on the generalized accuracy metric, while human performance is 96%.

Copied to clipboard

Kai-Wei Chang, He He, Robin Jia, Sameer Singh

PoMo: Generating Entity-Specific Post-Modifiers in Context (N19-1)

Challenge:	This tutorial aims to raise awareness of practical concerns about NLP robustness . it aims at addressing the weaknesses of NLP systems when faced with adversarial inputs and data with a distribution shift .
Approach:	This tutorial aims to bring awareness of practical concerns about NLP robustness . it reviews recent studies on analyzing the weakness of NLP systems when facing adversarial inputs .
Outcome:	This tutorial aims to bring awareness of practical concerns about NLP robustness . it will examine the weaknesses of NLP systems when faced with adversarial inputs and data with a distribution shift .

Copied to clipboard

Jun Seok Kang, Robert Logan, Zewei Chu, Yang Chen, Dheeru Dua, Kevin Gimpel, Sameer Singh, Niranjan Balasubramanian

Challenge:	Using crowdsourcing, we show that contextual relevance is necessary for accurate post-modifier generation.
Approach:	They introduce entity post-modifier generation as an instance of a collaborative writing task . they build a post- modifier dataset from news articles that provides contextually relevant information about the target entity.
Outcome:	The proposed system can generate a post-modifier phrase that provides contextually relevant information about the target entity.

Lost in Simulation: LLM-Simulated Users are Unreliable Proxies for Human Users in Agentic Evaluations (2026.acl-long)

Copied to clipboard

Preethi Seshadri, Samuel Cahyawijaya, Ayomide Odumakinde, Sameer Singh, Seraphina Goldfarb-Tarrant

Embedding Multimodal Relational Data for Knowledge Base Completion (D18-1)

Challenge:	Agentic benchmarks rely on LLM-simulated users to evaluate agent performance . however, the robustness, validity, and fairness of this approach remain unexamined .
Approach:	They investigate whether LLM-simulated users are reliable proxies for real human users . they find that agent success rates vary up to 9 percentage points across different LLMs .
Outcome:	The results show that simulated users underestimate success on challenging tasks while miscalibrate performance on moderately difficult tasks.

Copied to clipboard

Pouya Pezeshkpour, Liyan Chen, Sameer Singh

AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts (2020.emnlp-main)

Challenge:	Existing approaches focus on a finite set of entities, ignoring the variety of data types used in knowledge bases.
Approach:	They propose multimodal knowledge base embeddings that use different neural encoders for observed data and different neural decoders to learn embedded entities and multimodal data.
Outcome:	The proposed models outperform existing methods with 5-7% accuracy over existing methods.

Copied to clipboard

Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace, Sameer Singh

Perceptions of Linguistic Uncertainty by Language Models and Humans (2024.emnlp-main)

Challenge:	Pretrained language models have been successful when finetuned to downstream tasks . however, it is difficult to determine whether the knowledge that finetuning LMs contain is learned during the pretraining or the finetailing process.
Approach:	They propose a method to create prompts for a diverse set of tasks using a gradient-guided search.
Outcome:	The proposed method performs sentiment analysis and natural language inference without additional parameters and finetuning.

Copied to clipboard

Catarina Belém, Markelle Kelly, Mark Steyvers, Sameer Singh, Padhraic Smyth

Barack’s Wife Hillary: Using Knowledge Graphs for Fact-Aware Language Modeling (P19-1)

Challenge:	Prior work has shown that humans are well-attuned to the use of uncertainty expressions, exhibiting population-level agreement in mapping these expressions to numerical responses.
Approach:	They propose to map linguistic expressions of uncertainty to numerical responses by using a theory of mind approach to understand the uncertainty of another agent.
Outcome:	The proposed model can map expressions to probabilistic responses in a human-like manner, but different behavior depending on whether a statement is actually true or false.

Copied to clipboard

Robert Logan, Nelson F. Liu, Matthew E. Peters, Matt Gardner, Sameer Singh

Deep Adversarial Learning for NLP (N19-5)

Challenge:	Existing language models are only capable of remembering facts seen at training time, and have difficulty recalling them.
Approach:	They introduce a knowledge graph language model with mechanisms for selecting and copying facts from a Knowledge graph that are relevant to the context.
Outcome:	The proposed model outperforms a baseline language model in generating factual knowledge and generating sentences that require factual information.

Copied to clipboard

William Yang Wang, Sameer Singh, Jiwei Li

Generative Context Pair Selection for Multi-hop Question Answering (2021.emnlp-main)

Challenge:	Adversarial learning is a game-theoretic learning paradigm that has achieved huge successes in the field of Computer Vision recently.
Approach:	This tutorial introduces the foundations of deep adversarial learning and some practical problems and solutions in NLP.
Outcome:	This tutorial introduces the foundations of deep adversarial learning and some practical problems and solutions in NLP.

Copied to clipboard

Dheeru Dua, Cicero Nogueira dos Santos, Patrick Ng, Ben Athiwaratkun, Bing Xiang, Matt Gardner, Sameer Singh

Continued Pretraining for Better Zero- and Few-Shot Promptability (2022.emnlp-main)

Challenge:	Recent studies have shown that discriminative training results in models that exploit these underlying biases to achieve a better held-out performance, without learning the right way to reason.
Approach:	They propose a generative context selection model for multi-hop QA that reasons about how the given question could have been generated given a context pair and not just independent contexts.
Outcome:	The proposed model outperforms the state-of-the-art model on hotpotQA while being comparable to the state of the art answering performance on adversarial held-out set.

Copied to clipboard

Zhaofeng Wu, Robert L Logan IV, Pete Walsh, Akshita Bhagia, Dirk Groeneveld, Sameer Singh, Iz Beltagy

Successive Prompting for Decomposing Complex Questions (2022.emnlp-main)

Challenge:	Recent language model prompting methods can achieve high accuracy in zero- and few-shot settings while requiring few to no learned task-specific parameters.
Approach:	They propose to use a dedicated pretraining stage to improve promptability in zero-shot settings and few-shot tuning.
Outcome:	The proposed method improves promptability in zero- and few-shot settings, while the existing method yields subpar performance.

Copied to clipboard

Dheeru Dua, Shivanshu Gupta, Sameer Singh, Matt Gardner

Low-Resource NLP

Challenge:	Recent work leverages the capabilities of large language models (LMs) to perform complex question answering in a few-shot setting by generating latent reasoning steps before answering the question.
Approach:	They propose a method to iteratively break down a complex task into a simple task, solve it, and repeat the process until we get the final solution.
Outcome:	The proposed model achieves an improvement in F1 of 5% when compared with a state-of-the-art model with synthetic augmentations and few-shot version of the DROP dataset.

Obtaining Faithful Interpretations from Compositional Neural Networks (2020.acl-main)

Copied to clipboard

Sanjay Subramanian, Ben Bogin, Nitish Gupta, Tomer Wolfson, Sameer Singh, Jonathan Berant, Matt Gardner

Benchmarking Scalable Methods for Streaming Cross Document Entity Coreference (2021.acl-long)

Challenge:	Neural module networks (NMNs) are a popular approach for modeling compositionality but prior work implicitly assumed that the structure of the network modules provides a faithful explanation of the model’s reasoning.
Approach:	They propose to use auxiliary supervision to train a model with a structured model that can understand the reasoning process and make better choices for module architecture.
Outcome:	The proposed models on two datasets show that the proposed models do not provide a faithful explanation of model behaviour.

Copied to clipboard

Robert L Logan IV, Andrew McCallum, Sameer Singh, Dan Bikel

Information Extraction & Text Mining

Challenge:	Existing approaches to disambiguate mentions of named entities are limited . existing approaches omit details needed to ensure fair comparisons .
Approach:	They propose to use streaming CDC to disambiguate mentions of named entities . they compare a set of existing and new datasets to evaluate their models .
Outcome:	The proposed system is well-suited for processing streams of data where new entities are frequently introduced.

Cutting Down on Prompts and Parameters: Simple Few-Shot Learning with Language Models (2022.findings-acl)

Copied to clipboard

Robert Logan IV, Ivana Balazevic, Eric Wallace, Fabio Petroni, Sameer Singh, Sebastian Riedel

The Bias Amplification Paradox in Text-to-Image Generation (2024.naacl-long)

Challenge:	Prompting language models (LMs) with training examples and task descriptions has been seen as critical to recent successes in few-shot learning.
Approach:	They propose to fine tune masked language models with training examples and task descriptions to reduce prompt engineering by using null prompts.
Outcome:	The proposed prompts can be used to improve few-shot learning by finetuning only the bias terms while updating only 0.1% of the parameters.

Copied to clipboard

Preethi Seshadri, Sameer Singh, Yanai Elazar