Papers by Matt Gardner

Universal Adversarial Triggers for Attacking and Analyzing NLP (D19-1)

Challenge:	Existing approaches to generalize compositionally are inadequate, but there is no evidence for this.
Approach:	They propose a model-agnostic algorithm for subsampling instances with diverse structures from a labeled instance pool with structured outputs.
Outcome:	The proposed algorithm leads to comparable or better generalization than prior algorithms in 9 out of 10 dataset-split type pairs.

Copied to clipboard

Eric Wallace, Shi Feng, Nikhil Kandpal, Matt Gardner, Sameer Singh

Robustness in NLP

Challenge:	Using adversarial triggers, a model can produce a specific prediction . adversarial attacks are useful for evaluation and interpretation .
Approach:	They propose a gradient-guided search over tokens that finds short adversarial triggers that successfully trigger the target prediction.
Outcome:	The proposed algorithm finds short trigger sequences that successfully trigger the target prediction.

Paired Examples as Indirect Supervision in Latent Decision Models (2021.emnlp-main)

Copied to clipboard

Nitish Gupta, Sameer Singh, Matt Gardner, Dan Roth

Challenge:	a new method to learn compositional structured models is needed . end-task supervision provides only a weak indirect signal on values the latent decisions should take.
Approach:	They propose a way to leverage paired examples that provide stronger cues for learning latent decisions . they use a DROP dataset to acquire paired questions that provide strong cue signals .
Outcome:	The proposed approach improves compositional question answering on a DROP dataset.

Deep Contextualized Word Representations (N18-1)

Copied to clipboard

Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, Luke Zettlemoyer

Interpreting Predictions of NLP Models (2020.emnlp-tutorials)

Challenge:	a new type of deep contextualized word representation is proposed for language understanding problems . word vectors are learned functions of the internal states of a deep bidirectional language model .
Approach:	They propose a new type of deep contextualized word representation that models complex features of word use and how they vary across linguistic contexts.
Outcome:	The proposed representations improve the state of the art across six challenging NLP problems.

Copied to clipboard

Eric Wallace, Matt Gardner, Sameer Singh

IIRC: A Dataset of Incomplete Information Reading Comprehension Questions (2020.emnlp-main)

Challenge:	This tutorial will provide a background on interpretation techniques for neural NLP models.
Approach:	This tutorial will provide a background on interpretation techniques for NLP models . it will examine saliency maps, input perturbations, adversarial attacks and influence functions .
Outcome:	This tutorial will provide a background on interpretation techniques . examples-specific interpretations include saliency maps, input perturbations, adversarial attacks, influence functions .

Copied to clipboard

James Ferguson, Matt Gardner, Hannaneh Hajishirzi, Tushar Khot, Pradeep Dasigi

Linguistic Knowledge and Transferability of Contextual Representations (N19-1)

Challenge:	Existing reading comprehension tasks focus on questions for which the contexts provide all the information required to answer them, thus not evaluating a system’s performance at identifying a potential lack of sufficient information and locating sources for that information.
Approach:	They propose to use a dataset with 13K questions over paragraphs from English Wikipedia that provide only partial information to answer them, with the missing information occurring in one or more linked documents.
Outcome:	The proposed model achieves 31.1% F1 on the reading comprehension task, while estimated human performance is 88.4%.

Copied to clipboard

Nelson F. Liu, Matt Gardner, Yonatan Belinkov, Matthew E. Peters, Noah A. Smith

Benefits of Intermediate Annotations in Reading Comprehension (2020.acl-main)

Challenge:	Recent work has explored contextual word representations, which assign each word a vector that is a function of the entire input sequence.
Approach:	They compare pretrained word representations with 16 diverse probing tasks to examine their transferability.
Outcome:	The pretrained representations are successful across a diverse set of NLP tasks . the models are competitive with state-of-the-art models but fail on fine-grained tasks requiring fine-granular knowledge, the study finds .

Copied to clipboard

Dheeru Dua, Sameer Singh, Matt Gardner

Evaluating Question Answering Evaluation (D19-58)

Challenge:	a lack of intermediate supervision makes learning difficult for complex compositional reading comprehension datasets . lack of supervision makes the learning difficult, leading to the lack of correct latent reasoning steps.
Approach:	They propose to collect intermediate reasoning supervision along with the final answer during data collection . they find that this helps combat some of the biases introduced during the data collection process .
Outcome:	The use of intermediate reasoning supervision improves model performance for complex compositional datasets.

Copied to clipboard

Anthony Chen, Gabriel Stanovsky, Sameer Singh, Matt Gardner

On Importance Sampling-Based Evaluation of Latent Language Models (2020.acl-main)

Challenge:	Existing n-gram based QA metrics have a number of drawbacks and are not suitable for all extractive tasks.
Approach:	They propose to use BERTScore to evaluate translation for question answering (QA) they also explore whether existing n-gram based metrics are suitable for generative QA .
Outcome:	The proposed BERTScore metric fails to provide stronger correlation with human judgements .

Copied to clipboard

Robert L. Logan IV, Matt Gardner, Sameer Singh

Do NLP Models Know Numbers? Probing Numeracy in Embeddings (D19-1)

Challenge:	Existing approaches to evaluate language models using latent structures are intractable as they require marginalizing over the latent space.
Approach:	They propose to use importance sampling to evaluate latent language models . they elucidate subtle differences in how importance sampling is applied .
Outcome:	The proposed model performs better on tasks requiring structure and interpretability.

Copied to clipboard

Eric Wallace, Yizhong Wang, Sujian Li, Sameer Singh, Matt Gardner

Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus (2021.emnlp-main)

Challenge:	Existing models cannot capture numeracy, but they can be useful for complex reasoning tasks.
Approach:	They investigate numerical reasoning capabilities of a question-answering model . they probe token embedding methods on synthetic list maximum, number decoding, and addition tasks.
Outcome:	The proposed model excels on questions that require numerical reasoning, i.e., it already captures numeracy.

Copied to clipboard

Jesse Dodge, Maarten Sap, Ana Marasović, William Agnew, Gabriel Ilharco, Dirk Groeneveld, Margaret Mitchell, Matt Gardner

Challenge:	Large text corpora are often introduced with minimal documentation . documenting collection process, composition, intended uses, and other are key for structured, task-specific datasets.
Approach:	They propose to document a dataset created by applying filters to a single snapshot of Common Crawl.
Outcome:	The proposed dataset shows that blocklist filtering removes text from minority individuals and patents.

When to Use Multi-Task Learning vs Intermediate Fine-Tuning for Pre-Trained Encoder Transfer Learning (2022.acl-short)

Copied to clipboard

Orion Weller, Kevin Seppi, Matt Gardner

Enforcing Consistency in Weakly Supervised Semantic Parsing (2021.acl-short)

Challenge:	Transfer learning (TL) in natural language processing has seen a surge of interest in recent years . pre-trained models have shown impressive ability to transfer to novel tasks .
Approach:	They compare two different methods of transfer learning in natural language processing to find out which is better.
Outcome:	The proposed methods perform better when the target task has fewer instances than the supporting task and vice versa.

Copied to clipboard

Nitish Gupta, Sameer Singh, Matt Gardner

Break It Down: A Question Understanding Benchmark (2020.tacl-1)

Challenge:	Existing methods for training semantic parsers from only (utterance, denotation) supervision are challenging.
Approach:	They propose to use consistency between output programs for related inputs to reduce the impact of spurious programs.
Outcome:	The proposed formalisms improve model performance even without consistency-based training.

Copied to clipboard

Tomer Wolfson, Mor Geva, Ankit Gupta, Matt Gardner, Yoav Goldberg, Daniel Deutch, Jonathan Berant

QuaRTz: An Open-Domain Dataset of Qualitative Relationship Questions (D19-1)

Challenge:	Understanding natural language questions entails the ability to break down a question into the requisite steps for computing its answer.
Approach:	They introduce a Question Decomposition Meaning Representation (QDMR) for questions . they demonstrate that QDMRs can be annotated at scale using a hotpotQA dataset .
Outcome:	The proposed model outperforms several natural baselines in the open-domain question answering hotpotQA dataset and can be deterministically converted to a pseudo-SQL formal language.

Copied to clipboard

Oyvind Tafjord, Matt Gardner, Kevin Lin, Peter Clark

Challenge:	Unlike previous datasets, the general knowledge is textual and not tied to a fixed set of relationships.
Approach:	They introduce the first open-domain dataset, called QuaRTz, for reasoning about textual qualitative relationships.
Outcome:	The proposed dataset is the first open-domain dataset for reasoning about qualitative relationships.

ReCLIP: A Strong Zero-Shot Baseline for Referring Expression Comprehension (2022.acl-long)

Copied to clipboard

Sanjay Subramanian, William Merrill, Trevor Darrell, Matt Gardner, Sameer Singh, Anna Rohrbach

Comprehensive Multi-Dataset Evaluation of Reading Comprehension (D19-58)

Challenge:	Visual referring expression comprehension (ReC) models can be trained for a domain, but it remains unclear if they can be applied in a zero-shot manner to more complex tasks like ReC.
Approach:	They propose a method that repurposes CLIP, a state-of-the-art large-scale model, for training a referring expression comprehension model for a new visual domain.
Outcome:	The proposed model reduces the gap between zero-shot baselines from prior work and supervised models by as much as 29% on RefCOCOg, and on ReFGTA (video game imagery), and its relative improvement over supervised ReC models is 8%.

Copied to clipboard

Dheeru Dua, Ananth Gottumukkala, Alon Talmor, Sameer Singh, Matt Gardner

Multi-Step Inference for Reasoning Over Paragraphs (2020.emnlp-main)

Challenge:	Recent research aims to facilitate training and evaluation on several reading comprehension datasets at the same time.
Approach:	They propose an evaluation server that reports performance on seven diverse reading comprehension datasets and includes synthetic augmentations to test models' ability to handle out-of-domain questions.
Outcome:	The evaluation server performs on seven reading comprehension datasets, and collects and includes synthetic augmentations for these datasets to test models' ability to handle out-of-domain questions.

Copied to clipboard

Jiangming Liu, Matt Gardner, Shay B. Cohen, Mirella Lapata

Dynamic Sampling Strategies for Multi-Task Reading Comprehension (2020.acl-main)

Challenge:	Existing models for complex reasoning use symbols or black-box transformers . a compositional model can chain together free-form predicates and logical connectives .
Approach:	They propose a compositional model that finds relevant sentences and then chains them together using neural modules.
Outcome:	The proposed model improves performance on a recently-introduced dataset.

Copied to clipboard

Ananth Gottumukkala, Dheeru Dua, Sameer Singh, Matt Gardner

Impact of Pretraining Term Frequencies on Few-Shot Numerical Reasoning (2022.findings-emnlp)

Challenge:	Prior work focused on model architecture or generalization to held out datasets and largely passed over the particulars of the multi-task learning set up.
Approach:	They propose a dynamic sampling strategy that selects instances proportional to the model's current performance on a dataset relative to its single task performance.
Outcome:	The proposed model outperforms the best model on ORB, a recent multitask reading comprehension benchmark.

Copied to clipboard

Yasaman Razeghi, Robert L Logan IV, Matt Gardner, Sameer Singh

MedICaT: A Dataset of Medical Images, Captions, and Textual References (2020.findings-emnlp)

Challenge:	Pretrained language models have demonstrated ability to perform numerical reasoning by extrapolating from a few examples in few-shot settings.
Approach:	They investigate how well pretrained language models reason with terms less frequent in pretraining data.
Outcome:	The models are more accurate on instances whose terms are more prevalent, in some cases above 70% more accurate than the bottom 10%.

Copied to clipboard

Sanjay Subramanian, Lucy Lu Wang, Ben Bogin, Sachin Mehta, Madeleine van Zuylen, Sravanthi Parasa, Sameer Singh, Matt Gardner, Hannaneh Hajishirzi

Multimodality

Challenge:	Existing largescale datasets explicitly exclude compound figures . existing systems lack this ability to identify relevant subfigures .
Approach:	They propose a dataset of medical images in context that allows figure-to-text alignment . they use captions, inline references and manually annotated subfigures for compound figures .
Outcome:	The proposed dataset demonstrates the utility of inline references in image-text matching.

Compositional Questions Do Not Necessitate Multi-hop Reasoning (P19-1)

Copied to clipboard

Sewon Min, Eric Wallace, Sameer Singh, Matt Gardner, Hannaneh Hajishirzi, Luke Zettlemoyer

Coverage-based Example Selection for In-Context Learning (2023.findings-emnlp)

Challenge:	a single-hop reasoning model can solve much more of the dataset than previously thought.
Approach:	They propose a single-hop BERT-based RC model that achieves 67 F1 . they propose an evaluation setting where humans are not shown all paragraphs .
Outcome:	The proposed model achieves 67 F1—comparable to state-of-the-art multi-hop models.

Copied to clipboard

Shivanshu Gupta, Matt Gardner, Sameer Singh

Representing Schema Structure with Graph Neural Networks for Text-to-SQL Parsing (P19-1)

Challenge:	In-context learning (ICL) is a training-free paradigm of fewshot inference that can generalize to novel tasks by conditioning on a few task examples.
Approach:	They show that BERTScore-Recall (BSR) selects better examples that demonstrate more of the salient aspects of the test input.
Outcome:	The proposed model outperforms methods that leverage task or LLM-specific training on compositional tasks.

Copied to clipboard

Ben Bogin, Jonathan Berant, Matt Gardner

Multimodality

Challenge:	Semantic parsing to SQL has largely ignored the structure of the database schema . a recent study used a simple DB that was observed at both training and test time.
Approach:	They propose a semantic parser where the schema structure is encoded with a graph neural network and used at both encoding and decoding time.
Outcome:	The proposed parser improves from 33.8% to 39.4%, dramatically above the current state of the art, which is at 19.7%.

Reasoning Over Paragraph Effects in Situations (D19-58)

Copied to clipboard

Kevin Lin, Oyvind Tafjord, Peter Clark, Matt Gardner

Mitigating False-Negative Contexts in Multi-document Question Answering with Retrieval Marginalization (2021.emnlp-main)

Challenge:	a key component of reading a passage is the ability to apply knowledge gained from the passage to a new situation.
Approach:	They propose a benchmark for reading comprehension targeting Reasoning Over Paragraph Effects in Situations.
Outcome:	The proposed model performs slightly better than randomly guessing an answer of the correct type, but is below the human performance of 89.0%.

Copied to clipboard

Ansong Ni, Matt Gardner, Pradeep Dasigi

Easy, Reproducible and Quality-Controlled Data Collection with CROWDAQ (2020.emnlp-demos)

Challenge:	Question Answering models typically use retrieval and reasoning components to identify relevant information for reasoning.
Approach:	They propose a retrieval parameterization method that marginalizes unanswerable queries . they show that marginalization allows a model to mitigate false negatives in annotations .
Outcome:	The proposed model improves on two multi-document question answering datasets and shows that marginalization improves performance.

Copied to clipboard

Qiang Ning, Hao Wu, Pradeep Dasigi, Dheeru Dua, Matt Gardner, Robert L. Logan IV, Ana Marasović, Zhen Nie

Evidentiality-guided Generation for Knowledge-Intensive NLP Tasks (2022.naacl-main)

Challenge:	Efficient data collection is important for advancing research and building time-sensitive applications.
Approach:	They propose an open-source platform that standardizes the data collection pipeline . it includes customizable user interface components, automated annotator qualification, and saved pipelines .
Outcome:	The proposed platform simplifies data annotation significantly on diverse datasets . it can be used by researchers and engineers to improve reproducibility and minimize overhead .

Copied to clipboard

Akari Asai, Matt Gardner, Hannaneh Hajishirzi

Low-Resource NLP

Challenge:	Existing methods to augment retrieval-augmented generation models with retrievers often rely on spurious cues or generate hallucinations during inference.
Approach:	They propose a method to incorporate evidentiality of passages into training a retrieval-augmented generation model.
Outcome:	The proposed method outperforms its direct counterpart on all knowledge-intensive tasks.

Quoref: A Reading Comprehension Dataset with Questions Requiring Coreferential Reasoning (D19-1)

Copied to clipboard

Pradeep Dasigi, Nelson F. Liu, Ana Marasović, Noah A. Smith, Matt Gardner

Information Extraction & Text Mining

Challenge:	Existing reading comprehension benchmarks do not contain complex coreferential phenomena . obtaining questions focused on such phenomena is difficult because of lexical cues .
Approach:	They propose to use a crowdsourced dataset to examine the ability of models to resolve coreference among entities in Wikipedia paragraphs.
Outcome:	The proposed model performs significantly worse than humans on the reading comprehension benchmark . paragraphs and other longer texts typically make multiple references to the same entities .

Simple and Effective Multi-Paragraph Reading Comprehension (P18-1)

Copied to clipboard

Christopher Clark, Matt Gardner

MOCHA: A Dataset for Training and Evaluating Generative Reading Comprehension Metrics (2020.emnlp-main)

Challenge:	Existing question answering models cannot scale beyond short paragraphs, so adapting a model to document-level input is difficult.
Approach:	They propose a method of adapting neural paragraph-level question answering models to document input.
Outcome:	The proposed method achieves state-of-the-art on TriviaQA and SQuAD and a 10 point gain on SQuADA.

Copied to clipboard

Anthony Chen, Gabriel Stanovsky, Sameer Singh, Matt Gardner

Learning with Instance Bundles for Reading Comprehension (2021.emnlp-main)

Challenge:	Existing reading comprehension metrics rely on token overlap and are agnostic to the nuances of reading comprehension.
Approach:	They propose a benchmark for training and evaluating generative reading comprehension metrics: MOdeling Correctness with Human Annotations.
Outcome:	The proposed benchmark outperforms baseline metrics by 10 to 36 absolute Pearson points on held-out annotations.

Copied to clipboard

Dheeru Dua, Pradeep Dasigi, Sameer Singh, Matt Gardner

Global Reasoning over Database Structures for Text-to-SQL Parsing (D19-1)

Challenge:	a study shows that training reading comprehension models assumes that the training instances are independent and identically distributed . however, this assumption can cause the learner to ignore distinguishing cues between related or minimally different questions .
Approach:	They propose to normalize question-answer scores across neighborhoods of closely contrasting questions and/or answers by adding a cross entropy loss term to the supervision signal.
Outcome:	The proposed methods show up to 9% absolute gains in accuracy on two datasets.

Copied to clipboard

Ben Bogin, Matt Gardner, Jonathan Berant

Low-Resource NLP

Challenge:	Existing semantic parsers only select a set of database constants at training time . current models only consider local information, not global ones .
Approach:	They propose a semantic parser that globally reasons about the structure of the query to make a more contextually-informed selection of database constants.
Outcome:	The proposed model increases accuracy from 39.4% to 47.4% on a zero-shot semantic parsing dataset with complex databases.

Improving Compositional Generalization in Semantic Parsing (2020.findings-emnlp)

Copied to clipboard

Inbar Oren, Jonathan Herzig, Nitish Gupta, Matt Gardner, Jonathan Berant

CONDAQA: A Contrastive Reading Comprehension Dataset for Reasoning about Negation (2022.emnlp-main)

Challenge:	Generalization of models to out-of-distribution data has sparked substantial interest . compositional generalization is the ability to systematically generalize to test examples composed of components seen during training .
Approach:	They propose to extend compositional generalization in semantic parsing by using contextual representations and training attention to agree with pre-computed token alignments.
Outcome:	The proposed extensions improve compositional generalization on OOD compositions.

Copied to clipboard

Abhilasha Ravichander, Matt Gardner, Ana Marasovic

Evaluating Models’ Local Decision Boundaries via Contrast Sets (2020.findings-emnlp)

Challenge:	Negation is fundamental to human communication.
Approach:	They propose a dataset which requires reasoning about implications of negated statements in paragraphs . they collect paragraphs with diverse negation cues and crowdworkers ask questions about implications .
Outcome:	The first dataset in english requires reasoning about implications of negated statements in paragraphs . it features 14,182 question-answer pairs with over 200 unique negation cues based on crowd-workers . the best performing model achieves only 42% on consistency metric, well below human performance of 81%.

Copied to clipboard

Matt Gardner, Yoav Artzi, Victoria Basmov, Jonathan Berant, Ben Bogin, Sihao Chen, Pradeep Dasigi, Dheeru Dua, Yanai Elazar, Ananth Gottumukkala, Nitish Gupta, Hannaneh Hajishirzi, Gabriel Ilharco, Daniel Khashabi, Kevin Lin, Jiangming Liu, Nelson F. Liu, Phoebe Mulcaire, Qiang Ning, Sameer Singh, Noah A. Smith, Sanjay Subramanian, Reut Tsarfaty, Eric Wallace, Ally Zhang, Ben Zhou

Challenge:	Standard test sets for supervised learning evaluate in-distribution generalization but are misleading when a dataset has systematic gaps.
Approach:	They propose a more rigorous annotation paradigm for NLP that helps to close systematic gaps in the test data.
Outcome:	The proposed model performs significantly lower on contrast sets than on the original test sets—up to 25% in some cases.

COVR: A Test-Bed for Visually Grounded Compositional Generalization with Real Images (2021.emnlp-main)

Copied to clipboard

Ben Bogin, Shivanshu Gupta, Matt Gardner, Jonathan Berant

Visual Data in NLP

Challenge:	Existing benchmarks for visual-grounded models have focused on synthetic images . et al., 2018: compositional generalization is crucial for building models that generalize to new settings.
Approach:	They propose a test-bed for visually-grounded compositional generalization with real images.
Outcome:	The proposed test-bed enables compositional splits where models need to generalize to new concepts and compositions in a zero- or few-shot setting.

Structured Alignment Networks for Matching Sentences (D18-1)

Copied to clipboard

Yang Liu, Matt Gardner, Mirella Lapata

Textual Inference

Challenge:	Many tasks in natural language processing involve comparing two sentences to compute some notion of relevance, entailment, or similarity.
Approach:	They propose a model of structured alignments between sentences to compare two sentences by matching their latent structures.
Outcome:	The proposed model is differentiable and trained only on the matching objective.

On Making Reading Comprehension More Comprehensive (D19-58)

Copied to clipboard

Matt Gardner, Jonathan Berant, Hannaneh Hajishirzi, Alon Talmor, Sewon Min

TORQUE: A Reading Comprehension Dataset of Temporal Ordering Questions (2020.emnlp-main)

Challenge:	Getting machines to "understand" text is a vast and long-standing problem, made more challenging by the fact that it is not even clear what it means to understand text.
Approach:	They propose a question-based approach to machine reading comprehension that uses a natural language question to test a system's comprehension of a passage of text.
Outcome:	The proposed questions have surface cues or other biases that allow a model to shortcut the intended reasoning process.

Copied to clipboard

Qiang Ning, Hao Wu, Rujun Han, Nanyun Peng, Matt Gardner, Dan Roth

Machine Reading Comprehension

Challenge:	Current machine reading comprehension benchmarks have no questions that test temporal phenomena . a new study studies reading comprehension for temporal relations .
Approach:	They propose a reading comprehension benchmark built on news snippets and 21k human-generated questions querying temporal relationships.
Outcome:	The new reading comprehension benchmark TORQUE achieves an exact-match score of 51% on the test set . the benchmark is built on 3.2k news snippets with 21k human-generated questions .

Snoopy: An Online Interface for Exploring the Effect of Pretraining Term Frequencies on Few-Shot LM Performance (2022.emnlp-demos)

Copied to clipboard

Yasaman Razeghi, Raja Sekhar Reddy Mekala, Robert L Logan Iv, Matt Gardner, Sameer Singh

A Meta-framework for Spatiotemporal Quantity Extraction from Text (2022.acl-long)

Challenge:	Snoopy allows researchers to analyze the impact of the overlap between pretraining corpus and test data on model performance statistics.
Approach:	They propose to align terms in test instances with their frequency in the Pile to explore correlations between model accuracy and model size and number.
Outcome:	Snoopy allows researchers to analyze term frequency statistics in large language models on NLP benchmarks.

Copied to clipboard

Qiang Ning, Ben Zhou, Hao Wu, Haoruo Peng, Chuchu Fan, Matt Gardner

Information Extraction & Text Mining

Challenge:	a meta-framework for news events that extracts quantities from text is proposed . a previous work on news events focused on extracting event mentions, attributes, and relationships .
Approach:	They propose a meta-framework for solving the NLP problem of spatiotemporal quantity extraction . they demonstrate the framework is general and extensible, and shareable crowdsourcing pipeline and baseline models are used .
Outcome:	The proposed framework is general and extensible, the authors say . it can extract quantity from news streams, quickly respond to emergencies, investigate incidents .

Competency Problems: On Finding and Removing Artifacts in Language Data (2021.emnlp-main)

Copied to clipboard

Matt Gardner, William Merrill, Jesse Dodge, Matthew Peters, Alexis Ross, Sameer Singh, Noah A. Smith

DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs (N19-1)

Challenge:	Recent work in NLP has documented dataset artifacts, bias, and spurious correlations . how to tell which features have spurious instead of legitimate correlations is typically left unspecified .
Approach:	They propose a class of competency problems to formalize this notion into a classification . they show that realistic datasets will increasingly deviate from competency problems .
Outcome:	The proposed model can be used to show that models are inappropriately affected by these less extreme biases.

Copied to clipboard

Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, Matt Gardner

Tailor: Generating and Perturbing Text with Semantic Controls (2022.acl-long)

Challenge:	a large body of work has highlighted the brittleness of reading comprehension systems . a crowdsourced, adversarially-created, 55k-question benchmark requires a more comprehensive understanding of paragraphs .
Approach:	They propose a reading comprehension benchmark that requires Discrete Reasoning over the content of paragraphs.
Outcome:	The proposed benchmarks show that the best systems only achieve 38.4% F1 on the generalized accuracy metric, while human performance is 96%.

Copied to clipboard

Alexis Ross, Tongshuang Wu, Hao Peng, Matthew Peters, Matt Gardner

Challenge:	Existing studies train task-specific generators, relying on training a model for every perturbation.
Approach:	They propose a semantically-controlled text generation system that modifies sentences to match target attributes.
Outcome:	The proposed system produces textual outputs conditioned on control codes derived from semantic representations.

Iterative Search for Weakly Supervised Semantic Parsing (N19-1)

Copied to clipboard

Pradeep Dasigi, Matt Gardner, Shikhar Murty, Luke Zettlemoyer, Eduard Hovy

A Dataset of Information-Seeking Questions and Answers Anchored in Research Papers (2021.naacl-main)

Challenge:	Recent work has focused on training semantic parsers via weak supervision from denotations alone.
Approach:	They propose an iterative training algorithm that alternates between searching for consistent logical forms and maximizing the marginal likelihood of the retrieved ones.
Outcome:	The proposed algorithm outperforms the previous best systems on WikiTableQuestions and Cornell Natural Language Visual Reasoning (NLVR) iteratively train models that provide guidance to subsequent models to search for logical forms of increasing complexity, thus dealing with spuriousness.

Copied to clipboard

Pradeep Dasigi, Kyle Lo, Iz Beltagy, Arman Cohan, Noah A. Smith, Matt Gardner

Barack’s Wife Hillary: Using Knowledge Graphs for Fact-Aware Language Modeling (P19-1)

Challenge:	Existing information-seeking question answering datasets do not perform well on answering these questions . existing models that do well on other QA tasks do not do well answering these tasks .
Approach:	They present a dataset of 5049 questions over 1585 NLP papers . they use a question-seeking QA model that seeks information in the full text .
Outcome:	The proposed dataset underperforms existing models on other QA tasks by 27 F1 points . the focus is on document-grounded, information-seeking QA .

Copied to clipboard

Robert Logan, Nelson F. Liu, Matthew E. Peters, Matt Gardner, Sameer Singh

Learning from Task Descriptions (2020.emnlp-main)

Challenge:	Existing language models are only capable of remembering facts seen at training time, and have difficulty recalling them.
Approach:	They introduce a knowledge graph language model with mechanisms for selecting and copying facts from a Knowledge graph that are relevant to the context.
Outcome:	The proposed model outperforms a baseline language model in generating factual knowledge and generating sentences that require factual information.

Copied to clipboard

Orion Weller, Nicholas Lourie, Matt Gardner, Matthew E. Peters

Challenge:	Recent work in supervised NLP has shown significant progress in learning tasks from examples.
Approach:	They propose a framework for developing NLP systems that solve new tasks after reading their descriptions, synthesizing prior work in this area.
Outcome:	The proposed model achieves 12% on the new dataset, leaving a significant challenge for NLP researchers.

Generative Context Pair Selection for Multi-hop Question Answering (2021.emnlp-main)

Copied to clipboard

Dheeru Dua, Cicero Nogueira dos Santos, Patrick Ng, Ben Athiwaratkun, Bing Xiang, Matt Gardner, Sameer Singh

Matt Gardner, Pradeep Dasigi, Srinivasan Iyer, Alane Suhr, Luke Zettlemoyer

Challenge:	Recent studies have shown that discriminative training results in models that exploit these underlying biases to achieve a better held-out performance, without learning the right way to reason.
Approach:	They propose a generative context selection model for multi-hop QA that reasons about how the given question could have been generated given a context pair and not just independent contexts.
Outcome:	The proposed model outperforms the state-of-the-art model on hotpotQA while being comparable to the state of the art answering performance on adversarial held-out set.

Neural Semantic Parsing (P18-5)

Copied to clipboard

Multilinguality

Challenge:	Semantic parsing is the study of translating natural language utterances into machine-executable programs.
Approach:	They will describe the various approaches researchers have taken to translate natural language into a formal language . they will also discuss why much recent work has chosen to use standard programming languages instead of more linguistically-motivated representations.
Outcome:	This paper will describe the various approaches researchers have taken to translate natural language into a formal language.

Latent Compositional Representations Improve Systematic Generalization in Grounded Question Answering (2021.tacl-1)

Copied to clipboard

Ben Bogin, Sanjay Subramanian, Matt Gardner, Jonathan Berant

Successive Prompting for Decomposing Complex Questions (2022.emnlp-main)

Challenge:	Neural networks fail to generalize to out-of-distribution examples that contain new compositions.
Approach:	They propose a model that computes a representation and denotation for all question spans in a bottom-up, compositional manner using a CKY-style parser.
Outcome:	The proposed model achieves 96.1% accuracy on a challenging dataset compared to baseline models on . previous models failed to generalize to out-of-distribution examples .

Copied to clipboard

Dheeru Dua, Shivanshu Gupta, Sameer Singh, Matt Gardner

Low-Resource NLP

Challenge:	Recent work leverages the capabilities of large language models (LMs) to perform complex question answering in a few-shot setting by generating latent reasoning steps before answering the question.
Approach:	They propose a method to iteratively break down a complex task into a simple task, solve it, and repeat the process until we get the final solution.
Outcome:	The proposed model achieves an improvement in F1 of 5% when compared with a state-of-the-art model with synthetic augmentations and few-shot version of the DROP dataset.

Obtaining Faithful Interpretations from Compositional Neural Networks (2020.acl-main)

Copied to clipboard

Sanjay Subramanian, Ben Bogin, Nitish Gupta, Tomer Wolfson, Sameer Singh, Jonathan Berant, Matt Gardner