Papers by Greg Durrett

WiCE: Real-World Entailment for Claims in Wikipedia (2023.emnlp-main)

Challenge:	Existing evaluation metrics poorly approximate parser quality, says a new study . questions under discussion is a linguistic framework that views discourse as asking questions and answering them .
Approach:	They propose a framework for automatic evaluation of QUD parsing . they use a dataset of fine-grained evaluation of 2,190 QUD questions .
Outcome:	The proposed framework shows that satisfying constraints of QUD is still challenging for modern LLMs.

Copied to clipboard

Ryo Kamoi, Tanya Goyal, Juan Diego Rodriguez, Greg Durrett

Can Explanations Be Useful for Calibrating Black Box Models? (2022.acl-long)

Challenge:	Textual entailment models are increasingly used in fact-checking, presupposition verification in question answering, or summary evaluation.
Approach:	They propose a new fine-grained textual entailment dataset built on natural claim and evidence pairs extracted from Wikipedia that provides en-tailment judgments over sub-sentence units of the claim and a minimal subset of evidence sentences that support each subclaim.
Outcome:	The proposed dataset improves on multiple datasets at test time and shows that real claims involve verification and retrieval problems that existing models fail to address.

Copied to clipboard

Xi Ye, Greg Durrett

Explainability & Interpretability in NLP

Challenge:	Existing models are often used as black boxes to adapt to new domains, but there is no single recipe for making them work.
Approach:	They propose to use black box models to improve their performance on new domains by leveraging explanations of their behavior.
Outcome:	The proposed method improves model generalization performance on two tasks using explanations.

Compressive Summarization with Plausibility and Salience Modeling (2020.emnlp-main)

Copied to clipboard

Shrey Desai, Jiacheng Xu, Greg Durrett

Shortcomings of Question Answering Based Factuality Frameworks for Error Localization (2023.eacl-main)

Challenge:	a new method to learn which compressions to apply is based on syntactic rules for deleting spans . plausibility and salience are the two main criteria for determining which compression to apply . a recent study shows that the plausability model generally selects for grammatical and factual deletions compared to extractive methods .
Approach:	They propose to leave the decision about what to delete to two data-driven criteria . they show that plausibility and salience are the most important criteria if a span is deleted .
Outcome:	The proposed method achieves strong in-domain results on benchmark datasets and human evaluation shows that plausibility model generally selects for grammatical and factual deletions.

Copied to clipboard

Ryo Kamoi, Tanya Goyal, Greg Durrett

Training Dynamics for Text Summarization Models (2022.findings-acl)

Challenge:	Abstractive summarization systems often generate summaries with factual errors . many approaches to detect these errors have been proposed, but this capability has not been evaluated in past research .
Approach:	They propose to use question answering-based factuality metrics to detect errors in summaries . they find that QA-based frameworks fail to correctly identify error spans in generated summary .
Outcome:	The proposed methods outperform trivial exact match baselines in localizing errors in summaries.

Copied to clipboard

Tanya Goyal, Jiacheng Xu, Junyi Jessy Li, Greg Durrett

Su Wang, Eric Holgate, Greg Durrett, Katrin Erk

Challenge:	Pre-trained language models have shown impressive results when fine-tuned on large summarization datasets.
Approach:	They analyze the training dynamics for generation models, focusing on summarization . they find that a propensity to copy the input is learned early in the training process .
Outcome:	The proposed model learns at different stages of fine-tuning, the authors show . they show that factual errors are learnt in later stages, but not at high-loss tokens .

Picking Apart Story Salads (D18-1)

Copied to clipboard

Text Clustering

Challenge:	Story salads are mixtures of multiple documents that can be generated at scale . they exhibit challenging inference problems, and require global context and coherence .
Approach:	They propose to generate salads that exhibit challenging inference problems by exploiting the Wikipedia hierarchy . they propose a task where the objective is to group sentences from the same narratives .
Outcome:	The proposed task is based on a novel, challenging clustering task using Wikipedia . it is difficult to identify relevant information and assemble it into coherent narratives .

Robust Question Answering Through Sub-part Alignment (2021.naacl-main)

Copied to clipboard

Jifan Chen, Greg Durrett

Robustness in NLP

Challenge:	Current textual question answering models fail to generalize to out-of-domain settings.
Approach:	They propose to decompose question and context into smaller units and align them to find the answer.
Outcome:	The proposed model is more robust than the standard BERT QA model on adversarial and out-of-domain datasets.

Query-focused Scenario Construction (D19-1)

Copied to clipboard

Su Wang, Greg Durrett, Katrin Erk

Effective Use of Transformer Networks for Entity Tracking (D19-1)

Challenge:	Stronger neural network models and harder synthetic training settings are important to achieve high performance.
Approach:	They propose a query-based system that extracts compatible sets of events from news data . stronger neural network models and harder synthetic training settings are important to achieve high performance .
Outcome:	The proposed system outperforms baselines on a human-curated dataset of scenarios about real-world news topics.

Copied to clipboard

Aditya Gupta, Greg Durrett

Complementary Explanations for Effective In-Context Learning (2023.findings-acl)

Challenge:	Existing pre-trained language models for entity-related tasks are not able to handle the nuances of procedural text.
Approach:	They propose to use pre-trained transformer networks to track entities in procedural text by restructuring input to focus on a particular entity.
Outcome:	The proposed models outperform baseline models on ingredient detection in recipes and QA over scientific processes on two different tasks.

Copied to clipboard

Xi Ye, Srinivasan Iyer, Asli Celikyilmaz, Veselin Stoyanov, Greg Durrett, Ramakanth Pasunuru

Natural Language Deduction with Incomplete Information (2022.emnlp-main)

Challenge:	Large language models (LLMs) have remarkable capabilities in learning from expla- nations in prompts, but there has been limited understanding of exactly how these explana- tions function or why they are effective.
Approach:	They propose a maximal marginal relevance-based exemplar selection approach to construct exemplar sets that are both relevant and comple- mentary.
Outcome:	The proposed model improves in- context learning performance across three tasks on multiple LLMs.

Copied to clipboard

Zayne Sprague, Kaj Bostrom, Swarat Chaudhuri, Greg Durrett

Linguistically-Informed Specificity and Semantic Plausibility for Dialogue Generation (N19-1)

Challenge:	Existing systems for reasoning given incomplete information are inadequate . current approaches to reasoning are based on latent reasoning by large language models .
Approach:	They propose a system that generates a natural language "proof" by abductively inferring a premise from another premise and a conclusion.
Outcome:	The proposed system can handle the underspecified setting where not all premises are stated at the outset; additional assumptions need to be materialized to prove a claim.

Copied to clipboard

Wei-Jen Ko, Greg Durrett, Junyi Jessy Li

Dialogue Systems & Conversational Agents

Challenge:	Past work has focused on word frequency-based approaches to improving specificity, such as penalizing responses with only common words.
Approach:	They propose to rerank a sequence-to-sequence model to improve the informativeness, reasonableness, and grammatically of responses by using externally-trained classifiers targeting each of these factors.
Outcome:	The proposed model improves the informativeness, reasonableness, and grammatically of responses.

Is the Top Still Spinning? Evaluating Subjectivity in Narrative Understanding (2025.emnlp-main)

Copied to clipboard

Melanie Subbiah, Akankshya Mishra, Grace Kim, Liyan Tang, Greg Durrett, Kathleen McKeown

Challenge:	In many domains, determining faithfulness of a claim to a source document is a binary judgment . but, whether a document is factual or whether it is entailed given some input is highly subjective.
Approach:	They propose a task to manage the subjectivity involved with factuality judgments of ambiguous claims.
Outcome:	The proposed method improves the annotator agreement on faithfulness of a claim by 21%.

Connecting Attributions and QA Model Behavior on Realistic Counterfactuals (2021.emnlp-main)

Copied to clipboard

Xi Ye, Rohan Nair, Greg Durrett

Complex Claim Verification with Evidence Retrieved in the Wild (2024.naacl-long)

Challenge:	Recent research in interpretability of neural models has yielded numerous token attribution techniques, but it is hard to evaluate whether these explanations are faithful.
Approach:	They propose to use pairwise attributions to connect outputs to high-level model behavior to examine how well different attribution techniques align with this assumption on realistic counterfactuals in the case of reading comprehension (RC).
Outcome:	The proposed methods are better suited to RC than token-level attributions across different RC settings, and the best performance comes from a modification that was proposed to an existing pairwise attribution method.

Copied to clipboard

Jifan Chen, Grace Kim, Aniruddh Sriram, Greg Durrett, Eunsol Choi

Fact & Claim Verification

Challenge:	Prior work makes simplifying assumptions in retrieval that depart from real-world use cases: no access to evidence, access to curated evidence, or access to published evidence after a claim was made.
Approach:	They propose a pipeline to check claims using raw evidence from the web . they restrict their retriever to only search documents available prior to the claim's making .
Outcome:	The proposed method is based on a political claim dataset and shows that the evidence summary produced by the system is reliable and relevant to answering key questions.

Interpretable Entity Representations through Large-Scale Typing (2020.findings-emnlp)

Copied to clipboard

Yasumasa Onoe, Greg Durrett

Dissecting Generation Modes for Abstractive Summarization Models via Ablation and Attribution (2021.acl-long)

Challenge:	In traditional methods for natural language processing, entities are embedded in dense vector spaces with pre-trained models.
Approach:	They propose an approach to creating entity representations that are human readable and achieve high performance on entity-related tasks out of the box.
Outcome:	The proposed representations are vectors whose values correspond to posterior probabilities over fine-grained entity types, indicating the confidence of a typing model’s decision that the entity belongs to the corresponding type.

Copied to clipboard

Jiacheng Xu, Greg Durrett

Effective Use of Context in Noisy Entity Linking (D18-1)

Challenge:	Abstractive summarization models have made great strides in recent years, but little is known about how they actually form summaries and how to understand where their decisions come from.
Approach:	They propose a two-step method to interpret summarization model decisions by categorizing each decoder decision into one of several generation modes.
Outcome:	The proposed method can identify phrases the summarization model has memorized and determine where in the training pipeline this memorization happened, and study complex generation phenomena on a per-instance basis.

Copied to clipboard

David Mueller, Greg Durrett

Flexible Generation of Natural Language Deductions (2021.emnlp-main)

Challenge:	Effectively using an entity mention's context to disambiguate it is the crux of the entity linking task.
Approach:	They propose to use convolutional neural networks to extract cues from context to effectively disambiguate between closely related concepts.
Outcome:	The proposed model outperforms previous work on the WikilinksNED test set by 2.8% absolute.

Copied to clipboard

Kaj Bostrom, Xinyu Zhao, Swarat Chaudhuri, Greg Durrett

Natural Language Deduction through Search over Statement Compositions (2022.findings-emnlp)

Challenge:	Developing models that can make useful inferences from natural language premises has been a core goal in artificial intelligence since the field's early days.
Approach:	They propose a method for building models to generate deductive inferences from diverse natural language inputs without direct human supervision.
Outcome:	The proposed model is more accurate and flexible than baseline systems.

Copied to clipboard

Kaj Bostrom, Zayne Sprague, Swarat Chaudhuri, Greg Durrett

Discourse Analysis via Questions and Answers: Parsing Dependency Structures of Questions Under Discussion (2023.findings-acl)

Challenge:	Existing methods focus on the end-to-end discriminative version of this task, but less work has treated the generative version of the task.
Approach:	They propose a system that decomposes the task into separate steps coordinated by a search procedure and produces a tree of intermediate conclusions that faithfully reflects the system’s reasoning process.
Outcome:	The proposed system proves true statements while rejecting false ones on the EntailmentBank dataset with a 17% absolute higher step validity than the end-to-end T5 model.

Copied to clipboard

Wei-Jen Ko, Yating Wu, Cutter Dalton, Dananjay Srinivas, Greg Durrett, Junyi Jessy Li

Understanding Factual Errors in Summarization: Errors, Summarizers, Datasets, Error Detectors (2023.acl-long)

Challenge:	Existing discourse formalisms require large taxonomies of discourse relations to be accurate.
Approach:	They propose a linguistic framework for discourse analysis using questions under discussion . they propose qUD parser that derives a dependency structure of questions over full documents .
Outcome:	The proposed model is trained on a large, crowdsourced question-answering dataset.

Copied to clipboard

Liyan Tang, Tanya Goyal, Alex Fabbri, Philippe Laban, Jiacheng Xu, Semih Yavuz, Wojciech Kryscinski, Justin Rousseau, Greg Durrett

Understanding Dataset Design Choices for Multi-hop Reasoning (N19-1)

Challenge:	Abstractive summarization systems still include factual errors in generated summaries despite recent improvements in factuality detection .
Approach:	They aggregate factuality error annotations from nine existing datasets and stratify them according to the underlying summarization model.
Outcome:	The proposed method improves on the ChatGPT-based model and shows that it is not superior for all error types.

Copied to clipboard

Jifan Chen, Greg Durrett

From Distributional to Overton Pluralism: Investigating Large Language Model Alignment (2025.naacl-long)

Challenge:	Existing datasets that explicitly focus on multi-hop reasoning are lacking in learning multi-tasking.
Approach:	They propose to use sentence-factored models to solve multi-hop question answering tasks . they find spurious correlations in unmasked versions of WikiHop and HotpotQA .
Outcome:	The proposed datasets are used to test models on multi-hop question answering tasks.

Copied to clipboard

Thom Lake, Eunsol Choi, Greg Durrett

Learning to Denoise Distantly-Labeled Data for Entity Typing (N19-1)

Challenge:	a large language model's (LLM) output distribution is changed by an alignment process . a recent study shows that aligned models surface information that cannot be recovered from base models without fine-tuning.
Approach:	They analyze two aspects of the alignment process that change output distributions . they find alignment suppresses irrelevant and unhelpful content .
Outcome:	The proposed model can be imitated without fine-tuning by using in-context examples and lower-resolution semantic hints about response content.

Copied to clipboard

Yasumasa Onoe, Greg Durrett

Did they answer? Subjective acts and intents in conversational discourse (2021.naacl-main)

Challenge:	Distantly-labeled data can be used to scale up statistical models, but it is noisy . specialized probabilistic models can be employed to scale the training of models, however, they require sophisticated probabilistic inference for the training.
Approach:	They propose a method for denoising and denoising noisy data with supervised training.
Outcome:	The proposed method outperforms models trained on clean and denoised data on an ultra-fine entity typing task.

Copied to clipboard

Elisa Ferracane, Greg Durrett, Junyi Jessy Li, Katrin Erk

Natural Language Interfaces

Challenge:	Discourse signals are often implicit, leaving it up to the interpreter to draw inferences . current discourse data and frameworks ignore the social aspect, expecting only a single ground truth . elisa f. and her team present a dataset with multiple and subjective interpretations of English conversation .
Approach:	They present a first discourse dataset with multiple and subjective interpretations of English conversation . they show disagreements are nuanced and require a deeper understanding of contextual factors .
Outcome:	The proposed dataset shows disagreements are nuanced and require deeper understanding of contextual factors.

Neural Syntactic Preordering for Controlled Paraphrase Generation (2020.acl-main)

Copied to clipboard

Generating Literal and Implied Subquestions to Fact-check Complex Claims (2022.emnlp-main)

Challenge:	Existing approaches to paraphrasing natural language sentences are limited by the complexity of the task.
Approach:	They propose a framework for paraphrasing natural language sentences that uses syntactic transformations to softly "reorder" the source sentence and their proposed system is evaluated automatically and by humans .
Outcome:	The proposed model retains the quality of the baseline approaches while giving a substantial increase in the diversity of the generated paraphrases.

Copied to clipboard

Jifan Chen, Aniruddh Sriram, Eunsol Choi, Greg Durrett

Which questions should I answer? Salience Prediction of Inquisitive Questions (2024.emnlp-main)

Challenge:	Existing fact-checking systems are not reliable because it is unclear which parts of a claim are true and which are not.
Approach:	They propose to decompose a political claim into a comprehensive set of yes-no subquestions whose answers influence the veracity of the claim.
Outcome:	The proposed models can decompose a complex claim into a comprehensive set of yes-no subquestions whose answers influence the veracity of the claim.

Copied to clipboard

Yating Wu, Ritika Mangla, Alex Dimakis, Greg Durrett, Junyi Jessy Li

Modeling Fine-Grained Entity Types with Box Embeddings (2021.acl-long)

Challenge:	Recent work in NLP has taken advantage of question generation capabilities of LLMs to enhance a wide range of applications.
Approach:	They propose a salience predictor for inquisitive questions that is instruction-tuned . they show that highly salient questions are empirically more likely to be answered in the same article .
Outcome:	The proposed model is based on linguist-annotated salience scores of 1,766 questions . it shows that answering salient questions improves comprehension of the text .

Copied to clipboard

Yasumasa Onoe, Michael Boratko, Andrew McCallum, Greg Durrett

Evaluating Discourse in Structured Text Representations (P19-1)

Challenge:	Neural entity typing models typically represent fine-grained entity types as vectors in a high-dimensional space, but such spaces are not well-suited to modeling complex interdependencies.
Approach:	They propose to use box embeddings to embed types into a high-dimensional hyperrectangle space and then use it to hypothesize a type representation for the mention.
Outcome:	The proposed model captures latent type hierarchies better than a vector-based model on several entity typing benchmarks.

Copied to clipboard

Elisa Ferracane, Greg Durrett, Junyi Jessy Li, Katrin Erk

Assessing Out-of-Domain Language Model Performance from Few Examples (2023.eacl-main)

Challenge:	Discourse structure is integral to understanding a text and is useful in many NLP tasks.
Approach:	They propose a structured attention mechanism for text classification that derives a tree over a text, akin to an RST discourse tree.
Outcome:	The proposed model improves performance on multiple discourse-relevant tasks and datasets and ablation studies show it does little to capture discourse structure.

Copied to clipboard

Prasann Singhal, Jarad Forristal, Xi Ye, Greg Durrett

Embedding Time Expressions for Deep Temporal Ordering Models (P19-1)

Challenge:	Pretrained language models exhibit impressive generalization capabilities, but behave unpredictably under certain domain shifts.
Approach:	They propose to incorporate attributions into a few-shot model predicting out-of-domain (OOD) performance task to find out if models agree with pathological heuristics that may indicate worse generalization capabilities.
Outcome:	The proposed model-based model-learning model can perform better on a few-shot example set, and incorporate feature attributions to improve it.

Copied to clipboard

Conditional Generation of Temporally-ordered Event Sequences (2021.acl-long)

Representation Learning

Challenge:	Existing data-driven models fail to capture explicit temporal signals, such as dates and time windows.
Approach:	They propose a framework to infuse temporal awareness into data-driven models by learning a pre-trained model to embed timexes.
Outcome:	The proposed framework infuses temporal awareness into data-driven models by learning a pre-trained model to embed timexes.

Copied to clipboard

Shih-Ting Lin, Nathanael Chambers, Greg Durrett

Massive-scale Decoding for Text Generation using Lattices (2022.naacl-main)

Challenge:	a new model of narrative schema knowledge does not capture the temporal relationships between events . a temporal order model is able to unscramble event sequences without access to labeled temporal training data .
Approach:	They propose a temporal order-based model that can be flexibly applied to different tasks . they use a BART-based conditional generation model that captures temporal co-occurrence .
Outcome:	The proposed model outperforms existing models on temporal ordering and event infilling tasks.

Copied to clipboard

Jiacheng Xu, Siddhartha Jonnalagadda, Greg Durrett

SNaC: Coherence Error Detection for Narrative Summarization (2022.emnlp-main)

Challenge:	Conditional neural text generation models generate high-quality outputs, but often focus on a mode when what we really want is a diverse set of options.
Approach:	They propose a search algorithm to construct lattices encoding a massive number of generation options.
Outcome:	The proposed algorithm encodes thousands of diverse options that remain grammatical and high-quality into one lattice.

Copied to clipboard

Tanya Goyal, Junyi Jessy Li, Greg Durrett

Optimal Neural Program Synthesis from Multimodal Specifications (2021.findings-emnlp)

Challenge:	SNaC framework is used to evaluate long summaries, but it fails to identify gaps in coherence . nallapati and colleagues have developed a framework for fine-grained annotations of long summarizations .
Approach:	They propose a narrative coherence evaluation framework for fine-grained annotations of long summaries that can be used to evaluate coherent narratives.
Outcome:	The proposed framework can support future work in document summarization and coherence evaluation, the authors show .

Copied to clipboard

Xi Ye, Qiaochu Chen, Isil Dillig, Greg Durrett

Evaluating Factuality in Generation with Dependency-level Entailment (2020.findings-emnlp)

Challenge:	Existing methods for multimodal program synthesis combine noisy signals from the user with hard constraints on the program’s behavior.
Approach:	They propose an optimal neural synthesis approach where the goal is to find a program that satisfies user-provided constraints while also maximizing the program’s score with respect to a neural model.
Outcome:	The proposed approach outperforms prior state-of-the-art methods in terms of accuracy and efficiency and finds model-optimal programs more frequently.

Copied to clipboard

Neural Extractive Text Summarization with Syntactic Compression (D19-1)

Textual Inference

Challenge:	Existing text generation models produce text that is factually inconsistent with information in the input.
Approach:	They propose a new formulation of entailment that decomposes it at the level of dependency arcs.
Outcome:	The proposed model can identify factual errors better than sentence-level methods and question generation methods while localizing the erroneous parts of the generation.

Copied to clipboard

Jiacheng Xu, Greg Durrett

FALTE: A Toolkit for Fine-grained Annotation for Long Text Evaluation (2022.emnlp-demos)

Challenge:	Recent approaches to summarization are either selection-based extraction or generation-based abstraction.
Approach:	They propose a neural model for single-document summarization based on joint extraction and syntactic compression.
Outcome:	The proposed model outperforms an off-the-shelf compression module and its output generally remains grammatical.

Copied to clipboard

Tanya Goyal, Junyi Jessy Li, Greg Durrett

Benchmarking Multimodal Regex Synthesis with Complex Structures (2020.acl-main)

Challenge:	Existing tools to evaluate long text outputs are lacking in the field of NLP . human rating and error analysis remains a crucial component for any evaluation of long text generation.
Approach:	They propose a web-based toolkit to collect fine-grained error annotations for long texts . they use a taxonomy to identify errors and assign them to text spans .
Outcome:	The proposed tool can be used to evaluate the coherence of long generated summaries.

Copied to clipboard

Xi Ye, Qiaochu Chen, Isil Dillig, Greg Durrett

Explanation Selection Using Unlabeled Data for Chain-of-Thought Prompting (2023.emnlp-main)

Challenge:	Existing datasets for regex generation from natural language are limited in complexity . Existing regex synthesis datasets are simple and the language used to describe them is not diverse .
Approach:	They propose a dataset for regex generation from natural language that generates regexes using a probabilistic grammar and pre-defined macros.
Outcome:	The proposed dataset is compared to existing datasets for regex generation from natural language . it generates the regexes using a probabilistic grammar with pre-defined macros observed from real-world StackOverflow posts.

Copied to clipboard

Xi Ye, Greg Durrett

SynthesizRR: Generating Diverse Datasets with Retrieval Augmentation (2024.emnlp-main)

Challenge:	Recent work shows how to prompt large language models with explanations to obtain strong performance on textual reasoning tasks.
Approach:	They propose to optimize explanation-infused prompts in a blackbox fashion by using leave-one-out schemes and a two-stage framework.
Outcome:	The proposed method improves prompts over crowdworker annotations and naive search strategies.

Copied to clipboard

Abhishek Divekar, Greg Durrett

Information Retrieval

Challenge:	Prior approaches to synthesis use few-shot prompting, which relies on the LLM’s parametric knowledge to generate usable examples.
Approach:	They propose to use a dataset to generate examples of each label from the LLM.
Outcome:	The proposed model significantly improves lexical and semantic diversity, similarity to human-written text, and distillation performance, when compared to 32-shot prompting and four prior approaches.

Don’t Let Discourse Confine Your Model: Sequence Perturbations for Improved Event Language Models (2021.acl-short)

Copied to clipboard

Mahnaz Koupaee, Greg Durrett, Nathanael Chambers, Niranjan Balasubramanian

Spherical Latent Spaces for Stable Variational Autoencoders (D18-1)

Challenge:	Existing approaches to train event language models on text constrain them to follow discourse order of events.
Approach:	They propose a method to perturb event sequences so that they can relax model dependence on text order.
Outcome:	The proposed technique improves performance on applications and out-of-domain events data.

Copied to clipboard

Jiacheng Xu, Greg Durrett

MiniCheck: Efficient Fact-Checking of LLMs on Grounding Documents (2024.emnlp-main)

Challenge:	Variational autoencoders use a multivariate Gaussian latent variable to capture latent structure in data.
Approach:	They propose a variational autoencoder which uses a latent distribution instead of Gaussian . they find that the variational posterior averts the KL collapse by a fixed hyperparameter .
Outcome:	The von Mises-Fisher distribution averts the KL collapse and gives better likelihoods than Gaussian models across a range of modeling conditions.

Copied to clipboard

Liyan Tang, Philippe Laban, Greg Durrett

Annotating and Modeling Fine-grained Factuality in Summarization (2021.naacl-main)

Challenge:	Current methods for fact-checking are based on verifying each piece of a model against potential evidence using an LLM.
Approach:	They propose a method that builds small fact-checking models that have GPT-4-level performance but 400x lower cost.
Outcome:	The proposed model outperforms other models and reaches GPT-4 accuracy.

Copied to clipboard

Molecular Facts: Desiderata for Decontextualization in LLM Fact Verification (2024.findings-emnlp)

Challenge:	Recent abstractive summarization systems produce factual errors that are not faithful to the input . current methods are lacking in identifying what errors are most important to target .
Approach:	They use synthetic and human-labeled data to identify factual errors in summarization and train models on the factuality detection task.
Outcome:	The proposed model detects factual errors on word, dependency, and sentence levels.

Copied to clipboard

Anisha Gunjal, Greg Durrett

Discourse Comprehension: A Question Answering Framework to Represent Sentence Connections (2022.emnlp-main)

Challenge:	Large language models (LLMs) are increasingly used to combat hallucinations . granularity of fact-checking makes it difficult to fact- check larger chunks of text .
Approach:	They propose a method for generating molecular facts automatically using decontextuality and minimality.
Outcome:	The proposed method balances minimality with fact verification accuracy in ambiguous settings.

Copied to clipboard

Wei-Jen Ko, Cutter Dalton, Mark Simmons, Eliza Fisher, Greg Durrett, Junyi Jessy Li

SAGEViz: SchemA GEneration and Visualization (2023.emnlp-demo)

Challenge:	Existing systems for text comprehension are inadequate for more holistic comprehension of a discourse.
Approach:	They propose a new paradigm that captures both discourse and semantic links between sentences in the form of free-form, open-ended questions.
Outcome:	The proposed model captures discourse and semantic links between sentences in the form of free-form, open-ended questions.

Copied to clipboard

Sugam Devare, Mahnaz Koupaee, Gautham Gunapati, Sayontan Ghosh, Sai Vallurupalli, Yash Kumar Lal, Francis Ferraro, Nathanael Chambers, Greg Durrett, Raymond Mooney, Katrin Erk, Niranjan Balasubramanian

Prompted Opinion Summarization with GPT-3.5 (2023.findings-acl)

Challenge:	Schema induction involves creating a graph representation depicting how events unfold . supervised and few-shot approaches are not scalable and time-consuming .
Approach:	They propose a tool that utilizes human-AI collaboration to create and update complex schema graphs efficiently.
Outcome:	The proposed tool can generate schemas of better quality and be used by users in a variety of domains.

Copied to clipboard

Adithya Bhaskar, Alex Fabbri, Greg Durrett

Learning to Refine with Fine-Grained Natural Language Feedback (2024.findings-emnlp)

Challenge:	Recent years have seen several shifts in summarization research, including extractive models.
Approach:	They propose a pipeline method for applying GPT-3.5 to summarize user reviews . they propose three new metrics targeting faithfulness, factuality, and genericity .
Outcome:	The proposed methods perform well in opinion summarization, the authors show . they also show that standard evaluation metrics do not reflect this performance .

Copied to clipboard

Manya Wadhwa, Xinyu Zhao, Junyi Jessy Li, Greg Durrett

Less Likely Brainstorming: Using Language Models to Generate Alternative Hypotheses (2023.findings-acl)

Challenge:	Recent work has explored the capability of large language models to identify and correct errors in LLM-generated responses.
Approach:	They propose to combine refinement with feedback into three distinct competencies . step 1: Detect, Critique, Refine gives a fine-grained feedback about errors .
Outcome:	The proposed method outperforms existing refinement approaches and models not fine-tuned for factuality critiquing.

Copied to clipboard

Liyan Tang, Yifan Peng, Yanshan Wang, Ying Ding, Greg Durrett, Justin Rousseau

Modeling Complex Event Scenarios via Simple Entity-focused Questions (2023.eacl-main)

Challenge:	Existing methods to reduce cognitive errors in MRI interpretations do not work for generating less likely outputs.
Approach:	They propose a task that asks a model to generate outputs that humans think are relevant but less likely to happen.
Outcome:	The proposed method compares with several state-of-the-art controlled text generation models via automatic and human evaluations and shows that it reduces cognitive errors in interpreting MRI findings.

Copied to clipboard

Mahnaz Koupaee, Greg Durrett, Nathanael Chambers, Niranjan Balasubramanian

Challenge:	Event schemas describe a sequence of events in a particular context, but they are difficult to model with standard event language models.
Approach:	They propose a question-guided generation framework that generates events as answers to questions about participants.
Outcome:	The proposed framework provides better coverage of participants, diverse events within a domain, comparable perplexities for modeling event sequences, and more effective control for interactive schema generation.

Sketch-Driven Regular Expression Generation from Natural Language and Examples (2020.tacl-1)

Copied to clipboard

Xi Ye, Qiaochu Chen, Xinyu Wang, Isil Dillig, Greg Durrett

Can NLI Models Verify QA Systems’ Predictions? (2021.findings-emnlp)

Challenge:	Recent systems for converting natural language descriptions into regexes have achieved some success, but typically deal with short, formulaic text and can only produce simple regexe.
Approach:	They propose a framework for regex synthesis in a context where both natural language and examples are available.
Outcome:	The proposed framework achieves state-of-the-art on two prior datasets and a real-world dataset, which existing neural systems completely fail on.

Copied to clipboard

Jifan Chen, Eunsol Choi, Greg Durrett

Calibration of Pre-trained Transformers (2020.emnlp-main)

Challenge:	Recent question answering systems perform well on benchmark datasets, but are not always well-calibrated to spot spurious answers under distribution shifts.
Approach:	They propose to use natural language inference to verify whether answers are correct . they leverage large pre-trained models and recent prior datasets to construct powerful question conversion and decontextualization modules.
Outcome:	The proposed approach improves the confidence estimation of a QA model across different domains, evaluated in a selective QA setting.

Copied to clipboard

Shrey Desai, Greg Durrett

X-PARADE: Cross-Lingual Textual Entailment and Information Divergence across Paragraphs (2024.naacl-long)

Challenge:	Pre-trained Transformers dominate benchmark tasks but use a large number of self-attention heads across many layers in a way that is difficult to unpack.
Approach:	They analyze pre-trained Transformer models' posterior probabilities to determine whether they are calibrated for three tasks: natural language inference, paraphrase detection, and commonsense reasoning.
Outcome:	The models are calibrated in-domain and out-of-domain, and their calibration error out-domain can be as much as 3.5x lower.

Copied to clipboard

Juan Rodriguez, Katrin Erk, Greg Durrett

Inquisitive Question Generation for High Level Text Comprehension (2020.emnlp-main)

Challenge:	X-parade is the first cross-lingual dataset of paragraph-level information divergences and entailments . ability to recognize differences in meaning underlies many NLP tasks .
Approach:	They propose a cross-lingual dataset of paragraph-level information divergences . they use a dictionary to identify new or inferred information in Wikipedia pages .
Outcome:	The proposed dataset shows that the proposed methods fail to handle inferable information . the dataset contains fine-grained span-level annotations for content in different languages .

Copied to clipboard

Wei-Jen Ko, Te-yuan Chen, Yiyan Huang, Greg Durrett, Junyi Jessy Li

Model Agnostic Answer Reranking System for Adversarial Question Answering (2021.eacl-srw)

Challenge:	Existing data-driven questions generate questions that fill gaps in knowledge . a dataset of 19K questions is used to generate meaningful questions .
Approach:	They propose a dataset of 19K questions that are elicited while a person is reading a document.
Outcome:	The proposed model generates reasonable questions, but the task is challenging.

Copied to clipboard

Sagnik Majumder, Chinmoy Samant, Greg Durrett

Robustness in NLP

Challenge:	Existing methods for adversarial QA are often model specific and require retraining of the model . authors propose a simple method that can be applied directly to any QA model based on a model-agnostic approach .
Approach:	They propose a model-agnostic approach that explicitly reranks candidate answers . they use a QA model that scores candidates on the basis of content overlap with the question .
Outcome:	The proposed method outperforms state-of-the-art models on adversarial examples without retraining.

Entity Cloze By Date: What LMs Know About Unseen Entities (2022.findings-naacl)

Copied to clipboard

Yasumasa Onoe, Michael Zhang, Eunsol Choi, Greg Durrett

EEL: Efficiently Encoding Lattices for Reranking (2023.acl-long)

Challenge:	Existing literature provides benchmarks to measure LMs' knowledge about entities .
Approach:	They propose a framework to analyze what language models can infer about new entities that did not exist when they were pretrained.
Outcome:	The proposed framework shows that models more informed about the entities achieve lower perplexity on this benchmark.

Copied to clipboard

Prasann Singhal, Jiacheng Xu, Xi Ye, Greg Durrett

Can LMs Learn New Entities from Descriptions? Challenges in Propagating Injected Knowledge (2023.acl-long)

Challenge:	Existing methods for decoding conditional text are slow to apply to large numbers of hypotheses.
Approach:	They propose a method that can efficiently encode lattices of generated outputs using Transformers.
Outcome:	The proposed method can extract high-quality hypotheses from lattices with minimal degradation error compared to naive reranking methods.

Copied to clipboard

Yasumasa Onoe, Michael Zhang, Shankar Padmanabhan, Greg Durrett, Eunsol Choi

Understanding Neural Abstractive Summarization Models via Uncertainty (2020.emnlp-main)

Challenge:	Existing methods for updating knowledge show little propagation of injected knowledge.
Approach:	They propose to inject individual facts into LMs and evaluate whether they can propagate injected facts while not changing predictions on other contexts.
Outcome:	The proposed model can make inferences based on injected facts and propagate them . existing methods show little propagation of injected knowledge .

Copied to clipboard

Jiacheng Xu, Shrey Desai, Greg Durrett

Causal Graph based Event Reasoning using Semantic Relation Experts (2025.acl-long)

Challenge:	Recent advances in abstractive summarization have been fueled by the advent of large-scale Transformers pre-trained on autoregressive language modeling objectives.
Approach:	They analyze summarization decoders in both blackbox and whitebox ways by studying on the entropy, or uncertainty, of the model’s token-level predictions.
Outcome:	The proposed model generates tokens in a free-form manner, but this flexibility makes it difficult to interpret their behavior.

Copied to clipboard

Mahnaz Koupaee, Xueying Bai, Mudan Chen, Greg Durrett, Nathanael Chambers, Niranjan Balasubramanian

Byte Pair Encoding is Suboptimal for Language Model Pretraining (2020.findings-emnlp)

Challenge:	Recent advances in event reasoning have limited ability to accurately identify causal connections between events.
Approach:	They propose a collaborative approach to generate correct graphs and graphs to assist reasoning . they propose 'a causal chain of events' task that requires a causal link between events .
Outcome:	The proposed approach achieves competitive results with state-of-the-art models on forecasting and next event prediction tasks.

Copied to clipboard

Kaj Bostrom, Greg Durrett