Papers by Tanya Goyal
WiCE: Real-World Entailment for Claims in Wikipedia (2023.emnlp-main)
Copied to clipboard
| Challenge: | Textual entailment models are increasingly used in fact-checking, presupposition verification in question answering, or summary evaluation. |
| Approach: | They propose a new fine-grained textual entailment dataset built on natural claim and evidence pairs extracted from Wikipedia that provides en-tailment judgments over sub-sentence units of the claim and a minimal subset of evidence sentences that support each subclaim. |
| Outcome: | The proposed dataset improves on multiple datasets at test time and shows that real claims involve verification and retrieval problems that existing models fail to address. |
Shortcomings of Question Answering Based Factuality Frameworks for Error Localization (2023.eacl-main)
Copied to clipboard
| Challenge: | Abstractive summarization systems often generate summaries with factual errors . many approaches to detect these errors have been proposed, but this capability has not been evaluated in past research . |
| Approach: | They propose to use question answering-based factuality metrics to detect errors in summaries . they find that QA-based frameworks fail to correctly identify error spans in generated summary . |
| Outcome: | The proposed methods outperform trivial exact match baselines in localizing errors in summaries. |
Training Dynamics for Text Summarization Models (2022.findings-acl)
Copied to clipboard
| Challenge: | Pre-trained language models have shown impressive results when fine-tuned on large summarization datasets. |
| Approach: | They analyze the training dynamics for generation models, focusing on summarization . they find that a propensity to copy the input is learned early in the training process . |
| Outcome: | The proposed model learns at different stages of fine-tuning, the authors show . they show that factual errors are learnt in later stages, but not at high-loss tokens . |
Challenges in Trustworthy Human Evaluation of Chatbots (2025.findings-naacl)
Copied to clipboard
| Challenge: | apathetic or adversarial annotators can corrupt the reliability of open leaderboard rankings . human annotation is widely accepted as the gold standard for open-ended text generation tasks . |
| Approach: | They show that bad annotations can corrupt the reliability of open leaderboard rankings . they argue that human annotation is widely accepted as the gold standard . |
| Outcome: | The proposed algorithm can corrupt the reliability of open leaderboard rankings by up to 5 places. |
RefreshKV: Updating Small KV Cache During Long-form Generation (2025.acl-long)
Copied to clipboard
| Challenge: | Existing methods for generating long sequences of tokens are expensive and require memory and computation resources. |
| Approach: | They propose a method that alternates between full context attention and attention over a subset of input tokens during generation. |
| Outcome: | The proposed method achieves comparable speedup to eviction-based methods while improving performance for various long-form generation tasks. |
Memorization vs. Reasoning: Updating LLMs with New Knowledge (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing methods and benchmarks focus on entity substitutions, failing to capture the full breadth of complex real-world dynamics. |
| Approach: | They propose an automatic pipeline for simulating realistic knowledge updates reflected in an evidence corpus. |
| Outcome: | The proposed method outperforms prior continued pre-training (CPT) baselines on two LLM families and improves direct probing (memorization) results by 25.4%. |
Understanding Factual Errors in Summarization: Errors, Summarizers, Datasets, Error Detectors (2023.acl-long)
Copied to clipboard
Liyan Tang, Tanya Goyal, Alex Fabbri, Philippe Laban, Jiacheng Xu, Semih Yavuz, Wojciech Kryscinski, Justin Rousseau, Greg Durrett
| Challenge: | Abstractive summarization systems still include factual errors in generated summaries despite recent improvements in factuality detection . |
| Approach: | They aggregate factuality error annotations from nine existing datasets and stratify them according to the underlying summarization model. |
| Outcome: | The proposed method improves on the ChatGPT-based model and shows that it is not superior for all error types. |
The Progress Illusion: Revisiting meta-evaluation standards of LLM evaluators (2025.findings-emnlp)
Copied to clipboard
| Challenge: | LLM judges have gained popularity as an inexpensive and performant substitute for human evaluation. |
| Approach: | They revisit meta-evaluations of LLM evaluators under a setting that more closely aligns with practice by examining evaluers’ ability to distinguish test system pairs that are closer in capability. |
| Outcome: | The proposed meta-evaluation setting is significantly different from the use of human evaluations. |
Neural Syntactic Preordering for Controlled Paraphrase Generation (2020.acl-main)
Copied to clipboard
| Challenge: | Existing approaches to paraphrasing natural language sentences are limited by the complexity of the task. |
| Approach: | They propose a framework for paraphrasing natural language sentences that uses syntactic transformations to softly "reorder" the source sentence and their proposed system is evaluated automatically and by humans . |
| Outcome: | The proposed model retains the quality of the baseline approaches while giving a substantial increase in the diversity of the generated paraphrases. |
Embedding Time Expressions for Deep Temporal Ordering Models (P19-1)
Copied to clipboard
| Challenge: | Existing data-driven models fail to capture explicit temporal signals, such as dates and time windows. |
| Approach: | They propose a framework to infuse temporal awareness into data-driven models by learning a pre-trained model to embed timexes. |
| Outcome: | The proposed framework infuses temporal awareness into data-driven models by learning a pre-trained model to embed timexes. |
SNaC: Coherence Error Detection for Narrative Summarization (2022.emnlp-main)
Copied to clipboard
| Challenge: | SNaC framework is used to evaluate long summaries, but it fails to identify gaps in coherence . nallapati and colleagues have developed a framework for fine-grained annotations of long summarizations . |
| Approach: | They propose a narrative coherence evaluation framework for fine-grained annotations of long summaries that can be used to evaluate coherent narratives. |
| Outcome: | The proposed framework can support future work in document summarization and coherence evaluation, the authors show . |
One Thousand and One Pairs: A “novel” challenge for long-context language models (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing long-context evaluation methods measure surface-level retrieval capabilities, but do not assess performance on the more challenging task of synthesizing distant and underlying information. |
| Approach: | They propose a dataset of 1,001 minimally different pairs of true and false claims about 67 recently-published English fictional books. |
| Outcome: | The proposed model performs better on pairs that require only sentence-level retrieval vs. global reasoning . the proposed model also performs worse on speculative fiction books with extensive world-building . |
Evaluating Factuality in Generation with Dependency-level Entailment (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Existing text generation models produce text that is factually inconsistent with information in the input. |
| Approach: | They propose a new formulation of entailment that decomposes it at the level of dependency arcs. |
| Outcome: | The proposed model can identify factual errors better than sentence-level methods and question generation methods while localizing the erroneous parts of the generation. |
FALTE: A Toolkit for Fine-grained Annotation for Long Text Evaluation (2022.emnlp-demos)
Copied to clipboard
| Challenge: | Existing tools to evaluate long text outputs are lacking in the field of NLP . human rating and error analysis remains a crucial component for any evaluation of long text generation. |
| Approach: | They propose a web-based toolkit to collect fine-grained error annotations for long texts . they use a taxonomy to identify errors and assign them to text spans . |
| Outcome: | The proposed tool can be used to evaluate the coherence of long generated summaries. |
Annotating and Modeling Fine-grained Factuality in Summarization (2021.naacl-main)
Copied to clipboard
| Challenge: | Recent abstractive summarization systems produce factual errors that are not faithful to the input . current methods are lacking in identifying what errors are most important to target . |
| Approach: | They use synthetic and human-labeled data to identify factual errors in summarization and train models on the factuality detection task. |
| Outcome: | The proposed model detects factual errors on word, dependency, and sentence levels. |
HydraSum: Disentangling Style Features in Text Summarization with Multi-Decoder Models (2022.emnlp-main)
Copied to clipboard
| Challenge: | Abstractive summarization systems implicitly encode “decisions” about summary properties, but these are not enforced. |
| Approach: | They propose a new summarization architecture that extends existing models to a mixture-of-experts version with multiple decoders. |
| Outcome: | The proposed architecture outperforms baseline models in obtaining stylistically-diverse summaries by sampling from individual decoders or their mixtures. |
LitSearch: A Retrieval Benchmark for Scientific Literature Search (2024.emnlp-main)
Copied to clipboard
| Challenge: | Literature search questions pose significant challenges for modern retrieval systems . a lack of domain expertise and reasoning through lengthy papers is a challenge . |
| Approach: | They propose a retrieval benchmark for literature search queries using inline citations from papers and questions about recently published papers. |
| Outcome: | The proposed retrieval benchmarks outperform state-of-the-art retrieval models and reranking pipelines. |
DCRM: A Heuristic to Measure Response Pair Quality in Preference Optimization (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Recent research has attempted to associate preference optimization (PO) performance with the underlying preference datasets. |
| Approach: | They propose a best-of-N2 pairing method that selects response pairs with the highest DCRM to measure the quality of a response pair for PO. |
| Outcome: | The proposed method produces training datasets that can further improve models’ performance on AlpacaEval, MT-Bench, and Arena-Hard over the existing training sets. |