Papers by Wojciech Kryscinski
FOLIO: Natural Language Reasoning with First-Order Logic (2024.emnlp-main)
Copied to clipboard
Simeng Han, Hailey Schoelkopf, Yilun Zhao, Zhenting Qi, Martin Riddell, Wenfei Zhou, James Coady, David Peng, Yujie Qiao, Luke Benson, Lucy Sun, Alexander Wardle-Solano, Hannah Szabó, Ekaterina Zubova, Matthew Burtell, Jonathan Fan, Yixin Liu, Brian Wong, Malcolm Sailor, Ansong Ni, Linyong Nan, Jungo Kasai, Tao Yu, Rui Zhang, Alexander Fabbri, Wojciech Kryscinski, Semih Yavuz, Ye Liu, Xi Lin, Shafiq Joty, Yingbo Zhou, Caiming Xiong, Rex Ying, Arman Cohan, Dragomir Radev
| Challenge: | Existing benchmarks for logical reasoning in large language models lack language naturalness or limited complexity. |
| Approach: | They propose to use first-order logic annotations to evaluate logical reasoning capabilities of large language models. |
| Outcome: | The proposed dataset evaluates the FOL reasoning ability of supervised fine-tuning on medium-sized language models. |
What’s New? Summarizing Contributions in Scientific Literature (2023.eacl-main)
Copied to clipboard
| Challenge: | a growing number of academic articles are shared daily, making it difficult to keep up with the latest findings. |
| Approach: | They propose a task of disentangled paper summarization which generates separate summaries for papers and contexts to make it easier to identify key findings shared in articles. |
| Outcome: | The proposed task is more useful than traditional scientific paper summarization. |
SWiPE: A Dataset for Document-Level Simplification of Wikipedia Pages (2023.acl-long)
Copied to clipboard
| Challenge: | Prior work on document-level simplification has focused on sentence-level edits, while many desirable edits require document- level context. |
| Approach: | They propose a dataset that reconstructs the document-level editing process from English Wikipedia to paired Simple Wikipedia articles. |
| Outcome: | The proposed dataset reconstructs the document-level editing process from English Wikipedia (EW) articles to paired Simple Wikipedia (SEW) pages. |
Socratic Pretraining: Question-Driven Pretraining for Controllable Summarization (2023.acl-long)
Copied to clipboard
| Challenge: | Existing methods to control document controllable summarization lack abundant labeled data. |
| Approach: | They propose a question-driven, unsupervised pretraining objective to improve controllability in document controllable summarization tasks. |
| Outcome: | The proposed method outperforms pre-finetuning approaches on QMSum and SQuALITY. |
Understanding Factual Errors in Summarization: Errors, Summarizers, Datasets, Error Detectors (2023.acl-long)
Copied to clipboard
Liyan Tang, Tanya Goyal, Alex Fabbri, Philippe Laban, Jiacheng Xu, Semih Yavuz, Wojciech Kryscinski, Justin Rousseau, Greg Durrett
| Challenge: | Abstractive summarization systems still include factual errors in generated summaries despite recent improvements in factuality detection . |
| Approach: | They aggregate factuality error annotations from nine existing datasets and stratify them according to the underlying summarization model. |
| Outcome: | The proposed method improves on the ChatGPT-based model and shows that it is not superior for all error types. |
BOOKSUM: A Collection of Datasets for Long-form Narrative Summarization (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing text summarization datasets include short-form source documents that lack long-range causal and temporal dependencies and contain strong layout and stylistic biases. |
| Approach: | They propose a dataset for long-form narrative summarization that uses human written summaries on three levels of difficulty. |
| Outcome: | The proposed dataset covers documents from the literature domain, such as novels, plays and stories, and includes highly abstractive, human written summaries on three levels of difficulty. |
SummVis: Interactive Visual Analysis of Models, Data, and Evaluation for Text Summarization (2021.acl-demo)
Copied to clipboard
| Challenge: | despite advances in abstractive text summarization, the true performance and failure modes of modern neural models are not yet fully understood due to the black-box nature of neural models and unmanageable scale of recent datasets for manual analysis. |
| Approach: | They propose an open-source tool for visualizing abstractive summaries that enables fine-grained analysis of models, data, and evaluation metrics associated with text summarization. |
| Outcome: | The proposed tool can identify the shortcomings and failure modes of state-of-the-art summarization models. |
SummEdits: Measuring LLM Ability at Factual Reasoning Through The Lens of Summarization (2023.emnlp-main)
Copied to clipboard
Philippe Laban, Wojciech Kryscinski, Divyansh Agarwal, Alexander Fabbri, Caiming Xiong, Shafiq Joty, Chien-Sheng Wu
| Challenge: | Existing factual consistency benchmarks are inadequate to detect factual inconsistencies in LLMs. |
| Approach: | They propose a protocol for inconsistency detection benchmark creation and implement it in a 10-domain benchmark called SummEdits. |
| Outcome: | The proposed method is 20 times more cost-effective per sample and highly reproducible, as it estimates inter-annotator agreement at about 0.9. |
Exploring Neural Models for Query-Focused Summarization (2022.findings-naacl)
Copied to clipboard
| Challenge: | Recent work in Query-focused summarization lacks a comprehensive study of the broad space of applicable modeling methods. |
| Approach: | They propose to explore two general classes of methods for Query-focused summarization: extractive-abstractive solutions and end-to-end models. |
| Outcome: | The proposed models achieve state-of-the-art on the QMSum dataset, with a margin of 3.38 ROUGE-1, 3.72 ROUGe2 and 3.28 ROUGEL-L. |
Evaluating the Factual Consistency of Abstractive Text Summarization (2020.emnlp-main)
Copied to clipboard
| Challenge: | a weakly-supervised approach is needed to verify factual consistency . auxiliary span extraction tasks are useful for verifying factual consistent summaries . |
| Approach: | They propose a weakly-supervised approach for verifying factual consistency . they transfer the model to summaries generated by several neural models . |
| Outcome: | The proposed approach outperforms models trained with strong supervision on source documents and human evaluations. |
HydraSum: Disentangling Style Features in Text Summarization with Multi-Decoder Models (2022.emnlp-main)
Copied to clipboard
| Challenge: | Abstractive summarization systems implicitly encode “decisions” about summary properties, but these are not enforced. |
| Approach: | They propose a new summarization architecture that extends existing models to a mixture-of-experts version with multiple decoders. |
| Outcome: | The proposed architecture outperforms baseline models in obtaining stylistically-diverse summaries by sampling from individual decoders or their mixtures. |
Neural Text Summarization: A Critical Evaluation (D19-1)
Copied to clipboard
| Challenge: | Current approaches to text summarization use advanced attention and copying mechanisms, multi-task and multi-reward training techniques. |
| Approach: | They evaluate datasets, evaluation metrics, and models for text summarization . they highlight three primary shortcomings: 1) datasets leave task underconstrained; 2) models overfit layout biases . |
| Outcome: | The current evaluation protocol is weakly correlated with human judgment and does not account for factual correctness. |
Long Document Summarization with Top-down and Bottom-up Inference (2023.findings-eacl)
Copied to clipboard
| Challenge: | Recent models infer latent representations of words or tokens with a transformer encoder, which is bottom-up and thus does not capture long-distance context well. |
| Approach: | They propose a method to infer latent representations of words or tokens in documents . they assume a hierarchical structure of a document where top-level captures long range dependency . |
| Outcome: | The proposed model can summarize an entire book and achieve competitive performance on a wide range of document summarization benchmarks. |
CTRLsum: Towards Generic Controllable Text Summarization (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing summarization systems produce generic summaries that are disconnected from users’ preferences and expectations. |
| Approach: | They propose a generic framework to control generated summaries through a set of keywords. |
| Outcome: | The proposed framework is comparable or better than strong pretrained systems on three domains of summarization datasets and five control tasks. |
Improving the Faithfulness of Abstractive Summarization via Entity Coverage Control (2022.findings-naacl)
Copied to clipboard
| Challenge: | Abstractive summarization systems have been shown to be more prone to unfaithful facts . 30% of summaries generated by pre-trained language models suffer from hallucination . |
| Approach: | They propose a method to remedy entity-level extrinsic hallucinations with Entity Coverage Control . they first compute entity coverage precision and prepend the corresponding control code . a further fine-tuning is performed to unlock zero-shot summarization . |
| Outcome: | The proposed method leads to more faithful and salient abstractive summarization in fine-tuning and zero-shot settings. |