Papers by Shashi Narayan
HighRES: Highlight-based Reference-less Evaluation of Summarization (P19-1)
Copied to clipboard
| Challenge: | Existing methods for summarizing documents are inconsistent due to the difficulty of manual evaluation. |
| Approach: | They propose a method where summaries are evaluated by multiple annotators against the source document via manually highlighted salient content. |
| Outcome: | The proposed method improves inter-annotator agreement while highlighting differences among systems. |
Don’t Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization (D18-1)
Copied to clipboard
| Challenge: | Existing approaches to summarize documents are not extractive and require an abstractive approach. |
| Approach: | They propose a novel abstractive model which is conditioned on the article’s topics and based entirely on convolutional neural networks. |
| Outcome: | The proposed model outperforms an oracle extractive system and state-of-the-art abstractive approaches when evaluated automatically and by humans. |
On Faithfulness and Factuality in Abstractive Summarization (2020.acl-main)
Copied to clipboard
| Challenge: | Existing conditional text generation models produce unfaithful and unfaithed summaries . current models accomplish a high level of fluency and coherence . |
| Approach: | They propose to use pretrained models for document summarization to better understand hallucinations . they find that textual entailment measures better correlate with faithfulness . |
| Outcome: | The proposed models generate faithful and factual summaries as evaluated by humans. |
Query Refinement Prompts for Closed-Book Long-Form QA (2023.acl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) can answer questions and produce long-form texts, but the latter is difficult to evaluate since they are subjective in nature. |
| Approach: | They propose query refinement prompts that encourage LLMs to express multifacetedness and generate long-form answers covering multiple facets of the question. |
| Outcome: | The proposed model outperforms fully finetuned models in the closed-book setting and retrieve-then-generate open-book models. |
Multilingual Summarization with Factual Consistency Evaluation (2023.findings-acl)
Copied to clipboard
| Challenge: | Abstractive summarization models generate factually inconsistent summaries, reducing their utility for real-world applications. |
| Approach: | They propose to use data filtering and controlled generation to detect hallucinations in machine generated summaries. |
| Outcome: | The proposed models detect factual inconsistencies in machine generated summaries, but they focus on English only. |
A Thorough Evaluation of Task-Specific Pretraining for Summarization (2021.emnlp-main)
Copied to clipboard
| Challenge: | Previous work has used task-agnostic pretraining methods like masked language models or corrupted span prediction to improve performance on downstream tasks. |
| Approach: | They propose to use a task-agnostic pretraining to improve on low-resource tasks. |
| Outcome: | The proposed model can predict extracted gap sentences on summarization with a low resource and zero shot setup. |
Stepwise Extractive Summarization and Planning with Structured Transformers (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to extractive summarization use transformers to learn the structure of long inputs. |
| Approach: | They propose encoder-centric stepwise models for extractive summarization using structured transformers – HiBERT and Extended Transformers . |
| Outcome: | The proposed models outperform previous models on CNN/DailyMail extractive summarization and Rotowire table-to-text generation. |
Document Modeling with External Attention for Sentence Extraction (P18-1)
Copied to clipboard
Shashi Narayan, Ronald Cardenas, Nikos Papasarantopoulos, Shay B. Cohen, Mirella Lapata, Jiangsheng Yu, Yi Chang
| Challenge: | Document modeling is essential to a variety of natural language understanding tasks. |
| Approach: | They propose to use external information to improve document modeling for sentence extraction problems. |
| Outcome: | The proposed model outperforms baseline models on document summarization and answer selection tasks and achieves state-of-the-art results on WikiQA and NewsQA. |
MiRANews: Dataset and Benchmarks for Multi-Resource-Assisted News Summarization (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Current news summarization systems often contain 'extrinsic hallucinations', i.e. facts that are not present in the source document, which are often derived via world knowledge. |
| Approach: | They propose to use multiple supplementary resource documents to assist the task by pairing a single document with a human authored summary as the summary. |
| Outcome: | The proposed model reduces 55% of hallucinations when compared to single-document summarization models trained on the main article only. |
Learning to Plan and Generate Text with Citations (2024.acl-long)
Copied to clipboard
Constanza Fierro, Reinald Kim Amplayo, Fantine Huot, Nicola De Cao, Joshua Maynez, Shashi Narayan, Mirella Lapata
| Challenge: | Large language models (LLMs) are increasingly useful in information-seeking scenarios, ranging from answering simple questions to generating responses to search-like queries. |
| Approach: | They propose to use plan-based models to improve faithfulness, grounding, and controllability of generated content and its organization. |
| Outcome: | The proposed models improve faithfulness, grounding, and controllability of generated content and its organization. |
Planning with Learned Entity Prompts for Abstractive Summarization (2021.tacl-1)
Copied to clipboard
| Challenge: | a simple but flexible mechanism is used to ground the generation of abstractive summaries. |
| Approach: | They propose a mechanism to learn an intermediate plan to ground the generation of abstractive summaries. |
| Outcome: | The proposed model outperforms state-of-the-art methods for faithfulness on CNN and BillSum. |
Local String Transduction as Sequence Labeling (C18-1)
Copied to clipboard
| Challenge: | String transduction and sequence labeling are often treated as separate entities and often give treatment to different problems in NLP. |
| Approach: | They propose to reduce string transduction to sequence labeling by using a finite-state technique that uses string transducing and sequence labelling. |
| Outcome: | The proposed method performs better than seq2seq models and yields state-of-the-art results in several cases. |
Little Red Riding Hood Goes around the Globe: Crosslingual Story Planning and Generation with Large Language Models (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing work has demonstrated the effectiveness of planning for story generation exclusively in a monolingual setting focusing primarily on English. |
| Approach: | They propose a task of crosslingual story generation with planning to leverage the creative and reasoning capabilities of large pretrained language models to generate stories in multiple languages. |
| Outcome: | The proposed task combines planning and planning in a monolingual setting and demonstrates that plans which structure stories into three acts lead to more coherent and interesting narratives while allowing to explicitly control their content and structure. |
𝜇PLAN: Summarizing using a Content Plan as Cross-Lingual Bridge (2024.eacl-long)
Copied to clipboard
Fantine Huot, Joshua Maynez, Chris Alberti, Reinald Kim Amplayo, Priyanka Agrawal, Constanza Fierro, Shashi Narayan, Mirella Lapata
| Challenge: | Recent advances in abstractive summarization have focused on English, but more recently, with the advent of large pre-trained models, the task is becoming more complex. |
| Approach: | They propose an approach to cross-lingual summarization that uses an intermediate planning step as a cross-linguistic bridge. |
| Outcome: | The proposed approach achieves state-of-the-art in terms of informativeness and faithfulness on the XWikis dataset. |
Conditional Generation with a Question-Answering Blueprint (2023.tacl-1)
Copied to clipboard
Shashi Narayan, Joshua Maynez, Reinald Kim Amplayo, Kuzman Ganchev, Annie Louis, Fantine Huot, Anders Sandholm, Dipanjan Das, Mirella Lapata
| Challenge: | Neural generation models often struggle to identify which content units are salient. |
| Approach: | They propose a new conceptualization of text plans as a sequence of question-answer pairs . they propose QA blueprints as QA proxy for content selection and planning . |
| Outcome: | The proposed model improves existing datasets with QA blueprints as proxy for content selection and planning. |
Deep Learning Approaches to Text Production (N18-6)
Copied to clipboard
| Challenge: | Text production is a key component of many NLP applications . Claire Gardent is based in France and is pursuing research in text production . |
| Approach: | This tutorial will cover the fundamentals and state-of-the-art research on neural models for text production. |
| Outcome: | This tutorial will cover the fundamentals and the state-of-the-art research on neural models for text production. |
A Well-Composed Text is Half Done! Composition Sampling for Diverse Conditional Generation (2022.acl-long)
Copied to clipboard
Shashi Narayan, Gonçalo Simões, Yao Zhao, Joshua Maynez, Dipanjan Das, Michael Collins, Mirella Lapata
| Challenge: | Composition Sampling is a simple but effective method to generate diverse outputs for conditional generation of higher quality compared to previous stochastic decoding strategies. |
| Approach: | They propose a method to generate diverse outputs for conditional generation . they use a plan-based neural generation model that is trained to create a composition of the output and then generate by conditioning on it and the input. |
| Outcome: | The proposed method avoids text degeneration by first sampling a composition in the form of an entity chain and then using beam search to generate the best possible text grounded to this entity chain. |
On Uncertainty Calibration and Selective Generation in Probabilistic Neural Summarization: A Benchmark Study (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Modern deep models for summarization generate miscalibrated predictive uncertainty, compromising reliability and trustworthiness in real-world applications. |
| Approach: | They propose to use probabilistic methods to improve the uncertainty quality of neural summarization models by using three large-scale benchmarks with varying difficulty. |
| Outcome: | The proposed methods consistently improve the model’s generation and uncertainty quality, leading to improved selective generation performance (i.e., abstaining from low-quality summaries) in practice. |
Focus Attention: Promoting Faithfulness and Diversity in Summarization (2021.acl-long)
Copied to clipboard
| Challenge: | Currently, document summarization is challenging even for humans. |
| Approach: | They propose a focus attention mechanism which encourages decoders to generate tokens that are topically similar to the input document. |
| Outcome: | The proposed method outperforms top-k and nucleus sampling methods on the BBC extreme summarization task and is more accurate than focus attention-based models. |
Leveraging Pre-trained Checkpoints for Sequence Generation Tasks (2020.tacl-1)
Copied to clipboard
| Challenge: | Unsupervised pre-training of large neural models has revolutionized Natural Language Processing. |
| Approach: | They propose to use pre-trained checkpoints for Sequence Generation to initialize a Transformer-based sequence-to-sequence model that is compatible with these checkpoint. |
| Outcome: | The proposed model is compatible with pre-trained BERT, GPT-2, and RoBERTa checkpoints and achieves state-of-the-art results on Machine Translation, Text Summarization, Sentence Splitting, and Sentance Fusion. |
Data Augmentation for Low-Resource Dialogue Summarization (2022.findings-naacl)
Copied to clipboard
| Challenge: | DADS generates synthetic examples by replacing sections of text from input dialogue and summary while preserving the augmented summary to correspond to a viable summary for the simulated dialogue. |
| Approach: | They propose a Data Augmentation technique for low-resource Dialogue Summarization that uses pretrained language models to generate diverse alternatives. |
| Outcome: | The proposed method generates synthetic examples from a low-resource dataset . it produces topically diverse examples without introducing additional hallucinations . |
Jointly Extracting and Compressing Documents with Summary State Representations (N19-1)
Copied to clipboard
| Challenge: | Text summarization is an important NLP problem with a wide range of applications in data-driven industries. |
| Approach: | They propose a neural model that extracts sentences from a document and compresses them. |
| Outcome: | The proposed model generates concise and informa-tive summaries on CNN/DailyMail and Newsroom datasets and human evaluations show it outperforms existing methods. |
Ranking Sentences for Extractive Summarization with Reinforcement Learning (N18-1)
Copied to clipboard
| Challenge: | Abstractive summarization involves various text rewriting operations and has been identified as a sequence-to-sequence problem. |
| Approach: | They propose a novel algorithm which globally optimizes the ROUGE evaluation metric through a reinforcement learning objective. |
| Outcome: | The proposed algorithm outperforms state-of-the-art extractive and abstractive systems when evaluated automatically and by humans. |
Text-Blueprint: An Interactive Platform for Plan-based Conditional Generation (2023.eacl-demo)
Copied to clipboard
Fantine Huot, Joshua Maynez, Shashi Narayan, Reinald Kim Amplayo, Kuzman Ganchev, Annie Priyadarshini Louis, Anders Sandholm, Dipanjan Das, Mirella Lapata
| Challenge: | Recent work shows that conditional generation models can be useful to control the text generation process, leading to irrelevant, repetitive, and hallucinated content. |
| Approach: | They propose a web browser-based demonstration for query-focused summarization that uses a sequence of question-answer pairs as a blueprint plan for guiding text generation. |
| Outcome: | The proposed model can be used to generate query-focused summarization text using question-answer pairs as a blueprint plan. |
Privacy-preserving Neural Representations of Text (D18-1)
Copied to clipboard
| Challenge: | a specific type of attack is used to characterize the privacy of neural representations for NLP tasks, in the context of privacy protection. |
| Approach: | They propose several defense methods based on modified training objectives and characterize the tradeoff between privacy and the utility of neural representations. |
| Outcome: | The proposed defenses improve the privacy of neural representations and characterize the tradeoff between privacy and utility of representations. |