Lexical Repetitions Lead to Rote Learning: Unveiling the Impact of Lexical Overlap in Train and Test Reference Summaries (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Ideal summarization models should generalize to novel summary-worthy content without remembering reference training summaries by rote. |
| Approach: | They propose to partition test set based on lexical similarity of reference test summaries with training summary to determine model competencies. |
| Outcome: | The proposed evaluation protocol improves generalization and generalization on novel test cases while maintaining average performance. |
Similar Papers
Balancing Lexical and Semantic Quality in Abstractive Summarization (2023.acl-short)
Copied to clipboard
| Challenge: | Existing methods to reduce exposure bias in sequence-to-sequence models are underexplored. |
| Approach: | They propose a method to re-rank sequence-to-sequence neural models to reduce exposure bias. |
| Outcome: | The proposed method achieves an 89.67 BERTScore on the CNN/DailyMail and XSum datasets. |
Breaking NLI Systems with Sentences that Require Simple Lexical Inferences (P18-2)
Copied to clipboard
| Challenge: | a new test set shows the deficiency of state-of-the-art models in inferences that require lexical and world knowledge. |
| Approach: | They create a new NLI test set that shows the deficiency of state-of-the-art models in inferences that require lexical and world knowledge. |
| Outcome: | The new examples are simpler than the SNLI test set, but the state-of-the-art systems perform poorly on it. |
How well do you know your summarization datasets? (2021.findings-acl)
Copied to clipboard
| Challenge: | State-of-the-art summarization systems are trained on massive datasets scraped from the web. |
| Approach: | They manually analyse 600 samples from three popular summarization datasets . they use a six-class typology which captures different noise types and degrees of summarizing difficulty. |
| Outcome: | The proposed model performs better on large datasets than on the current models. |
SummEval: Re-evaluating Summarization Evaluation (2021.tacl-1)
Copied to clipboard
Alexander R. Fabbri, Wojciech Kryściński, Bryan McCann, Caiming Xiong, Richard Socher, Dragomir Radev
| Challenge: | a lack of comprehensive studies on evaluation metrics for text summarization hinders progress . a new study aims to improve evaluation metrics that correlate with human judgments . |
| Approach: | They propose to re-evaluate automatic evaluation metrics and share a toolkit for evaluation . they hope to promote a more complete evaluation protocol for text summarization . |
| Outcome: | The proposed evaluation metrics are inconsistent with existing evaluation protocols. |
Lexical Generalization Improves with Larger Models and Longer Training (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Excessive utilization of lexical overlap heuristics can lead to failure on challenging inputs. |
| Approach: | They analyze the use of lexical overlap heuristics in natural language inference, paraphrase detection, and reading comprehension using a contrastive dataset. |
| Outcome: | The proposed model can be used to improve performance on a wide range of tasks, but it is often wrong. |
What Have We Achieved on Text Summarization? (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for text summarization have been investigated, but there are still gaps between them and human professionals. |
| Approach: | They analyze 8 major sources of errors on 10 representative summarization models manually. |
| Outcome: | Aiming to gain more understanding of summarization systems with respect to their strengths and limitations on a fine-grained syntactic and semantic level, we use 8 major sources of errors on 10 representative summarizing models. |
A Tale of Evaluating Factual Consistency: Case Study on Long Document Summarization Evaluation (2025.findings-acl)
Copied to clipboard
| Challenge: | Despite the recent progress for summarization models in producing fluent summaries, they still encounter challenges when long sequences of generated texts and inputs (over thousands of words) need to be evaluated. |
| Approach: | They conduct a systematic analysis of factual-consistency evaluation systems across four long-document datasets and examine the relationship between sentence-level and summary-level model performance. |
| Outcome: | The proposed models can achieve higher recall in error detection for older summaries, yet struggle with false positives and fine-grained error detection. |
Annotating and Modeling Fine-grained Factuality in Summarization (2021.naacl-main)
Copied to clipboard
| Challenge: | Recent abstractive summarization systems produce factual errors that are not faithful to the input . current methods are lacking in identifying what errors are most important to target . |
| Approach: | They use synthetic and human-labeled data to identify factual errors in summarization and train models on the factuality detection task. |
| Outcome: | The proposed model detects factual errors on word, dependency, and sentence levels. |
Split and Rephrase: Better Evaluation and Stronger Baselines (P18-2)
Copied to clipboard
| Challenge: | a dataset mapping a complex sentence to a sequence of sentences conveying the same meaning is challenging in NLP. |
| Approach: | They propose a neural split and a copy-mechanism to break a complex sentence into several shorter sentences that convey the same meaning. |
| Outcome: | The proposed model outperforms the baseline model by 8.68 BLEU and further improves on the task. |
Proceedings of the 2nd Workshop on New Frontiers in Summarization (D19-54)
Copied to clipboard
| Challenge: | EMNLP 2017 is a workshop on enhancing natural language processing's ability to produce concise, fluent summaries. |
| Approach: | the workshop provides a forum for cross-fertilization of ideas towards automatic summarization . four invited speakers will be present at the workshop . |
| Outcome: | the workshop aims to provide a forum for cross-fertilization of ideas towards automatic summarization. |