Analyzing and Evaluating Faithfulness in Dialogue Summarization (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies on faithfulness of text summarization have not been conducted on abstractive summarizing. |
| Approach: | They propose a method to evaluate faithfulness of dialogue summarization models by multi-choice questions. |
| Outcome: | The proposed method can facilitate the development of dialogue summarization systems. |
Similar Papers
DialSummEval: Revisiting Summarization Evaluation for Dialogues (2022.naacl-main)
Copied to clipboard
| Challenge: | Current models for dialogue summarization have flaws that may not be well exposed by frequently used metrics such as ROUGE. |
| Approach: | They propose to re-evaluate 18 categories of metrics in terms of four dimensions: coherence, consistency, fluency and relevance, as well as a unified human evaluation of various models for the first time. |
| Outcome: | The proposed dataset will be used to evaluate 18 categories of metrics in terms of coherence, consistency, fluency and relevance, and a unified human evaluation of various models for the first time. |
STORYSUMM: Evaluating Faithfulness in Story Summarization (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for evaluating abstractive summarization are lacking in faithfulness evaluation. |
| Approach: | They propose a dataset that measures faithfulness of LLM summaries with localized errors and faithfulness labels for evaluation methods. |
| Outcome: | The proposed method does not achieve more than 70% accuracy on this task. |
On Faithfulness and Factuality in Abstractive Summarization (2020.acl-main)
Copied to clipboard
| Challenge: | Existing conditional text generation models produce unfaithful and unfaithed summaries . current models accomplish a high level of fluency and coherence . |
| Approach: | They propose to use pretrained models for document summarization to better understand hallucinations . they find that textual entailment measures better correlate with faithfulness . |
| Outcome: | The proposed models generate faithful and factual summaries as evaluated by humans. |
Exploring the Factual Consistency in Dialogue Comprehension of Large Language Models (2024.naacl-long)
Copied to clipboard
| Challenge: | LLMs generate responses following user's instructions, which requires high dialogue comprehension ability. |
| Approach: | They propose to evaluate LLMs' dialogue comprehension ability using a dialogue summarization task to derive factual questions from the generated summaries and use them as a more flexible measurement of dialogue comprehension. |
| Outcome: | The proposed model reduces the error rate by 11% on the dialogue summarization task. |
CONFIT: Toward Faithful Dialogue Summarization with Linguistically-Informed Contrastive Fine-tuning (2022.naacl-main)
Copied to clipboard
Xiangru Tang, Arjun Nair, Borui Wang, Bingyao Wang, Jai Desai, Aaron Wade, Haoran Li, Asli Celikyilmaz, Yashar Mehdad, Dragomir Radev
| Challenge: | Factual inconsistencies in generated summaries severely limit the practical applications of abstractive dialogue summarization. |
| Approach: | They propose a typology of factual errors to better understand hallucinations generated by current models and a contrastive fine-tuning strategy to improve the factual consistency and overall quality of summaries. |
| Outcome: | The proposed model significantly reduces all kinds of factual errors on both SAMSum dialogue summarization and AMI meeting summarizing datasets. |
LongEval: Guidelines for Human Evaluation of Faithfulness in Long-form Summarization (2023.eacl-main)
Copied to clipboard
| Challenge: | Human evaluation is labor-intensive, expensive to scale, and difficult to design. |
| Approach: | They propose a set of guidelines for human evaluation of faithfulness in long-form summaries that address the following challenges: (1) How can we achieve high inter-annotator agreement on faithfulness scores? (2) How can our annotator minimize workload while maintaining accurate faithfulness? |
| Outcome: | The proposed framework reduces inter-annotator variance in faithfulness scores while minimizing annotator workload while maintaining accuracy. |
SWING: Balancing Coverage and Faithfulness for Dialogue Summarization (2023.findings-eacl)
Copied to clipboard
Kung-Hsiang Huang, Siffi Singh, Xiaofei Ma, Wei Xiao, Feng Nan, Nicholas Dingwall, William Yang Wang, Kathleen McKeown
| Challenge: | Existing approaches to dialogue summarization rely on features of conversation data. |
| Approach: | They propose to use natural language inference models to improve coverage and faithfulness . they use fine-grained training signals to encourage model to generate missing content . |
| Outcome: | The proposed model achieves higher faithfulness and coverage while maintaining conciseness compared to prior methods. |
Towards Faithfully Interpretable NLP Systems: How Should We Define and Evaluate Faithfulness? (2020.acl-main)
Copied to clipboard
| Challenge: | Current approaches to interpretability evaluation focus on faithfulness criteria . current approaches focus on readability, plausibility and faithfulness . |
| Approach: | They argue that current binary definition of faithfulness sets unrealistic standards . they argue that a more graded definition would be of greater practical utility . |
| Outcome: | The proposed approach is based on three assumptions and lacks a graded definition of faithfulness. |
Human-in-the-loop Abstractive Dialogue Summarization (2023.findings-acl)
Copied to clipboard
| Challenge: | Abstractive dialogue summarization systems are trained to maximize the likelihood of human-written summaries, but there is still a huge gap in generating high-quality summary as determined by humans. |
| Approach: | They propose to incorporate different levels of human feedback into the training process . they ask humans to highlight salient information to be included in summaries . |
| Outcome: | The proposed model captures human-written summaries and compares them with state-of-the-art models on multiple datasets. |
FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization (2020.acl-main)
Copied to clipboard
| Challenge: | Existing automatic metrics do not capture errors in abstractive summarization models. |
| Approach: | They propose an automatic question answering metric for faithfulness that leverages recent advances in reading comprehension. |
| Outcome: | The proposed metric has significantly higher correlation with human faithfulness scores on highly abstracted summaries. |