Leveraging Entailment Judgements in Cross-Lingual Summarisation (2024.findings-acl)
Copied to clipboard
| Challenge: | Synthetically created cross-lingual summarisation datasets are prone to include document-summary pairs where the reference summary is unfaithful to the corresponding document. |
| Approach: | They propose to use off-the-shelf cross-lingual Natural Language Inference to evaluate faithfulness of reference and model generated summaries and use unlikelihood loss to teach a model about unfaithful summary sequences. |
| Outcome: | The proposed approach evaluates faithfulness of reference and model generated summaries and uses unlikelihood loss to teach a model about unfaithful summary sequences. |
Similar Papers
Improving Faithfulness by Augmenting Negative Summaries from Fake Documents (2022.emnlp-main)
Copied to clipboard
| Challenge: | Current abstractive summarization systems tend to hallucinate unfaithful content . however, the most common method does not disentangle factual errors from other errors. |
| Approach: | They propose a back-translation-style approach to augment negative samples that mimic factual errors made by the model. |
| Outcome: | The proposed method improves faithfulness without sacrificing informativeness . it incorporates negative samples into training, and produces faithful/unfaithful summaries . |
Fine-Grained Natural Language Inference Based Faithfulness Evaluation for Diverse Summarisation Tasks (2024.eacl-long)
Copied to clipboard
| Challenge: | Existing approaches to evaluate summary faithfulness are sub-optimal due to the granularity level considered for premises and hypotheses. |
| Approach: | They propose a novel approach that uses a variable premise size and simplifies summary sentences into shorter hypotheses. |
| Outcome: | The proposed model performs better on diverse summarisation tasks than existing models. |
With a Little Push, NLI Models can Robustly and Efficiently Predict Faithfulness (2023.acl-short)
Copied to clipboard
| Challenge: | Conditional language models generate unfaithful output that is not supported by their input . this jeopardizes trust in real-world applications, raising a need for automatic faithfulness metrics. |
| Approach: | They propose to augment conditional language models with robust inference procedures to improve faithfulness. |
| Outcome: | The proposed approach outperforms existing models on the TRUE benchmark. |
Detecting and Mitigating Hallucinations in Multilingual Summarisation (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing faithfulness metrics for abstractive summarisation models focus on English . metric mFACT is best suited to detect hallucinations in cross-lingual transfer . |
| Approach: | They propose a method to evaluate the faithfulness of non-English summaries by translation-based transfer from multiple English faithfulness metrics. |
| Outcome: | The proposed method reduces hallucinations in cross-lingual transfer by weighing the loss of each training example by its faithfulness score. |
On Faithfulness and Factuality in Abstractive Summarization (2020.acl-main)
Copied to clipboard
| Challenge: | Existing conditional text generation models produce unfaithful and unfaithed summaries . current models accomplish a high level of fluency and coherence . |
| Approach: | They propose to use pretrained models for document summarization to better understand hallucinations . they find that textual entailment measures better correlate with faithfulness . |
| Outcome: | The proposed models generate faithful and factual summaries as evaluated by humans. |
CLIFF: Contrastive Learning for Improving Faithfulness and Factuality in Abstractive Summarization (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for generating abstractive summarization are inconsistent and rely on heuristically created data for error handling. |
| Approach: | They propose a contrastive learning formulation that leverages both positive and negative summaries to train summarization systems that are better at distinguishing between them. |
| Outcome: | The proposed learning framework produces more factual summaries than strong comparisons with post error correction, entailment-based reranking, and unlikelihood training. |
Evaluating Factuality in Cross-lingual Summarization (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing evaluation metrics for monolingual summarization require translation to evaluate the factuality of cross-lingual summmarization. |
| Approach: | They propose to analyze cross-lingual factuality by collecting annotations and generated summaries from models at summary level and sentence level. |
| Outcome: | The proposed dataset shows that over 50% of generated summaries contain factual errors with different characteristics from monolingual summarization. |
Falsesum: Generating Document-level NLI Examples for Recognizing Factual Inconsistency in Summarization (2022.naacl-main)
Copied to clipboard
| Challenge: | Neural abstractive summarization models generate factually inconsistent summaries . previous work has introduced the task of recognizing factual inconsistency as a downstream application of natural language inference (NLI). |
| Approach: | They propose a data generation pipeline that enables a task-oriented approach to detect factual inconsistencies in abstractive summarization models. |
| Outcome: | The proposed model improves the state-of-the-art performance across four benchmarks for recognizing factual inconsistency in generated summaries. |
Mixed-Lingual Pre-training for Cross-lingual Summarization (2020.aacl-main)
Copied to clipboard
| Challenge: | Cross-lingual summarization (CLS) aims at producing a summary in the target language for an article in the source language. |
| Approach: | They propose a mixed-lingual pre-training scheme that leverages both cross-lingual tasks such as translation and monolingual tasks like masked language models. |
| Outcome: | The proposed model improves on the translation and masked language models with no task-specific components and saves memory. |
Understanding Translationese in Cross-Lingual Summarization (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing datasets involve translation, but translationese is distinguished from original text . previous studies have shown that translationeses in CLS are not a problem in training sets . |
| Approach: | They propose to use cross-lingual summarization to generate a concise summary in a target language from a document in . existing datasets typically involve translation in their creation, but the translated text is distinguished from the original written in that language. |
| Outcome: | The proposed method systematically investigates how translationese affects CLS model evaluation and performance when it appears in source documents or target summaries. |