Factual or Contextual? Disentangling Error Types in Entity Description Generation (2023.acl-long)
Copied to clipboard
| Challenge: | Existing evaluation practices only distinguish between model generated referring expressions being accurate (ground-truth) versus inaccurate (not groundtruth). |
| Approach: | They propose to integrate indicators for factual inconsistencies and contextual incongruities into automated evaluations of language models to assess the differences in error types across familiar vs unfamiliar entities. |
| Outcome: | The proposed evaluation paradigm disentangles factuality and congruity errors in natural contexts. |
Similar Papers
Entity-level Factual Consistency of Abstractive Text Summarization (2021.eacl-main)
Copied to clipboard
Feng Nan, Ramesh Nallapati, Zhiguo Wang, Cicero Nogueira dos Santos, Henghui Zhu, Dejiao Zhang, Kathleen McKeown, Bing Xiang
| Challenge: | Existing models exhibit entity hallucination, generating names of entities that are not present in the source document. |
| Approach: | They propose to use entity-level factual consistency to improve model quality . they propose to filter the training data to reduce entity hallucination problem . |
| Outcome: | The proposed model can reduce the entity hallucination problem by filtering the training data. |
Does Your Model Classify Entities Reasonably? Diagnosing and Mitigating Spurious Correlations in Entity Typing (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing entity typing models are subject to spurious correlations due to shortcuts and biased training. |
| Approach: | They propose a method to augment existing model biases by combining spurious correlations with debiasedcounterparts to improve generalization. |
| Outcome: | The proposed method improves generalization of different entity typing models on the original and debiased test sets. |
Evaluating Factuality in Text Simplification (2022.acl-long)
Copied to clipboard
| Challenge: | Automated simplification models aim to make input texts more readable without altering their meaning. |
| Approach: | They propose a taxonomy of errors that are used to analyze simplification models . they propose to use simplification methods to make input texts more readable . |
| Outcome: | The proposed models introduce errors that are not captured by existing evaluation metrics. |
NonFactS: NonFactual Summary Generation for Factuality Evaluation in Document Summarization (2023.findings-acl)
Copied to clipboard
| Challenge: | Pre-trained abstractive summarization models generate fluent summaries that are inconsistent with context document and contain nonfactual information. |
| Approach: | They propose a data generation model that synthesizes nonfactual summaries using human annotations. |
| Outcome: | The proposed model can generate nonfactual summaries and generalize to out-of-domain documents. |
Annotating and Modeling Fine-grained Factuality in Summarization (2021.naacl-main)
Copied to clipboard
| Challenge: | Recent abstractive summarization systems produce factual errors that are not faithful to the input . current methods are lacking in identifying what errors are most important to target . |
| Approach: | They use synthetic and human-labeled data to identify factual errors in summarization and train models on the factuality detection task. |
| Outcome: | The proposed model detects factual errors on word, dependency, and sentence levels. |
Merging Facts, Crafting Fallacies: Evaluating the Contradictory Nature of Aggregated Factual Claims in Long-Form Generations (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing factuality metrics cannot evaluate paragraphs with ambiguous entities, authors show . |
| Approach: | They propose a new metric to evaluate the factuality of long-form generations from large language models. |
| Outcome: | The proposed metric can assess the factuality of people biographies with entity ambiguity better than FActScore. |
Factuality of Large Language Models: A Survey (2024.emnlp-main)
Copied to clipboard
Yuxia Wang, Minghan Wang, Muhammad Arslan Manzoor, Fei Liu, Georgi Georgiev, Rocktim Das, Preslav Nakov
| Challenge: | Large language models (LLMs) are factually incorrect, which limits their applicability in real-world scenarios. |
| Approach: | They analyze existing work to identify major challenges and their associated causes . they propose to evaluate LLMs using a variety of measures to mitigate factual errors . |
| Outcome: | The proposed methods are based on a variety of datasets and proposed strategies to mitigate factual errors. |
Understanding Factual Errors in Summarization: Errors, Summarizers, Datasets, Error Detectors (2023.acl-long)
Copied to clipboard
Liyan Tang, Tanya Goyal, Alex Fabbri, Philippe Laban, Jiacheng Xu, Semih Yavuz, Wojciech Kryscinski, Justin Rousseau, Greg Durrett
| Challenge: | Abstractive summarization systems still include factual errors in generated summaries despite recent improvements in factuality detection . |
| Approach: | They aggregate factuality error annotations from nine existing datasets and stratify them according to the underlying summarization model. |
| Outcome: | The proposed method improves on the ChatGPT-based model and shows that it is not superior for all error types. |
Understanding Factuality in Abstractive Summarization with FRANK: A Benchmark for Factuality Metrics (2021.naacl-main)
Copied to clipboard
| Challenge: | Modern summarization models generate fluent but often factually unreliable outputs. |
| Approach: | They propose to use human annotations to identify different categories of factual errors and benchmark factuality metrics to improve summarization evaluation. |
| Outcome: | The proposed method identifies the proportion of different categories of factual errors and benchmarks their human judgements as well as their specific strengths and weaknesses. |
FactKB: Generalizable Factuality Evaluation using Language Models Enhanced with Factual Knowledge (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing factuality evaluation models are not robust, especially with respect to entity and relation errors in new domains. |
| Approach: | They propose a new approach to factuality evaluation that is generalizable across domains . they propose entities-specific facts, facts extracted from external knowledge bases and facts constructed compositionally through knowledge base walks. |
| Outcome: | The proposed model achieves state-of-the-art on two in-domain news summarization benchmarks and on three out-of domain scientific literature datasets. |