Papers by Alessandro Scirè
FENICE: Factuality Evaluation of summarization based on Natural language Inference and Claim Extraction (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent advances in text summarization have shown remarkable performance, but a significant number of summaries exhibit factual inconsistencies, such as hallucinations. |
| Approach: | They propose a factuality-oriented metric that evaluates text summarization for accuracy . they use a human annotation process to examine the accuracy of automatically generated summaries . |
| Outcome: | The proposed metric sets a new state-of-the-art on AGGREFACT, the de-facto benchmark for factuality evaluation. |
Semantic Role Labeling Meets Definition Modeling: Using Natural Language to Describe Predicate-Argument Structures (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to Semantic Role Labeling rely on discrete labels to classify predicate senses and their arguments. |
| Approach: | They propose a generalized formulation of Semantic Role Labeling that leverages Definition Modeling to describe predicate-argument structures using natural language definitions instead of discrete labels. |
| Outcome: | The proposed model can describe predicate-argument structures using natural language definitions instead of discrete labels. |
Right Answer, Wrong Score: Uncovering the Inconsistencies of LLM Evaluation in Multiple-Choice Question Answering (2025.findings-acl)
Copied to clipboard
| Challenge: | Multiple-choice question answering tasks are one of the most commonly used tasks for evaluating Large Language Models (LLMs). |
| Approach: | They analyze whether existing answer extraction methods are aligned with human judgment and how they are influenced by answer constraints in the prompt across different domains. |
| Outcome: | The proposed evaluation strategies can be inconsistent with human judgment, and can lead to inaccurate and misleading comparisons. |
Echoes from Alexandria: A Large Resource for Multilingual Book Summarization (2023.findings-acl)
Copied to clipboard
| Challenge: | Recent research in text summarization has focused on news stories, where texts are typically short and have strong layout features. |
| Approach: | They propose a resource for multilingual book summarization that uses a new extractive-then-abstractive baseline to compare the results. |
| Outcome: | The proposed resource is the largest and first to be multilingual, featuring 5 languages and 25 language pairs. |