QEVA: A Reference-Free Evaluation Metric for Narrative Video Summarization with Multimodal Question Answering (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing video-to-text summarization evaluation methods depend heavily on human-written reference summaries. |
| Approach: | They propose a reference-free metric evaluating candidate summaries directly against source videos through multimodal question answering. |
| Outcome: | The proposed metric assesses candidate summaries directly against source videos through multimodal question answering. |
Similar Papers
Measuring What Matters Beyond Text: Evaluating Multimodal Summaries by Quality, Alignment, and Diversity (2026.findings-acl)
Copied to clipboard
| Challenge: | MLLMs have facilitated multimodal summarization with multimodal outputs, but their evaluation is fragmented . MM-Eval integrates assessments of textual quality, cross-modal alignment, and visual diversity . |
| Approach: | They propose a unified evaluation framework that integrates assessments of textual quality, cross-modal alignment, and visual diversity. |
| Outcome: | The proposed framework improves over heuristic aggregation baselines and provides an interpretable, reference-weak framework for comparative evaluation of multimodal summaries. |
Automatic, Meta and Human Evaluation for Multimodal Summarization with Multimodal Output (2024.naacl-long)
Copied to clipboard
| Challenge: | Multimodal summarization with multimodal output (MSMO) has attracted increasing research interest . evaluation is an emerging yet underexplored research topic . |
| Approach: | They propose a framework that studies three research questions of MSMO evaluation . they propose an automatic evaluation metric and a meta-evaluation benchmark dataset . |
| Outcome: | The proposed evaluation metric and human-annotated meta-evaluation benchmark are used to assess the quality of evaluation metrics and show the framework is effective. |
Towards Question-Answering as an Automatic Metric for Evaluating the Content Quality of a Summary (2021.tacl-1)
Copied to clipboard
| Challenge: | Existing text overlap based evaluation metrics are limited to matching tokens, either lexically or via embeddings. |
| Approach: | They propose a metric to evaluate the content quality of a summary using question-answering (QA) QA-based methods directly measure a summary’s information overlap with a reference, making them fundamentally different from text overlap metrics. |
| Outcome: | The proposed metric outperforms current state-of-the-art metrics on most evaluations using benchmark datasets while being competitive on others due to limitations of state- of-the art models. |
QuestEval: Summarization Asks for Fact-based Evaluation (2021.emnlp-main)
Copied to clipboard
Thomas Scialom, Paul-Alexis Dray, Sylvain Lamprier, Benjamin Piwowarski, Jacopo Staiano, Alex Wang, Patrick Gallinari
| Challenge: | Existing evaluation metrics for summarization evaluation are limited and do not correlate well with human judgments. |
| Approach: | They propose to extend existing evaluation metrics to include question answering models to assess whether a summary contains all relevant information in its source document. |
| Outcome: | The proposed framework significantly improves the correlation with human judgments over four evaluation dimensions. |
What Is That Talk About? A Video-to-Text Summarization Dataset for Scientific Presentations (2025.acl-long)
Copied to clipboard
Dongqi Liu, Chenxi Whitehouse, Xi Yu, Louis Mahon, Rohit Saxena, Zheng Zhao, Yifu Qiu, Mirella Lapata, Vera Demberg
| Challenge: | VISTA dataset contains 18,599 recorded AI conference presentations . large multimodal models exhibit reduced performance in scientific contexts, study shows . |
| Approach: | They propose a dataset specifically designed for video-to-text summarization in scientific domains. |
| Outcome: | This paper compares the performance of large models with human models and shows that they improve on human models. |
SummEval: Re-evaluating Summarization Evaluation (2021.tacl-1)
Copied to clipboard
Alexander R. Fabbri, Wojciech Kryściński, Bryan McCann, Caiming Xiong, Richard Socher, Dragomir Radev
| Challenge: | a lack of comprehensive studies on evaluation metrics for text summarization hinders progress . a new study aims to improve evaluation metrics that correlate with human judgments . |
| Approach: | They propose to re-evaluate automatic evaluation metrics and share a toolkit for evaluation . they hope to promote a more complete evaluation protocol for text summarization . |
| Outcome: | The proposed evaluation metrics are inconsistent with existing evaluation protocols. |
MM-AVS: A Full-Scale Dataset for Multi-modal Summarization (2021.naacl-main)
Copied to clipboard
| Challenge: | Multimodal summarization materials lacking a holistic organization by integrating resources from various modalities. |
| Approach: | They propose a multimodal article and video summarization dataset that integrates resources from different modalities. |
| Outcome: | The proposed dataset validates the important assistance role of external information for multimodal summarization. |
MSMO: Multimodal Summarization with Multimodal Output (D18-1)
Copied to clipboard
| Challenge: | Existing studies show that multimodal summarization can improve user satisfaction for informativeness of summaries by using information in visual modality. |
| Approach: | They propose a task to generate text and select the most relevant image from the multimodal input and a novel multimodal automatic evaluation method to evaluate multimodal outputs. |
| Outcome: | The proposed method improves user satisfaction by 12.4% compared to the current system . |
Detecting and Mitigating Challenges in Zero-Shot Video Summarization with Video LLMs (2025.findings-acl)
Copied to clipboard
Luca Cagliero, Lorenzo Vaiani, Eliana Pastor, Alkis Koudounas, Elena Baralis, Vittorio Mazzia, Sandro Pollastrini, Thomas Gueudre, Manuel Giollo, Daniele Amberti, Yue Wu
| Challenge: | Video Large Language Models (VLLMs) exhibit impressive zero-shot capabilities in video analysis, but their performance varies significantly depending on the LLM prompt, the characteristics of the video, and the properties of the training data and LLM architecture. |
| Approach: | They propose to use Chain-of-Thought prompting to inject knowledge extracted by external, lightweight models into video summarization benchmarks to evaluate their performance. |
| Outcome: | The proposed solutions improve summarization performance by injecting knowledge extracted by external, lightweight models. |
QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarization (2022.naacl-main)
Copied to clipboard
| Challenge: | Existing studies on text summarization factual consistency are divided into two categories . entailment-based and question answering-based metrics are the most efficient . |
| Approach: | They propose an optimized QA-based metric that improves factual consistency by 14% . they compare entailment-based and QA metrics to find the best fit . |
| Outcome: | The proposed metric outperforms the best performing entailment-based metric on the SummaC factual consistency benchmark. |