MULSUM: A Multimodal Summarization System with Vis-Aligner and Diversity-Aware Image Selection (2026.eacl-long)
Copied to clipboard
| Challenge: | Existing systems that condense text and images into concise, faithful digests are inefficient and require large fusion transformers. |
| Approach: | They propose a framework that uses image embeddings to generate a visually informed text summary and a Diversity-Aware Image Selector to maximize images-relevance to the summary. |
| Outcome: | The proposed framework outperforms baselines on automatic metrics such as ROUGE and human evaluation shows that selected images act as explanatory evidence rather than ornamental add-ons. |
Similar Papers
Exploiting Pseudo Image Captions for Multimodal Summarization (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to multimodal summarization with multimodal output (MSMO) lack reference images for training, and exposure of image captions during training is inconsistent with MSMO’s task settings. |
| Approach: | They propose a coarse-to-fine image-text alignment mechanism to identify the most relevant sentence of each image in a document, resembling the role of image captions in capturing visual knowledge. |
| Outcome: | The proposed method sets up state-of-the-art on all intermodality and intramodality metrics and improves on image recommendation precision. |
Measuring What Matters Beyond Text: Evaluating Multimodal Summaries by Quality, Alignment, and Diversity (2026.findings-acl)
Copied to clipboard
| Challenge: | MLLMs have facilitated multimodal summarization with multimodal outputs, but their evaluation is fragmented . MM-Eval integrates assessments of textual quality, cross-modal alignment, and visual diversity . |
| Approach: | They propose a unified evaluation framework that integrates assessments of textual quality, cross-modal alignment, and visual diversity. |
| Outcome: | The proposed framework improves over heuristic aggregation baselines and provides an interpretable, reference-weak framework for comparative evaluation of multimodal summaries. |
Towards Visually Grounded Multimodal Summarization via Cross-Modal Transformer and Gated Attention (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for multimodal summarization often inject shallow visual features into deep models, leading to representational mismatches and weak cross-modal grounding. |
| Approach: | They propose a framework that performs text summarization and representative image selection . a deep visual processor aligns the visual encoder with the language model at corresponding depths . |
| Outcome: | The proposed framework produces more accurate, visually grounded summaries and selects more representative images. |
MSMO: Multimodal Summarization with Multimodal Output (D18-1)
Copied to clipboard
| Challenge: | Existing studies show that multimodal summarization can improve user satisfaction for informativeness of summaries by using information in visual modality. |
| Approach: | They propose a task to generate text and select the most relevant image from the multimodal input and a novel multimodal automatic evaluation method to evaluate multimodal outputs. |
| Outcome: | The proposed method improves user satisfaction by 12.4% compared to the current system . |
mRedditSum: A Multimodal Abstractive Summarization Dataset of Reddit Threads with Images (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing summarization datasets do not cover multimodal discussions, multiple modalities, or both . mRedditSum consists of 3,033 discussion threads and images with human-written summaries. |
| Approach: | They propose a multimodal discussion summarization dataset that annotates 3,033 discussion threads with a human-written summary. |
| Outcome: | The proposed method outperforms existing models and serves as competitive baseline for future work. |
REFINESUMM: Self-Refining MLLM for Generating a Multimodal Summarization Dataset (2024.acl-long)
Copied to clipboard
| Challenge: | generating accurate and faithful multimodal summaries is challenging due to lack of appropriate multimodal datasets . large language models excel at synthesizing key information from diverse sources, but lack of adequate multimodal data sets for fine-tuning . |
| Approach: | They propose a dataset specifically designed for image-text multimodal summarization . they generate summaries from Wikipedia sections and corresponding images and evaluate them . |
| Outcome: | The proposed dataset improves summary quality by training a critic model on human annotations and using its predictions to remove low-quality summaries. |
Multimodal Sentence Summarization via Multimodal Selective Encoding (2020.coling-main)
Copied to clipboard
| Challenge: | Existing methods for generating summary from text and image ignore that the image can improve the ability of the encoder to identify highlights of a news event or document. |
| Approach: | They propose a multimodal selective gate network that takes reciprocal relationships between textual and multi-level visual features into account to select highlights of the event. |
| Outcome: | The proposed model can generate summary for a given sentence-image pair using visual signals . it can also capture highlights embedded in the image more accurately, the authors show . |
MM-AVS: A Full-Scale Dataset for Multi-modal Summarization (2021.naacl-main)
Copied to clipboard
| Challenge: | Multimodal summarization materials lacking a holistic organization by integrating resources from various modalities. |
| Approach: | They propose a multimodal article and video summarization dataset that integrates resources from different modalities. |
| Outcome: | The proposed dataset validates the important assistance role of external information for multimodal summarization. |
Leveraging Entity Information for Cross-Modality Correlation Learning: The Entity-Guided Multimodal Summarization (2024.findings-acl)
Copied to clipboard
| Challenge: | Multimodal Summarization with Multimodal Output (MSMO) is a new approach to produce a multimodal summary that integrates both text and relevant images. |
| Approach: | They propose an Entity-Guided Multimodal Summarization model that integrates both text and relevant images to produce a multimodal summary. |
| Outcome: | The proposed model integrates text-image and entity-image information and refines image selection through knowledge distillation from a pre-trained vision-language model. |
Enhancing Large Language Models for Scientific Multimodal Summarization with Multimodal Output (2025.coling-industry)
Copied to clipboard
| Challenge: | Scientific publications are becoming more multimedia, containing both text and visual content. |
| Approach: | They propose a framework for Scientific Multimodal Summarization with Multimodal Output . it leverages the power of large language models and extends its capability to cross-modal understanding . |
| Outcome: | The proposed framework outperforms uni- and multi-modality methods on two new datasets . it leverages the power of large language models and extends its capability to cross-modal understanding . |