| Challenge: | Existing work on multimodal summarization does not consider the topic of the content. |
| Approach: | They propose a topic-aware MS system which performs two tasks simultaneously: differentiating images into "on-topic" and "off-topic". |
| Outcome: | The proposed system outperforms the state-of-the-art approach by 1.7 % in ROUGE-L metric. |
Similar Papers
MSMO: Multimodal Summarization with Multimodal Output (D18-1)
Copied to clipboard
| Challenge: | Existing studies show that multimodal summarization can improve user satisfaction for informativeness of summaries by using information in visual modality. |
| Approach: | They propose a task to generate text and select the most relevant image from the multimodal input and a novel multimodal automatic evaluation method to evaluate multimodal outputs. |
| Outcome: | The proposed method improves user satisfaction by 12.4% compared to the current system . |
Neural Multimodal Topic Modeling: A Comprehensive Evaluation (2024.lrec-main)
Copied to clipboard
| Challenge: | Neural topic models can find coherent and diverse topics in textual data, but they are limited in dealing with multimodal datasets. |
| Approach: | They propose two new topic modeling solutions and two new evaluation metrics for document multimodality. |
| Outcome: | The proposed models generate coherent and diverse topics on a rich dataset. |
Pay More Attention to Images: Numerous Images-Oriented Multimodal Summarization (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing multimodal summarization approaches struggle with scenarios involving multiple images as input. |
| Approach: | They propose a task to generate multimodal summaries by integrating multiple images as input . they propose 'multimodal information evaluation' method that measures differences between generated summary and input based on multimodal input - and compares various methods . |
| Outcome: | The proposed method correlates more closely with human judgments than five widely used metrics . |
Leveraging Entity Information for Cross-Modality Correlation Learning: The Entity-Guided Multimodal Summarization (2024.findings-acl)
Copied to clipboard
| Challenge: | Multimodal Summarization with Multimodal Output (MSMO) is a new approach to produce a multimodal summary that integrates both text and relevant images. |
| Approach: | They propose an Entity-Guided Multimodal Summarization model that integrates both text and relevant images to produce a multimodal summary. |
| Outcome: | The proposed model integrates text-image and entity-image information and refines image selection through knowledge distillation from a pre-trained vision-language model. |
Automatic, Meta and Human Evaluation for Multimodal Summarization with Multimodal Output (2024.naacl-long)
Copied to clipboard
| Challenge: | Multimodal summarization with multimodal output (MSMO) has attracted increasing research interest . evaluation is an emerging yet underexplored research topic . |
| Approach: | They propose a framework that studies three research questions of MSMO evaluation . they propose an automatic evaluation metric and a meta-evaluation benchmark dataset . |
| Outcome: | The proposed evaluation metric and human-annotated meta-evaluation benchmark are used to assess the quality of evaluation metrics and show the framework is effective. |
VMSMO: Learning to Generate Multimodal Summary for Video-based News Articles (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies show that multimodal news can significantly improve users' sense of satisfaction for informativeness. |
| Approach: | They propose a task of Video-based Multimodal Summarization with Multimodal Output to solve this problem. |
| Outcome: | The proposed method can generate multimodal summaries with a single input . it can model the temporal dependency of video with semantic meaning of article . |
Multimodal Sentence Summarization via Multimodal Selective Encoding (2020.coling-main)
Copied to clipboard
| Challenge: | Existing methods for generating summary from text and image ignore that the image can improve the ability of the encoder to identify highlights of a news event or document. |
| Approach: | They propose a multimodal selective gate network that takes reciprocal relationships between textual and multi-level visual features into account to select highlights of the event. |
| Outcome: | The proposed model can generate summary for a given sentence-image pair using visual signals . it can also capture highlights embedded in the image more accurately, the authors show . |
Topic-Guided Abstractive Multi-Document Summarization (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies on multi-document summarization (MDS) focus on extractive and abstractive approaches to create a fluent and concise summary for a collection of thematically related documents. |
| Approach: | They propose a novel abstractive MDS model that represents multiple documents as a heterogeneous graph and then applies a graph-to-sequence framework to generate summaries. |
| Outcome: | The proposed model outperforms state-of-the-art models on Rouge scores and human evaluation, while learning high-quality topics. |
A Modular Approach for Multimodal Summarization of TV Shows (2024.acl-long)
Copied to clipboard
| Challenge: | In this paper, we address the task of summarizing television shows, which touches key areas in AI research. |
| Approach: | They propose a modular approach where separate components perform specialized sub-tasks . they propose atomic facts to measure precision and recall of generated summaries . |
| Outcome: | The proposed method produces higher quality summaries than comparison models on a recently released dataset. |
SCCS: Semantics-Consistent Cross-domain Summarization via Optimal Transport Alignment (2023.findings-acl)
Copied to clipboard
Jielin Qiu, Jiacheng Zhu, Mengdi Xu, Franck Dernoncourt, Trung Bui, Zhaowen Wang, Bo Li, Ding Zhao, Hailin Jin
| Challenge: | Existing methods for multimodal summarization ignore the structure and semantics of the whole video and article. |
| Approach: | They propose a semantic-consistent cross-domain summarization model that extracts features from video and article and uses fusion methods to select representative one. |
| Outcome: | The proposed model produces high-quality multimodal summaries on three MSMO datasets. |