A Challenging Multimodal Video Summary: Simultaneously Extracting and Generating Keyframe-Caption Pairs from Video (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to summarize video content have only considered video and image data, and the trend towards multimodal video summarization is changing. |
| Approach: | They propose a multimodal video summarization task setting and a dataset to train and evaluate the task. |
| Outcome: | The proposed task is useful as a practical application and presents a highly challenging problem worthy of study. |
Similar Papers
VMSMO: Learning to Generate Multimodal Summary for Video-based News Articles (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies show that multimodal news can significantly improve users' sense of satisfaction for informativeness. |
| Approach: | They propose a task of Video-based Multimodal Summarization with Multimodal Output to solve this problem. |
| Outcome: | The proposed method can generate multimodal summaries with a single input . it can model the temporal dependency of video with semantic meaning of article . |
MSMO: Multimodal Summarization with Multimodal Output (D18-1)
Copied to clipboard
| Challenge: | Existing studies show that multimodal summarization can improve user satisfaction for informativeness of summaries by using information in visual modality. |
| Approach: | They propose a task to generate text and select the most relevant image from the multimodal input and a novel multimodal automatic evaluation method to evaluate multimodal outputs. |
| Outcome: | The proposed method improves user satisfaction by 12.4% compared to the current system . |
Multimodal Pretraining for Dense Video Captioning (2020.aacl-main)
Copied to clipboard
| Challenge: | a billion hours of videos are being watched on YouTube every day . videos are difficult to skim through, making it harder to quickly target the relevant part(s) of a video. |
| Approach: | They propose to use a video timeline tag dataset to generate time-stamped annotations for videos . they propose to pretrain and finetune captioning models using YouCook2 and ViTT . |
| Outcome: | The proposed model generalizes well and is robust over a wide variety of instructional videos. |
Abstractive Multi-Video Captioning: Benchmark Dataset Construction and Extensive Evaluation (2024.lrec-main)
Copied to clipboard
| Challenge: | Abstractive multi-video captioning focuses on abstracting multiple videos with natural language. |
| Approach: | They propose a task that generates an abstract caption of shared video content . they propose end-to-end and cascade approaches to abstractive multi-video captioning . |
| Outcome: | The proposed task generates an abstract caption of shared content in a video group containing multiple videos. |
Exploiting Pseudo Image Captions for Multimodal Summarization (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to multimodal summarization with multimodal output (MSMO) lack reference images for training, and exposure of image captions during training is inconsistent with MSMO’s task settings. |
| Approach: | They propose a coarse-to-fine image-text alignment mechanism to identify the most relevant sentence of each image in a document, resembling the role of image captions in capturing visual knowledge. |
| Outcome: | The proposed method sets up state-of-the-art on all intermodality and intramodality metrics and improves on image recommendation precision. |
MM-AVS: A Full-Scale Dataset for Multi-modal Summarization (2021.naacl-main)
Copied to clipboard
| Challenge: | Multimodal summarization materials lacking a holistic organization by integrating resources from various modalities. |
| Approach: | They propose a multimodal article and video summarization dataset that integrates resources from different modalities. |
| Outcome: | The proposed dataset validates the important assistance role of external information for multimodal summarization. |
MLASK: Multimodal Summarization of Video-based News Articles (2023.findings-eacl)
Copied to clipboard
| Challenge: | Recent studies on multimodal summarization have shown that the benefits of pre-training and using additional modalities in the input are not orthogonal. |
| Approach: | They propose to use a dataset to train a multimodal article summarization model by automatically crawling several news websites. |
| Outcome: | The proposed dataset can be used to model multimodal summarization by training a Transformer-based neural model. |
Hierarchical3D Adapters for Long Video-to-text Summarization (2023.findings-eacl)
Copied to clipboard
| Challenge: | a recent study shows that multimodal summarization is not efficient for long inputs and outputs. |
| Approach: | They extend a TV episode transcript summarization dataset and create a multimodal variant by collecting full-length videos. |
| Outcome: | The proposed model can be tuned to perform multimodal summarization tasks efficiently using adapter modules augmented with a hierarchical structure while tuning only 3.8% of model parameters. |
Assist Non-native Viewers: Multimodal Cross-Lingual Summarization for How2 Videos (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing multimodal summarization methods are limited to monolingual videos . a proposed task aims to generate cross-lingual summaries from multimodal inputs . |
| Approach: | They propose a task to generate cross-lingual summaries from multimodal inputs of videos . they propose fusion network that integrates multimodal and cross-linguistic information . |
| Outcome: | The proposed task outperforms existing methods on a reorganized How2 dataset on the reorganized How2 data set. |
Multimodal Sentence Summarization via Multimodal Selective Encoding (2020.coling-main)
Copied to clipboard
| Challenge: | Existing methods for generating summary from text and image ignore that the image can improve the ability of the encoder to identify highlights of a news event or document. |
| Approach: | They propose a multimodal selective gate network that takes reciprocal relationships between textual and multi-level visual features into account to select highlights of the event. |
| Outcome: | The proposed model can generate summary for a given sentence-image pair using visual signals . it can also capture highlights embedded in the image more accurately, the authors show . |