| Challenge: | Existing datasets focused on pre-selected human activities, whereas social media videos contain a great diversity of topics. |
| Approach: | They propose a large-scale dataset for video description as a new challenge for multi-sentence video description. |
| Outcome: | The proposed dataset contains 20k videos with 123k sentences, temporally aligned to the video. |
Similar Papers
Tell Me Again! a Large-Scale Dataset of Multiple Summaries for the Same Story (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing approaches to represent narratives on short-form texts are limited as narrative semantics are an open class. |
| Approach: | They propose to use Wikipedia summaries as a proxy for entire stories or for analysis of the summary itself. |
| Outcome: | The proposed dataset contains 96,831 individual summaries across 29,505 stories. |
A Challenging Multimodal Video Summary: Simultaneously Extracting and Generating Keyframe-Caption Pairs from Video (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to summarize video content have only considered video and image data, and the trend towards multimodal video summarization is changing. |
| Approach: | They propose a multimodal video summarization task setting and a dataset to train and evaluate the task. |
| Outcome: | The proposed task is useful as a practical application and presents a highly challenging problem worthy of study. |
Multilingual Synopses of Movie Narratives: A Dataset for Vision-Language Story Understanding (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Story video-text alignment is a core task in computational story understanding, but its progress has been held back by the scarcity of manually annotated video- text correspondences and the heavy concentration on English narrations of Hollywood movies. |
| Approach: | They construct a multilingual video story dataset with 13,166 movie summary videos from 7 languages and manual annotations of fine-grained video-text correspondences. |
| Outcome: | The proposed approach outperforms the SOTA methods on clip accuracy and Sentence IoU scores. |
From Long Videos to Engaging Clips: A Human-Inspired Video Editing Framework with Multimodal Narrative Understanding (2025.emnlp-industry)
Copied to clipboard
Xiangfeng Wang, Xiao Li, Yadong Wei, null Songxueyu, Yang Song, null Xiaxiaoqiang, Fangrui Zeng, Zaiyi Chen, null Liuliu, Gu Xu, Tong Xu
| Challenge: | Existing methods for video editing rely on textual cues from ASR transcripts and segment selection, often neglecting rich visual context. |
| Approach: | They propose a human-inspired automatic video editing framework that leverages multimodal narrative understanding to address these limitations. |
| Outcome: | The proposed framework outperforms existing baselines across general and advertisement-oriented editing tasks. |
A Video Is Worth 4096 Tokens: Verbalize Videos To Understand Them In Zero Shot (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing annotated training datasets hinder development of supervised learning models for multimedia content . lack of annotating benchmarks hinders development of models with satisfactory performance . a recent study shows that large language models have zero-shot performance in multimedia understanding . |
| Approach: | They propose to verbalize long videos to generate their descriptions in natural language . they then perform video-understanding tasks on the generated story as opposed to the original video . |
| Outcome: | The proposed method achieves better results than baselines for video understanding. |
BigVideo: A Large-scale Video Subtitle Translation Dataset for Multimodal Machine Translation (2023.findings-acl)
Copied to clipboard
Liyan Kang, Luyang Huang, Ningxin Peng, Peihao Zhu, Zewei Sun, Shanbo Cheng, Mingxuan Wang, Degen Huang, Jinsong Su
| Challenge: | Existing datasets focus on captions describing images or videos, which are not large and diverse enough. |
| Approach: | They propose a large-scale video subtitle translation dataset to facilitate multi-modality machine translation. |
| Outcome: | The proposed dataset is 10 times larger than the widely used *How2* and *VaTeX* datasets. |
Synchronized Video Storytelling: Generating Video Narrations with Structured Storyline (2024.acl-long)
Copied to clipboard
| Challenge: | Existing studies on dense video captioning and video story generation have made some progress, but in practical applications, we typically require synchronized narrations for ongoing visual scenes. |
| Approach: | They propose a task of Synchronized Video Storytelling to generate synchronized narrations for videos using a benchmark dataset with rich annotations. |
| Outcome: | The proposed framework can generate narrations with the guidance of the generated or predefined storyline and human evaluations validate the effectiveness. |
Multimodal Pretraining for Dense Video Captioning (2020.aacl-main)
Copied to clipboard
| Challenge: | a billion hours of videos are being watched on YouTube every day . videos are difficult to skim through, making it harder to quickly target the relevant part(s) of a video. |
| Approach: | They propose to use a video timeline tag dataset to generate time-stamped annotations for videos . they propose to pretrain and finetune captioning models using YouCook2 and ViTT . |
| Outcome: | The proposed model generalizes well and is robust over a wide variety of instructional videos. |
VIEWS: Entity-Aware News Video Captioning (2024.emnlp-main)
Copied to clipboard
Hammad Ayyubi, Tianqi Liu, Arsha Nagrani, Xudong Lin, Mingda Zhang, Anurag Arnab, Feng Han, Yukun Zhu, Xuande Feng, Kevin Zhang, Jialu Liu, Shih-Fu Chang
| Challenge: | Existing video captioning benchmarks and models produce generic captions for videos that lack specific identification of individuals, locations, or organizations. |
| Approach: | They propose a task of directly summarizing news videos into captions that are entity-aware . they validate the effectiveness of their approach across three video captioning models . |
| Outcome: | The proposed approach is effective across three video captioning models. |
MM-AVS: A Full-Scale Dataset for Multi-modal Summarization (2021.naacl-main)
Copied to clipboard
| Challenge: | Multimodal summarization materials lacking a holistic organization by integrating resources from various modalities. |
| Approach: | They propose a multimodal article and video summarization dataset that integrates resources from different modalities. |
| Outcome: | The proposed dataset validates the important assistance role of external information for multimodal summarization. |