| Challenge: | Existing models for video dense captioning learn video segments and generate captions without considering transcripts. |
| Approach: | They propose a model to generate procedure captions from narrated instructional videos . they extract procedures by a cross-modality module and generate captions by encoding video frames and transcripts within each extracted procedure. |
| Outcome: | The proposed model can extract procedures from narrated instructional videos and generate procedure captions by encoding video frames and transcripts. |
Similar Papers
End-to-end Dense Video Captioning as Sequence Generation (2022.coling-1)
Copied to clipboard
| Challenge: | Existing methods for dense video captioning use a two-stage generative process . but, more complex tasks are not able to fully utilize this powerful paradigm . |
| Approach: | They propose to model two subtasks of dense video captioning as one sequence generation task and predict the events and the corresponding descriptions. |
| Outcome: | Experiments on YouCook2 and ViTT show that the proposed model can be used on any video platform. |
DeCEMBERT: Learning from Noisy Instructional Videos via Dense Captions and Entropy Minimization (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing methods to train models on unlabeled web videos are noisy and temporally misaligned . authors propose a method that adds captions and constrained attention loss to improve performance . |
| Approach: | They propose a method that adds captions from video frames as auxiliary text input to provide visual cues for learning better video and language associations. |
| Outcome: | The proposed method outperforms state-of-the-art methods on video-and-language tasks . it adds captions and constrained attention loss to improve model performance . |
Multimodal Pretraining for Dense Video Captioning (2020.aacl-main)
Copied to clipboard
| Challenge: | a billion hours of videos are being watched on YouTube every day . videos are difficult to skim through, making it harder to quickly target the relevant part(s) of a video. |
| Approach: | They propose to use a video timeline tag dataset to generate time-stamped annotations for videos . they propose to pretrain and finetune captioning models using YouCook2 and ViTT . |
| Outcome: | The proposed model generalizes well and is robust over a wide variety of instructional videos. |
Multimodal Intent Discovery from Livestream Videos (2022.findings-naacl)
Copied to clipboard
Adyasha Maharana, Quan Tran, Franck Dernoncourt, Seunghyun Yoon, Trung Bui, Walter Chang, Mohit Bansal
| Challenge: | Existing models for instructional video understanding struggle to understand abstract intents . identifying procedural intent within instructional videos is a challenging task . |
| Approach: | They propose to extract instructional intent from software instructional livestreams by using a multimodal cascaded cross-attention model that integrates weaker and noisier video signals with more discriminative text signals. |
| Outcome: | The proposed model improves on baseline models and compares it to existing models. |
Retrieval-augmented Video Encoding for Instructional Captioning (2023.findings-acl)
Copied to clipboard
| Challenge: | Instructional videos provide a detailed multimodal context of each procedure in instruction. key-object degeneracy is a problem for machine systems, causing incorrect captions. |
| Approach: | They propose a retrieval-based framework to augment the model representations in the presence of key-object degeneracy. |
| Outcome: | The proposed framework can be extended over baselines using modalities with key-object degeneracy. |
TutorialVQA: Question Answering Dataset for Tutorial Videos (2020.lrec-1)
Copied to clipboard
| Challenge: | a new question answering task on instructional videos is needed due to their verbose nature . factoid questions are only a small part of what people actually want to ask on video contents . |
| Approach: | They propose a question answering task on instructional videos based on video transcripts . they use a dataset consisting of 6,000 manually collected triples of (video, question, answer span) |
| Outcome: | The proposed task focuses on screencast tutorial videos pertaining to an image editing program. |
Towards Fine-grained Audio Captioning with Multimodal Contextual Fusion (2026.acl-long)
Copied to clipboard
| Challenge: | Existing methods for audio captioning lack fine-grained detail and contextual accuracy due to limited unimodal or superficial information. |
| Approach: | They propose a two-stage automated pipeline that uses pretrained models to extract contextual cues from video . a large language model synthesizes these inputs to generate detailed and context-aware captions . |
| Outcome: | The proposed method is scalable and generates detailed and context-aware captions on large-scale audio datasets. |
Video Paragraph Captioning as a Text Summarization Task (2021.acl-short)
Copied to clipboard
| Challenge: | Existing methods for video paragraph captioning use ground-truth event segments. |
| Approach: | They propose a video paragraph captioning task that generates coherent paragraphs without ground-truth event segments. |
| Outcome: | The proposed framework outperforms existing methods on two popular datasets. |
Task-Specific Information Decomposition for End-to-End Dense Video Captioning (2025.acl-long)
Copied to clipboard
| Challenge: | Existing methods for label assignment fail to ensure relevance of localization information to descriptions. |
| Approach: | They propose a decomposed dense video captioning framework that derives localization and captioning queries from event queries, enabling task-specific representations while maintaining inter-task collaboration. |
| Outcome: | Experiments on YouCook2 and ActivityNet Captions show that the proposed framework achieves state-of-the-art performance. |
Sali4Vid: Saliency-Aware Video Reweighting and Adaptive Caption Retrieval for Dense Video Captioning (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent work proposes end-to-end models but suffer from limitations . prior work focused on generating captions from long video streams . |
| Approach: | They propose a saliency-aware framework that localizes events and generates captions for each event. |
| Outcome: | The proposed framework achieves state-of-the-art results on YouCook2 and ViTT. |