Sali4Vid: Saliency-Aware Video Reweighting and Adaptive Caption Retrieval for Dense Video Captioning (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent work proposes end-to-end models but suffer from limitations . prior work focused on generating captions from long video streams . |
| Approach: | They propose a saliency-aware framework that localizes events and generates captions for each event. |
| Outcome: | The proposed framework achieves state-of-the-art results on YouCook2 and ViTT. |
Similar Papers
Task-Specific Information Decomposition for End-to-End Dense Video Captioning (2025.acl-long)
Copied to clipboard
| Challenge: | Existing methods for label assignment fail to ensure relevance of localization information to descriptions. |
| Approach: | They propose a decomposed dense video captioning framework that derives localization and captioning queries from event queries, enabling task-specific representations while maintaining inter-task collaboration. |
| Outcome: | Experiments on YouCook2 and ActivityNet Captions show that the proposed framework achieves state-of-the-art performance. |
End-to-end Dense Video Captioning as Sequence Generation (2022.coling-1)
Copied to clipboard
| Challenge: | Existing methods for dense video captioning use a two-stage generative process . but, more complex tasks are not able to fully utilize this powerful paradigm . |
| Approach: | They propose to model two subtasks of dense video captioning as one sequence generation task and predict the events and the corresponding descriptions. |
| Outcome: | Experiments on YouCook2 and ViTT show that the proposed model can be used on any video platform. |
Dense-Caption Matching and Frame-Selection Gating for Temporal Localization in VideoQA (2020.acl-main)
Copied to clipboard
| Challenge: | Recent years have witnessed a paradigm shift in the way we get our information, and a lot of it. |
| Approach: | They propose a video question answering model which integrates multi-modal input sources and finds temporally relevant information to answer questions. |
| Outcome: | The proposed model outperforms the state-of-the-art on a TVQA dataset. |
Hierarchical Context-aware Network for Dense Video Event Captioning (2021.acl-long)
Copied to clipboard
| Challenge: | Existing technologies for dense video event captioning generate fine-grained captions for all events in a long untrimmed video. |
| Approach: | They propose a hierarchical context-aware network for dense video event captioning to capture context from various aspects. |
| Outcome: | The proposed model outperforms the existing model on youcook2 and activitynet . it generates coherent captions for events in a long untrimmed video . |
VIEWS: Entity-Aware News Video Captioning (2024.emnlp-main)
Copied to clipboard
Hammad Ayyubi, Tianqi Liu, Arsha Nagrani, Xudong Lin, Mingda Zhang, Anurag Arnab, Feng Han, Yukun Zhu, Xuande Feng, Kevin Zhang, Jialu Liu, Shih-Fu Chang
| Challenge: | Existing video captioning benchmarks and models produce generic captions for videos that lack specific identification of individuals, locations, or organizations. |
| Approach: | They propose a task of directly summarizing news videos into captions that are entity-aware . they validate the effectiveness of their approach across three video captioning models . |
| Outcome: | The proposed approach is effective across three video captioning models. |
Multimodal Pretraining for Dense Video Captioning (2020.aacl-main)
Copied to clipboard
| Challenge: | a billion hours of videos are being watched on YouTube every day . videos are difficult to skim through, making it harder to quickly target the relevant part(s) of a video. |
| Approach: | They propose to use a video timeline tag dataset to generate time-stamped annotations for videos . they propose to pretrain and finetune captioning models using YouCook2 and ViTT . |
| Outcome: | The proposed model generalizes well and is robust over a wide variety of instructional videos. |
ViLL-E: Video LLM Embeddings for Retrieval (2026.acl-long)
Copied to clipboard
| Challenge: | Video Large Language Models excel at video understanding tasks where outputs are textual . however, they underperform specialized embedding-based models in Retrieval tasks . |
| Approach: | They propose a video-LLM-based model with an embedding generation mechanism that allows the model to "think longer" for complex videos and stop early for easy ones. |
| Outcome: | The proposed model outperforms specialized embedding-based models in video understanding tasks while remaining competitive on VideoQA tasks. |
DeCEMBERT: Learning from Noisy Instructional Videos via Dense Captions and Entropy Minimization (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing methods to train models on unlabeled web videos are noisy and temporally misaligned . authors propose a method that adds captions and constrained attention loss to improve performance . |
| Approach: | They propose a method that adds captions from video frames as auxiliary text input to provide visual cues for learning better video and language associations. |
| Outcome: | The proposed method outperforms state-of-the-art methods on video-and-language tasks . it adds captions and constrained attention loss to improve model performance . |
See More, Store Less: Memory-Efficient Resolution for Video Moment Retrieval (2026.findings-eacl)
Copied to clipboard
| Challenge: | Existing video moment retrieval methods rely on sparse frame sampling, risking information loss. |
| Approach: | a new video-based framework enhances memory efficiency while maintaining high information resolution . SMORE uses query-guided captions to encode semantics aligned with user intent . |
| Outcome: | a new framework improves memory efficiency while maintaining high information resolution . it achieves state-of-the-art performance on QVHighlights, Charades-STA, and ActivityNet-Captions benchmarks . |
The Devil is in the Distributions: Explicit Modeling of Scene Content is Key in Zero-Shot Video Captioning (2026.findings-eacl)
Copied to clipboard
| Challenge: | Existing methods for zero-shot video captioning focus on one key aspect of the scene and ignore the rest of the visual input. |
| Approach: | They propose a novel textual prompting strategy for zero-shot video captioning that uses a category-aware retrieval mechanism to promote prompt diversity while ensuring visual relevance. |
| Outcome: | The proposed method outperforms existing methods on in-domain and cross-domain settings. |