Jiaze Li, Yaya Shi, Zongyang Ma, Haoran Xu, Yandong.bai Yandong.bai, Huihui Xiao, Ruiwen Kang, Fan Yang, Tingting Gao, Di Zhang
| Challenge: | Recent advances in Video Large Language Models have led to rapid development, significantly enhancing the capture of overall video semantics and achieving remarkable performance in general video understanding tasks. |
| Approach: | They propose a large-scale instance-motion-aware video instruction-tuning dataset iMOVE that utilizes Event-awful Spatiotemporal Efficient Modeling to retain informative instance spatiotemporal motion details while maintaining computational efficiency. |
| Outcome: | The proposed model excels in video temporal understanding and general video understanding. |
Similar Papers
TUNA: Comprehensive Fine-grained Temporal Understanding Evaluation on Dense Dynamic Videos (2025.acl-long)
Copied to clipboard
Fanheng Kong, Jingyuan Zhang, Hongzhi Zhang, Shi Feng, Daling Wang, Linhao Yu, Xingguang Ji, Yu Tian, V. W., Fuzheng Zhang
| Challenge: | Existing benchmarks for video understanding often focus on specific aspects, overlooking the holistic nature of video content. |
| Approach: | They propose a temporal-oriented benchmark for fine-grained understanding on dense dynamic videos with two complementary tasks: captioning and QA. |
| Outcome: | The proposed model performs well on diverse video scenarios and dynamic videos, with interpretable and robust evaluation criteria. |
VideoINSTA: Zero-shot Long Video Understanding via Informative Spatial-Temporal Reasoning with LLMs (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Long video understanding presents unique challenges due to the complexity of reasoning over extended timespans. |
| Approach: | They propose a framework VideoINSTA to leverage large language models for video understanding . they propose 'event-based temporalreasoning' and 'content-based spatial reasoning' |
| Outcome: | The proposed model significantly improves state-of-the-art on three long video question-answering benchmarks. |
Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding (2023.emnlp-demo)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are capable of understanding multi-modal content, but textonly human-computer interaction is not sufficient for many application scenarios. |
| Approach: | They propose a video-to-text generation task and a multi-modal framework that bootstraps cross-modal training from frozen pre-trained visual & audio encoders and frozen LLMs. |
| Outcome: | The proposed framework can understand both visual and auditory content in video and generate meaningful responses grounded in the visual and audio information presented in the videos. |
VideoQA-TA: Temporal-Aware Multi-Modal Video Question Answering (2025.coling-main)
Copied to clipboard
| Challenge: | Existing methods for video question answering align visual or textual features directly with large language models, limiting the deep semantic association between modalities and hindering a comprehensive understanding of interactions within spatial and temporal contexts. |
| Approach: | They propose a temporal-aware framework for multi-modal video question answering that aligns videos and questions at fine-grained levels. |
| Outcome: | The proposed framework improves reasoning ability and accuracy of videoQA by aligning videos and questions at fine-grained levels. |
VideoPASTA: 7K Preference Pairs That Matter for Video-LLM Alignment (2025.emnlp-main)
Copied to clipboard
| Challenge: | Video-language models excel at understanding video content but struggle with spatial relationships, temporal ordering, and cross-frame continuity. |
| Approach: | They propose a framework that trains video-LLMs to distinguish accurate representations from carefully crafted adversarial examples. |
| Outcome: | Experiments show that VideoPASTA improves performance without human annotation or captioning . the framework can be used on various state-of-the-art video-LLMs with no human annotation . |
VIEWS: Entity-Aware News Video Captioning (2024.emnlp-main)
Copied to clipboard
Hammad Ayyubi, Tianqi Liu, Arsha Nagrani, Xudong Lin, Mingda Zhang, Anurag Arnab, Feng Han, Yukun Zhu, Xuande Feng, Kevin Zhang, Jialu Liu, Shih-Fu Chang
| Challenge: | Existing video captioning benchmarks and models produce generic captions for videos that lack specific identification of individuals, locations, or organizations. |
| Approach: | They propose a task of directly summarizing news videos into captions that are entity-aware . they validate the effectiveness of their approach across three video captioning models . |
| Outcome: | The proposed approach is effective across three video captioning models. |
TESTA: Temporal-Spatial Token Aggregation for Long-form Video-Language Understanding (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Experimental results show that TESTA reduces the number of visual tokens by 75% and thus accelerates video encoding. |
| Approach: | They propose a method to condense video semantics by aggregating similar frames and patches within each frame. |
| Outcome: | The proposed method reduces visual tokens by 75% and accelerates video encoding. |
Sali4Vid: Saliency-Aware Video Reweighting and Adaptive Caption Retrieval for Dense Video Captioning (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent work proposes end-to-end models but suffer from limitations . prior work focused on generating captions from long video streams . |
| Approach: | They propose a saliency-aware framework that localizes events and generates captions for each event. |
| Outcome: | The proposed framework achieves state-of-the-art results on YouCook2 and ViTT. |
HERMES: KV Cache as Hierarchical Memory for Efficient Streaming Video Understanding (2026.acl-long)
Copied to clipboard
| Challenge: | Existing models struggle to maintain stable understanding performance and low GPU memory overhead. |
| Approach: | They propose a training-free architecture for real-time and accurate understanding of video streams . HERMES reuses a compact KV cache, enabling efficient streaming understanding . |
| Outcome: | The proposed architecture achieves 10 faster TTFT compared to prior SOTA. |
See More, Store Less: Memory-Efficient Resolution for Video Moment Retrieval (2026.findings-eacl)
Copied to clipboard
| Challenge: | Existing video moment retrieval methods rely on sparse frame sampling, risking information loss. |
| Approach: | a new video-based framework enhances memory efficiency while maintaining high information resolution . SMORE uses query-guided captions to encode semantics aligned with user intent . |
| Outcome: | a new framework improves memory efficiency while maintaining high information resolution . it achieves state-of-the-art performance on QVHighlights, Charades-STA, and ActivityNet-Captions benchmarks . |