Beyond the Last Frame: Process-aware Evaluation for Generative Video Reasoning (2026.acl-long)
Copied to clipboard
| Challenge: | Existing evaluation frameworks often rely on single-frame assessments, which can lead to outcome-hacking. |
| Approach: | They propose a process-aware evaluation paradigm that uses a hierarchical rubric to evaluate the validity of the intermediate steps and the final result. |
| Outcome: | The proposed model achieves POC@1.0 only about 20% and exhibits significant outcome-hacking. |
Similar Papers
Let’s Think Frame by Frame with VIP: A Video Infilling and Prediction Dataset for Evaluating Video Chain-of-Thought (2023.emnlp-main)
Copied to clipboard
Vaishnavi Himakunthala, Andy Ouyang, Daniel Rose, Ryan He, Alex Mei, Yujie Lu, Chinmay Sonar, Michael Saxon, William Wang
| Challenge: | Existing studies show vision-language systems can reason about images using natural language, but their capacity for video reasoning remains underexplored. |
| Approach: | They propose to frame video reasoning as the sequential understanding of a small number of keyframes, thereby leveraging the power and robustness of vision-language systems' capacity to reason about images using natural language. |
| Outcome: | The proposed models can generate multiple intermediate keyframes and predict future keyframe, and they perform poorly on GPT-4, GPT-3, and VICUNA. |
VChain: Chain-of-Visual-Thought for Reasoning in Video Generation (2026.findings-acl)
Copied to clipboard
| Challenge: | Recent video generation models struggle to synthesize complex dynamics with a coherent chain of consequences. |
| Approach: | They propose a framework that injects visual reasoning signals from multimodal models into video generation. |
| Outcome: | a new framework that leverages multimodal models to generate sparse keyframes significantly improves quality of generated videos. |
Beyond Outcome Verification: Verifiable Process Reward Models for Structured Reasoning (2026.findings-acl)
Copied to clipboard
| Challenge: | Recent work on reinforcement learning with verifiable rewards (RLVR) has shown that large language models can be substantially improved using outcome-level verification signals. |
| Approach: | They propose a framework where intermediate reasoning steps are checked by deterministic, rule-based verifiers. |
| Outcome: | The proposed framework achieves 20% higher F1 than state-of-the-art models and 6.5% higher than verifiable outcome rewards, with substantial gains in evidence grounding and logical coherence. |
Generative Frame Sampler for Long Video Understanding (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing video large language models (LMMs) employ an impedance of thousands of frames to understand long videos. |
| Approach: | They propose a plug-and-play module integrated with VideoLLMs to facilitate efficient lengthy video perception. |
| Outcome: | The proposed module boosts the performance of open-source VideoLLMs and proprietary assistants on long-form video benchmarks. |
TUNA: Comprehensive Fine-grained Temporal Understanding Evaluation on Dense Dynamic Videos (2025.acl-long)
Copied to clipboard
Fanheng Kong, Jingyuan Zhang, Hongzhi Zhang, Shi Feng, Daling Wang, Linhao Yu, Xingguang Ji, Yu Tian, V. W., Fuzheng Zhang
| Challenge: | Existing benchmarks for video understanding often focus on specific aspects, overlooking the holistic nature of video content. |
| Approach: | They propose a temporal-oriented benchmark for fine-grained understanding on dense dynamic videos with two complementary tasks: captioning and QA. |
| Outcome: | The proposed model performs well on diverse video scenarios and dynamic videos, with interpretable and robust evaluation criteria. |
TC-Bench: Benchmarking Temporal Compositionality in Conditional Video Generation (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing video generation models struggle to interpret compositional changes and synthesize components across different time steps. |
| Approach: | They propose a temporal compositionality benchmark that uses text prompts and ground truth videos to evaluate compositional changes in video. |
| Outcome: | The proposed benchmark can be used for text-to-video and image-to video generation. |
VideoPro: Adaptive Program Reasoning for Long Video Understanding (2026.acl-long)
Copied to clipboard
Chenglin Li, Feng Han, Yikun Wang, Ruilin Li, Shuai Dong, Haowen Hou, Haitao Li, Qianglong Chen, Feng Tao, Jingqi Tong, Yin Zhang, Jiaqi Wang
| Challenge: | Existing methods for understanding long videos are limited due to the sparsity of visual evidence relevant to a given query. |
| Approach: | They propose a framework that enables VideoLLMs to reason over long videos and refine their predictions through executable programs. |
| Outcome: | The proposed framework outperforms existing methods across long-video understanding benchmarks. |
StreamingEval: A Unified Evaluation Framework towards Realistic Streaming Video Understanding (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing research on streaming video understanding focuses on isolated aspects of visual understanding, but ignores practical deployability under realistic resource constraints. |
| Approach: | They propose a framework to evaluate streaming video understanding capabilities under realistic constraints. |
| Outcome: | StreamingEval benchmarks offline and online video models under a standardized protocol . it evaluates visual encoding efficiency, text decoding latency and task performance . |
FIFA: Unified Faithfulness Evaluation Framework for Text-to-Video and Video-to-Text Generation (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing evaluation methods for VideoMLLMs are limited to one task and fail to assess hallucinations in open-ended, free-form responses. |
| Approach: | They propose a unified framework that extracts comprehensive descriptive facts and models their semantic dependencies via a Spatio-Temporal Semantic Dependency Graph. |
| Outcome: | The proposed framework aligns more closely with human judgment than existing evaluation methods and improves factual consistency in both text and video generation. |
MAViS: A Multi-Agent Framework for Long-Sequence Video Storytelling (2026.eacl-long)
Copied to clipboard
| Challenge: | Experimental results show that MAViS achieves state-of-the-art performance in assistive capability, visual quality, and video expressiveness. |
| Approach: | They propose an end-to-end multi-agent collaborative framework for long-sequence video storytelling that orchestrates specialized agents across multiple stages. |
| Outcome: | The proposed framework achieves state-of-the-art performance in assistive capability, visual quality, and video expressiveness. |