Bingchan Zhao, Xinyi Liu, Zhuocheng Yu, Tongchen Yang, Yifan Song, Mingyu Jin, Sujian Li, Yizhou Wang
| Challenge: | Existing video captioning models fail to capture nuanced semantics of videos . existing models generate coarse descriptions of human motions, resulting in poor quality . |
| Approach: | They construct a fine-grained human motion video captioning dataset named BoFiT and a model that generates fine-grain descriptions of human motions via prompting. |
| Outcome: | The proposed model outperforms existing models on comprehensive metrics. |
Similar Papers
Motion Generation from Fine-grained Textual Descriptions (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing models for motion generation from textual descriptions are limited to coarse-grained descriptions. |
| Approach: | They build a large-scale language-motion dataset specializing in fine-grained textual descriptions . they feed it with step-by-step instructions with pseudo-code compulsory checks . quantitative evaluation shows that the model outperforms MotionDiffuse in generating spatially or chronologically composite motions . |
| Outcome: | The proposed model outperforms existing models in generating human motion sequences from textual descriptions by a large margin. |
Diving Deep into the Motion Representation of Video-Text Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing video-text models that capture motion in videos are lacking in quality. |
| Approach: | They propose a method to improve motion understanding in video-text models by utilizing motion descriptions to capture motion in action videos. |
| Outcome: | The proposed pipeline improves motion understanding on two action datasets and shows that it is effective on fine-grained motion descriptions. |
HAIC: Improving Human Action Understanding and Generation with Better Captions for Multi-modal Large Language Models (2025.acl-long)
Copied to clipboard
Xiao Wang, Jingyun Hua, Weihong Lin, Yuanxing Zhang, Fuzheng Zhang, Jianlong Wu, Di Zhang, Liqiang Nie
| Challenge: | Existing studies have shown that high-quality video captions can improve MLLMs' performance on videos involving human actions. |
| Approach: | They propose a data annotation pipeline to collect videos featuring clear human actions from the Internet and annotate them in a standardized caption format that uses human attributes to distinguish individuals. |
| Outcome: | The proposed pipeline combines two datasets to evaluate human action understanding. |
Finer: Investigating and Enhancing Fine-Grained Visual Concept Recognition in Large Vision Language Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in instruction-tuned Large Vision-Language Models (LVLMs) have imbued the models with the ability to generate high-level, image-grounded explanations with ease. |
| Approach: | They propose to use a multiple granularity attribute-centric benchmark and training mixture to evaluate LVLMs’ fine-grained visual comprehension ability. |
| Outcome: | The proposed model improves on LLaVa-1.5, InstructBLIP and GPT-4V and demonstrates that they struggle to generate descriptive visual attributes based on a concept that appears within an input image despite their prominent zero-shot image captioning ability. |
YouMakeup: A Large-Scale Domain-Specific Multimodal Dataset for Fine-Grained Semantic Comprehension (D19-1)
Copied to clipboard
| Challenge: | Multimodal semantic comprehension has attracted increasing research interest recently such as visual question answering and caption generation. |
| Approach: | They propose to use a large-scale multimodal instructional video dataset to support fine-grained comprehension research in specific domain. |
| Outcome: | The proposed dataset contains 2,800 videos from YouTube, spanning more than 420 hours in total. |
Towards Fine-Grained and Multi-Granular Contrastive Language-Speech Pre-training (2026.acl-long)
Copied to clipboard
Yifan Yang, Bing Han, Hui Wang, Wei Wang, Ziyang Ma, Long Zhou, Zengrui Jin, Guanrou Yang, Tianrui Wang, Xu Tan, Xie Chen
| Challenge: | Existing models for fine-grained speaking styles are limited in terms of accuracy, coverage, and naturalness. |
| Approach: | They propose a model that pre-trains with coarse captions and annotates with a pipeline that grounds captions in audio. |
| Outcome: | The proposed model outperforms existing models with fine-grained style annotations . it integrates global and fine-granular supervision, enabling unified representations based on the proposed model . |
GenProve: Learning to Generate Text with Fine-Grained Provenance (2026.acl-long)
Copied to clipboard
| Challenge: | Existing methods for large language models (LLMs) are coarse-grained and fail to distinguish between direct quotes and complex reasoning. |
| Approach: | They propose a framework that combines supervised fine-tuning and group relative policy optimization to generate fluent answers while simultaneously producing sentence-level provenance triples. |
| Outcome: | The proposed framework outperforms 14 strong large language models in joint evaluation. |
Multimodal Pretraining for Dense Video Captioning (2020.aacl-main)
Copied to clipboard
| Challenge: | a billion hours of videos are being watched on YouTube every day . videos are difficult to skim through, making it harder to quickly target the relevant part(s) of a video. |
| Approach: | They propose to use a video timeline tag dataset to generate time-stamped annotations for videos . they propose to pretrain and finetune captioning models using YouCook2 and ViTT . |
| Outcome: | The proposed model generalizes well and is robust over a wide variety of instructional videos. |
How Much Do Large Language Models Know about Human Motion? A Case Study in 3D Avatar Control (2025.findings-emnlp)
Copied to clipboard
| Challenge: | a new study explores the human motion knowledge of Large Language Models (LLMs) using 3D avatar control. |
| Approach: | They use 20 representative motion instructions to interpolate LLMs into avatar animations . they find they are strong at interpreting high-level body movements but struggle with precise body part positioning . |
| Outcome: | The proposed model is strong at interpreting high-level body movements but struggles with precise body part positioning. |
Video Caption Dataset for Describing Human Actions in Japanese (2020.lrec-1)
Copied to clipboard
| Challenge: | Existing video caption datasets for English have no equivalent for Japanese . authors evaluated two methods to obtain benchmark results . |
| Approach: | They propose to use Japanese video captions to describe human actions . they evaluated two different methods to obtain benchmark results . |
| Outcome: | The proposed dataset evaluates two different methods to obtain benchmark results . it shows that the generation methods can specify "who does what and where" |