CapArena: Benchmarking and Analyzing Detailed Image Captioning in the LLM Era (2025.findings-acl)
Copied to clipboard
Kanzhi Cheng, Wenpo Song, Jiaxin Fan, Zheng Ma, Qiushi Sun, Fangzhi Xu, Chenyang Yan, Nuo Chen, Jianbing Zhang, Jiajun Chen
| Challenge: | Image captioning has been a challenge for vision-language researchers for decades . current VLMs focus on tasks like visual question answering (YA) but image captioning is not as advanced as expected. |
| Approach: | They evaluate VLMs' performance on image captioning using human annotations . they find that some metrics show high caption-level agreement with humans . |
| Outcome: | The proposed model outperforms open-source models on image captioning . it achieves 93.4% correlation with human rankings at $4 per test . |
Similar Papers
VELA: An LLM-Hybrid-as-a-Judge Approach for Evaluating Long Image Captions (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing evaluation metrics for image captioning are primarily designed for short captions and are not suitable for long captions. |
| Approach: | They propose an automatic evaluation metric for long captions developed within a novel LLM-Hybrid-as-a-Judge framework. |
| Outcome: | The proposed metric outperforms existing metrics and achieves superhuman performance on LongCap-Arena. |
CLAIR: Evaluating Image Captions with Large Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing measures for image caption evaluation fail to capture dimensions of similarity . a novel method that leverages the zero-shot language modeling capabilities of large language models (LLMs) demonstrates a stronger correlation with human judgments of caption quality compared to existing measures. |
| Approach: | They propose a method that leverages the zero-shot language modeling capabilities of large language models to evaluate captions. |
| Outcome: | The proposed method shows a stronger correlation with human judgments of caption quality compared to other measures. |
Putting Captions to the Test: Evaluating Video Caption Quality through Multiple-Choice Question Answering (2026.acl-long)
Copied to clipboard
| Challenge: | Existing metrics for video captioning are based on text-based comparisons with ground-truth references. |
| Approach: | They propose a reference-free benchmark that assesses video captions based on their utility . they will release the benchmark to facilitate reproducible research . |
| Outcome: | The proposed benchmark improves on human-verified, fine-grained questions . it correlates significantly better with human judgments than existing metrics . |
Bridging the Visual Gap: Fine-Tuning Multimodal Models with Knowledge-Adapted Captions (2025.naacl-long)
Copied to clipboard
| Challenge: | Recent work focuses on training vision-language models with long, detailed image captions, but small-scale VLMs struggle to balance the richness of these captions with the risk of hallucinations. |
| Approach: | They propose an evaluation framework that breaks down generated captions into individual propositions, assessing each in isolation. |
| Outcome: | The proposed framework outperforms baselines in both automatic metrics and human evaluations on small-scale vision-language models with long, detailed captions. |
LOTUS: A Leaderboard for Detailed Image Captioning from Quality to Societal Bias and User Preferences (2025.acl-industry)
Copied to clipboard
Yusuke Hirota, Boyi Li, Ryo Hachiuma, Yueh-Hua Wu, Boris Ivanovic, Marco Pavone, Yejin Choi, Yu-Chiang Frank Wang, Yuta Nakashima, Chao-Han Huck Yang
| Challenge: | Large Vision-Language Models (LVLMs) have transformed image captioning . existing evaluations lack standardized criteria and a standardized evaluation framework . |
| Approach: | They propose a leaderboard for evaluating detailed captions that addresses three main gaps in existing evaluations: lack of standardized criteria, bias-aware assessments, and user preference considerations. |
| Outcome: | The proposed model evaluates caption quality, descriptiveness, risks, and societal biases while tailoring criteria to user preferences. |
Evaluating Multimodal Large Language Models on Video Captioning via Monte Carlo Tree Search (2025.acl-long)
Copied to clipboard
Linhao Yu, Xingguang Ji, Yahui Liu, Fanheng Kong, Chenxi Sun, Jingyuan Zhang, Hongzhi Zhang, V. W., Fuzheng Zhang, Deyi Xiong
| Challenge: | Existing benchmarks and evaluation protocols suffer from inadequate or homogeneous creation of key points, exorbitant cost of data creation, and limited evaluation scopes. |
| Approach: | They propose an automatic framework which leverages Monte Carlo Tree Search to construct numerous and diverse descriptive sentences that thoroughly represent video content in an iterative way. |
| Outcome: | The proposed framework improves MCTS-VCB and DREAM-1K on video captioning tasks by 25.0% and 16.3% respectively. |
Enhancing Large Vision-Language Models with Ultra-Detailed Image Caption Generation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing pipelines for generating high-quality, ultra-detailed image captions are limited by the scarcity of image caption data. |
| Approach: | They propose a pipeline for generating high-quality, ultra-detailed image captions that integrates both pre-processing and post-processor stages. |
| Outcome: | The proposed pipeline improves LVLMs' perception and cognitive abilities across multiple vision-language benchmarks. |
VLURes: Benchmarking Long-Text Grounding and Cross-Lingual Robustness in Vision Language Models (2026.findings-acl)
Copied to clipboard
| Challenge: | ***VLURes** provides a practical testbed for long-text grounding and multilingual robustness in web-realistic agent settings. |
| Approach: | They propose a multilingual benchmark for evaluating vision-language models under long-text grounding. |
| Outcome: | ***VLURes** provides a testbed for long-text grounding and multilingual robustness in web-realistic agent settings. |
Evaluation Metrics in the Era of GPT-4: Reliably Evaluating Large Language Models on Sequence to Sequence Tasks (2023.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) evaluation is a patchy and inconsistent landscape . established automatic evaluation metrics are poor surrogates, correlating weakly with human judgement. |
| Approach: | They propose to use both automatic and human evaluation to evaluate generative LLMs on three NLP benchmarks: text summarisation, text simplification and grammatical error correction. |
| Outcome: | The proposed model outperforms many popular models according to human reviewers on the majority of metrics, while scoring much worse when using classic automatic evaluation metrics. |
TIGEr: Text-to-Image Grounding for Image Caption Evaluation (D19-1)
Copied to clipboard
Ming Jiang, Qiuyuan Huang, Lei Zhang, Xin Wang, Pengchuan Zhang, Zhe Gan, Jana Diesner, Jianfeng Gao
| Challenge: | Existing metrics based on text-level comparisons fail to assess the quality of captions produced by machines. |
| Approach: | They propose to use a machine-learned text-image grounding model to measure the accuracy of machine-generated captions and their correlation with human judgments. |
| Outcome: | The proposed metric has higher consistency with human judgments and is more accurate than existing metrics. |