Papers with captioning
LAVIS: A One-stop Library for Language-Vision Intelligence (2023.acl-demo)
Copied to clipboard
| Challenge: | a new open-source library for language-vision research and applications is available for free. |
| Approach: | They introduce LAVIS, an open-source deep learning library for LAnguage-VISion research and applications. |
| Outcome: | The proposed library is open-source and highly extensible and configurable. |
Improving Image Captioning via Predicting Structured Concepts (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies on image captioning ignore the relationship between concepts . current methods for image caption generation ignore this relationship . |
| Approach: | They propose a structured concept predictor to predict concepts and their structures . they integrate these predictions into captioning to enhance visual signals . |
| Outcome: | The proposed approach improves image captioning performance by using semantic concepts as a bridge between images and texts. |
Enhancing Descriptive Image Captioning with Natural Language Inference (2021.acl-short)
Copied to clipboard
| Challenge: | Existing captioning models tend to generate generic captions, but generating descriptive captions is important. |
| Approach: | They propose a novel approach to encourage captioning models to produce more detailed captions using natural language inference. |
| Outcome: | The proposed method outperforms baseline models on MSCOCO metrics on descriptiveness and descriptiveness. |
Are Scene Graphs Good Enough to Improve Image Captioning? (2020.aacl-main)
Copied to clipboard
| Challenge: | Existing image captioning models rely on object detection features to generate image descriptions, but they are noisy. |
| Approach: | They propose to use scene graphs to introduce information about object relations into captioning to improve image descriptions. |
| Outcome: | The proposed model improves image caption quality by 3.3 CIDEr compared to a strong Bottom-Up Top-Down baseline. |
Finding and Editing Multi-Modal Neurons in Pre-Trained Transformers (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to identify key neurons for interpretability of multi-modal large language models are unclear. |
| Approach: | They propose a method to identify key neurons for interpretability by multi-modal large language models. |
| Outcome: | The proposed method improves conventional works upon efficiency and applied range by removing needs of costly gradient computation. |
TUNA: Comprehensive Fine-grained Temporal Understanding Evaluation on Dense Dynamic Videos (2025.acl-long)
Copied to clipboard
Fanheng Kong, Jingyuan Zhang, Hongzhi Zhang, Shi Feng, Daling Wang, Linhao Yu, Xingguang Ji, Yu Tian, V. W., Fuzheng Zhang
| Challenge: | Existing benchmarks for video understanding often focus on specific aspects, overlooking the holistic nature of video content. |
| Approach: | They propose a temporal-oriented benchmark for fine-grained understanding on dense dynamic videos with two complementary tasks: captioning and QA. |
| Outcome: | The proposed model performs well on diverse video scenarios and dynamic videos, with interpretable and robust evaluation criteria. |
Selectively Answering Visual Questions (2024.findings-acl)
Copied to clipboard
| Challenge: | Large multi-modal models (LMMs) are capable of visual question answering (VQA) with unprecedented accuracy. |
| Approach: | They propose a calibration score that can be used to quantify uncertainty in visual question answering models. |
| Outcome: | The proposed calibration score is better calibrated than in text-only models for in-context learning. |
Interactive Key-Value Memory-augmented Attention for Image Paragraph Captioning (2020.coling-main)
Copied to clipboard
| Challenge: | Existing approaches to image paragraph captioning ignore the past alignment information, resulting in repetitive captioning and incomplete captioning. |
| Approach: | They propose an Interactive key-value Memory-augmented Attention model for image paragraph captioning to keep track of attention history along with update-chain of decoder state. |
| Outcome: | Extensive experiments on a benchmark dataset demonstrate the effectiveness of the proposed model. |
OLIVE: Object Level In-Context Visual Embeddings (2024.acl-long)
Copied to clipboard
| Challenge: | Existing vision-language models lack fine-grained object-level understanding and grounding . existing models implicitly align text tokens with image patch tokens, which is ineffective for embedding alignment at the same granularity and introduces noisy spurious background features. |
| Approach: | They propose a method to prompt large language models with in-context visual object vectors . this method allows for controllable object-level reasoning . |
| Outcome: | The proposed method achieves competitive referring object classification and captioning performance while offering zero-shot generalization and robustness to visually challenging contexts. |
Connecting the Dots between Audio and Text without Parallel Data through Visual Knowledge Transfer (2022.naacl-main)
Copied to clipboard
| Challenge: | Existing methods for learning audio-text connections rely on parallel audio- text data . a new approach allows for the representation of environmental soundscapes without using parallel data - a challenge for many applications . |
| Approach: | They propose a model that induces Audio-Text alignment without using parallel audio-text data. |
| Outcome: | The proposed model outperforms the current state-of-the-art for audio classification tasks with no audio-text data by 2.2% on the ESC50 and US8K tasks. |
INSTRUCTSCORE: Towards Explainable Text Generation Evaluation with Automatic Feedback (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to evaluate the quality of language generation do not provide explicit explanation of their verdicts. |
| Approach: | They propose a fine-grained explainable evaluation metric for text generation that harnesses human instruction and implicit knowledge of GPT-4 to fine-tune it. |
| Outcome: | The proposed model outperforms all other unsupervised metrics on translation, captioning, data-to-text, and commonsense generation tasks. |
Dense Procedure Captioning in Narrated Instructional Videos (P19-1)
Copied to clipboard
| Challenge: | Existing models for video dense captioning learn video segments and generate captions without considering transcripts. |
| Approach: | They propose a model to generate procedure captions from narrated instructional videos . they extract procedures by a cross-modality module and generate captions by encoding video frames and transcripts within each extracted procedure. |
| Outcome: | The proposed model can extract procedures from narrated instructional videos and generate procedure captions by encoding video frames and transcripts. |
CapArena: Benchmarking and Analyzing Detailed Image Captioning in the LLM Era (2025.findings-acl)
Copied to clipboard
Kanzhi Cheng, Wenpo Song, Jiaxin Fan, Zheng Ma, Qiushi Sun, Fangzhi Xu, Chenyang Yan, Nuo Chen, Jianbing Zhang, Jiajun Chen
| Challenge: | Image captioning has been a challenge for vision-language researchers for decades . current VLMs focus on tasks like visual question answering (YA) but image captioning is not as advanced as expected. |
| Approach: | They evaluate VLMs' performance on image captioning using human annotations . they find that some metrics show high caption-level agreement with humans . |
| Outcome: | The proposed model outperforms open-source models on image captioning . it achieves 93.4% correlation with human rankings at $4 per test . |
MM-CRITIC: A Holistic Evaluation of Large Multimodal Models as Multimodal Critique (2025.findings-emnlp)
Copied to clipboard
| Challenge: | e MM-CRITIC is a holistic benchmark for evaluating the critique ability of Large Multimodal Models (LMMs) covering 8 main task types and over 500 tasks, covering 4471 samples. |
| Approach: | They introduce a holistic benchmark for evaluating the critique ability of Large Multimodal Models across multiple dimensions: basic, correction, and comparison. |
| Outcome: | The proposed benchmark covers 8 main task types and over 500 tasks and is composed of 4471 samples. |
Bridging Semantic and Modality Gaps in Zero-Shot Captioning via Retrieval from Synthetic Data (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for zero-shot image captioning fail to bridge the semantic gap between images and textual inputs. |
| Approach: | They propose a retrieval-based framework that leverages only existing synthetic image-text pairs as its search corpus to bridge the gap when using synthetic data for captioning. |
| Outcome: | The proposed method bridges the semantic gap between a synthetic image and its input text . it extracts image-related textual descriptions to mitigate the modality gap during decoding . |
Shaking Up VLMs: Comparing Transformers and Structured State Space Models for Vision & Language Modeling (2024.emnlp-main)
Copied to clipboard
| Challenge: | a task-agnostic visual encoding yields minimal performance gains on grounding, but Transformers outperform Mamba at in-context multimodal retrieval. |
| Approach: | They propose to replace Transformers in Visual Language Models with Mamba, a structured state space model that demonstrates promising performance in sequence modeling. |
| Outcome: | The proposed model outperforms Transformers-based models in captioning, question answering, and reading comprehension. |
MVP: Enhancing Video Large Language Models via Self-supervised Masked Video Prediction (2026.acl-long)
Copied to clipboard
| Challenge: | Recent research has attempted to transfer reinforcement learning paradigms to Video Large Language Models (MLLMs) but these methods lack explicit supervision for intrinsic temporal coherence and inter-frame correlations. |
| Approach: | They propose a novel post-training objective: Masked Video Prediction (MVP) that requires the model to reconstruct a masked continuous segment from a set of challenging distractors and employs Group Relative Policy Optimization (GRPO) with a fine-grained reward function to enhance the model's understanding of video context and temporal properties. |
| Outcome: | The proposed model improves video reasoning capabilities by reinforcing temporal reasoning and causal understanding. |
GLA: Grounding Large Language Models in Molecular Hierarchy for Chemical Understanding (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing molecule-language models obscure the hierarchical organization of chemical semantics . Existing models rely on linear or uniform encodings, causing structural distortion . |
| Approach: | They propose a framework that integrates intrinsic molecular topology into large language models. |
| Outcome: | The proposed framework improves on cross-modal retrieval, captioning, and property prediction benchmarks. |
Revisiting Audio-language Pretraining for Learning General-purpose Audio Representation (2026.acl-long)
Copied to clipboard
| Challenge: | Existing methods for learning general-purpose audio representations are limited in scope and coverage of audio attributes. |
| Approach: | They propose to use a 10.7M caption dataset to compare ALP with captioning . they find that contrastive learning yields competitive, transferable representations . |
| Outcome: | The proposed model yields competitive, transferable representations, while captioning exhibits better scalability. |
VideoPASTA: 7K Preference Pairs That Matter for Video-LLM Alignment (2025.emnlp-main)
Copied to clipboard
| Challenge: | Video-language models excel at understanding video content but struggle with spatial relationships, temporal ordering, and cross-frame continuity. |
| Approach: | They propose a framework that trains video-LLMs to distinguish accurate representations from carefully crafted adversarial examples. |
| Outcome: | Experiments show that VideoPASTA improves performance without human annotation or captioning . the framework can be used on various state-of-the-art video-LLMs with no human annotation . |
CaTS-Bench: Can Language Models Describe Time Series? (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing time series captioning benchmarks rely on fully synthetic or generic captions . authors propose a pipeline for generating high-fidelity synthetic captions, which is validated . |
| Approach: | They propose a benchmark for Context-aware Time Series reasoning across 11 diverse domains . they evaluate leading Vision-Language Models on their benchmark . |
| Outcome: | The proposed benchmark evaluates 1746 human-rewritten captions and shows they perform better than open-source models. |
PolyAudio: Advancing Multi-Audio Reasoning in Large Audio Language Models with Interleaved Multi-Audio Contexts (2026.findings-acl)
Copied to clipboard
Sonal Kumar, Sreyan Ghosh, Yueqian Lin, S Sakshi, Ashish Seth, Yiran Chen, Ramani Duraiswami, Dinesh Manocha
| Challenge: | Large Audio Language Models have shown impressive performance on single-clip tasks . however, their ability to reason over interleaved multi-audio contexts remains limited . |
| Approach: | They propose a LALM that targets multi-audio understanding via instruction tuning rather than massive-scale pre-training. |
| Outcome: | The proposed model outperforms baseline models on multi-audio tasks while maintaining robustness. |