Towards Efficient and Robust VQA-NLE Data Generation with Large Vision-Language Models

Challenge:	Existing methods for creating a vision question-answering with natural language explanations rely on human annotations that are time-consuming and costly.
Approach:	They propose a method that generates high-quality natural language explanations using LVLMs by using visual prompts.
Outcome:	The proposed method generates high-quality synthetic VQA-NLE datasets 20x faster than human annotations with minimal decrease in qualitative metrics.

Similar Papers

Overview

Challenge

Approach

Outcome

Using Natural Language Explanations to Improve Robustness of In-context Learning (2024.acl-long)

Copied to clipboard

Xuanli He, Yuxiang Wu, Oana-Maria Camburu, Pasquale Minervini, Pontus Stenetorp

Responsible & Trustworthy NLP

Challenge:	Recent studies show that large language models excel in many tasks via in-context learning (ICL). However, ICL struggles to execute complex tasks such as arithmetic, commonsense, and symbolic reasoning.
Approach:	They propose to augment ICL with natural language explanations (NLEs) to produce further NLEs on adversarial datasets.
Outcome:	The proposed approach yields more accurate results than zero-shot-ICL and using only human-generated NLEs on eight adversarial datasets.

Self-Rewarding Large Vision-Language Models for Optimizing Prompts in Text-to-Image Generation (2025.findings-acl)

Copied to clipboard

Hongji Yang, Yucheng Zhou, Wencheng Han, Jianbing Shen

Language Models

Challenge:	Existing methods for rewriting text-to-image models require specialized vocabulary . a new approach uses large vision language models to optimize text-based models .
Approach:	They propose a prompt optimization framework that rephrases a user prompt into a text-to-image model by using large vision language models as solver and reward model.
Outcome:	The proposed model outperforms existing models on two popular datasets.

All You May Need for VQA are Image Captions (2022.naacl-main)

Copied to clipboard

Soravit Changpinyo, Doron Kukliansy, Idan Szpektor, Xi Chen, Nan Ding, Radu Soricut

Multimodality

Challenge:	Visual Question Answering (VQA) has benefited from increasingly sophisticated models, but has not enjoyed the same level of engagement in terms of data creation.
Approach:	They propose a method that automatically derives VQA examples at volume by leveraging existing image-caption annotations combined with neural models for textual question generation.
Outcome:	The proposed method improves state-of-the-art zero-shot accuracy by double digits and achieves robustness that lacks in the same model trained on human-annotated VQA data.

NLKI: A Lightweight Natural Language Knowledge Integration Framework for Improving Small VLMs in Commonsense VQA Tasks (2025.findings-emnlp)

Copied to clipboard

Aritra Dutta, Swapnanil Mukherjee, Deepanway Ghosal, Somak Aditya

Responsible & Trustworthy NLP

Challenge:	Small vision-language models lag behind their larger generative counterparts due to lack of knowledge.
Approach:	They propose a framework that integrates commonsense knowledge into small vision-language models . the framework retrieves natural language facts and prompts an LLM to craft natural language explanations .
Outcome:	The proposed framework retrieves natural language facts and prompts an LLM to craft natural language explanations.

Leakage-Adjusted Simulatability: Can Models Generate Non-Trivial Explanations of Their Behavior in Natural Language? (2020.findings-emnlp)

Copied to clipboard

Peter Hase, Shiyue Zhang, Harry Xie, Mohit Bansal

Responsible & Trustworthy NLP

Challenge:	Existing models that generate NL explanations for tasks have been evaluated on the basis of surface-level similarities to human explanations, both through automatic metrics like BLEU and human evaluations.
Approach:	They propose to use a model as a proxy for a human observer to evaluate NL explanations from the model simulatability perspective.
Outcome:	The proposed model-generated explanations are evaluated on the basis of surface-level similarities to human explanations, both through automatic metrics like BLEU and human evaluations.

Do explanations make VQA models more predictable to a human? (D18-1)

Copied to clipboard

Arjun Chandrasekaran, Viraj Prabhu, Deshraj Yadav, Prithvijit Chattopadhyay, Devi Parikh

Explainability & Interpretability in NLP

Challenge:	Existing explanations of a model's behavior are not used in interactive tasks like Visual Question Answering (VQA).
Approach:	They analyze existing explanations and their role in making a VQA model more predictable to a human by using human-in-the-loop approaches that treat the model as a black-box.
Outcome:	The proposed explanations make a model more predictable to humans, whereas human-in-the-loop approaches treat it as a black-box do.

Enhancing Large Vision-Language Models with Ultra-Detailed Image Caption Generation (2025.emnlp-main)

Copied to clipboard

Yu Zeng, Yukun Qi, Yiming Zhao, Xikun Bao, Lin Chen, Zehui Chen, Shiting Huang, Jie Zhao, Feng Zhao

Multimodality

Challenge:	Existing pipelines for generating high-quality, ultra-detailed image captions are limited by the scarcity of image caption data.
Approach:	They propose a pipeline for generating high-quality, ultra-detailed image captions that integrates both pre-processing and post-processor stages.
Outcome:	The proposed pipeline improves LVLMs' perception and cognitive abilities across multiple vision-language benchmarks.

Towards Cross-Lingual Explanation of Artwork in Large-scale Vision Language Models (2025.findings-naacl)

Copied to clipboard

Shintaro Ozaki, Kazuki Hayashi, Yusuke Sakai, Hidetaka Kamigaito, Katsuhiko Hayashi, Taro Watanabe

Multilinguality

Challenge:	LVLMs are increasingly capable of responding in multiple languages . however, there is a lack of evaluation tools for LVLs that handle multiple languages.
Approach:	They used an extended dataset in multiple languages to evaluate LVLMs' ability to generate explanations in multiple language combinations.
Outcome:	The proposed dataset in multiple languages evaluates LVLMs' ability to generate explanations in other languages.

On Efficient Language and Vision Assistants for Visually-Situated Natural Language Understanding: What Matters in Reading and Reasoning (2024.emnlp-main)

Copied to clipboard

Geewook Kim, Minjoon Seo

Multimodality

Challenge:	Recent advances in language and vision assistants have showcased impressive capabilities but suffer from a lack of transparency, limiting broader research and reproducibility.
Approach:	They propose to redefine the design of vision-language models by identifying key components and creating efficient models with constrained inference costs.
Outcome:	The proposed models achieve significant improvements in inference throughput while maintaining high performance.

Filling the Image Information Gap for VQA: Prompting Large Language Models to Proactively Ask Questions (2023.findings-emnlp)

Copied to clipboard

Ziyue Wang, Chi Chen, Peng Li, Yang Liu

Language Models

Challenge:	Large Language Models (LLMs) demonstrate impressive reasoning ability and the maintenance of world knowledge in natural language tasks.
Approach:	They propose a framework that enables LLMs to ask relevant questions to uncover more details in the image, along with filters for refining the generated information.
Outcome:	The proposed framework boosts the performance of baseline methods by 2.15% on OK-VQA and achieves consistent improvements across different LLMs.