Papers with VLMs
Understanding Figurative Meaning through Explainable Visual Entailment (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing models for visual entailment and visual question-answering have limited ability to understand figurative meaning in images and captions. |
| Approach: | They propose a task framing the figurative meaning understanding problem as an explainable visual entailment task where the model has to predict whether the image entitles a caption and justify the predicted label with a textual explanation. |
| Outcome: | The proposed dataset contains 6,027 image, caption, label, explanation instances covering five diverse figurative phenomena. |
SCoPE VLM: Selective Context Processing for Efficient Document Navigation in Vision-Language Models (2026.eacl-long)
Copied to clipboard
| Challenge: | Existing methods for document understanding are memory-intensive and impractical for local deployments. |
| Approach: | They propose a document navigation expert that leverages a Chain of Scroll mechanism to selectively and recursively navigate documents, focusing exclusively on relevant segments. |
| Outcome: | The proposed method reduces memory usage and effectively models human-like reading behaviors. |
Z3D: Zero-Shot 3D Visual Grounding from Images (2026.acl-short)
Copied to clipboard
Nikita Drozdov, Andrey Lemeshko, Nikita Gavrilov, Anton Konushin, Danila Rukhovich, Maksim Kolodiazhnyi
| Challenge: | 3D visual grounding (3DVG) aims to localize objects in a 3D scene based on natural language queries. |
| Approach: | They propose a zero-shot 3D visual grounding pipeline that operates on multi-view images without geometric supervision and without object priors. |
| Outcome: | Experiments on ScanRefer and Nr3D show that the proposed method outperforms existing methods. |
JEEM: Vision-Language Understanding in Four Arabic Dialects (2026.findings-eacl)
Copied to clipboard
Karima Kadaoui, Hanin Atwany, Hamdan Al-Ali, Abdelrahman Mohamed, Ali Mekky, Sergei Tilga, Natalia Fedorova, Ekaterina Artemova, Hanan Aldarmaki, Yova Kementchedjhieva
| Challenge: | Existing evaluation datasets feature Western-centric images and English text, while their non-English counterparts are often derived from the latter. |
| Approach: | They propose to evaluate Vision-Language Models (VLMs) on visual understanding across four Arabic-speaking countries: Jordan, The Emirates, Egypt, and Morocco. |
| Outcome: | The proposed model underperforms in visual understanding and dialect-specific generation across four Arabic-speaking countries. |
HGCLIP: Exploring Vision-Language Models with Graph Representations for Hierarchical Understanding (2025.coling-main)
Copied to clipboard
| Challenge: | Object categories are typically organized into a multi-granularity taxonomic hierarchy . traditional uni-modal approaches focus primarily on image features, revealing limitations in complex scenarios. |
| Approach: | They propose a framework that combines vision-language models with a deeper exploitation of the hierarchy. |
| Outcome: | The proposed framework shows significant improvements on 11 diverse visual recognition benchmarks. |
MARCH: Multi-Agent Radiology Clinical Hierarchy for CT Report Generation (2026.acl-short)
Copied to clipboard
| Challenge: | Automated 3D radiology report generation suffers from clinical hallucinations and lacks the iterative verification characteristic of clinical workflows. |
| Approach: | They propose a multi-agent framework that emulates the professional hierarchy of radiology departments and assigns specialized roles to distinct agents. |
| Outcome: | The proposed framework outperforms state-of-the-art models in clinical fidelity and linguistic accuracy on the RadGenome-ChestCT dataset. |
OpenFMNav: Towards Open-Set Zero-Shot Object Navigation via Vision-Language Foundation Models (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing methods for object navigation are limited to household datasets with close-set objects, and they lack the ability to generalize to new environments in a zero-shot manner. |
| Approach: | They propose a framework that leverages reasoning abilities of large vision language models to extract proposed objects from natural language instructions that meet the user’s demand. |
| Outcome: | The proposed framework surpasses baselines on all metrics and can be used in a HM3D ObjectNav benchmark. |
Pre-Deployment Advertisement Ranking under Data Scarcity via Context-Aware Criteria Generation with VLMs (2026.acl-industry)
Copied to clipboard
Kyungho Kim, Yeonje Choi, Gyurim Hwang, Sejin Chung, Hongseok Lee, Myeong Ho Song, Yeongho Kim, Sunwoo Kim, Jongha Lee, Juyeon Kim, Kijung Shin
| Challenge: | Existing VLMs perform well on general multimodal tasks, but limited labeled data makes them difficult to apply to real-world business decisions. |
| Approach: | They propose a new task that aims to rank ads for a target brand prior to deployment . they propose 'brand-specific ad ranking' which uses brand-specific effectiveness . |
| Outcome: | The proposed task outperforms baselines on 10 brands on real-world advertising data. |
MiSCHiEF: A Benchmark in Minimal-Pairs of Safety and Culture for Holistic Evaluation of Fine-Grained Image-Caption Alignment (2026.eacl-short)
Copied to clipboard
Sagarika Banerjee, Tangatar Madi, Advait Swaminathan, Jolie Nguyen, Shivank Garg, Kevin Zhu, Vasu Sharma
| Challenge: | Fine-grained image-caption alignment is crucial for vision-language models in socially critical contexts. |
| Approach: | They present a benchmarking dataset for fine-grained image-caption alignment in safety and culture contexts. |
| Outcome: | The proposed benchmarks show that models perform better at confirming correct pairs than rejecting incorrect ones on dual alignment tasks. |
Parameter-free and Accessible Prompt Learning to Enhance Adversarial Robustness for Pre-trained Vision-Language Models (2025.naacl-long)
Copied to clipboard
| Challenge: | Large pre-trained Vision-Language Models (VLMs) have revolutionized downstream vision-language tasks including classification, object detection, and segmentation. |
| Approach: | They propose to search for text prompts at the word level rather than optimizing continuous textual embeddings to boost adversarial robustness. |
| Outcome: | Experiments show that the proposed method outperforms hand-engineered prompts with average gains of +4.9% and +5.8%. |
When More Words Say Less: Decoupling Length and Specificity in Image Description Evaluation (2026.acl-short)
Copied to clipboard
| Challenge: | Vision-language models are increasingly used to produce textual descriptions of visual content. |
| Approach: | They propose to disentangle description specificity from description length . they find people prefer more specific descriptions regardless of length based on their own subjective preferences . |
| Outcome: | The proposed model shows that people prefer more specific descriptions regardless of length. |
Adapting Vision-Language Models for E-commerce Understanding at Scale (2026.eacl-industry)
Copied to clipboard
Matteo Nulli, Vladimir Orshulevich, Tala Bazazo, Christian Herold, Michael Kozielski, Marcin Mazur, Szymon Tuzel, Cees G. M. Snoek, Seyyed Hadi Hashemi, Omar Javed, Yannick Versley, Shahram Khadivi
| Challenge: | Existing approaches to adapt VLMs to attribute-centric, multi-image, and noisy data are limited. |
| Approach: | They propose a novel evaluation suite that incorporates deep product understanding, strict instruction following, and dynamic attribute extraction. |
| Outcome: | The proposed model improves e-commerce performance while preserving broad multimodal capabilities. |
On the Additive Compositionality of Task Vectors in Vision–Language Models (2026.eacl-short)
Copied to clipboard
| Challenge: | In-context learning (ICL) in large language models (LLMs) has been shown to operate through task vectors, but its extension to vision-language models (VLMs) remains underexplored. |
| Approach: | They construct visual reasoning tasks with clearly defined subtasks and extract task vectors from few-shot demonstrations. |
| Outcome: | The proposed model can be extended to vision-language models (VLMs) by adding the vectors of its constituent subtasks. |
Do Video Language Models really understand the video contexts? (2025.naacl-srw)
Copied to clipboard
| Challenge: | Recent advances in VideoQA performance have shown that visual language models are effective but the processes of understanding and reasoning in VLMs remain under-explored. |
| Approach: | They propose a framework that incorporates a fine-grained question generation and answering process to measure how well VLMs understand video question answering tasks. |
| Outcome: | The proposed framework incorporates a fine-grained question generation and answering process to measure how well the responses generated by VLMs align with what the model understands. |
Do Vision-Language Models Understand Compound Nouns? (2024.naacl-short)
Copied to clipboard
| Challenge: | Open-vocabulary vision-language models (CLIP) are emerging as a promising new paradigm for text-to-image retrieval. |
| Approach: | They propose a benchmark to evaluate the effectiveness of open-vocabulary vision-language models (CLIP) for text-to-image retrieval using contrastive loss. |
| Outcome: | The proposed framework improves CN understanding of CLIP by 8.25% on Compun. |
Think Before You Act: A Two-Stage Framework for Mitigating Gender Bias Towards Vision-Language Tasks (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing vision-language models focus on salient attributes but ignore contextualized nuances, resulting in gender bias. |
| Approach: | They propose a task-agnostic generation framework to mitigate gender bias in vision-language models. |
| Outcome: | The proposed framework can mitigate gender bias in vision-language models . it yields all-sided but gender-obfuscated narratives, which prevents concentration on localized image features, especially gender attributes. |
DOSE: Data Selection for Multi-Modal LLMs via Off-the-Shelf Models (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing data filtering methods are expensive because they are trained on the same data they are meant to screen. |
| Approach: | They propose to use off-the-shelf pretrained models that have never seen the target data to select training samples for larger and stronger multimodal models without task-specific training. |
| Outcome: | The proposed method can achieve comparable or even better results than those trained on the full dataset in standard VQA and math benchmarks. |
Chart-based Reasoning: Transferring Capabilities from LLMs to VLMs (2024.findings-naacl)
Copied to clipboard
Victor Carbune, Hassan Mansoor, Fangyu Liu, Rahul Aralikatte, Gilles Baechler, Jindong Chen, Abhanshu Sharma
| Challenge: | Visual language models (VLMs) are achieving increasingly strong performance on multimodal tasks. |
| Approach: | They propose to transfer reasoning capabilities from large-language models to VLMs by constructing a 20x larger dataset and a larger dataset to improve general reasoning capabilities. |
| Outcome: | The proposed model outperforms larger models without an upstream OCR system while keeping inference time constant. |
GlyphPattern: An Abstract Pattern Recognition for Vision-Language Models (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing benchmarks for abstract pattern recognition are easier because they do not involve a natural language description of the pattern. |
| Approach: | They present a dataset that pairs human-written descriptions of visual patterns with three visual presentation styles. |
| Outcome: | The proposed benchmark pairs human-written and human-verified patterns with three visual presentation styles. |
MovieCORE: COgnitive REasoning in Movies (2025.emnlp-main)
Copied to clipboard
Gueter Josmy Faure, Min-Hung Chen, Jia-Fong Yeh, Ying Cheng, Hung-Ting Su, Yung-Hao Tang, Shang-Hong Lai, Winston H. Hsu
| Challenge: | MovieCORE is a video question answering dataset that focuses on surface-level comprehension. |
| Approach: | They propose a video question-answer dataset that uses large language models as thought agents to generate and refine high-quality question-anchor pairs. |
| Outcome: | The proposed model improves model reasoning capabilities post-training by 25% . the proposed model is based on a large language model and is scalable to a wide range of tasks . |
CoLLaVO: Crayon Large Language and Vision mOdel (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing Large Language Models (LLMs) and instruction tuning have been used to drive the evolution of Vision Language Model (VLM) towards a versatile general-purpose model. |
| Approach: | They propose a learning strategy of Dual QLoRA to preserve object-level image understanding without forgetting it during visual instruction tuning, thereby achieving a significant leap in numerous VL benchmarks in a zero-shot setting. |
| Outcome: | The proposed model outperforms closed-source models on vision language tasks and achieves a significant leap in numerous benchmarks. |
VGA: Vision GUI Assistant - Minimizing Hallucinations through Image-Centric Fine-Tuning (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing Large Vision-Language Models (VLMs) often overly rely on internal text-based knowledge while neglecting visual inputs. |
| Approach: | They propose a model that balances attention image and text to enhance interpretation and reduce hallucinations by using a visual input. |
| Outcome: | The proposed model improves interpretation and reduces hallucinations by balancing attention image and text to enhance interpretation and reduction of hallucinosity. |
DRISHTIKON: A Multimodal Multilingual Benchmark for Testing Language Models’ Understanding on Indian Culture (2025.emnlp-main)
Copied to clipboard
Arijit Maji, Raghvendra Kumar, Akash Ghosh, null Anushka, Nemil Shah, Abhilekh Borah, Vanshika Shah, Nishant Mishra, Sriparna Saha
| Challenge: | DRISHTIKON is a first-of-its-kind multimodal and multilingual benchmark centered exclusively on Indian culture. |
| Approach: | They evaluate a wide range of vision-language models across zero-shot and chain-of-thought settings and use them to evaluate cultural understanding of generative AI systems. |
| Outcome: | The DRISHTIKON dataset covers 15 languages, all states and union territories, and incorporating over 64,000 aligned text-image pairs. |
From Behavioral Performance to Internal Competence: Interpreting Vision-Language Models with VLM-Lens (2025.emnlp-demos)
Copied to clipboard
Hala Sheta, Eric Haoran Huang, Shuyu Wu, Ilia Alenabi, Jiajun Hong, Ryker Lin, Ruoxi Ning, Daniel Wei, Jialin Yang, Jiawei Zhou, Ziqiao Ma, Freda Shi
| Challenge: | Existing vision-language models are based on exactmatch based accuracy and its derivations to evaluate performance. |
| Approach: | They propose a toolkit that supports systematic benchmarking, analysis, and interpretation of vision-language models by extracting intermediate outputs from any layer during the forward pass of open-source VLMs. |
| Outcome: | The proposed toolkit supports 16 state-of-the-art base VLMs and their over 30 variants and is extensible to accommodate new models without changing the core logic. |
Colorism in Multimodal AI: An Empirical Exploration of Socioeconomic Linguistic Bias in Text-to-Image Generation (2026.eacl-srw)
Copied to clipboard
| Challenge: | Socioeconomic inequalities worldwide are deeply linked to ethnoracial hierarchies and stereotypes, argues a new study. |
| Approach: | They use a Monk Skin Tone scale to benchmark VLMs and annotators . they then use linguistic cues to vary skin-tone representations in text-to-image generation . |
| Outcome: | The study compares 3 small VLMs and 60 human annotators on the monk skin tone scale with 210 occupations and produces over 2,500 portraits across 3 large VLM models. |
KAFA: Rethinking Image Ad Understanding with Knowledge-Augmented Feature Adaptation of Vision-Language Models (2023.acl-industry)
Copied to clipboard
| Challenge: | Image ad understanding is a crucial task with wide real-world applications, but is under-explored in the machine learning community due to the lack of foundational vision-language models (VLMs) . |
| Approach: | They propose a simple feature adaptation strategy to fuse multimodal information for image ads and further empower it with knowledge of real-world entities. |
| Outcome: | The proposed strategy fuses multimodal information for image ads and empowers it with knowledge of real-world entities. |
Seeing is Believing, but How Much? A Comprehensive Analysis of Verbalized Calibration in Vision-Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to quantify uncertainty are limited in vision-language models . however, current models display notable miscalibration across diverse tasks and settings . |
| Approach: | They evaluate verbalized confidence in vision-language models using visual reasoning . they propose a prompting strategy that improves confidence alignment in multimodal settings . |
| Outcome: | The proposed method improves confidence alignment across multimodal settings. |
Jailbreak Large Vision-Language Models Through Multi-Modal Linkage (2025.acl-long)
Copied to clipboard
| Challenge: | Existing methods to jailbreak large vision-language models fail against cutting-edge models such as GPT-4o, despite having undergone safety alignment training. |
| Approach: | They propose a new framework for jailbreaking large vision-language models that uses an encryption-decryption process to mitigate the over-exposure of harmful information. |
| Outcome: | The proposed framework jailbreaks GPT-4o with 99.40% success rates on SafeBench, 98.81% on MM-SafeBench and 99.07% on HADES-Dataset. |
sudo rm -rf agentic_security (2025.acl-industry)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are increasingly used as computer-use agents . authors present a novel attack framework that bypasses refusal-trained safeguards . |
| Approach: | They propose a new attack framework that bypasses refusal-trained safeguards in LLMs . SUDO iteratively refines its attacks based on a built-in refusal feedback . authors highlight need for robust, context-aware safeguards if LLM is to be used . |
| Outcome: | The proposed framework bypasses refusal-trained safeguards in commercial agents . it achieves a stark attack success rate of 24.41% (with no refinement) and up to 41.33% (by iterative refinement). |
StarFlow: Generating Structured Workflow Outputs From Sketch Images (2026.eacl-long)
Copied to clipboard
Patrice Bechard, Chao Wang, Amirhossein Abaskohi, Juan A. Rodriguez, Christopher Pal, David Vazquez, Spandana Gella, Sai Rajeswar, Perouz Taslakian
| Challenge: | Despite being widely used, building workflows can be complex, often requiring manual configuration through low-code platforms or visual programming tools. |
| Approach: | They propose a framework for generating structured workflow outputs from sketches using vision-language models to automate the process. |
| Outcome: | The proposed framework outperforms large vision-language models in the task of generating structured workflow outputs from sketches and diagrams. |
IndiFoodVQA: Advancing Visual Question Answering and Reasoning with a Knowledge-Infused Synthetic Data Generation Pipeline (2024.findings-eacl)
Copied to clipboard
| Challenge: | Large Vision Language Models lack domain-specific data for reasoning on complex problems. |
| Approach: | They propose to use explicit knowledge-infused questions, answers, and reasons to answer and reason upon the questions. |
| Outcome: | The proposed model improves by 25% over the baseline model. |
Spectra: A Mechanistic Interpretability Library for Vision-Language Models (2026.acl-demo)
Copied to clipboard
| Challenge: | Existing interpretability tools for visionlanguage models are limited to activation probing and saving. |
| Approach: | They propose a library specifically designed for mechanistic interpretability of visionlanguage models that provides unified abstractions for activation patching, attention pattern analysis, and meta-functions across diverse VLM architectures. |
| Outcome: | The proposed library handles architecture-specific complexities while maintaining a simple, high-level interface. |
Act as you think: Reinforcing Consistent Reasoning in Medical Visual Question Answering (2026.acl-long)
Copied to clipboard
Songtao Jiang, Yuan Wang, Ruizhe Chen, Yan Zhang, Ruilin Luo, Bohan Lei, Yeying Jin, Sibo Song, ZhiBo Yang, Jimeng Sun, Jian Wu, Zuozhu Liu
| Challenge: | Recent advances have improved the accuracy of medical visual question answering (Med-VQA) however, the high stakes nature of the medical domain has precipitated a shift towards interpretability and transparency of reasoning processes. |
| Approach: | They propose a reinforcement learning from verifiable rewards framework that rewards internal consistency and logical coherence. |
| Outcome: | The proposed framework rewards internal consistency and logical coherence, and is highly versatile, the authors show. |
Improve Vision Language Model Chain-of-thought Reasoning (2025.acl-long)
Copied to clipboard
Ruohong Zhang, Bowen Zhang, Yanghao Li, Haotian Zhang, Zhiqing Sun, Zhe Gan, Yinfei Yang, Ruoming Pang, Yiming Yang
| Challenge: | Current training recipes often rely on datasets dominated by short annotations with limited rationales, hindering the models' ability to generalize to tasks requiring comprehensive reasoning. |
| Approach: | They propose a two-stage post-training strategy that augments short answers with CoT reasoning generated by GPT-4o, enhancing the VLM's CoT capabilities through fine-tuning. |
| Outcome: | The proposed strategy enhances the model's CoT capabilities through fine-tuning and reinforcement learning. |
SCAN: Semantic Document Layout Analysis for Textual and Visual Retrieval-Augmented Generation (2026.findings-eacl)
Copied to clipboard
| Challenge: | Recent research indicates that using VLMs yields better RAG performance, but processing rich documents remains a challenge. |
| Approach: | They propose a VLM-friendly approach that enhances both textual and visual RAG systems. |
| Outcome: | The proposed approach outperforms conventional methods and commercial document processing solutions. |
VividMed: Vision Language Model with Versatile Visual Grounding for Medicine (2025.naacl-long)
Copied to clipboard
| Challenge: | Vision Language Models (VLMs) have demonstrated promise in generating visually grounded responses, but their application in the medical domain is hindered by unique challenges. |
| Approach: | They propose a vision language model with versatile visual grounding for medicine that generates semantic segmentation masks and instance-level bounding boxes. |
| Outcome: | The proposed model can generate semantic segmentation masks and instance-level bounding boxes, and accommodates various imaging modalities, including both 2D and 3D data. |
MATE: Meet At The Embedding - Connecting Images with Long Texts (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Recent advances in Vision Language Models (VLMs) focus on aligning images with short descriptive captions. |
| Approach: | They propose a method that combines VLMs with Large Language Models to efficiently align images with long texts without additional text pairs. |
| Outcome: | The proposed method bridges the gap between VLM and LLM without additional image-long text pairs. |
CAST: Cross-modal Alignment Similarity Test for Vision Language Models (2025.coling-main)
Copied to clipboard
| Challenge: | Vision Language Models (VLMs) are typically evaluated with Visual Question Answering tasks which assess a model’s understanding of scenes. |
| Approach: | They propose to use visual question answering (VQA) to assess a model's understanding of scenes to probe for self-consistency across modalities. |
| Outcome: | The proposed test does not focus on objective accuracy but rather on whether VLMs are internally consistent in their outputs. |
Evaluating Vision-Language Models for Emotion Recognition (2025.findings-naacl)
Copied to clipboard
| Challenge: | Large Vision-Language Models (VLMs) have been used for objective multimodal reasoning tasks for decades. |
| Approach: | They present a comprehensive evaluation of large vision-language models for recognizing evoked emotions from images. |
| Outcome: | The proposed model performs well in evoked emotion recognition task and is robust to human errors. |
ColorSwap: A Color and Word Order Dataset for Multimodal Evaluation (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent work reveals that vision and language models struggle to comprehend fine grained distinctions in images. |
| Approach: | They propose a dataset to assess multimodal models' ability to match objects with their colors. |
| Outcome: | The proposed model performs well in visual questionanswering, text-to-image generation and word-order understanding tasks. |
CARES: Context-Aware Resolution Selector for VLMs (2026.acl-long)
Copied to clipboard
| Challenge: | Large vision–language models process images at native or high resolution to remain effective across tasks. |
| Approach: | They propose a lightweight preprocessing module that predicts the minimum sufficient input resolution for large vision–language models. |
| Outcome: | CARES predicts when a pre-trained VLM's response converges to its peak ability to answer correctly, reducing compute by up to 80%. |
Enhancing Fine-Grained Image Classifications via Cascaded Vision Language Models (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Fine-grained image classification is a challenge for vision-language models (VLMs) such as CLIP, which struggle to distinguish between semantically similar classes due to insufficient supervision for fine-grain tasks. |
| Approach: | They propose a framework that harnesses the complementary strengths of both CLIP-like and LVLMs to tackle these challenges. |
| Outcome: | The proposed framework outperforms existing models on multiple fine-grained datasets, particularly the Stanford Cars dataset. |
A Structured Framework for Evaluating and Enhancing Interpretive Capabilities of Multimodal LLMs in Culturally Situated Tasks (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Using a zero-shot classification model, we extracted multi-dimensional evaluative features from human expert critiques and used them to evaluate selected VLMs such as Llama, Qwen, or Gemini. |
| Approach: | They constructed a quantitative framework for Chinese painting critique by extracting multi-dimensional evaluative features from human expert critiques using a zero-shot classification model. |
| Outcome: | The framework was constructed by extracting features from human critiques using a zero-shot classification model. |
TopViewRS: Vision-Language Models as Top-View Spatial Reasoners (2024.emnlp-main)
Copied to clipboard
| Challenge: | Top-view perspective is a typical way in which humans read and reason over different types of maps, but spatial reasoning capabilities of modern VLMs in this setup remain unattested and underexplored. |
| Approach: | They introduce a top-view spatial reasoning dataset and use it to evaluate VLMs across 4 perception and reasoning tasks with different levels of complexity. |
| Outcome: | The proposed model can understand and reason over spatial relations from the top view and can be controlled at different granularities of spatial reasoning. |
EcoDoc: A Cost-Efficient Multimodal Document Processing System for Enterprises Using LLMs (2025.acl-industry)
Copied to clipboard
| Challenge: | Recent advances in Retrieval-Augmented Generation (RAG) frameworks and Vision-Language Models (VLMs) have improved retrieval performance on multimodal documents by processing pages as images. |
| Approach: | They propose a cost-effective multimodal document processing system that dynamically selects the processing modalities for each page as an image or text based on page characteristics and query intent. |
| Outcome: | The proposed system reduces average query processing latency by 2.29 and cost by up to 10 . it reduces cost and latency while maintaining high performance on large scale deployments . |
Enhancing Advanced Visual Reasoning Ability of Large Language Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in Vision-Language (VL) research have sparked new benchmarks for complex visual reasoning, challenging models’ advanced reasoning ability. |
| Approach: | They propose a novel multi-modal in-context learning methodology to enhance LLMs’ contextual understanding and reasoning. |
| Outcome: | The proposed model achieves SOTA performance among all visual reasoning tasks and achieves a 'higher level of accuracy' than previous models. |
VIVA: A Benchmark for Vision-Grounded Decision-Making with Human Values (2024.emnlp-main)
Copied to clipboard
| Challenge: | Recent large vision language models (VLMs) have demonstrated remarkable intelligence and proficiency across diverse tasks. |
| Approach: | They propose a benchmark for VIsion-grounded decision-making driven by human VA. |
| Outcome: | The proposed model can make decisions under a vision-depicted situation using human values and human values. |
TechING: Towards Real World Technical Image Understanding via VLMs (2026.findings-eacl)
Copied to clipboard
| Challenge: | Modern day vision language models struggle when it comes to understanding technical diagrams . a large synthetically generated corpus is needed to train and evaluate VLMs on hand-drawn images . |
| Approach: | They propose a large synthetically generated corpus for training VLMs and evaluate them on hand-drawn images. |
| Outcome: | The proposed model improves ROUGE-L performance of Llama 3.2 11B-instruct by 2.14x on synthetic images on real-world images. |
Automating eHMI Action Design with LLMs for Automated Vehicle Communication (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Currently, eHMIs employ predefined text messages and manually designed actions to perform these messages . this limits the real-world deployment of ehMIs, where adaptability in dynamic scenarios is essential. |
| Approach: | They propose a pipeline that integrates large language models and 3D renderers to generate executable actions for controlling eHMIs and rendering action clips. |
| Outcome: | The proposed pipeline integrates large language models and 3D renderers to generate executable actions for controlling eHMIs and rendering action clips. |
Vision-Language Models Align with Human Neural Representations in Concept Processing (2026.eacl-long)
Copied to clipboard
| Challenge: | Recent studies suggest that transformer-based vision-language models capture the multimodality of concept processing in the human brain. |
| Approach: | They analysed multiple VLMs employing different strategies to integrate visual and textual modalities, along with language-only counterparts. |
| Outcome: | The transformer-based vision-language models outperform language-only models in two experimental conditions, while only some outperformed the language-based models. |
Defeating Cerberus: Privacy-Leakage Mitigation in Vision Language Models (2026.findings-eacl)
Copied to clipboard
Boyang Zhang, Istemi Ekin Akkus, Ruichuan Chen, Alice Dethise, Klaus Satzke, Ivica Rimac, Yang Zhang
| Challenge: | Existing models that process multiple modalities of data have been used for multimodal tasks, but their advanced capabilities raise privacy concerns. |
| Approach: | They propose a method to modify the model’s internal states associated with PII-related content and to reduce the risk of PI I leakage by modifying the model's internal state. |
| Outcome: | The proposed method achieves on average 93.3% refusal rate for various PII-related tasks with minimal impact on unrelated model performances. |
Finding Culture-Sensitive Neurons in Vision-Language Models (2026.eacl-long)
Copied to clipboard
| Challenge: | Vision-language models struggle on culturally situated inputs, study shows . despite impressive performance, many VLMs struggle on such culturally grounded inputs . |
| Approach: | They propose a new margin-based selector to identify neurons associated with cultural selectivity . they also introduce a model-dependent decoder to identify such neurons . |
| Outcome: | The proposed model outperforms probability- and entropy-based methods in identifying neurons associated with cultural selectivity. |
Image Embedding Sampling Method for Diverse Captioning (2025.emnlp-main)
Copied to clipboard
| Challenge: | Currently, large-scale captioning models are less accessible for resource-constrained applications such as mobile devices and assistive technologies. |
| Approach: | They propose a training-free framework that enhances caption diversity and informativeness by explicitly attending to distinct image regions using a comparably small VLM as the backbone. |
| Outcome: | The proposed framework achieves comparable performance to larger models on MSCOCO, Flickr30k, and Nocaps test datasets while maintaining strong image-caption relevancy and semantic integrity with the human-annotated captions. |
Breach in the Shield: Unveiling the Vulnerabilities of Large Language Models (2026.eacl-long)
Copied to clipboard
| Challenge: | Large Language Models and Vision Language Model (LLMs) have impressive performance across a wide range of tasks, yet remain vulnerable to external and internal perturbations. |
| Approach: | They propose a stability measure called FI, First order local Influence, which quantifies the sensitivity of individual parameters and input dimensions. |
| Outcome: | The proposed stability measure measures the sensitivity of parameters and input dimensions across LLMs and VLMs. |
HiPrune: Hierarchical Attention for Efficient Token Pruning in Vision-Language Models (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for visual token pruning lack insight into the intrinsic property of the vision encoder . et al., 2017: 99.3% of task accuracy with only 1/3 of the tokens. |
| Approach: | They propose a model-agnostic token pruning method that trains without training . they propose 'HiPrune' method which prunes visual tokens according to their attention . |
| Outcome: | The proposed method achieves 99.3% of task accuracy with only 1/3 of the tokens . it reduces inference FLOPs by 58.7% and maintains 99.99% accuracy with 2/9 tokens. |
Too Many Frames, Not All Useful: Efficient Strategies for Long-Form Video QA (2026.eacl-long)
Copied to clipboard
| Challenge: | Recent studies leverage large language models (LLMs) in LVQA benchmarks, achieving exceptional performance while relying on vision language models to convert all visual content into natural language. |
| Approach: | They propose a modular and training-free framework that leverages large language models to generate a small subset of informative frames tailored to each question. |
| Outcome: | The proposed framework achieves state-of-the-art performance among similar models across four benchmark LVQA datasets: EgoSchema, NExT-QA, IntentQA, VideoMME. |
TRACE: A Framework for Analyzing and Enhancing Stepwise Reasoning in Vision-Language Models (2026.eacl-long)
Copied to clipboard
| Challenge: | Evaluating large vision-language models has focused on final-answer correctness, but this metric is often insufficient and misleading. |
| Approach: | They propose a framework that decomposes complex multimodal tasks into Auxiliary Reasoning Sets (ARS) ARS decomposition reveals how consistently a model reasons across sub-questions with structured dependencies. |
| Outcome: | a new framework improves diagnostic evaluation of large vision-language models . it decomposes complex multimodal tasks into auxiliary reasoning sets with structured dependencies . the framework pinpoints reasoning failures and exposes errors overlooked by standard evaluation . |
With Ears to See and Eyes to Hear: Sound Symbolism Experiments with Multimodal Large Language Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models and Vision Language Model (VLMs) have demonstrated aptitude as potential substitutes for human participants in psycholinguistic experiments. |
| Approach: | They examine whether large language models and vision language models implicitly understand sound-based phenomena via orthography and imagery alone. |
| Outcome: | The proposed models demonstrate sound symbolism and ability to "hear" using language and vision modules. |
WorldCuisines: A Massive-Scale Benchmark for Multilingual and Multicultural Visual Question Answering on Global Cuisines (2025.naacl-long)
Copied to clipboard
Genta Indra Winata, Frederikus Hudi, Patrick Amadeus Irawan, David Anugraha, Rifki Afina Putri, Wang Yutong, Adam Nohejl, Ubaidillah Ariq Prathama, Nedjma Ousidhoum, Afifa Amriani, Anar Rzayev, Anirban Das, Ashmari Pramodya, Aulia Adila, Bryan Wilie, Candy Olivia Mawalim, Cheng Ching Lam, Daud Abolade, Emmanuele Chersoni, Enrico Santus, Fariz Ikhwantri, Garry Kuwanto, Hanyang Zhao, Haryo Akbarianto Wibowo, Holy Lovenia, Jan Christian Blaise Cruz, Jan Wira Gotama Putra, Junho Myung, Lucky Susanto, Maria Angelica Riera Machin, Marina Zhukova, Michael Anugraha, Muhammad Farid Adilazuarda, Natasha Christabelle Santosa, Peerat Limkonchotiwat, Raj Dabre, Rio Alexander Audino, Samuel Cahyawijaya, Shi-Xiong Zhang, Stephanie Yulia Salim, Yi Zhou, Yinxuan Gui, David Ifeoluwa Adelani, En-Shiun Annie Lee, Shogo Okada, Ayu Purwarianti, Alham Fikri Aji, Taro Watanabe, Derry Tanti Wijaya, Alice Oh, Chong-Wah Ngo
| Challenge: | Vision Language Models struggle with cultural-specific knowledge, especially in languages other than English and in underrepresented cultural contexts. |
| Approach: | They propose a visual question answering (VQA) dataset with text-image pairs across 30 languages and dialects and a training dataset. |
| Outcome: | The proposed model performs better with correct location context, but struggles with adversarial contexts and predicting specific regional cuisines and languages. |
PlanGPT-VL: Enhancing Urban Planning with Domain-Specific Vision-Language Models (2025.emnlp-industry)
Copied to clipboard
| Challenge: | Existing Vision-Language Models (VLMs) fail to analyze planning maps . specialized visual representations of land use zones, transportation networks, and development policies are needed to interpret complex planning maps. |
| Approach: | They propose a domain-specific VLM tailored for urban planning maps that employs three innovations: PlanAnno-V framework for high-quality VQA data synthesis, Critical Point Thinking (CPT) and PlanBench-V benchmark for systematic evaluation. |
| Outcome: | The new model outperforms general-purpose VLMs on planning map interpretation tasks. |
DRIVINGVQA: A Dataset for Interleaved Visual Chain-of-Thought in Real-World Driving Scenarios (2026.findings-eacl)
Copied to clipboard
| Challenge: | Chain-of-thought (CoT) prompting is a prompting strategy that improves reasoning in large language models, but its effectiveness in vision-language models remains limited due to over-reliance on textual cues and memorized knowledge. |
| Approach: | They propose a visual question-answering dataset derived from driving theory exams that incorporates textual explanations with visual tokens extracted from entities relevant to the reasoning process. |
| Outcome: | The proposed approach outperforms chain-of-thought prompting in large language models and vision-language models in real-world scenarios. |
DashboardQA: Benchmarking Multimodal Agents for Question Answering on Interactive Dashboards (2026.findings-eacl)
Copied to clipboard
Aaryaman Kartha, Ahmed Masry, Mohammed Saidul Islam, Thinh Lang, Shadikur Rahman, Ridwan Mahbub, Mizanur Rahman, Mahir Ahmed, Md Rizwan Parvez, Enamul Hoque, Shafiq Joty
| Challenge: | Existing question-answering benchmarks for data visualizations focus on static charts instead of interactive dashboards. |
| Approach: | They propose a benchmark to assess how vision-language GUI agents comprehend and interact with real-world dashboards. |
| Outcome: | The first benchmark explicitly designed to assess how vision-language GUI agents comprehend and interact with real-world dashboards. |
Teaching Vision-Language Models to Ask: Resolving Ambiguity in Visual Questions (2025.acl-long)
Copied to clipboard
| Challenge: | Existing research addresses ambiguous visual questions by rephrasing questions, but it fails to address the inherently interactive nature of user interactions with visual language models (VLMs). Existing studies focus on re-phrase questions, and lack of a benchmark to assess VLMs’ capacity for resolving ambiguities through interaction. |
| Approach: | They propose a visual question answering task that provides a natural language answer to a question based on a given image and an automated pipeline to generate ambiguity-clarification question pairs. |
| Outcome: | The proposed benchmark targets three common categories of ambiguity in visual question answering (VQA) context and encompasses various VQA scenarios. |
MatViX: Multimodal Information Extraction from Visually Rich Articles (2025.naacl-long)
Copied to clipboard
Ghazal Khalighinejad, Sharon Scott, Ollie Liu, Kelly L. Anderson, Rickard Stureborg, Aman Tyagi, Bhuwan Dhingra
| Challenge: | Existing methods for multimodal information extraction are limited due to the multimodal nature of scientific articles and complex interconnections between data points. |
| Approach: | They propose a benchmark to extract structured information from scientific articles . they use curated JSON files extracted from text, tables, and figures . |
| Outcome: | The proposed benchmark is based on 324 full-length research articles and 1,688 complex structured JSON files curated by experts in polymer nanocomposites and biodegradation. |
Unraveling and Mitigating Safety Alignment Degradation of Vision-Language Models (2025.findings-acl)
Copied to clipboard
Qin Liu, Chao Shang, Ling Liu, Nikolaos Pappas, Jie Ma, Neha Anna John, Srikanth Doss, Lluis Marquez, Miguel Ballesteros, Yassine Benajiba
| Challenge: | LLaVA-7B demonstrated a decline in safety alignment ability on multi-modal inputs compared to its LLM backbone. |
| Approach: | They propose a method to recover alignment ability from LLM backbone while preserving functional capabilities of VLMs. |
| Outcome: | The proposed framework recovers alignment ability that is inherent in the LLM backbone with minimal impact on fluency and linguistic capabilities of pre-trained VLMs. |
CAPSTONE: Composable Attribute‐Prompted Scene Translation for Zero‐Shot Vision–Language Reasoning (2025.emnlp-industry)
Copied to clipboard
| Challenge: | CAPSTONE transforms visual inputs into structured text prompts that can be interpreted by a frozen Large Language Model (LLM). |
| Approach: | They propose a plug-and-play framework that transforms off-the-shelf vision models into structured text prompts that can be interpreted by a frozen Large Language Model (LLM). |
| Outcome: | The proposed framework outperforms fully trained VLMs on the POPE dataset while the 4B model achieves competitive results. |
NegVQA: Can Vision Language Models Understand Negation? (2025.findings-acl)
Copied to clipboard
| Challenge: | NegVQA is a visual question answering (VQA) benchmark consisting of 7,379 two-choice questions covering diverse negation scenarios and image-question distributions. |
| Approach: | They propose a visual question answering benchmark consisting of 7,379 two-choice questions covering diverse negation scenarios and image-question distributions. |
| Outcome: | The proposed model fails to correctly interpret negation, leading to critical errors in interactive AI systems. |
Language-to-Space Programming for Training-Free 3D Visual Grounding (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for 3D visual grounding have been proposed, but they are limited by the scarcity of 3D vision-language datasets and the high cost of annotations. |
| Approach: | They propose a method for training-free 3D visual grounding that uses LLM-generated codes to analyze 3D spatial relations among objects. |
| Outcome: | The proposed method achieves 52.9% accuracy on the Nr3D benchmark and significantly reduces grounding time and token costs. |
Journey Before Destination: On the importance of Visual Faithfulness in Slow Thinking (2026.eacl-long)
Copied to clipboard
| Challenge: | Existing evaluations for visual hallucinations are narrow. |
| Approach: | They propose a framework that decomposes reasoning chains into perception versus reasoning steps and uses off-the-shelf VLM judges for step-level faithfulness. |
| Outcome: | The proposed framework reduces Unfaithful Perception Rate while preserving final-answer accuracy. |
NTSEBENCH: Cognitive Reasoning Benchmark for Vision Language Models (2025.findings-naacl)
Copied to clipboard
| Challenge: | Recent advances in large language models have demonstrated their strong performance on IQ test questions, achieving high scores across many languages. |
| Approach: | They propose a dataset to evaluate cognitive multimodal reasoning and problem-solving skills of large models. |
| Outcome: | The proposed dataset contains 2,728 multiple-choice questions and 4,642 images spanning 26 categories. |
AVA: Attentive VLM Agent for Mastering StarCraft II (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing StarCraft II benchmarks rely on abstract state representations that deviate from human perception . Existing systems rely only on abstract representations, creating an artificial gap between how humans process battlefield information and limiting ecological validity of learned behaviors. |
| Approach: | They introduce AVACraft, the first multimodal benchmark environment for complex decision-making in StarCraft II. |
| Outcome: | The AVACraft benchmark supports both traditional and modern multi-agent reinforcement learning paradigms. |
Decompose and Compare Consistency: Measuring VLMs’ Answer Reliability via Task-Decomposition Consistency Comparison (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for estimating uncertainty using answer likelihoods or prompt-based confidence generation often suffer from overconfidence and confirmation biases. |
| Approach: | They propose to use Decompose and Compare Consistency (DeCC) to measure the reliability of a VLM's direct answer and indirect answers by decomposing the question into sub-questions and reasoning over the sub-answers. |
| Outcome: | Experiments on six vision-language tasks with three VLMs show that DeCC achieves better correlation with task accuracy compared to existing methods. |
BLEnD-Vis: Benchmarking Multimodal Cultural Understanding in Vision Language Models (2026.eacl-long)
Copied to clipboard
Bryan Chen Zhengyu Tan, Weihua Zheng, Zhengyuan Liu, Nancy F. Chen, Hwaran Lee, Kenny Tsu Wei Choo, Roy Ka-Wei Lee
| Challenge: | Existing evaluations assess static recall or isolated visual grounding, leaving unanswered whether VLMs possess robust and transferable cultural understanding. |
| Approach: | They propose a multimodal, multicultural benchmark to evaluate the robustness of everyday cultural knowledge in vision-language models across linguistic rephrasings and visual modalities. |
| Outcome: | ‘BLEnD-Vis‘ constructs 313 culturally grounded question templates spanning 16 regions and generates three aligned multiple-choice formats. |
Sparkle: Mastering Basic Spatial Capabilities in Vision Language Models Elicits Generalization to Spatial Reasoning (2025.findings-emnlp)
Copied to clipboard
Yihong Tang, Ao Qu, Zhaokai Wang, Dingyi Zhuang, Zhaofeng Wu, Wei Ma, Shenhao Wang, Yunhan Zheng, Zhan Zhao, Jinhua Zhao
| Challenge: | Currently, vision-language models excel in many downstream tasks but struggle with spatial reasoning, which is crucial for navigation and interaction with physical environments. |
| Approach: | They propose a framework that generates synthetic data to provide targeted supervision for VLMs across these basic spatial capabilities. |
| Outcome: | The proposed framework disentangles 2D spatial reasoning into three core components: direction comprehension, distance estimation, and localization. |
Look Less, Reason More: Rollout-Guided Adaptive Pixel-Space Reasoning (2026.acl-long)
Copied to clipboard
| Challenge: | Recent work has shown promise by incorporating pixel-level visual information into the reasoning process, enabling VLMs to access high-resolution visual details during their thought process. |
| Approach: | They propose a framework that dynamically determines necessary pixel-level operations based on the input query. |
| Outcome: | The proposed model achieves 73.4% accuracy on HR-Bench 4K while maintaining a tool usage ratio of only 20.1%, improving accuracy and reducing tool usage by 66.5% compared to the previous methods. |
Once Correct, Still Wrong: Counterfactual Hallucination in Multilingual Vision-Language Models (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing hallucination benchmarks rarely test this failure mode outside Western contexts and English. |
| Approach: | They propose a multimodal benchmark built from images spanning 17 MENA countries . they use a CFHR-based test to measure hallucination beyond raw accuracy . |
| Outcome: | The proposed model is based on images from 17 MENA countries . it measures counterfactual acceptance conditioned on correctly answering the true statement. |
Vision Language Model Helps Private Information De-Identification in Vision Data (2025.findings-acl)
Copied to clipboard
| Challenge: | Visual Language Models (VLMs) have gained popularity due to their ability to solve imagerelated tasks. |
| Approach: | They propose a framework to enhance privacy awareness of visual language models . they use a specialized instruction-tuning dataset and a tailored training methodology . |
| Outcome: | The proposed framework outperforms existing approaches in handling private information. |
CIVET: Systematic Evaluation of Understanding in VLMs (2025.findings-emnlp)
Copied to clipboard
Massimo Rizzoli, Simone Alghisi, Olha Khomyn, Gabriel Roccabruna, Seyed Mahed Mousavi, Giuseppe Riccardi
| Challenge: | Current Vision-Language Models can accurately recognize only a limited set of basic object properties; 3) they struggle to understand basic relations among objects. |
| Approach: | They propose a framework that evaluates VLMs on exhaustive sets of stimuli, free from annotation noise, dataset-specific biases, and uncontrolled scene complexity. |
| Outcome: | The proposed framework addresses the lack of standardized systematic evaluation for assessing VLMs’ understanding, enabling researchers to test hypotheses with statistical rigor. |
MM-R3: On (In-)Consistency of Vision-Language Models (VLMs) (2025.findings-acl)
Copied to clipboard
| Challenge: | a flurry of research has been conducted on the performance of state-of-the-art (SoTA) Vision Language Models (VLMs) on a variety of tasks. |
| Approach: | They propose a benchmarking tool to analyze performance of SoTA Vision Language Models (VLMs) on three tasks: Question Rephrasing, Image Restyling, and Context Reasoning. |
| Outcome: | The proposed model achieves absolute improvements of 5.7% and 12.5% on widely used VLMs such as BLIP-2 and LLaVa 1.5M in terms of consistency over their existing counterparts. |
Your Vision-Language Model Itself Is a Strong Filter: Towards High-Quality Instruction Tuning with Data Selection (2024.findings-acl)
Copied to clipboard
Ruibo Chen, Yihan Wu, Lichang Chen, Guodong Liu, Qi He, Tianyi Xiong, Chenxi Liu, Junfeng Guo, Heng Huang
| Challenge: | Existing data selection methods for instruction-following large language models rely on unreliable scores or use downstream tasks for selection. |
| Approach: | They propose a method that utilizes the VLM itself as a filter to select high-quality instruction-tuning data. |
| Outcome: | The proposed method can reach better results compared to full data settings with merely about 15% samples and can achieve superior performance against competitive baselines. |
Why Vision Language Models Struggle with Visual Arithmetic? Towards Enhanced Chart and Geometry Understanding (2025.findings-acl)
Copied to clipboard
| Challenge: | Vision Language Models struggle with visual arithmetic, seemingly simple tasks like object counting or length comparison, which are essential for relevant complex tasks like chart understanding and geometric reasoning. |
| Approach: | They propose a novel post-training strategy inspired by Piaget’s theory of cognitive development that trains VLMs to recognize invariant properties under visual transformations. |
| Outcome: | The proposed approach outperforms supervised fine-tuning methods while requiring 60% less training data. |
Multimodal Cognitive Reframing Therapy via Multi-hop Psychotherapeutic Reasoning (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing studies have focused on text-based cognitive reframing, but neglected the importance of non-verbal evidence in real-life therapy. |
| Approach: | They propose a dataset that pairs each GPT-4-generated dialogue with an image that reflects the virtual client’s facial expressions to better mirror real psychotherapy, where facial expression leads to interpreting implicit emotional evidence. |
| Outcome: | The proposed approach outperforms existing methods with LLMs and vision-language models and provides more thoughtful and empathetic suggestions. |
Spec-o3: A Tool-Augmented Vision-Language Agent for Rare Celestial Object Candidate Vetting via Automated Spectral Inspection (2026.acl-long)
Copied to clipboard
| Challenge: | Spec-o3 is a tool-augmented vision-language agent that performs astronomer-aligned spectral inspection. |
| Approach: | They propose a tool-augmented vision-language agent that performs astronomer-aligned spectral inspection via interleaved multimodal chain-of-thought reasoning. |
| Outcome: | Spec-o3 outperforms traditional visual inspection methods on rare-object inspection tasks. |
AdaV: Adaptive Text-visual Redirection for Vision-Language Models (2025.findings-acl)
Copied to clipboard
| Challenge: | Vision-language models often generate excessive visual tokens, leading to poor performance . a novel training-free visual token pruning method is proposed to improve performance despite the computational cost associated with VLMs. |
| Approach: | They propose a training-free visual token pruning method that reduces biased token pruning . they plan to open-source the code upon publication . |
| Outcome: | The proposed method reduces biased token pruning and enhances model robustness with limited visual token budget. |
ExStrucTiny: A Benchmark for Schema-Variable Structured Information Extraction from Document Images (2026.eacl-long)
Copied to clipboard
Mathieu Sibue, Andrés Muñoz Garza, Samuel Mensah, Pranav Shetty, Zhiqiang Ma, Xiaomo Liu, Manuela Veloso
| Challenge: | Existing models for structured information extraction are limited by narrow entity ontologies, simple queries, or homogeneous document types. |
| Approach: | They propose a benchmark dataset for structured Information Extraction (IE) from document images . they analyze open and closed VLMs on this benchmark . |
| Outcome: | The proposed model can perform fine-grained structured extraction across document types and schemas. |
World to Code: Multi-modal Data Generation via Self-Instructed Compositional Captioning and Filtering (2024.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in Vision-Language Models and the scarcity of high-quality multi-modal alignment data have inspired numerous researches on synthetic VLM data generation. |
| Approach: | They propose a multi-modal data construction pipeline that organizes the final output into a Python code format. |
| Outcome: | The proposed pipeline improves visual question answering and visual grounding benchmarks across different VLMs. |
What’s Missing in Vision-Language Models? Probing Their Struggles with Causal Order Reasoning (2026.eacl-long)
Copied to clipboard
| Challenge: | Existing benchmarks often include a mixture of reasoning questions, making it difficult to truly assess VLMs’ causal reasoning abilities. |
| Approach: | They propose two new benchmarks specifically designed to isolate and rigorously evaluate VLMs’ causal reasoning abilities. |
| Outcome: | The proposed benchmarks show that vision-language models perform poorly on causal reasoning tasks, often only marginally surpassing random guessing. |
WikiSeeker: Rethinking the Role of Vision-Language Models in Knowledge-Based Visual Question Answering (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for Knowledge-Based Visual Question Answering rely on images as the retrieval key, and often overlook or misplace the role of Vision-Language Models (VLMs) |
| Approach: | They propose a multi-modal RAG framework that assigns VLMs two specialized agents: a Refiner and an Inspector. |
| Outcome: | Experiments on EVQA, InfoSeek, and M2KR show that the proposed framework achieves state-of-the-art performance with significant improvements in both retrieval accuracy and answer quality. |
OS-Genesis: Automating GUI Agent Trajectory Construction via Reverse Task Synthesis (2025.acl-long)
Copied to clipboard
Qiushi Sun, Kanzhi Cheng, Zichen Ding, Chuanyang Jin, Yian Wang, Fangzhi Xu, Zhenyu Wu, Chengyou Jia, Liheng Chen, Zhoumianze Liu, Ben Kao, Guohao Li, Junxian He, Yu Qiao, Zhiyong Wu
| Challenge: | Graphical User Interface (GUI) agents powered by Vision-Language Models (VLMs) have demonstrated human-like computer control capability. |
| Approach: | They propose a GUI data synthesis pipeline that reverse engineers GUI trajectory construction process by executing pre-defined tasks. |
| Outcome: | The proposed GUI data synthesis pipeline overcomes the bottlenecks of previous methods that rely on pre-defined tasks and limited data diversity. |
From the Least to the Most: Building a Plug-and-Play Visual Reasoner via Data Synthesis (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing models consisting of multiple steps of visual and language processing are limited in the visual and visual processing community . a visual reasoner is a plug-and-play approach that can be used to improve VLMs' reasoning abilities. |
| Approach: | They propose a least-to-most visual reasoning paradigm that divides a question into sub-questions and invokes external tools for resolving sub-questions. |
| Outcome: | The proposed method can improve four VLMs on four VQA benchmarks. |
What Media Frames Reveal About Stance: A Dataset and Study about Memes in Climate Change Discourse (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Media framing is a method of shaping public perceptions of issues, but the interaction between stance and media frame remains unexplored. |
| Approach: | They propose to use a dataset of climate-change memes annotated with stance and media frames to conceptualize and computationally explore this interaction. |
| Outcome: | The proposed dataset includes 1,184 climate-change memes sourced from 47 subreddits and enables analysis of frame prominence over time and communities. |
HALP: Detecting Hallucinations in Vision-Language Models without Generating a Single Token (2026.eacl-long)
Copied to clipboard
| Challenge: | Existing methods for detection of hallucinations operate after text generation, making intervention costly and untimely. |
| Approach: | They examine whether hallucination risk can instead be predicted before any token is generated by probing a model's internal representations in a single forward pass. |
| Outcome: | The proposed model can detect hallucinations before token generation, while query-token representations can be more accurate. |
POP: Prefill-Only Pruning for Efficient Large Model Inference (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing structured pruning methods suffer from significant accuracy degradation . Existing pruning methods are expensive and require specialized hardware and kernels to perform . |
| Approach: | They propose a stage-agnostic pruning approach that overlooks asymmetric roles between prefill and decode stages. |
| Outcome: | The proposed pruning approach achieves 1.37 speedup in prefill latency with minimal performance loss. |
Visually-Guided Policy Optimization for Multimodal Reasoning (2026.acl-long)
Copied to clipboard
| Challenge: | Existing RLVRs lack visual faithfulness due to text-dominated reasoning . a novel framework to reinforce visual focus during policy optimization is proposed . |
| Approach: | They propose a framework to reinforce visual focus during policy optimization using visual attention compensation mechanism. |
| Outcome: | The proposed framework exhibits better visual activation and superior performance in multimodal reasoning and visual-dependent tasks. |
DreamSync: Aligning Text-to-Image Generation with Image Understanding Feedback (2025.naacl-long)
Copied to clipboard
Jiao Sun, Deqing Fu, Yushi Hu, Su Wang, Royi Rassin, Da-Cheng Juan, Dana Alon, Charles Herrmann, Sjoerd Van Steenkiste, Ranjay Krishna, Cyrus Rashtchian
| Challenge: | Text-to-Image models (T2I) still struggle to produce images that are both aesthetically pleasing and faithful to the user’s input text. |
| Approach: | They propose a training algorithm that trains T2I models to be faithful to the input text. |
| Outcome: | The proposed model improves both the semantic alignment and aesthetic appeal of two diffusion-based T2I models, evidenced by multiple benchmarks (+1.7% on TIFA, +2.9% on DSG1K, +3.4% on VILA aesthetic). |
GOBench: Stage-Wise Diagnostics and the Visual Paradox in Multimodal Graph Optimization (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing benchmarks fail to represent multimodal problem specifications, score outcomes only and cannot localize where failures occur along the modeling pipeline. |
| Approach: | They propose a Graph Optimization benchmark that aligns multiple modalities with solver-derived oracles and a diagnostic protocol that evaluates intermediate artifacts as well as end results. |
| Outcome: | Graph Optimization benchmark (GOBench) evaluates intermediate artifacts as well as end results . vision reliably increases inference cost, while reliability impact is regime-dependent . current benchmarks fail to represent multimodal problem specifications, fail to localize failures . |
Multimodal Fact-Checking with Vision Language Models: A Probing Classifier based Solution with Embedding Strategies (2025.coling-main)
Copied to clipboard
| Challenge: | Existing fact-checking systems that use text and image information are susceptible to fake news spread by social media platforms. |
| Approach: | They propose a neural probing classifier based on multimodality and embeddings from text and image encoders to represent multimodal content for fact-checking. |
| Outcome: | The proposed classifier outperforms KNN and SVM baselines in leveraging extracted embeddings, highlighting its effectiveness for multimodal fact-checking. |
Iterative Self-Correction for Text-Driven Person Re-Identification with Large Vision-Language Models (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for Person Re-Identification (ReID) adopt a static "one-pass" paradigm, converting images to text once for retrieval. |
| Approach: | They propose a framework that reformulates ReID as an iterative "Think-and-Refine" process. |
| Outcome: | The proposed framework outperforms state-of-the-art methods in complex occlusion scenarios. |
Open-Vocabulary Federated Learning with Multimodal Prototyping (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing studies assume the label space of training data and test data is identical. |
| Approach: | They propose a framework for adaptation to a federated learning (FL) query that uses arbitrary unknown classes. |
| Outcome: | The proposed framework exploits the knowledge learned from seen classes and robustifies the adapted framework to unseen categories. |
Do Images Speak Louder than Words? Investigating the Effect of Textual Misinformation in VLMs (2026.eacl-long)
Copied to clipboard
| Challenge: | Existing studies have shown that Vision-Language Models have robust multimodal reasoning capabilities, but their robustness against textual misinformation remains under-explored. |
| Approach: | They propose to use visual-question-answering (VQA) prompts to generate persuasive prompts that deliberately conflict with visual evidence to test their models. |
| Outcome: | The proposed framework shows that models are vulnerable to misleading prompts, and show an average performance drop of over 48.2% after only one round of persuasive conversation. |
Advancing General Multimodal Capability of Vision-language Models with Pyramid-descent Visual Position Encoding (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to encode visual positions inhibit the performance of vision-language Models (VLMs) however, language constitutes only one aspect of communication. |
| Approach: | They propose a method to assign visual position indexes from the periphery to the center and expand the central receptive field incrementally to enhance the perception of visual tokens within VLMs. |
| Outcome: | The proposed method reduces the relative distance between interrelated visual elements and instruction tokens, promoting a more rational allocation of attention weights and allowing for a multi-granularity perception of visual elements. |
Benchmarking Vision Language Models for Cultural Understanding (2024.emnlp-main)
Copied to clipboard
Shravan Nayak, Kanishk Jain, Rabiul Awal, Siva Reddy, Sjoerd Steenkiste, Lisa Hendricks, Karolina Stanczak, Aishwarya Agrawal
| Challenge: | Recent multimodal vision-language models have shown impressive performance in tasks such as image-to-text generation, visual question answering, and image captioning. |
| Approach: | They propose a visual question-answering benchmark to assess VLMs' cultural understanding of various facets of culture from 11 countries across 5 continents. |
| Outcome: | The visual question-answering benchmark aims to assess VLMs' cultural understanding across regions. |
Self-Correction is More than Refinement: A Learning Framework for Visual and Language Reasoning Tasks (2025.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have shown remarkable abilities, but they invariably generate flawed responses. |
| Approach: | They propose a self-correction approach that instructs VLMs to refine their outputs by allowing them to learn from their self-generated self-reference data without external feedback. |
| Outcome: | The proposed approach enables VLMs to learn from their self-generated self-correction data without relying on external feedback, facilitating self-improvement. |
MEENA (PersianMMMU): Multimodal-Multilingual Educational Exams for N-level Assessment (2026.findings-eacl)
Copied to clipboard
Omid Ghahroodi, Arshia Hemmat, Marzia Nouri, Seyed Mohammad Hadi Hosseini, Doratossadat Dastgheib, Mohammad Vali Sanian, Alireza Sahebi, Reihaneh Zohrabi, Mohammad Hossein Rohban, Ehsaneddin Asgari, Mahdieh Soleymani Baghshah
| Challenge: | Recent advances in large vision-language models have primarily focused on English, with limited attention given to other languages. |
| Approach: | They propose a dataset to evaluate Persian VLMs across scientific, reasoning, and human-level understanding tasks. |
| Outcome: | The proposed model performs well across scientific reasoning, reasoning, and human-level understanding tasks in Persian and English. |
RATE-Nav: Region-Aware Termination Enhancement for Zero-shot Object Navigation with Vision-Language Models (2025.findings-acl)
Copied to clipboard
| Challenge: | Object navigation is a fundamental task in embodied artificial intelligence. |
| Approach: | They propose a region-aware Termination-Enhanced method that incorporates visual language models and exploration rates to enable efficient termination. |
| Outcome: | The proposed method achieves a success rate of 67.8% and an SPL of 31.3% on the HM3D dataset. |
ChartVerse: Scaling Chart Reasoning via Reliable Programmatic Synthesis from Scratch (2026.acl-long)
Copied to clipboard
Zheng Liu, Honglin Lin, Xiaoyang Wang, Xin Gao, Yu Li, Mengzhang Cai, Yun Zhu, Zhanping Zhong, Qizhi Pei, Zhuoshi Pan, Xiaoran Shang, Conghui He, Bin Cui, Wentao Zhang, Lijun Wu
| Challenge: | Existing open-source vision language models lack high-quality training data for chart reasoning . current models are simplistic and repetitive, while associated QA pairs are prone to hallucinations . |
| Approach: | They propose a framework to synthesize complex charts and reliable reasoning data from scratch. |
| Outcome: | Experimental results show that ChartVerse-8B surpasses existing models in QA and difficulty . lack of high-quality training data hampers development of open-source models . |
DrawEduMath: Evaluating Vision Language Models with Expert-Annotated Students’ Hand-Drawn Math Images (2025.naacl-long)
Copied to clipboard
| Challenge: | DrawEduMath examines the ability of vision language models to handle real-world math problems, such as those encountered in classrooms and tutoring sessions. |
| Approach: | They present DrawEduMath, an English-language dataset of 2,030 images of students’ handwritten responses to math problems. |
| Outcome: | The proposed model can be used to evaluate teachers' QA pairs and 44,362 synthetic QAs derived from teachers' descriptions. |
Towards Automatic Evaluation for Image Transcreation (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing methods for evaluating image transcreation have relied on human evaluation. |
| Approach: | They propose a suite of automatic evaluation metrics inspired by machine translation metrics . they identify cultural relevance, semantic equivalence and visual similarity as critical dimensions of image transcreation . |
| Outcome: | The proposed evaluation metrics agree with human ratings across 7 countries. |
SpaRE: Enhancing Spatial Reasoning in Vision-Language Models with Synthetic Data (2025.acl-long)
Copied to clipboard
| Challenge: | Vision-language models struggle with spatial reasoning, a skill that humans excel at. |
| Approach: | They propose to use a spatial-reasoning Enhanced (SpaRE) VLM to improve spatial reasoning in visual question answering and robotics. |
| Outcome: | The proposed model achieves a 49% performance gain on the What's Up benchmark while maintaining strong results on general tasks. |
Does Chain-of-Thought Reasoning Help Mobile GUI Agents? An Empirical Study (2026.findings-acl)
Copied to clipboard
| Challenge: | Reasoning capabilities have improved vision-language models in domains like math, coding, and visual question-answering, but their impact on real-world applications remains unclear. |
| Approach: | They evaluate six pairs of VLMs by comparing their base and reasoning-enhanced versions across static and interactive benchmarks. |
| Outcome: | The reasoning-enhanced models perform better on static and interactive benchmarks than non-reasoning models. |
DiscoSG: Towards Discourse-Level Text Scene Graph Parsing through Iterative Graph Refinement (2025.emnlp-main)
Copied to clipboard
| Challenge: | Current approaches typically merge sentence-level parsing outputs for discourse input, resulting in fragmented graphs and degraded downstream performance. |
| Approach: | They propose a task for discourse-level text scene graph parsing that merges sentence-level outputs for discourse input and propose 'DiscoSG' a dataset of 400 expert-annotated and 8,430 synthesised multi-sentence caption-graph pairs is used to test the new task. |
| Outcome: | The proposed task improves SPICE by 30% over the baseline while achieving 86 faster inference than existing models. |
CROPE: Evaluating In-Context Adaptation of Vision and Language Models to Culture-Specific Concepts (2025.naacl-long)
Copied to clipboard
| Challenge: | Recent Vision and Language models have shown impressive performance across benchmarks . however, frontier models lack cultural awareness and can affect global cultural diversity . |
| Approach: | They propose a visual question answering benchmark to probe the knowledge of culture-specific concepts and evaluate the capacity for cultural adaptation through contextual information. |
| Outcome: | The proposed model shows large performance disparities between culture-specific and common concepts in the parametric setting. |
Advancement in Graph Understanding: A Multimodal Benchmark and Fine-Tuning of Vision-Language Models (2024.acl-long)
Copied to clipboard
| Challenge: | Graph data organizes complex relationships and interactions between objects . Graph neural networks (GNNs) are becoming more popular in graph learning . |
| Approach: | They propose a new paradigm for interactive and instructional graph data understanding and reasoning . they first evaluate the capabilities of public VLMs in graph learning from multiple aspects . |
| Outcome: | The proposed model achieves an accuracy increase of 5%-15% compared to baseline models . the best-performing model achieve scores comparable to Gemini in GPT-asissted Evaluation . |
VLR-Bench: Multilingual Benchmark Dataset for Vision-Language Retrieval Augmented Generation (2025.coling-main)
Copied to clipboard
Hyeonseok Lim, Dongjae Shin, Seohyun Song, Inho Won, Minjun Kim, Junghun Yuk, Haneol Jang, KyungTae Lim
| Challenge: | Existing evaluation datasets for external knowledge-based VQA lack a capability to determine which passage is useful for answering queries. |
| Approach: | They propose a visual question answering benchmark for vision language models based on retrieval augmented generation (RAG) the proposed benchmark includes five input passages, a capability lacking in previous research. |
| Outcome: | The proposed benchmark includes five input passages and is validated using the state-of-the-art Llama3-based VLM, the Llava-Llamama-3 model. |
CityCube: Benchmarking Cross-view Spatial Reasoning on Vision-Language Models in Urban Environments (2026.acl-long)
Copied to clipboard
Haotian Xu, Yue Hu, Zhengqiu Zhu, Chen Gao, Ziyou Wang, Junreng Rao, Wenhao Lu, Weishi Li, Quanjun Yin, Yong Li
| Challenge: | Existing benchmarks focus on indoor or street settings, overlooking challenges of open-ended urban spaces. |
| Approach: | They propose a benchmark to probe cross-view spatial reasoning capabilities of current VLMs in urban settings. |
| Outcome: | The citycube benchmark examines the performance of current vision-language models in urban environments. |
AbsVis – Benchmarking How Humans and Vision-Language Models “See” Abstract Concepts in Images (2025.emnlp-main)
Copied to clipboard
| Challenge: | Abstract concepts like mercy and peace lack clear visual grounding, and therefore challenge humans and models to provide suitable image representations. |
| Approach: | They propose a dataset of 675 images annotated with 14,175 concept–explanation attributions from humans and two Vision-Language Models where each concept is accompanied by a textual explanation. |
| Outcome: | The proposed dataset compares human and VLM attributions in terms of diversity, abstractness, and alignment, and shows that overlapping concepts are most preferred. |
VisPCO: Visual Token Pruning Configuration Optimization via Budget-Aware Pareto-Frontier Learning for Vision-Language Models (2026.acl-long)
Copied to clipboard
| Challenge: | Existing methods for visual token pruning rely on predefined configurations without determining whether they achieve optimal performance. |
| Approach: | They propose a framework that formulates visual token pruning as a Pareto configuration optimization problem to automatically identify optimal configurations. |
| Outcome: | The proposed framework approximates the empirical Pareto frontier obtained through grid search and generalizes well across pruning methods and VLM architectures. |
Beyond Words: Exploring Cultural Value Sensitivity in Multimodal Models (2025.findings-naacl)
Copied to clipboard
| Challenge: | Using large vision-language models to understand cultural contexts is a critical area of research. |
| Approach: | They conduct a thorough evaluation of multimodal models at different scales, focusing on their alignment with cultural values. |
| Outcome: | The proposed models show that they exhibit sensitivity to cultural values but their performance is highly context-dependent. |
Does Acceleration Cause Hidden Instability in Vision Language Models? Uncovering Instance-Level Divergence Through a Large-Scale Empirical Study (2025.emnlp-main)
Copied to clipboard
| Challenge: | Current acceleration evaluations focus on minimal overall performance degradation . however, accelerated models can exhibit significant changes in instance-level predictions . |
| Approach: | They investigate whether accelerated vision-Language Models can still give the same answers as before . they found that accelerated models changed original answers up to 20% of the time . |
| Outcome: | The results show that accelerated models changed their original answers up to 20% of the time. |
VLM Is a Strong Reranker: Advancing Multimodal Retrieval-augmented Generation via Knowledge-enhanced Reranking and Noise-injected Training (2025.findings-emnlp)
Copied to clipboard
| Challenge: | a significant drawback of Vision-language Models is their reliance on static training data, leading to outdated information and limited contextual awareness. |
| Approach: | They propose a framework with knowledge-enhanced reranking and noise-injected training to improve the VLM's ranking ability. |
| Outcome: | The proposed framework is based on a simple yet effective instruction template and is able to induce its ranking ability and serve it as a reranker to precisely filter the top-k retrieved images. |
“I See What You Did There”: Can Large Vision-Language Models Understand Multimodal Puns? (2026.acl-long)
Copied to clipboard
Naen Xu, Jiayi Sheng, Changjiang Li, Chunyi Zhou, Yuyuan Li, Tianyu Du, Jun Wang, Zhihui Fu, Jinbao Li, Shouling Ji
| Challenge: | Puns are a common form of rhetorical wordplay that exploits polysemy and phonetic similarity to create humor. |
| Approach: | They propose a multimodal pun generation pipeline and a model to evaluate their understanding of puns. |
| Outcome: | The proposed benchmark improves the understanding of multimodal puns by 16.5% in the F1 test. |
Sparrow: Text-Anchored Window Attention with Visual-Semantic Glimpsing for Speculative Decoding in Video LLMs (2026.acl-long)
Copied to clipboard
| Challenge: | Existing speculative decoding models face performance collapse due to key-value cache explosion and context window mismatches. |
| Approach: | They propose a framework that offloads visual computation to the target model by using hidden state reuse. |
| Outcome: | The proposed framework achieves an average speedup of 2.82x even with 25k visual tokens . |
Language-Guided Temporal Token Pruning for Efficient VideoLLM Processing (2025.emnlp-main)
Copied to clipboard
| Challenge: | Current models struggle with long-form videos due to the quadratic complexity of attention mechanisms. |
| Approach: | They propose a model-agnostic framework that leverages temporal cues from queries to prune video tokens. |
| Outcome: | The proposed framework reduces computation by 65% while preserving 97-99% of original performance. |
Value-Spectrum: Quantifying Preferences of Vision-Language Models via Value Decomposition in Social Media Contexts (2025.acl-long)
Copied to clipboard
| Challenge: | Recent advances in Vision-Language Models (VLMs) have broadened the scope of multimodal applications, but evaluations often neglect abstract dimensions such as personality traits and human values. |
| Approach: | They propose a Visual Question Answering (VQA) benchmark based on Schwartz’s value dimensions that capture core human values guiding people’s preferences and actions. |
| Outcome: | The proposed model can be used to evaluate visual question answering (VQA) tasks and to simulate diverse personas. |
VIPHY: Probing “Visible” Physical Commonsense Knowledge (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies have demonstrated that vision-language models can retain and generalize knowledge, but they do not measure their ability to retain it. |
| Approach: | They build an automatic pipeline to derive a knowledge resource for calibrating and probing vision-language models. |
| Outcome: | The proposed model outperforms the pretrained model on size and spatial tasks. |
MAPWise: Evaluating Vision-Language Models for Advanced Map Queries (2025.naacl-long)
Copied to clipboard
| Challenge: | Vision-language models excel at tasks requiring joint understanding of visual information and natural language. |
| Approach: | They propose to use choropleth maps to answer questions from three geographical regions in the United States, India, China as question templates. |
| Outcome: | The proposed model outperforms other models in the area of visual language and visual question answering. |
MuSe: Multi-Stage Graph Reasoning via Vision-Language Models (2026.acl-long)
Copied to clipboard
| Challenge: | Graph Neural Networks (GNNs) and graph transformers are inadequate for tasks with limited generalization. |
| Approach: | They propose a multi-stage graph reasoning framework based on vision-language models that incrementally samples and visualizes task-relevant subgraphs. |
| Outcome: | The proposed framework outperforms existing benchmarks in Graph-related tasks. |
Beyond Single-View Detection: A Dual-Space Reasoning Framework for Interpretable Harmful Meme Understanding (2026.acl-long)
Copied to clipboard
| Challenge: | Existing methods for identifying harmful memes rely on modal alignment or black-box classifiers . BPDMoE-Hate provides visual explanations for viewpoint selection and hierarchical structuring . |
| Approach: | They propose a framework that conceptualizes harmful meme detection as a process of "viewpoint decoupling and hierarchical fusion" they propose BPDMoE-Hate, which generates adversarial binary perspectives via VLMs and incorporates an adaptive viewpoint gating to facilitate viewpoint selection. |
| Outcome: | The proposed framework surpasses existing methods in performance and provides visual explanations for viewpoint selection and hierarchical structuring. |
Aligning VLM Assistants with Personalized Situated Cognition (2025.acl-long)
Copied to clipboard
Yongqi Li, Shen Zhou, Xiaohu Li, Xin Miao, Jintao Wen, Mayi Xu, Jianhao Chen, Birong Pan, Hankun Kang, Yuanyuan Zhu, Ming Zhong, Tieyun Qian
| Challenge: | Existing studies on vision-language models aligned with general human objectives have not been successful because people with diversified backgrounds have different cognition even in the same situation. |
| Approach: | They propose to characterize individuals based on the sociological concept of Role-Set and then evaluate their actions to see whether personalized alignment is achieved. |
| Outcome: | The proposed framework constructs a cognition-aware and action-based reward model for personalized alignment. |
TCPO: Thought-Centric Preference Optimization for Effective Embodied Decision-making (2025.emnlp-main)
Copied to clipboard
Kechen Jiao, Zhirui Fang, Jiahao Liu, Bei Li, Qifan Wang, Xinyu Liu, Junhao Ruan, Zhongjian Qiao, Yifan Zhu, Yaxin Xu, Jingang Wang, Xiu Li
| Challenge: | Existing post-SFT methods for embodied AI are constrained by sparse rewards and action-only optimization, resulting in low sample efficiency, poor consistency, and model degradation. |
| Approach: | They propose to integrate Thought-Centric Preference Optimization (TCPO) into embodied decision-making by transforming sparse reward signals into richer step sample pairs. |
| Outcome: | The proposed approach achieves an average success rate of 26.67% in the ALFWorld environment, and a 6% improvement over RL4VLM. |
Visual Self-Fulfilling Alignment: Shaping Safety-Oriented Personas via Threat-Related Images (2026.acl-long)
Copied to clipboard
| Challenge: | Existing methods require explicit safety labels or contrastive data, yet visual inputs enable harmful outputs. |
| Approach: | They propose a visual self-fulfilling alignment mechanism that fine-tunes vision-language models on neutral VQA tasks without any safety labels. |
| Outcome: | The proposed approach reduces attack success rate, improves response quality, and mitigates over-refusal while preserving general capabilities. |
Multi3Hate: Multimodal, Multilingual, and Multicultural Hate Speech Detection with Vision–Language Models (2025.naacl-long)
Copied to clipboard
| Challenge: | a new study shows that cultural background significantly affects multimodal hate speech moderation models . a limited dataset excludes multi-modal forms of hate and excludes non-English-speaking cultures . the lowest pairwise label agreement between the USA and India is due to cultural factors . |
| Approach: | They use a multimodal and multilingual parallel hate speech dataset to examine cultural differences . they find that cultural background significantly affects multimodal hate speech annotation . |
| Outcome: | The proposed dataset shows that cultural background significantly affects multimodal hate speech annotation. |
CROP: Contextual Region-Oriented Visual Token Pruning (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing VLMs process entire images, leading to excessive visual tokens . redundant image information also introduces a large number of visual token, requiring much higher memory and computation in VLM. |
| Approach: | They propose a framework to prune visual tokens using localization and pruning . they propose CROP to locate local image regions relevant to the query . |
| Outcome: | The proposed framework outperforms existing visual token pruning methods on a wide range of tasks. |
EchoVLM: Dynamic Mixture-of-Experts Vision-Language Model for Universal Ultrasound Intelligence (2026.acl-long)
Copied to clipboard
| Challenge: | Ultrasound is the preferred early cancer screening modality due to non-ionizing radiation, cost-effectiveness, and real-time imaging. |
| Approach: | They propose to use ultrasound-tailored vision-language models with a mixture-of-experts architecture to train ultrasound-specific knowledge across seven anatomical systems. |
| Outcome: | The proposed model outperforms Qwen2-VL by 7.58 BLEU-1 and 3.45 ROUGE-1 points in report generation. |
How Do LLMs and VLMs Understand Viewpoint Rotation Without Vision? An Interpretability Study (2026.acl-long)
Copied to clipboard
Zhen Yang, Ping Jian, Zhongbin Guo, Zuming Zhang, Chengzhi Li, Yonghong Deng, Xinyue Zhang, Wenpeng Lu
| Challenge: | Existing studies on spatial intelligence from the perspective of visual-spatial intelligence have not explored whether visual intelligence alone is sufficient to endow models with spatial intelligence. |
| Approach: | They propose to use a linguistic perspective to investigate spatial intelligence from a theoretical perspective. |
| Outcome: | The proposed model performs poorly on the proposed dataset while human can easily achieve 100% accuracy. |
AI Knows Where You Are: Exposure, Bias, and Inference in Multimodal Geolocation with KoreaGEO (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing benchmarks show coarse granularity, linguistic bias, and a neglect of multimodal privacy risks. |
| Approach: | They propose a benchmark for visual-language models that analyzes social photos to assess location privacy risks. |
| Outcome: | The proposed benchmarks show coarse granularity, linguistic bias, and neglect of privacy risks. |
PII-VisBench: Evaluating Personally Identifiable Information Safety in Vision Language Models Along a Continuum of Visibility (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing evaluations of PII leakage ignore how a subject’s online presence affects privacy alignment. |
| Approach: | They propose a benchmark that evaluates safety through the continuum of online presence by stratifying 200 subjects into four visibility categories: high, medium, low, and zero. |
| Outcome: | The proposed model stratifies 200 subjects into four visibility categories based on the extent and nature of their information available online. |
R-VLM: Region-Aware Vision Language Model for Precise GUI Grounding (2025.findings-acl)
Copied to clipboard
Joonhyung Park, Peng Tang, Sagnik Das, Srikar Appalaraju, Kunwar Yashraj Singh, R. Manmatha, Shabnam Ghadar
| Challenge: | Existing vision-only GUI agents ground elements from large and cluttered screenshots, requiring them to process substantial irrelevant information that compromises their accuracy. |
| Approach: | They propose a visual agent model for GUI automation that leverages zoomed-in region proposals for precise element localization. |
| Outcome: | The proposed approach improves state-of-the-art grounding accuracy by 13% across diverse GUI platforms on the GUI grounding benchmarks ScreenSpot and AgentStudio. |
Vision-Language Models Mistake Head Orientation for Gaze Direction: Nonverbal Conversation Cues (2026.findings-acl)
Copied to clipboard
Zory Zhang, Pinyuan Feng, Bingyang Wang, Tianwei Zhao, Suyang Yu, Qingying Gao, Hokin Deng, Ziqiao Ma, Yijiang Li, Dezhi Luo
| Challenge: | Where someone looks is a nonverbal communication cue that children and adults readily use. |
| Approach: | They used 1,360 real-world photos to construct evaluation stimuli for Vision-Language Models (VLMs) they found a substantial performance gap between VLMs and humans . |
| Outcome: | The proposed model outperforms existing models in predicting gaze direction using head orientation rather than eye appearance. |
ProgressLM: Towards Progress Reasoning in Vision-Language Models (2026.acl-long)
Copied to clipboard
| Challenge: | Existing models for task progress estimation lack long-horizon and dynamic reasoning . estimating how much of a task has been completed requires long-term reasoning based on partial information. |
| Approach: | They propose a benchmark for evaluating progress reasoning from a single observation . they instantiate a two-stage paradigm that combines episodic retrieval with mental simulation . |
| Outcome: | The proposed benchmark improves on 14 VLMs on a small scale and shows common failure patterns. |
Bridging the Visual Gap: Fine-Tuning Multimodal Models with Knowledge-Adapted Captions (2025.naacl-long)
Copied to clipboard
| Challenge: | Recent work focuses on training vision-language models with long, detailed image captions, but small-scale VLMs struggle to balance the richness of these captions with the risk of hallucinations. |
| Approach: | They propose an evaluation framework that breaks down generated captions into individual propositions, assessing each in isolation. |
| Outcome: | The proposed framework outperforms baselines in both automatic metrics and human evaluations on small-scale vision-language models with long, detailed captions. |
What Users Leave Unsaid: Under-Specified Queries Limit Vision-Language Models (2026.findings-acl)
Copied to clipboard
Dasol Choi, Guijin Son, Hanwool Lee, Minhyuk Kim, Hyunwoo Ko, Teabin Lim, Eungyeol Ahn, Jungwhan Kim, Seunghyeok Hong, Youngsook Song
| Challenge: | HAERAE-Vision benchmarks feature clear, explicit prompts but are often informal and underspecified . state-of-the-art models achieve under 50% on original queries, compared to GPT-5 and Gemini 2.5 Pro . |
| Approach: | They propose a benchmark of 653 real-world visual questions from Korean online communities . they find that even state-of-the-art models achieve under 50% on original queries . |
| Outcome: | HAERAE-Vision benchmarks from Korean online communities yield 1,306 query variants . state-of-the-art models achieve under 50% on original queries, compared with smaller models . authors show that query explicitation alone yields 8 to 22 point improvements . |
AdDriftBench: A Benchmark for Detecting Data Drift and Label Drift in Short Video Advertising (2025.findings-emnlp)
Copied to clipboard
Yinghao Song, Xiangji Zeng, Shuai Cui, Lu Sun, Zhaowei Liu, Yuan Yuan, Yulu Wang, Hai Zhou, Zhaohan Gong
| Challenge: | Short video advertising scenarios present unique challenges due to data drift (DD) and label drift (LD). |
| Approach: | They propose to use data drift and label drift to evaluate models under rapidly shifting content distributions and labeling scenarios to assess their generalization capabilities. |
| Outcome: | The proposed model performs moderately in short video advertising contexts, particularly in handling fine-grained semantics and adapting to shifting instructions. |
If CLIP Could Talk: Understanding Vision-Language Model Representations Through Their Preferred Concept Descriptions (2024.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies assume that VLMs prioritize visual attributes to represent concepts. |
| Approach: | They propose a novel approach to characterize features important for VLMs using reinforcement learning. |
| Outcome: | The proposed approach characterizes features that are important for VLMs . it shows that spurious descriptions have a major role in VLM representations despite providing no helpful information. |
Hybrid Self-evolving Structured Memory for Computer-Use Agents (2026.findings-acl)
Copied to clipboard
| Challenge: | despite advances in vision–language models, real-world computer-use tasks remain challenging due to long-horizon workflows, diverse interfaces, and frequent intermediate errors. |
| Approach: | They propose a graph-based memory that couples discrete symbolic nodes with continuous trajectory embeddings. |
| Outcome: | The proposed system outperforms closed-source models in Qwen2.5-VL-7B and Gemini2.5-Pro-Vision on desktop and mobile platforms. |
Self-play through Computational Runtimes improves Chart Reasoning (2025.findings-acl)
Copied to clipboard
| Challenge: | Vision-language models (VLMs) achieve impressive zero-shot performance on multimodal reasoning tasks. |
| Approach: | They propose a self-play programming interface which leverages visual reasoning models to generate code to decompose a complex visual reasoning task in sub-tasks and use it as a tool to solve decomposed tasks. |
| Outcome: | The proposed approach outperforms baselines on difficult chart reasoning benchmarks. |
3D-Aware Vision-Language Models Fine-Tuning with Geometric Distillation (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Vision-Language Models (VLMs) have shown remarkable performance on diverse visual and linguistic tasks, yet they remain limited in their understanding of 3D spatial structures. |
| Approach: | They propose a framework that injects human-inspired geometric cues into pretrained VLMs . they use sparse correspondences, relative depth relations and dense cost volumes . |
| Outcome: | The proposed framework outperforms existing methods on vision-language reasoning and 3D perception benchmarks. |
AgentThink: A Unified Framework for Tool-Augmented Chain-of-Thought Reasoning in Vision-Language Models for Autonomous Driving (2025.findings-emnlp)
Copied to clipboard
Kangan Qian, Sicong Jiang, Yang Zhong, Ziang Luo, Zilin Huang, Tianze Zhu, Kun Jiang, Mengmeng Yang, Zheng Fu, Jinyu Miao, Yining Shi, He Zhe Lim, Li Liu, Tianbao Zhou, Hongyi Wang, Huang Yu, Yifei Hu, Guang Li, Guang Chen, Hao Ye, Lijun Sun, Diange Yang
| Challenge: | Vision-Language Models struggle with hallucinations, inefficient reasoning, and limited real-world validation hinders accurate perception and robust step-by-step reasoning. |
| Approach: | AgentThink integrates Chain-of-Thought reasoning with dynamic, agent-style tool invocation for autonomous driving tasks. |
| Outcome: | Experiments on the DriveLMM-o1 benchmark show AgentThink significantly boosts overall reasoning scores by 53.91% and enhances answer accuracy by 33.54% . |
What Do VLMs NOTICE? A Mechanistic Interpretability Pipeline for Gaussian-Noise-free Text-Image Corruption and Evaluation (2025.naacl-long)
Copied to clipboard
| Challenge: | Vision-Language Models (VLMs) have gained prominence due to their success in solving complex cross-modal tasks. |
| Approach: | They propose a Gaussian-Noise-free pipeline for mechanistic interpretability in VLMs that introduces Semantic Image Pairs corruption, the first visual counterpart to Symmetric Token Replacement for text. |
| Outcome: | The proposed pipeline identifies a set of “universal attention heads” in BLIP and LLaVA that consistently contribute across different tasks and modalities. |
Investigating Cross-Modal Skill Injection: Scenarios, Methods, and Hyperparameters (2026.acl-long)
Copied to clipboard
| Challenge: | Existing research lacks systematic analysis of the applicability and methodology of cross-modal skill injection. |
| Approach: | They investigate the applicability and methodology of cross-modal skill injection by integrating a domain-expert LLM into a VLM. |
| Outcome: | The proposed method enables transfer of domain-specific expertise from Large Language Models (LLMs) to VLMs without incurring additional training data requirements or significant computational overhead. |
Can Textual Unlearning Solve Cross-Modality Safety Alignment? (2024.findings-emnlp)
Copied to clipboard
Trishna Chakraborty, Erfan Shayegani, Zikui Cai, Nael Abu-Ghazaleh, M. Salman Asif, Yue Dong, Amit Roy-Chowdhury, Chengyu Song
| Challenge: | integrating new modalities into large language models creates new attack surface . existing safety training techniques like SFT and RLHF are not feasible in multi-modal settings . |
| Approach: | They explore whether unlearning in the textual domain can be effective for cross-modality safety alignment. |
| Outcome: | The proposed approach reduces the Attack Success Rate (ASR) to less than 8% and preserves the utility. |
Guiding Medical Vision-Language Models with Diverse Visual Prompts: Framework Design and Comprehensive Exploration of Prompt Variations (2025.naacl-long)
Copied to clipboard
| Challenge: | Current vision-language models lack the ability to focus on specific areas designated by humans . a new framework that integrates medical entity extraction, visual prompt generation, and dataset adaptation is proposed to improve visual prompt-guided fine-tuning. |
| Approach: | They propose to use visual prompts to guide and enhance formation of region-specific attention. |
| Outcome: | The proposed framework outperforms state-of-the-art large vision-language models on medical datasets. |
MobileVLM: A Vision-Language Model for Better Intra- and Inter-UI Understanding (2024.findings-emnlp)
Copied to clipboard
Qinzhuo Wu, Weikai Xu, Wei Liu, Tao Tan, Liujian Liujianfeng, Ang Li, Jian Luan, Bin Wang, Shuo Shang
| Challenge: | Recent mobile AI agents based on VLMs lack basic mobile capabilities due to their pre-trained nature. |
| Approach: | They propose a mobile AI agent based on VLMs that includes additional pre-training stages to enhance both intra- and inter-UI understanding. |
| Outcome: | The proposed model outperforms existing VLMs on the Chinese mobile dataset Mobile3M . |
RoboFailRing: Retrieval-Augmented and Language Grounding Failure Detection for VLM-enabled Robotic Manipulation (2026.acl-long)
Copied to clipboard
| Challenge: | RoboFailRing enables timely failure detection during task execution and enhances reasoning accuracy of VLMs. |
| Approach: | They propose a robot-based failure detection system that enables timely failure detection . they evaluate a large-scale simulated dataset and provide a grounded failure report . |
| Outcome: | The proposed method achieves rapid failure detection and returns similarity-based decision on large-scale simulated failures. |
Can you SPLICE it together? A Human Curated Benchmark for Probing Visual Reasoning in VLMs (2025.findings-emnlp)
Copied to clipboard
Mohamad Ballout, Okajevo Wilfred, Seyedalireza Yaghoubi, Nohayr Muhammad Abdelmoneim, Julius Mayer, Elia Bruni
| Challenge: | SPLICE is a benchmark designed to probe event-based reasoning across multiple dimensions. |
| Approach: | They introduce a human-curated benchmark to probe event-based reasoning across multiple dimensions. |
| Outcome: | The proposed benchmark includes 3,381 human-filtered videos spanning 12 categories and 180 sub-categories . results show that state-of-the-art vision-language models struggle to match human performance . |
Prompting Scientific Names for Zero-Shot Species Recognition (2023.emnlp-main)
Copied to clipboard
| Challenge: | We use visionlanguage models (VLMs) to recognize images of common objects in a zero-shot fashion, but it is underexplored how to use CLIP for zero- shot species recognition of highly specialized concepts. |
| Approach: | They propose a method to translate scientific names to common English names and use them in prompts to improve their performance. |
| Outcome: | The proposed method performs poorly for species recognition with prompts that use scientific names, e.g., “a photo of Lepus Timidus” (which is a scientific name in Latin) and additionally use them in the prompts. |
ALDEN: Reinforcement Learning for Active Navigation and Evidence Gathering in Long Documents (2026.acl-long)
Copied to clipboard
| Challenge: | Visually rich documents (VRDs) combine text, tables, and figures within complex, semantically structured layouts. |
| Approach: | They propose a multi-turn reinforcement learning framework that fine-tunes VLMs as interactive agents capable of actively navigating long, visually rich documents. |
| Outcome: | The proposed framework achieves state-of-the-art on five long-document benchmarks. |
BiasDora: Exploring Hidden Biased Associations in Vision-Language Models (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies on social biases focus on a limited set of documented associations, such as gender-profession or race-crime. |
| Approach: | They propose to examine hidden, implicit bias associations across 9 bias dimensions by probing VLMs to uncover hidden, unexamined associations. |
| Outcome: | The proposed methods reveal that biases vary in negativity, toxicity, and extremity. |
VAQUUM: Are Vague Quantifiers Grounded in Visual Data? (2025.findings-acl)
Copied to clipboard
| Challenge: | a dataset containing 20,300 human ratings on quantified statements is used to evaluate the appropriateness of vague quantifiers in visual contexts. |
| Approach: | They use a visual-language-models-based dataset to evaluate the appropriateness of vague quantifiers. |
| Outcome: | The proposed model is based on a visual-visual-language-model-based dataset . it shows that the model is compatible with humans when producing or judging vague quantifiers . |
Beyond Screenshots: Evaluating VLMs’ Understanding of UI Animations (2026.findings-acl)
Copied to clipboard
| Challenge: | Recent studies of Vision Language Models (VLMs) for UI understanding have focused primarily on static screenshots, leaving it unclear how well these models handle dynamic UI animations. |
| Approach: | They evaluate UI animation models' ability to perceive animation effects and interpret animation meaning . they use motion, context, and perceptual cues to probe factors affecting VLM performance . |
| Outcome: | The proposed model can detect primitive motion, but its interpretation is inconsistent . the proposed model is based on 300 annotated UI animation videos . |
From Mimesis to Metamorphosis: Evolving VLM Judges via In-Context Comparing and Knowledge Internalization (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to subjective assessment are inconsistent and inconsistent due to inconsistent scales and inherent preference biases. |
| Approach: | They propose a framework that operationalizes subjective assessment as comparative analysis and internalizes it via Language Buttons. |
| Outcome: | The proposed framework achieves state-of-the-art performance across multiple benchmarks and is scale-steerable. |
Light-weight Fine-tuning Method for Defending Adversarial Noise in Pre-trained Medical Vision-Language Models (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing fine-tuning algorithms for vision-language models are restricted by patient privacy concerns and can contain imperceptible noise. |
| Approach: | They propose a framework to mitigate adversarial noise and mitigate upstream noise during fine-tuning. |
| Outcome: | The proposed framework improves model robustness and transferability while decreasing noise levels negatively impact downstream performance. |
ProcWorld: Benchmarking Large Model Planning in Reachability-Constrained Environments (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing benchmarks for embodied spatial reasoning and long-term planning are non-trivial due to the combinatorial complexity of long-horizon abstract reasoning. |
| Approach: | They propose a large-scale benchmark for partially observable embodied spatial reasoning and long-term planning with large language models and vision language models. |
| Outcome: | The proposed model performs better in 16 task types, 5,000 rooms, and over 10 million evaluation trajectories with diverse data distribution. |
ViCor: Bridging Visual Understanding and Commonsense Reasoning with Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for visual commonsense reasoning (VCR) use pre-trained large language models and pre-training visionlanguage models. |
| Approach: | They propose a collaborative approach where pre-trained LLMs serve as problem classifiers to analyze problem category and either use VLMs to answer directly or actively instruct LLM to gather relevant visual elements to support potential commonsense inferences. |
| Outcome: | The proposed approach outperforms all other methods without in-domain fine-tuning on two VCR benchmark datasets. |
When Seeing Overrides Knowing: Disentangling Knowledge Conflicts in Vision-Language Models (2026.acl-long)
Copied to clipboard
| Challenge: | Vision-language models combine visual and textual information to perform complex tasks. conflicts between internal knowledge and external visual input can lead to hallucinations and unreliable predictions. |
| Approach: | They propose to use a dataset to identify attention heads that deliberately contradict internal commonsense knowledge to resolve cross-modal conflicts. |
| Outcome: | The proposed model can be manipulated to find out which visual inputs are conflicting . the model can then be orientated towards internal parametric knowledge or visual information . |
Repairs in a Block World: A New Benchmark for Handling User Corrections with Multi-Modal Language Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | In dialogue, the addressee may misunderstand the speaker and respond erroneously. |
| Approach: | They collect, analyse, and publicly release a dataset of multi-modal TPR sequences in dialogue . they evaluate several state-of-the-art Vision and Language Models across multiple settings . |
| Outcome: | The proposed model underperforms in a human-robot interaction task compared to humans . the proposed model can benefit from specialised losses targeting relevant tokens . |
Curr-ReFT: Overcoming Training Bottlenecks in Small-scale Vision-Language Models via Curriculum Reinforcement Finetuning (2025.findings-emnlp)
Copied to clipboard
| Challenge: | State-of-the-art vision-language models require massive scaling that limits practical deployment. |
| Approach: | They propose to use supervised fine-tuning to train small-scale vision-language models but face out-of-domain collapse when trained with traditional supervised learning (SFT). |
| Outcome: | Experiments show that curr-reFT achieves state-of-the-art performance across visual tasks in both in- and out-of domain settings and benchmarks. |
MASSV: Multimodal Adaptation and Self-Data Distillation for Speculative Decoding of Vision-Language Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Speculative decoding of vision-language models provides a novel way to accelerate language model inference by enabling a lightweight draft model to propose multiple tokens that a larger target model verifies simultaneously. |
| Approach: | They propose a technique that allows a lightweight draft model to propose multiple tokens that a larger target model verifies simultaneously. |
| Outcome: | The proposed technique increases accepted length by 30% and delivers speedups of up to 1.46x compared to conventional text-only drafting baselines on visually-grounded tasks. |
For-Value: Efficient Forward-Only Data Valuation for finetuning LLMs and VLMs (2026.acl-long)
Copied to clipboard
Wenlong Deng, Qi Zeng, Jiaming Zhang, Minghui Chen, Zixin Ding, Christos Thrampoulidis, Boying Gong, Xiaoxiao Li
| Challenge: | Existing methods for data valuation rely on gradient computations, making them prohibitive for billion-parameter models. |
| Approach: | They propose a forward-only data valuation framework that enables efficient batch-scalable value estimation while maintaining effectiveness. |
| Outcome: | The proposed framework matches or outperforms gradient-based baselines in detecting influential data and mislabeled data while achieving significant efficiency improvements. |
Enhancing Adverse Drug Event Detection with Multimodal Dataset: Corpus Creation and Model Development (2024.findings-acl)
Copied to clipboard
| Challenge: | ADEs are a serious public health concern and cost healthcare systems billions of dollars . despite advancements in healthcare, ADE detection remains a significant challenge . |
| Approach: | They propose a multimodal adverse drug event detection dataset that merges ADE-related textual information with visual aids to enhance patient safety. |
| Outcome: | The proposed dataset integrates ADE-related textual information with visual aids to improve patient safety and healthcare accessibility. |
Prometheus-Vision: Vision-Language Model as a Judge for Fine-Grained Evaluation (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing metrics for long-form text outputs are prone to biases and scaling up is expensive. |
| Approach: | They propose to evaluate VLMs with VLM feedback dataset . they use 15K customized score rubrics to train Prometheus-Vision . |
| Outcome: | The proposed model shows highest correlation with human evaluators and GPT-4V among open-source models. |
Believing without Seeing: Quality Scores for Contextualizing Vision-Language Model Explanations (2026.acl-long)
Copied to clipboard
| Challenge: | Prior work has found that explanations can easily convince users that inaccurate VLM predictions are correct. |
| Approach: | They propose to evaluate two complementary qualities of VLM-generated explanations via two quality scoring functions to improve their accuracy. |
| Outcome: | The proposed explanations improve accuracy on the A-OKVQA, VizWiz, and MMMU-Pro tasks by 11.1%, including a 15.4% reduction in falsely believing incorrect predictions. |
Medical Adaptation of Large Language and Vision-Language Models: Are We Making Progress? (2024.emnlp-main)
Copied to clipboard
| Challenge: | Several studies claim that domain-adaptive pretraining improves performance on downstream medical tasks. |
| Approach: | They compare medical LLMs and VLMs against their corresponding base models . they find that medical Lms outperform their base models in 12.1% of cases . |
| Outcome: | The proposed models outperform their base models on medical questions and tasks in 12.1% of cases and reach a tie in 49.8% of cases. |
Emma-X: An Embodied Multimodal Action Model with Grounded Chain of Thought and Look-ahead Spatial Reasoning (2025.acl-long)
Copied to clipboard
| Challenge: | Visual-Language-Action models lack the ability to generate actionable policies tailored to specific robotic embodiments. |
| Approach: | They propose an embodied multimodal action model with Grounded Chain of Thought and Look-ahead Spatial Reasoning that enhances spatial reasoning and task planning. |
| Outcome: | The proposed model improves on existing baselines in tasks requiring spatial reasoning and grounding reasoning. |
Visualizing Dialogues: Enhancing Image Selection through Dialogue Understanding with Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for dialogue-to-image retrieval are constrained by pre-trained vision language models. |
| Approach: | They leverage the reasoning capabilities of large language models to predict potential features in images to be shared based on dialogue context. |
| Outcome: | The proposed method outperforms existing methods significantly in terms of Recall@k. |
VIGNETTE: Socially Grounded Bias Evaluation for Vision-Language Models (2026.acl-long)
Copied to clipboard
| Challenge: | Existing studies on VLM bias focus on portrait-style images and gender-occupation associations . existing studies ignore broader and more complex social stereotypes and their implied harm . |
| Approach: | They propose a large-scale VQA benchmark for evaluating bias in vision-language models . they use a question-answering framework that spans factuality, perception, stereotyping, and decision making . |
| Outcome: | The proposed framework examines bias in vision-language models using 30M+ images . findings reveal subtle, multifaceted, and surprising stereotypical patterns . |
More Than Meets the Eye: Measuring the Semiotic Gap in Vision-Language Models via Semantic Anchorage (2026.acl-long)
Copied to clipboard
| Challenge: | Vision-Language Models excel at photorealistic generation, but struggle to represent abstract meanings. |
| Approach: | They propose a benchmark that replaces high-fidelity visual detail with schematic iconicity by generating paired, sense-anchored visualizations for literal and idiomatic readings. |
| Outcome: | The proposed benchmark replaces high-fidelity visual detail with schematic iconicity by generating paired, sense-anchored visualizations for literal and idiomatic readings. |
CapArena: Benchmarking and Analyzing Detailed Image Captioning in the LLM Era (2025.findings-acl)
Copied to clipboard
Kanzhi Cheng, Wenpo Song, Jiaxin Fan, Zheng Ma, Qiushi Sun, Fangzhi Xu, Chenyang Yan, Nuo Chen, Jianbing Zhang, Jiajun Chen
| Challenge: | Image captioning has been a challenge for vision-language researchers for decades . current VLMs focus on tasks like visual question answering (YA) but image captioning is not as advanced as expected. |
| Approach: | They evaluate VLMs' performance on image captioning using human annotations . they find that some metrics show high caption-level agreement with humans . |
| Outcome: | The proposed model outperforms open-source models on image captioning . it achieves 93.4% correlation with human rankings at $4 per test . |
Adversary-Aware DPO: Enhancing Safety Alignment in Vision Language Models via Adversarial Training (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Adversary-aware DPO (ADPO) is a training framework that explicitly considers adversary. |
| Approach: | a new framework integrates adversarial training into a pre-trained large language model to enhance safety alignment . adversary-aware DPO provides a framework that explicitly considers adversary . |
| Outcome: | a new training framework outperforms baselines in safety alignment and general utility of large language models. |
IdealGPT: Iteratively Decomposing Vision and Language Reasoning via Large Language Models (2023.findings-emnlp)
Copied to clipboard
Haoxuan You, Rui Sun, Zhecan Wang, Long Chen, Gengyu Wang, Hammad Ayyubi, Kai-Wei Chang, Shih-Fu Chang
| Challenge: | Existing approaches to decompose VL reasoning rely on domain-specific sub-question decomposing models. |
| Approach: | They propose a framework that iteratively decomposes VL reasoning using large language models. |
| Outcome: | The proposed framework outperforms existing models on multiple VL reasoning tasks. |
Scaling Vision-Language Models with Sparse Mixture of Experts (2023.findings-emnlp)
Copied to clipboard
| Challenge: | a study explores the effectiveness of mixture-of-experts (MoE) techniques in scaling vision-language models . alayrac and colleagues demonstrate the effectiveness and performance of MoE in scaling VLMs . |
| Approach: | They propose to use sparsely-gated mixture-of-experts techniques to scale vision-language models . they show that MoE can achieve state-of the-art performance over dense models a range of benchmarks . |
| Outcome: | The proposed approach achieves state-of-the-art performance over dense models of equivalent computational cost. |
Selective “Selective Prediction”: Reducing Unnecessary Abstention in Vision-Language Reasoning (2024.findings-acl)
Copied to clipboard
Tejas Srinivasan, Jack Hessel, Tanmay Gupta, Bill Yuchen Lin, Yejin Choi, Jesse Thomason, Khyathi Chandu
| Challenge: | ReCoVERR reduces the over-abstention of a vision-language system with low tolerance for inaccurate predictions without increasing the error rate of the system’s predictions. |
| Approach: | They propose an inference-time algorithm to reduce the over-abstention of a selective vision-language system without increasing the error rate of the system’s predictions. |
| Outcome: | ReCoVERR reduces the over-abstention of a vision-language system without increasing the error rate of the system’s predictions. |
VADE: Visual Attention Guided Hallucination Detection and Elimination (2025.findings-acl)
Copied to clipboard
| Challenge: | Vision Language Models (VLMs) are prone to hallucinations, generating outputs that lack grounding in the actual visual data. |
| Approach: | They propose a sequence modelling approach to learn complex sequential patterns from transformer attention maps. |
| Outcome: | The proposed approach achieves an average PR-AUC of 80% in hallucination detection on M-HalDetect and an 5% improvement in hallucinosis mitigation on MSCOCO. |
Collaborative Multi-Agent Scripts Generation for Enhancing Imperfect-Information Reasoning in Murder Mystery Games (2026.findings-acl)
Copied to clipboard
| Challenge: | Vision-language models have shown impressive capabilities in perceptual tasks . however, they degrade in complex multi-hop reasoning under multi-player game settings . |
| Approach: | They propose a multi-agent framework for evaluating and synthesizing role-driven game scripts . they use curated and synthetic datasets to model uncertainty and deception . |
| Outcome: | The proposed model significantly boosts the performance of VLMs in narrative reasoning and hidden fact extraction under uncertain, adversarial, and socially complex conditions. |
Why do LLaVA Vision-Language Models Reply to Images in English? (2024.findings-emnlp)
Copied to clipboard
Musashi Hinck, Carolin Holtermann, Matthew Olson, Florian Schneider, Sungduk Yu, Anahita Bhiwandiwalla, Anne Lauscher, Shao-Yen Tseng, Vasudev Lal
| Challenge: | Including an image in a multimodal query significantly increases the likelihood of the model returning an English response regardless of the language of the query. |
| Approach: | They propose a two-pronged approach that combines extensive ablation of the design space with a mechanistic analysis of the models’ internal representations of image and text inputs. |
| Outcome: | The proposed approach reduces the multilingual error by switching the language backbone for a bilingual language model. |
Shaking Up VLMs: Comparing Transformers and Structured State Space Models for Vision & Language Modeling (2024.emnlp-main)
Copied to clipboard
| Challenge: | a task-agnostic visual encoding yields minimal performance gains on grounding, but Transformers outperform Mamba at in-context multimodal retrieval. |
| Approach: | They propose to replace Transformers in Visual Language Models with Mamba, a structured state space model that demonstrates promising performance in sequence modeling. |
| Outcome: | The proposed model outperforms Transformers-based models in captioning, question answering, and reading comprehension. |
Vision-Language Model Fine-Tuning via Simple Parameter-Efficient Modification (2024.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in fine-tuning Vision-Language Models have seen the success of prompt tuning and adapter tuning. |
| Approach: | They propose a method to fine-tune CLIP without introducing any overhead of extra parameters. |
| Outcome: | The proposed method improves CLIP by 7.27% average harmonic mean accuracy. |
Boosting Multi-modal Keyphrase Prediction with Dynamic Chain-of-Thought in Vision-Language Models (2025.emnlp-main)
Copied to clipboard
Qihang Ma, Shengyu Li, Jie Tang, Dingkang Yang, null Chenshaodong, Yingyi Zhang, Chao Feng, Ran Jiao
| Challenge: | Multi-modal keyphrase prediction (MMKP) aims to produce concise, informative phrases that capture the essence of cross-modal inputs. |
| Approach: | They propose to use vision-language models to generate conclusive phrases using multiple modalities of input information. |
| Outcome: | The proposed methods outperform existing methods on absence and unseen scenarios and overestimate model capability due to overlap in training tests. |
LLM-empowered Dynamic Prompt Routing for Vision-Language Models Tuning under Long-Tailed Distributions (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Pre-trained vision-language models (VLMs) often suffer from bias in class-imbalanced scenes. |
| Approach: | They propose a multi-dimensional dynamic prompt routing framework that integrates a knowledge base for classes spanning multiple visual-semantic dimensions. |
| Outcome: | The proposed framework achieves comparable results with current SOTA methods on long-tailed benchmarks, including CIFAR-LT, ImageNet-LT and Places-LT. |
Your Reasoning Benchmark May Not Test Reasoning: Revealing Perception Bottleneck in Abstract Reasoning Benchmarks (2026.acl-long)
Copied to clipboard
| Challenge: | Abstraction and Reasoning Corpus and ARC-AGI are widely used to assess progress in artificial intelligence. |
| Approach: | They propose a two-stage pipeline that separates perception and reasoning . they propose to test this pipeline against standard end-to-end one-stage evaluation . |
| Outcome: | The proposed pipeline separates perception and reasoning, and isolates reasoning from bottlenecks. |
ActPlan-1K: Benchmarking the Procedural Planning Ability of Visual Language Models in Household Activities (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) have been adopted to process textual task description and accomplish procedural planning in embodied AI tasks because of their powerful reasoning ability. |
| Approach: | They propose to evaluate the planning ability of large language models and multi-modal counterfactual vision language models (VLMs) using a multi-factual household activity simulator and a chatGPT task description to evaluate their reasoning ability. |
| Outcome: | The proposed benchmark evaluates the planning ability of multi-modal and counterfactual vision language models on a household activity simulator and a chatGPT task description. |
Semantic Token Reweighting for Interpretable and Controllable Text Embeddings in CLIP (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Despite the varying significance of textual elements within a sentence depending on the context, efforts to account for variation of importance in constructing text embeddings have been lacking. |
| Approach: | They propose a framework for Semantic Token Reweighting to build Interpretable text embeddings which incorporates controllability as well. |
| Outcome: | The proposed framework improves the text encoding process in CLIP by differentially weighting semantic elements based on contextual importance, enabling finer control over emphasis responsive to data-driven insights and user preferences. |
Argus: Benchmarking and Enhancing Vision-Language Models for 3D Radiology Report Generation (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing work on 3D radiograph report generation focuses on 2D images, but 3D medical images provide more comprehensive diagnostic information. |
| Approach: | They propose a comprehensive training recipe for building high-performing VLMs for 3DRRG using a publicly available 3D CT-report dataset. |
| Outcome: | The proposed model achieves superior performance across different model sizes and input 3D medical image resolutions. |
Can VLMs Recall Factual Associations From Visual References? (2025.findings-emnlp)
Copied to clipboard
| Challenge: | a systematic deficiency in the multimodal grounding of Vision Language Models is identified . VLMs can recall factual associations when provided a textual reference to an entity . |
| Approach: | They identify a systematic deficiency in the multimodal grounding of Vision Language Models . they show that VLMs struggle to link their internal knowledge of an entity with its image representation . |
| Outcome: | The study shows that VLMs struggle to link internal knowledge with image representations . the findings provide recommendations for future research . |
Vulnerability of Text-to-Image Models to Prompt Template Stealing: A Differential Evolution Approach (2025.findings-acl)
Copied to clipboard
Yurong Wu, Fangwen Mu, Qiuhong Zhang, Jinjing Zhao, Xinrun Xu, Lingrui Mei, Yang Wu, Lin Shi, Junjie Wang, Zhiming Ding, Yiwei Wang
| Challenge: | Prompt trading has emerged as a significant intellectual property concern in recent years, where vendors entice users by showcasing sample images before selling prompt templates that can generate similar images. |
| Approach: | They propose a prompt-stealing benchmark consisting of 50 templates and 450 images organized into Easy and Hard difficulty levels. |
| Outcome: | The proposed method outperforms baseline methods with an average improvement of over 10%. |
Incorporating Structured Representations into Pretrained Vision & Language Models Using Scene Graphs (2023.emnlp-main)
Copied to clipboard
Roei Herzig, Alon Mendelson, Leonid Karlinsky, Assaf Arbelle, Rogerio Feris, Trevor Darrell, Amir Globerson
| Challenge: | Vision and language models (VLMs) have demonstrated remarkable zero-shot (ZS) performance in a variety of tasks. |
| Approach: | They propose to integrate structured annotations into visual and textual representations to improve VLMs' understanding of compositional scenes. |
| Outcome: | The proposed method improves VLMs on multiple VL datasets with only a mild degradation in ZS capabilities. |
Images Speak Louder than Words: Understanding and Mitigating Bias in Vision-Language Model from a Causal Mediation Perspective (2024.emnlp-main)
Copied to clipboard
| Challenge: | Current methods to learn biases from the perspective of model components are limited by their complexity and performance. |
| Approach: | They propose a framework that incorporates causal mediation analysis to measure and map the pathways of bias generation and propagation within vision-language and multimodal tasks. |
| Outcome: | The proposed framework is applicable to a wide range of vision-language and multimodal tasks and reduces bias by 22.03% and 9.04% in the MSCOCO and PASCAL-SENTENCE datasets. |
VisualEDU: A Benchmark for Assessing Coding and Visual Comprehension through Educational Problem-Solving Video Generation (2025.findings-emnlp)
Copied to clipboard
| Challenge: | VisualEDU is a benchmark to evaluate VLMs' ability to produce coherent video from text . advanced proprietary models show promise, but struggle with increasing task complexity . |
| Approach: | VisualEDU is a benchmark to evaluate VLMs' ability to produce coherent video from text . it integrates meta-prompt learning, visual and code feedback, and a drawing toolkit to enhance output quality. |
| Outcome: | VisualEDU is a benchmark to evaluate VLMs' ability to produce coherent video from text . it integrates meta-prompt learning, visual and code feedback, and a drawing toolkit to improve output quality. |
BANMIME : Misogyny Detection with Metaphor Explanation on Bangla Memes (2025.emnlp-main)
Copied to clipboard
Md Ayon Mia, Akm Moshiur Rahman Mazumder, Khadiza Sultana Sayma, Md Fahim, Md Tahmid Hasan Fuad, Muhammad Ibrahim Khan, Akmmahbubur Rahman
| Challenge: | Existing studies have explored hate speech and general meme classification, but the nuanced identification of misogyny in Bangla memes remains underexplored. |
| Approach: | They propose a Bangla misogynistic meme dataset that includes misos, humor, metaphors and detailed human-written explanations. |
| Outcome: | The proposed dataset is the first comprehensive dataset of misogynistic Bangla memes . it includes misos, humor categories, metaphor localization, and detailed human-written explanations based on 2,000 culturally grounded samples . |
Retrieval-augmented GUI Agents with Generative Guidelines (2025.emnlp-main)
Copied to clipboard
| Challenge: | GUI agents powered by vision-language models struggle with real-world tasks due to their complex nature and limited training data. |
| Approach: | They propose a lightweight vision-language model that leverages web tutorials at inferencetime to synthesize GUI agents. |
| Outcome: | The proposed agent outperforms baseline GUI agents and surpasses other inference baselines by 2.6% to 13.3% across two model sizes. |
FTibSuite: A Comprehensive Resource Suite for Tibetan Vision–Language Modeling (2026.findings-acl)
Copied to clipboard
| Challenge: | FTibSuite provides an end-to-end training-and-evaluation workflow for vision–language models . Tibetan is underserved due to the lack of infrastructure for reproducible training and evaluation. |
| Approach: | They propose a resource-centric workflow for Tibetan VLMs that provides an end-to-end training-and-evaluation workflow and human-verified multimodal annotations. |
| Outcome: | FTibSuite provides an end-to-end training-and-evaluation workflow and human-verified multimodal annotations. |
AI Sees Your Location—But With A Bias Toward The Wealthy World (2025.emnlp-main)
Copied to clipboard
| Challenge: | Visual Language Models (VLMs) have shown remarkable performance across various tasks, particularly in recognizing geographic information from images. |
| Approach: | They propose to use 1,200 images paired with detailed geographic metadata to evaluate VLMs' performance. |
| Outcome: | The models achieve 53.8% accuracy in city prediction, but exhibit significant biases in regional tasks. |
Iterative Prompt Refinement for Safer Text-to-Image Generation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing safety methods for text-to-image models ignore the images produced . this can result in unsafe outputs or unnecessary changes to already safe prompts . |
| Approach: | They propose an iterative prompt refinement algorithm that uses Vision Language Models to analyze prompts and generated images. |
| Outcome: | The proposed method improves safety while maintaining user intent and reliability comparable to existing methods. |
Does Visual Grounding Enhance the Understanding of Embodied Knowledge in Large Language Models? (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Despite significant progress in multimodal language models, it remains unclear whether visual grounding enhances their understanding of embodied knowledge compared to text-only models. |
| Approach: | They propose to assess vision-language models’ perceptual abilities across different sensory modalities through vector comparison and question-answering tasks with over 1,700 questions. |
| Outcome: | The proposed benchmark assesses the models’ perceptual abilities across different sensory modalities through vector comparison and question-answering tasks with over 1,700 questions. |
MegaPairs: Massive Data Synthesis for Universal Multimodal Retrieval (2025.acl-long)
Copied to clipboard
Junjie Zhou, Yongping Xiong, Zheng Liu, Ze Liu, Shitao Xiao, Yueze Wang, Bo Zhao, Chen Jason Zhang, Defu Lian
| Challenge: | despite the growing demand for multimodal retrieval, there is a lack of training data. |
| Approach: | They propose a data synthesis method that leverages vision language models and open-domain images to generate high-quality data. |
| Outcome: | The proposed method outperforms baseline models on 70 more datasets and can scale up. |
Med-VRAgent: A Framework for Medical Visual Reasoning-Enhanced Agents (2025.emnlp-main)
Copied to clipboard
| Challenge: | Visual Language Models (VLMs) have shown strong performance in tasks like radiology report generation but struggle with hallucinations, vague descriptions, Inconsistent logic and poor localization. |
| Approach: | They propose a framework for medical visual reasoning based on Visual Guidance and Self-Reward paradigms and Monte Carlo Tree Search to improve the model's visual reasoning capabilities. |
| Outcome: | The proposed framework outperforms existing models on multiple medical VQA benchmarks. |
PunMemeCN: A Benchmark to Explore Vision-Language Models’ Understanding of Chinese Pun Memes (2025.emnlp-main)
Copied to clipboard
| Challenge: | Pun memes combine wordplay with visual elements to create humor, irony, or other rhetorical effects. |
| Approach: | They propose a benchmark to assess Chinese pun memes' processing capabilities across three progressive tasks: pun meme detection, sentiment analysis, and chat-driven meme response. |
| Outcome: | The proposed model can detect pun memes, analyze sentiments, and respond to chats, while ignoring homophone wordplay. |
Reasoning Paths with Reference Objects Elicit Quantitative Spatial Reasoning in Large Vision-Language Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | Despite recent advances in visual language models, their ability to quantitatively reason about object sizes and distances remains underexplored. |
| Approach: | They propose a manually annotated benchmark of 241 questions designed for quantitative spatial reasoning and a zero-shot prompting technique that encourages VLMs to use reference objects as visual cues. |
| Outcome: | The proposed technique improves the performance of the top-performing VLMs by 19 points when a reasoning path using a reference object emerges naturally in the response. |
FAER: Benchmarking VLMs for Failure-Aware Embodied Reasoning (2026.findings-acl)
Copied to clipboard
| Challenge: | Visual-language models (VLMs) are the core component of embodied agents in perceiving the environment and making decisions. |
| Approach: | They propose a failure-aware benchmark to evaluate the performance of visual language models (VLMs) in long-horizon tasks. |
| Outcome: | The proposed benchmark evaluates the performance of 16 widely utilized VLMs and 4 LLMs for FAER tasks. |
Granular Privacy Control for Geolocation with Vision Language Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | Vision Language Models (VLMs) are rapidly advancing in their capability to answer information-seeking questions. |
| Approach: | They develop a benchmark to evaluate the ability of VLMs to moderate geolocation dialogues with users. |
| Outcome: | a new benchmark evaluates the ability of VLMs to moderate geolocation conversations with users. |
ContextBLIP: Doubly Contextual Alignment for Contrastive Image Retrieval from Linguistically Complex Descriptions (2024.findings-acl)
Copied to clipboard
Honglin Lin, Siyu Li, Guoshun Nan, Chaoyue Tang, Xueting Wang, Jingxin Xu, Rong Yankai, Zhouzhili Zhouzhili, Yutong Gao, Qimei Cui, Xiaofeng Tao
| Challenge: | Existing approaches to image retrieval from contextual descriptions (IRCD) lag behind human performance in IRCD. |
| Approach: | They propose a method that relies on a doubly contextual alignment scheme for challenging IRCD. |
| Outcome: | The proposed method can yield comparable results with GPT-4V, despite fewer parameters. |
MVTamperBench: Evaluating Robustness of Vision-Language Models (2025.findings-acl)
Copied to clipboard
Amit Agarwal, Srikant Panda, Angeline Charles, Hitesh Laxmichand Patel, Bhargava Kumar, Priyaranjan Pattnayak, Taki Hasan Rafi, Tejaswini Kumar, Hansa Meghwani, Karan Gupta, Dong-Kyu Chae
| Challenge: | Multimodal Large Language Models (MLLMs) have been a key advance in video understanding but their vulnerability to adversarial tampering remains underexplored. |
| Approach: | They evaluate MLLMs against five prevalent tampering techniques to assess their robustness . they use a tampered video format to examine the vulnerability of ML models . |
| Outcome: | The benchmark evaluates MLLMs against five prevalent tampering techniques based on 19 video manipulation tasks. |
e5-omni: Explicit Cross-modal Alignment for Omni-modal Embeddings (2026.findings-acl)
Copied to clipboard
| Challenge: | Recent omni-modal embeddings rely heavily on implicit alignment from pretrained visionlanguage models. |
| Approach: | They propose a lightweight explicit alignment recipe that adapts off-the-shelf VLMs into robust omni-modal embedding models. |
| Outcome: | The proposed model improves on MMEB-V2 and AudioCaps with a lightweight explicit alignment recipe. |
Unraveling the Truth: Do VLMs really Understand Charts? A Deep Dive into Consistency and Robustness (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Chart question answering (CQA) is a crucial area of Visual Language Understanding. |
| Approach: | They evaluate the robustness and consistency of current Visual Language Models on a dataset encompassing diverse question categories and chart formats. |
| Outcome: | The proposed models handle varying levels of chart and question complexity and are robust across different visual representations of the same underlying data. |
V-ALPHASOCIAL: Benchmark and Self-Reflective Chain-of-Thought Generation for Visual Social Commonsense Reasoning (2025.findings-acl)
Copied to clipboard
Zongyu Lin, Zhikun Xu, Xiaohan Song, Yixin Wan, Xingcheng Yao, Tsung-Han Lin, Selina Song, Pranav Subbaraman, Ben Zhou, Kai-Wei Chang, Yizhou Sun
| Challenge: | Social commonsense reasoning is a multimodal task that requires both textual and visual cues. |
| Approach: | They propose a method that integrates visual cues into social commonsense reasoning tasks. |
| Outcome: | The proposed method improves social commonsense reasoning on a multimodal foundation model. |
Mixture-of-Prompt-Experts for Multi-modal Semantic Understanding (2024.lrec-main)
Copied to clipboard
| Challenge: | Multimodal semantic understanding is crucial for developing machines capable of interpreting complex interplay of text and visual information. |
| Approach: | They propose a multi-modal soft prompt framework that integrates three experts of soft prompts . they propose sarcasm detection and sentiment analysis tasks that are critical for few-shot learning . |
| Outcome: | The proposed model outperforms the 8.2B model InstructBLIP with 2% parameters . it significantly outperformed other prompt methods on VLMs or task-specific methods . |
Deep Temporal Reasoning in Video Language Models: A Cross-Linguistic Evaluation of Action Duration and Completion through Perfect Times (2025.acl-long)
Copied to clipboard
| Challenge: | Experimental results show that video-language models struggle to mirror human-like temporal reasoning grounded in video . Sequential events are not simply arranged chronologically; rather, one event triggers the next upon reaching its completion. |
| Approach: | They propose a quadrilingual dataset to assess temporal reasoning in video-language models . they pair everyday activity videos with event completion labels and perfectivity distractors . |
| Outcome: | The perfect times dataset examines whether video-language models comprehend temporal dynamics . it combines everyday activity videos with event completion labels and perfectivity distractors . results show that state-of-the-art models struggle to mirror human-like temporal reasoning . |
OS-Symphony: A Holistic Framework for Robust and Generalist Computer-Using Agents (2026.acl-long)
Copied to clipboard
Bowen Yang, Kaiming Jin, Zhenyu Wu, Zhaoyang Liu, Qiushi Sun, Zehao Li, JingJing Xie, Zhoumianze Liu, Fangzhi Xu, Kanzhi Cheng, Yian Wang, Qingyun Li, Yu Qiao, Zun Wang, Zichen Ding
| Challenge: | Vision-Language Models (VLMs) lack visual-aware tutorial retrieval and historical visual context curation and pruning. |
| Approach: | They propose a framework that integrates an orchestrator and a Reflection-Memory Agent for robust automation. |
| Outcome: | Experimental results show that OS-Symphony delivers substantial performance gains across model scales. |
PALM: Few-Shot Prompt Learning for Audio Language Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | Audio-Language Models (ALMs) have recently achieved remarkable success in zero-shot audio recognition tasks, which match features of audio waveforms with class-specific text prompt features. |
| Approach: | They propose a method which optimizes the feature space of the text encoder branch and optimizes audio waveform features with text prompt features. |
| Outcome: | The proposed method outperforms existing methods while being less demanding. |
See2Refine: Vision-Language Feedback Improves LLM-Based eHMI Action Designers (2026.acl-long)
Copied to clipboard
| Challenge: | External Human-Machine Interfaces (eHMIs) are emerging as promising solutions to address this communication gap. |
| Approach: | They propose a framework that uses vision-language models (VLMs) for perceptual evaluation as automated visual feedback to improve an LLM-based eHMI action designer. |
| Outcome: | The proposed framework outperforms prompt-only LLM designers and manually specified baselines in three eHMI modalities and multiple LLM model sizes. |
Preserving Multi-Modal Capabilities of Pre-trained VLMs for Improving Vision-Linguistic Compositionality (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing fine-tuning approaches for compositional understanding compromise performance in zero-shot multi-modal tasks. |
| Approach: | They propose a method to enhance compositional understanding in pre-trained vision and language models without sacrificing performance in zero-shot multi-modal tasks. |
| Outcome: | The proposed method achieves compositionality on par with state-of-the-art models and retains strong multi-modal capabilities. |
GUICourse: From General Vision Language Model to Versatile GUI Agent (2025.acl-long)
Copied to clipboard
Wentong Chen, Junbo Cui, Jinyi Hu, Yujia Qin, Junjie Fang, Yue Zhao, Chongyi Wang, Jun Liu, Guirong Chen, Yupeng Huo, Yuan Yao, Yankai Lin, Zhiyuan Liu, Maosong Sun
| Challenge: | Graphical User Interfaces (GUIs) are a pivotal medium for human-computer interaction. |
| Approach: | They propose a series of datasets for training visual-based GUI agents using general VLMs. |
| Outcome: | The proposed GUICourse datasets show that even a small-sized GUI agent performs better on GUI tasks. |
Evaluating Visual and Cultural Interpretation: The K-Viscuit Benchmark with Human-VLM Collaboration (2025.acl-long)
Copied to clipboard
| Challenge: | Existing approaches to creating inclusive vision-language models rely on human annotators, making it labor-intensive and creating cognitive burdens. |
| Approach: | They propose a semi-automated framework for constructing cultural VLM benchmarks . they use an annotated sample of Korean culture to generate questions . |
| Outcome: | The proposed framework is based on a Korean culture dataset and shows that open-source models lag behind proprietary ones in understanding Korean culture. |
Leveraging Large Models to Evaluate Novel Content: A Case Study on Advertisement Creativity (2025.emnlp-main)
Copied to clipboard
| Challenge: | Evaluating creativity is challenging, even for humans, because of its subjectivity and complex cognitive processes. |
| Approach: | They propose a set of tasks to break down visual advertisement creativity into atypicality and originality with fine-grained annotations by humans. |
| Outcome: | The proposed tasks demonstrate the promise and challenges of using VLMs for automated creativity assessment. |
MTabVQA: Evaluating Multi-Tabular Reasoning of Language Models in Visual Space (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing benchmarks address single tables or non-visual data, leaving a critical gap . MTabVQA comprises 3,745 complex question-answer pairs . |
| Approach: | They propose a benchmark specifically designed for multi-tabular visual question answering that measures the ability to parse diverse table images and correlate information across them. |
| Outcome: | The proposed benchmarks show that fine-tuning VLMs with MTabVQA-Instruct significantly improves their reasoning abilities. |
Attention as Selector: Unlocking VLM Attention for Long Document Page Retrieval (2026.acl-long)
Copied to clipboard
| Challenge: | Existing page-level retrieval methods lack query–page interaction before similarity scoring . Existing methods require large-scale datasets to align visual and textual embeddings . |
| Approach: | They propose a retrieval framework that utilizes attention mechanisms inside VLMs for page selection. |
| Outcome: | The proposed retrieval framework outperforms embedding-based retrieval methods on four long-document benchmarks. |
Ascending the Infinite Ladder: Benchmarking Spatial Deformation Reasoning in Vision-Language Models (2026.acl-long)
Copied to clipboard
Jiahuan Zhang, Shunwen Bai, Tianheng Wang, KaiWen Guo, Zijia Song, Hanqing WU, Guozheng Rao, Kai Han, Kaicheng Yu
| Challenge: | Existing benchmarks explore aspects of threedimensional spatial reasoning and visual-language reasoning in dynamic environments, but they are unable to perform well on 3D spatial deformation reasoning. |
| Approach: | They propose to use a ladder competition format to assess the model's spatial deformation reasoning abilities to determine its performance. |
| Outcome: | The proposed framework assesses the performance of Vision-Language Models in spatial deformation reasoning tasks. |
Eliciting In-Context Learning in Vision-Language Models for Videos Through Curated Data Distributional Properties (2024.emnlp-main)
Copied to clipboard
| Challenge: | Emergent In-context Learning on Videos induces in-contact learning over video and text . eILeV-trained models outperform other off-the-shelf VLMs in few-shot video narration for novel, rare actions. |
| Approach: | They implement Emergent In-context Learning on Videos (EILeV) that induces in-contact learning over video and text by capturing key properties of pre-training data. |
| Outcome: | The proposed training paradigm outperforms off-the-shelf VLMs in few-shot video narration for novel, rare actions. |
Creating a Lens of Chinese Culture: A Multimodal Dataset for Chinese Pun Rebus Art Understanding (2025.findings-acl)
Copied to clipboard
Tuo Zhang, Tiantian Feng, Yibin Ni, Mengqin Cao, Ruying Liu, Kiana Avestimehr, Katharine Butler, Yanjun Weng, Mi Zhang, Shrikanth Narayanan, Salman Avestimehr
| Challenge: | a new study examines the performance of large vision-language models in understanding art . the Pun Rebus Art Dataset is a multimodal dataset for art understanding rooted in traditional Chinese culture . |
| Approach: | They propose a multimodal dataset for art understanding deeply rooted in traditional Chinese culture . they aim to facilitate the development of VLMs that can better understand culturally specific content . |
| Outcome: | The proposed dataset shows that state-of-the-art VLMs struggle with these tasks . the data will facilitate the development of VLM models that can better understand culturally specific content . |
ChemVLR: Prioritizing Reasoning in Perception for Chemical Vision-Language Understanding (2026.findings-acl)
Copied to clipboard
| Challenge: | Currently, vision-Language Models are optimized for direct visual question-answering tasks. |
| Approach: | They propose a visual-language-based VLM that prioritizes reasoning within the perception process. |
| Outcome: | The proposed model outperforms existing models and domain-specific open-source models in the chemical domain. |
Preserving Language Capabilities in Vision-Language Models via Representation Regulation (2026.findings-acl)
Copied to clipboard
| Challenge: | Vision-Language Models (VLMs) provide a unified framework to process both text-only and vision-language tasks. |
| Approach: | They propose a method to reduce the distance between visual and textual representations by introducing a Representation Distribution Difference (RDD) loss. |
| Outcome: | Empirical evidence shows that finetuning VLMs on vision-language data has degraded language capabilities. |
Mind the Gesture: Evaluating AI Sensitivity to Culturally Offensive Non-Verbal Gestures (2025.acl-long)
Copied to clipboard
| Challenge: | a dataset of 288 gesture-country pairs is used to evaluate AI systems' cultural awareness of offensive gestures and nonverbal signs. |
| Approach: | They use a dataset of 288 gesture-country pairs annotated for offensiveness, cultural significance, and contextual factors across 25 gestures and 85 countries. |
| Outcome: | The proposed dataset analyzes 288 gesture-country pairs across 25 gestures and 85 countries. |
CaMMT: Benchmarking Culturally Aware Multimodal Machine Translation (2025.findings-emnlp)
Copied to clipboard
Emilio Villa-Cueva, Sholpan Bolatzhanova, Diana Turmakhan, Kareem Elzeky, Henok Biadglign Ademtew, Alham Fikri Aji, Vladimir Araujo, Israel Abebe Azime, Jinheon Baek, Frederico Belcavello, Fermin Cristobal, Jan Christian Blaise Cruz, Mary Dabre, Raj Dabre, Toqeer Ehsan, Naome A Etori, Fauzan Farooqui, Jiahui Geng, Guido Ivetta, Thanmay Jayakumar, Soyeong Jeong, Zheng Wei Lim, Aishik Mandal, Sofía Martinelli, Mihail Minkov Mihaylov, Daniil Orel, Aniket Pramanick, Sukannya Purkayastha, Israfel Salazar, Haiyue Song, Tiago Timponi Torrent, Debela Desalegn Yadeta, Injy Hamed, Atnafu Lambebo Tonja, Thamar Solorio
| Challenge: | a human-curated benchmark of over 5,800 triples of images is used to evaluate multimodal translation systems. |
| Approach: | They introduce a human-curated benchmark of over 5,800 triples of images along with parallel captions in English and regional languages. |
| Outcome: | The results show that visual context improves translation quality in culturally-specific items . |
AIM-CoT: Active Information-driven Multimodal Chain-of-Thought for Vision-Language Reasoning (2026.acl-long)
Copied to clipboard
| Challenge: | Existing methods for I-MCoT fail to capture dynamic needs of vision-language models . existing methods rely on attention signals, which are unreliable under severe granularity imbalance between brief textual query and informative image. |
| Approach: | They propose a framework that integrates specially selected visual evidence into the context of Vision-Language Models (VLMs) they propose 'AIM-CoT' to improve evidence selection and insertion triggering . |
| Outcome: | Experiments across three benchmarks and four backbones demonstrate the proposed framework’s consistent superiority. |
Socratic-MCTS: Test-Time Visual Reasoning by Asking the Right Questions (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent research in vision-language models has centered around the possibility of equipping them with implicit long-form chain-of-thought reasoning via distillation and reinforcement learning. |
| Approach: | They propose a Monte Carlo Tree Search-inspired algorithm that injects subquestion–subanswer pairs into the model’s output stream to elicit hidden knowledge and induce long reasoning traces. |
| Outcome: | The proposed method yields a 2% improvement on MMMU-PRO, including a significant 9% gain in Liberal Arts. |
DocLens: A Tool-Augmented Multi-Agent Framework for Long Visual Document Understanding (2026.acl-long)
Copied to clipboard
| Challenge: | Existing approaches to localizing evidence from long visual documents fail on a fundamental challenge: evidence localization. |
| Approach: | They propose a tool-augmented multi-agent framework that “zooms in” on evidence like a lens. |
| Outcome: | The proposed framework achieves state-of-the-art performance on MMLongBench-Doc and FinRAGBench-V, surpassing even human experts. |
mrCAD: Multimodal Communication to Refine Computer-aided Designs (2025.findings-emnlp)
Copied to clipboard
William P McCarthy, Saujas Vaduguru, Karl D.d. Willis, Justin Matejka, Judith E Fan, Daniel Fried, Yewen Pu
| Challenge: | generative AI excels at creating artifacts in a single turn but can struggle to make precise refinements that match our design intent. |
| Approach: | They propose to use multi-turn interactions to iterate and refine computer-aided designs (CADs) they use text and drawing to communicate with each other over multiple rounds of interaction . |
| Outcome: | mrCAD consists of 6,082 communication games, 15,163 instruction-execution rounds, played between 1,092 pairs of humans. |
Token-level Inference-Time Alignment for Vision-Language Models (2026.findings-acl)
Copied to clipboard
Kejia Chen, Junjun Zheng, Jiawen Zhang, Manxi Lin, Xiao Pan, Jiacong Hu, Jian Lou, Zunlei Feng, Mingli Song
| Challenge: | Vision-Language Models (VLMs) often prioritize linguistic fluency over visual fidelity . despite widespread adoption, VLMs often exhibit a critical failure mode: hallucination . |
| Approach: | They propose a framework for Token-level Inference-Time Alignment that steers the decoding process without updating the base model parameters. |
| Outcome: | The proposed framework improves performance on 13 benchmarks across architectures . it boosts LLaVA-1.5-7B by 8.6% on MMVet and achieves a 74.0 MMStar score . |
Looking Beyond the Pixels: Evaluating Visual Metaphor Understanding in VLMs (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Visual metaphors are a complex vision–language phenomenon that requires both perceptual and conceptual reasoning to understand. |
| Approach: | They introduce a visual metaphor dataset featuring 2177 synthetic and 350 human-annotated images and benchmark several SOTA VLMs on two tasks: Visual Metaphor Captioning (VMC) and Visual Metamorphosis VQA (VM-VQA). |
| Outcome: | The proposed model outperforms standard few-shot baselines on visual metaphors and VM-VQA tasks. |
HomoGraphAdapter: A Homogeneous Graph Neural Network as an Effective Adapter for Vision-Language Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing adaptation methods overlook structural knowledge between text and image modalities or create overly complex graphs containing redundant information for alignment. |
| Approach: | They propose a method to adapt visual models to downstream tasks using text and image modalities. |
| Outcome: | The proposed method improves classification accuracy by 1.51% for 1-shot and 0.74% for 16-shot on 11 datasets. |
SteerVLM: Robust Model Control through Lightweight Activation Steering for Vision Language Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | SteerVLM is a lightweight steering module designed to guide Vision-Language Models (VLMs) towards outputs that better adhere to desired instructions. |
| Approach: | They propose a lightweight steering module that learns from latent embeddings of paired prompts encoding target and converse behaviors to dynamically adjust activations connecting the language modality with image context. |
| Outcome: | The proposed steering module outperforms existing intervention techniques on steering and hallucination mitigation benchmarks for VLMs. |
SmartTrim: Adaptive Tokens and Attention Pruning for Efficient Vision-Language Models (2024.lrec-main)
Copied to clipboard
Zekun Wang, Jingchang Chen, Wangchunshu Zhou, Haichao Zhu, Jiafeng Liang, Liping Shan, Ming Liu, Dongliang Xu, Qing Yang, Bing Qin
| Challenge: | Experimental results show that SmartTrim accelerates the original model by 2-3 times with minimal performance degradation. |
| Approach: | They propose an adaptive acceleration framework which prunes redundant token representations and attention heads within each layer of the original model. |
| Outcome: | The proposed framework accelerates the original model by 2-3 times with minimal performance degradation across vision-language tasks. |
LPOI: Listwise Preference Optimization for Vision Language Models (2025.acl-long)
Copied to clipboard
| Challenge: | Existing methods for aligning large VLMs with human preferences often overfit to textual information or exacerbate hallucinations. |
| Approach: | They propose an object-aware listwise preference optimization for reducing hallucinations in VLMs . they mask a critical object in an image and interpolate the masked region to form more complete images . |
| Outcome: | The proposed method outperforms existing methods in reducing hallucinations and enhancing performance on MMHalBench, AMBER, and Object HalBench. |
RG-VQA: Leveraging Retriever-Generator Pipelines for Knowledge Intensive Visual Question Answering (2025.findings-emnlp)
Copied to clipboard
Settaluri Lakshmi Sravanthi, Pulkit Agarwal, Debjyoti Mondal, Rituraj Singh, Subhadarshi Panda, Ankit Mishra, Kiran Pradeep, Srihari K B, Godawari Sudhakar Rao, Pushpak Bhattacharyya
| Challenge: | Existing methods to improve the reasoning capabilities of VQA systems are limited due to complexity of graph neural networks and end-to-end training. |
| Approach: | They propose a method to integrate Dense Passage Retrievers with Vision Language Models to boost the reasoning capabilities of VQA systems. |
| Outcome: | The proposed method outperforms human accuracy and GPT-4 in the ScienceQA dataset. |
Jigsaw-Puzzles: From Seeing to Understanding to Reasoning in Vision-Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing vision-language models lack spatial reasoning capability, despite their ability to comprehend spatial arrangements and model structural relations. |
| Approach: | They propose a benchmark to evaluate vision-language models' spatial perception, structural understanding, and reasoning capabilities by minimizing reliance on domain-specific knowledge. |
| Outcome: | The proposed benchmark is based on 1,100 carefully curated real-world images with high spatial complexity. |
Selective Test-Time Debiasing for CLIP via Reward Gating (2026.acl-long)
Copied to clipboard
| Challenge: | Existing methods for debiasing use uniform bias corrections across all input queries . weak debiases retains bias in sensitive queries, while weak dealiases in biased ones . |
| Approach: | They propose a framework that selectively applies debiasing based on input sensitivity . RG-TTA adaptively triggers fairness regularization based upon bias sensitivity of each input . |
| Outcome: | Experiments show that debiasing improves zero-shot performance while maintaining fairness . weak debiased queries distort semantically meaningful information while weak ones fail to mitigate stereotypes . |
Don’t Act Blindly: Robust GUI Automation via Action-Effect Verification and Self-Correction (2026.acl-long)
Copied to clipboard
Yuzhe Zhang, Xianwei Xue, Xingyong Wu, Mengke Chen, Chen Liu, Xinran He, Run Shao, Feiran Liu, Huanmin Xu, Qiutong Pan, Haiwei Wang
| Challenge: | Existing GUI agents assume deterministic environment responses, generating actions without verifying whether previous operations succeeded. |
| Approach: | They propose a GUI agent that explicitly models action outcomes and recovery under noisy environments. |
| Outcome: | The proposed agent reduces failure loops and improves recovery success in noisy environments while maintaining competitive standard task performance. |
FOCUS: Evaluating Pre-trained Vision-Language Models on Underspecification Reasoning (2025.acl-long)
Copied to clipboard
| Challenge: | a new dataset evaluates whether vision-language models have underspecification reasoning abilities . underspecifications are often left incomplete or vague, and are often ignored for mutual understanding . |
| Approach: | They propose a probing dataset to evaluate whether VLMs have underspecification reasoning . they find that pre-trained vision-language models lack this ability . |
| Outcome: | The proposed probing dataset shows that pre-trained vision-language models lack underspecification reasoning abilities. |
M2-TabFact: Multi-Document Multi-Modal Fact Verification with Visual and Textual Representations of Tabular Data (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing fact-checking systems that can reason over structured data are inefficient compared to humans. |
| Approach: | They propose a multi-modal table-based fact verification task that requires reasoning over visual and textual representations of structured data. |
| Outcome: | The proposed model can reason over visual and textual representations of structured data. |
Weaving Context Across Images: Improving Vision-Language Models through Focus-Centric Visual Chains (2025.acl-long)
Copied to clipboard
| Challenge: | Existing vision-language models struggle to disentangle information scattered across complex visual inputs, leading to performance degradation. |
| Approach: | They propose a focus-centric visual chain paradigm that enhances VLMs’ perception, comprehension, and reasoning abilities in multi-image scenarios. |
| Outcome: | The proposed approach achieves average performance gains of 3.16% and 2.24% across two distinct model architectures, without compromising the general vision-language capabilities. |
Mixed Signals: Decoding VLMs’ Reasoning and Underlying Bias in Vision-Language Conflict (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Vision-language models have demonstrated impressive performance by effectively integrating visual and textual information to solve complex tasks. |
| Approach: | They build upon existing benchmarks to create five datasets containing mismatched image-text pairs and examine how they reason over visual and textual data . |
| Outcome: | The proposed model reasoned over visual and textual data in real-world applications but not in the visual and visual descriptions. |
Spec-VLA: Speculative Decoding for Vision-Language-Action Models with Relaxed Acceptance (2025.emnlp-main)
Copied to clipboard
| Challenge: | Visual Language Models (VLMs) have significant parameter size and autoregressive (AR) decoding nature impose considerable computational demands on VLA models. |
| Approach: | They propose a framework to relax acceptance utilizing the relative distances represented by the action tokens of the VLA model. |
| Outcome: | Empirical results show that the proposed framework improves the speed of the prediction task by 44%. |
VLURes: Benchmarking Long-Text Grounding and Cross-Lingual Robustness in Vision Language Models (2026.findings-acl)
Copied to clipboard
| Challenge: | ***VLURes** provides a practical testbed for long-text grounding and multilingual robustness in web-realistic agent settings. |
| Approach: | They propose a multilingual benchmark for evaluating vision-language models under long-text grounding. |
| Outcome: | ***VLURes** provides a testbed for long-text grounding and multilingual robustness in web-realistic agent settings. |
Can Vision Language Models Understand Mimed Actions? (2025.findings-acl)
Copied to clipboard
Hyundong Justin Cho, Spencer Lin, Tejas Srinivasan, Michael Saxon, Deuksin Kwon, Natali T. Chavez, Jonathan May
| Challenge: | Nonverbal communication (NVC) is an integral part of human language, but it has been overlooked in natural language processing research. |
| Approach: | They propose a multimodal multimodal recognition task that uses a corpus of mimed gestures to evaluate their understanding of NVC. |
| Outcome: | The proposed task is based on 86 unique gestures with perturbations applied to avatar, background, and viewpoint for evaluating recognition robustness. |
PlotGen-Bench: Evaluating VLMs on Generating Visualization Code from Diverse Plots across Multiple Libraries (2026.findings-acl)
Copied to clipboard
| Challenge: | PlotGen-Bench evaluates vision-language models' ability to generate executable visualization code from plots under realistic and complex visualization requirements. |
| Approach: | They propose a benchmark to evaluate plot-to-code generation in vision-language models . they use Matplot, Matplos, Mat3D, Mat4D, and Mat4E to evaluate their performance . |
| Outcome: | The proposed benchmark covers 9 major categories, 30 subcategories, and 3 core tasks . it covers 2D, 3D and animated plots across 5 widely used visualization libraries. |
Multimodal Unlearning Across Vision, Language, Video, and Audio: Survey of Methods, Datasets, and Benchmarks (2026.findings-acl)
Copied to clipboard
| Challenge: | Survey aims to identify challenges of multimodal unlearning for vision, language, audio and video . retraining after deletion requests or policy updates is often impractical, survey finds . |
| Approach: | They propose to enable selective removal across modalities while retaining overall utility. |
| Outcome: | This study compares models with existing models to identify weaknesses and improves performance. |
SemVink: Advancing VLMs’ Semantic Understanding of Optical Illusions via Visual Global Thinking (2025.emnlp-main)
Copied to clipboard
| Challenge: | Vision-language models excel in semantic tasks but fail at detecting hidden content . current architectures prioritize abstract reasoning over low-level visual operations . |
| Approach: | They propose a benchmark to test vision-language models that can detect hidden content . they propose HC-Bench to scale images to low resolutions to unlock 99% accuracy . |
| Outcome: | HC-Bench shows that leading VLMs achieve near-zero accuracy even with explicit prompting . et al.: current models prioritize abstract reasoning over low-level visual operations . they urge a shift toward hybrid models bridging gap between computational vision and human cognition . |
On the Fine-Grained Planning Abilities of VLM Web Agents (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Vision-Language Models (VLMs) have shown promise as web agents, yet their planning has been overlooked. |
| Approach: | They propose to examine VLMs’ ability to understand temporal relationships within web contexts and assess plans of actions across diverse scenarios. |
| Outcome: | The proposed models exhibit limited performance in the above skills and are not reliable to function as web agents. |
Performance Gap in Entity Knowledge Extraction Across Modalities in Vision Language Models (2025.acl-long)
Copied to clipboard
| Challenge: | Vision-language models excel at extracting and reasoning about information from images, yet their capacity to leverage internal knowledge about specific entities remains underexplored. |
| Approach: | They propose a dataset which allows separating entity recognition and question answering . they hypothesize that this decline arises from limitations in how information flows from image tokens to query tokens. |
| Outcome: | The proposed model performance drops when the entity is presented visually rather than textually. |
Almieyar-Oryx-BloomBench: A Bilingual Multimodal Benchmark for Cognitively Informed Evaluation of Vision-Language Models (2026.findings-acl)
Copied to clipboard
Mohammad Mahdi Abootorabi, Omid Ghahroodi, Anas Madkoor, Marzia Nouri, Doratossadat Dastgheib, Ehsaneddin Asgari
| Challenge: | Existing evaluations focus on piecemeal or disconnected tasks, obscuring critical cognitive weaknesses and providing little insight for targeted improvement. |
| Approach: | They propose a bilingual, cognitively human-grounded multimodal benchmark for VLMs that evaluates six levels of cognition through carefully designed image–question–answer tasks. |
| Outcome: | The proposed framework ensures scalability, cultural inclusivity, and linguistic fidelity. |
Benchmarking Vision-Language Models on Chinese Ancient Documents: From OCR to Knowledge Reasoning (2026.findings-acl)
Copied to clipboard
Haiyang Yu, Yuchuan Wu, Fan Shi, Jinghui Lu, Ke Niu, Xiaodong Ge, Minghan Zhuo, Jingqun Tang, Bin Li
| Challenge: | Existing document benchmarks focus on English printed texts or simplified Chinese . current vision-language models struggle with visual complexity and poor adaptability . |
| Approach: | They propose a benchmark to evaluate Chinese ancient documents' visual/linguistic complexity . ancient documents are valuable cultural heritage, but they face challenges in digitization and understanding . |
| Outcome: | the first benchmark for Chinese ancient documents evaluates VLMs from OCR to knowledge reasoning . ancient documents carry thousands of years of Chinese history and culture . traditional methods only scan images, while current models struggle with visual complexity . |
VISA: Retrieval Augmented Generation with Visual Source Attribution (2025.acl-long)
Copied to clipboard
| Challenge: | Existing approaches to retrieval-augmented generation primarily link generated content to document-level references, making it difficult for users to locate evidence among multiple content-rich retrieved documents. |
| Approach: | They propose a novel approach that combines answer generation with visual source attribution by leveraging large vision-language models to identify evidence and highlight exact regions that support the generated answers with bounding boxes in the retrieved document screenshots. |
| Outcome: | The proposed approach identifies evidence and highlights exact regions that support the generated answers with bounding boxes in the retrieved document screenshots. |
Symmetrical Visual Contrastive Optimization: Aligning Vision-Language Models with Minimal Contrastive Images (2025.acl-long)
Copied to clipboard
| Challenge: | Recent studies have shown that Large Vision-Language Models (VLMs) tend to neglect image content and over-rely on language-model priors, resulting in errors in visually grounded tasks and hallucinations. |
| Approach: | They propose a novel finetuning objective that steers the model toward capturing important visual details and aligning them with corresponding text tokens. |
| Outcome: | The proposed method achieves up to 22% reduction in hallucinations and significant gains in vision-centric and general tasks while maintaining or improving the model's general abilities. |
From Charts to Fair Narratives: Uncovering and Mitigating Geo-Economic Biases in Chart-to-Text (2025.emnlp-main)
Copied to clipboard
Ridwan Mahbub, Mohammed Saidul Islam, Mir Tafseer Nayeem, Md Tahmid Rahman Laskar, Mizanur Rahman, Shafiq Joty, Enamul Hoque
| Challenge: | Existing VLMs produce more positive descriptions for high-income countries compared to middle- or low-income nations, even when country attribution is the only variable changed. |
| Approach: | They propose to automate the process by generating textual summaries of charts using vision-language models to understand how a country’s economic status influences the sentiment of generated summary. |
| Outcome: | The proposed model amplifys geo-economic biases in 6,000 chart-country pairs from six widely used vision-language models to understand how a country’s economic status influences the sentiment of generated summaries. |
InquireMobile: Teaching VLM-based Mobile Agent to Request Human Assistance via Reinforcement Fine-Tuning (2026.acl-long)
Copied to clipboard
Qihang Ai, Pi Bu, Yue Cao, Yingyao Wang, Jihao Gu, Jingxuan Xing, Zekun Zhu, Wei Jiang, Zhicheng Zheng, Jun Song, Yuning Jiang
| Challenge: | Recent advances in Vision-Language Models (VLMs) have enabled mobile agents to perceive and interact with real-world mobile environments based on human instructions. |
| Approach: | They propose a vision-language model that actively seeks human confirmation at critical decision points and a model inspired by reinforcement learning. |
| Outcome: | The proposed model achieves an improvement of 46.8% in inquiry success rate and the best overall success rate among existing baselines on InquireBench. |
Cross-Modal Masked Compositional Concept Modeling for Enhancing Visio-Linguistic Compositionality (2026.acl-long)
Copied to clipboard
| Challenge: | a contrastive learning approach for vision-language models is needed to capture compositional information. |
| Approach: | They propose a framework that masks compositional concepts in one modality and reconstructs them conditioned on full contextual information from the other . |
| Outcome: | The proposed framework enhances compositionality in visual language models and improves their ability to capture syntactic structure and linguistic information. |
A Parameter-Efficient and Fine-Grained Prompt Learning for Vision-Language Models (2025.acl-long)
Copied to clipboard
| Challenge: | Current vision-language models extract semantic information from large-scale cross-modal associations, limiting performance and efficiency. |
| Approach: | They propose a detail-oriented prompt learning method to implement fine-grained multi-modal semantic alignment with merely 0.25M trainable parameters. |
| Outcome: | The proposed method implements fine-grained multi-modal semantic alignment with merely 0.25M trainable parameters. |
MemeIntel: Explainable Detection of Propagandistic and Hateful Memes (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for label detection and explanation generation have been limited in understanding complex issues . identifying propaganda and hate in memes is essential for combating misinformation and minimizing harm . |
| Approach: | They propose an explanation-enhanced dataset for propaganda memes in Arabic and hateful memes on English to solve these tasks. |
| Outcome: | The proposed model outperforms the current state-of-the-art in label detection and explanation generation. |
Are Vision-Language Models Safe in the Wild? A Meme-Based Benchmark Study (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing safety evaluations rely on artificial images to evaluate vision-language models . a recent study found that memes are more effective at bypassing safety measures than synthetic or typographic images. |
| Approach: | They propose a benchmark pairing meme images with harmful and benign instructions . they assess multiple VLMs across single and multi-turn interactions . |
| Outcome: | The proposed benchmark pairs real meme images with harmful and benign instructions. |
Generalizable Prompt Tuning for Audio-Language Models via Semantic Expansion (2026.findings-acl)
Copied to clipboard
| Challenge: | Prompt tuning has achieved remarkable progress in vision–language models, but its generalization ability in ALMs remains underexplored. |
| Approach: | They propose a plug-and-play framework that regularizes the prompt embedding space . they propose introducing a semantic expansion loss with margin constraints that promote compactness . |
| Outcome: | The proposed framework regularizes the prompt embedding space by incorporating semantic neighbors generated by large language models. |
Scalable Vision Language Model Training via High Quality Data Curation (2025.acl-long)
Copied to clipboard
| Challenge: | SAIL-VL models achieve the highest average score in 18 widely used VLM benchmarks in our evaluation, with the 2B model takes the top position over VLMs of comparable sizes on OpenCompass 2024. |
| Approach: | They introduce an open-source vision language model (VLM) series that can be trained using high-quality data. |
| Outcome: | The proposed model achieves the highest average score in 18 widely used VLM benchmarks, with the 2B model taking the top position over VLMs of comparable sizes on OpenCompass 2024. |
Unlocking Human-Like Visible Logic: How Logic Diagrams Boost Logic Reasoning in Large Language Models? (2026.findings-acl)
Copied to clipboard
| Challenge: | Recent advances in large language models (LLMs) have demonstrated their remarkable capabilities in natural language understanding and generation, but they struggle with formal logical reasoning. |
| Approach: | They propose to incorporate visual logic diagrams into LLMs’ reasoning workflows to enhance their performance on formal logic tasks. |
| Outcome: | The proposed model improves on syllogistic and conditional reasoning with programmatically generated Venn, Euler, and Linear diagrams. |
Design Choices for Extending the Context Length of Visual Language Models (2025.acl-long)
Copied to clipboard
| Challenge: | Existing open-source Visual Language Models lack systematic exploration into extending their context length, and commercial models often provide limited details. |
| Approach: | They propose to extend Visual Language Models (VLMs) to 128K lengths and open-source the code, data, and models. |
| Outcome: | The proposed model is based on the Qwen-VL series model and is competitive with commercial model GPT-4V. |
CoV: Chain-of-View Prompting for Spatial Reasoning (2026.findings-acl)
Copied to clipboard
Haoyu Zhao, Akide Liu, Zeyu Zhang, Weijie Wang, Feng Chen, Ruihan Zhu, Gholamreza Haffari, Bohan Zhuang
| Challenge: | Embodied question answering requires collecting context that is distributed across multiple viewpoints . most recent vision–language models (VLMs) are constrained to a fixed and finite set of input views . |
| Approach: | They propose a training-free, test-time reasoning framework that transforms a VLM into an active viewpoint reasoner through a coarse-to-fine exploration process. |
| Outcome: | The proposed framework improves LLM-Match performance by 11.98% on four mainstream VLMs. |
VISaGE: Understanding Visual Generics and Exceptions (2025.emnlp-main)
Copied to clipboard
| Challenge: | atypical evaluation instances disrupt incontext instance understanding and in-weight conceptual knowledge. |
| Approach: | They propose to use a dataset to analyze atypical visual and textual images to test their models. |
| Outcome: | The proposed model is based on a dataset consisting of typical and exceptional images. |
microCLIP: Unsupervised CLIP Adaptation via Coarse-Fine Token Fusion for Fine-Grained Image Classification (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing UA methods for fine-grained image classification rely on coarse-grain visual tokens, which misses fine spatial details. |
| Approach: | They propose a label-free self-training framework that adapts visual features and LLMderived text prototypes using fine-grained cues. |
| Outcome: | The proposed framework improves alignment between finegrained visual regions and rich textual descriptions while updating only layer norms and a tiny head. |
VIDA: A Visual Intent-driven Design Assistant for Proactive Multimodal Clarification (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing vision-language models fail to provide accurate and complete answers to user requests . a new strategy-aware design assistant is developed to help designers create proactive, visually grounded, and strategically prioritized clarification questions. |
| Approach: | They propose a visual intent-driven design assistant to generate proactive, visually grounded, and strategically prioritized clarification questions. |
| Outcome: | The proposed assistant improves the strategic alignment score by 20.59% over baselines and restores visual grounding capabilities lost during fine-tuning. |
KRETA: A Benchmark for Korean Reading and Reasoning in Text-Rich VQA Attuned to Diverse Visual Contexts (2025.emnlp-main)
Copied to clipboard
| Challenge: | Understanding and reasoning over text within visual contexts poses a significant challenge for Vision-Language Models. |
| Approach: | They propose a benchmark for Korean Reading and rEasoning in Text-rich VQA Attuned to diverse visual contexts to address this challenge. |
| Outcome: | The proposed benchmark is tailored for Korean reading and rEasoning in text-rich VQA attuned to diverse visual contexts. |
Concept-pedia: a Wide-coverage Semantically-annotated Multimodal Dataset (2025.emnlp-main)
Copied to clipboard
| Challenge: | Current evaluations for Vision-language Models remain heavily anchored to ImageNet . |
| Approach: | They propose a large-scale semantically-annotated multimodal resource that extends the range of visual concepts, including diverse abstract categories. |
| Outcome: | The proposed model expands the range of visual concepts, including diverse abstract categories. |
Fico: Evaluating Vision-Language Models under Visual Fidelity and Compression at Scale (2026.findings-acl)
Copied to clipboard
| Challenge: | Visual text compression is emerging paradigm for rendering text as images for processing by vision-language models. |
| Approach: | They propose a benchmark to assess VLM robustness under dense visual inputs. |
| Outcome: | Evaluating 13 general-purpose VLMs and 3 OCR-specialized models reveals performance drops sharply under increased density or reduced resolution; cross-task transfer between OCR, NIAH, and VQA is limited; and VQ is comparatively robust because low-level details are lost before high-level semantics. |
CHURRO: Making History Readable with an Open-Weight Large Vision-Language Model for High-Accuracy, Low-Cost Historical Text Recognition (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing vision-language models are not equipped to read diverse languages and scripts found in historical materials. |
| Approach: | They propose to train an open-weight vision-language model for historical text recognition on CHURRO-DS, the largest historical text-recognition dataset to date. |
| Outcome: | The proposed model outperforms existing vision-language models on CHURRO-DS, the largest historical text recognition dataset to date. |
Can VLMs Predict Future States? Bootstrapping World Models from Inverse Dynamics (2026.findings-acl)
Copied to clipboard
| Challenge: | unified vision–language models (VLMs) struggle to generate physically plausible transitions between frames from instructions. |
| Approach: | They find that VLMs struggle to generate physically plausible transitions between frames from instructions. |
| Outcome: | The proposed model outperforms state-of-the-art image editing models on Aurora-Bench . it achieves the best average human evaluation across all subsets of Aurora-bench compared with other models . |
On-policy Reinforcement Fine-tuning with Offline reward for Multi-step Embodied Planning (2026.acl-long)
Copied to clipboard
| Challenge: | Embodied planning requires agents to make coherent multi-step decisions based on dynamic visual observations and verbal goals. |
| Approach: | They propose an On-policy Reinforcement fine-tuning framework with offline rewards for Embodied Task Planning that preserves generalization benefits of RFT while addressing costly interaction and sparse rewards. |
| Outcome: | The proposed framework outperforms closed-source and online-RL methods on EmbodiedBench, a recent benchmark for interactive embodied tasks. |
VisTW: Benchmarking Vision-Language Models for Taiwanese Mandarin in Taiwan (2026.findings-acl)
Copied to clipboard
| Challenge: | Vision-Language Models (VLMs) struggle in Taiwanese Mandarin environments due to region-specific orthographic and cultural context. |
| Approach: | They propose a human-grounded purity penalty for character mixing under Taiwanese-Mandarin-style prompts. |
| Outcome: | The proposed model outperforms the strongest open-weight baseline by 22 percentage points on dialogue tasks. |
Measuring Social Bias in Vision-Language Models with Face-Only Counterfactuals from Real Photos (2026.acl-long)
Copied to clipboard
| Challenge: | Vision-Language Models (VLMs) are increasingly deployed in socially consequential settings . attribution under visual confounding is a central challenge in measuring social bias . |
| Approach: | They propose a face-only counterfactual evaluation paradigm that isolates demographic effects while preserving real-image realism. |
| Outcome: | The proposed paradigm isolates demographic effects while preserving real-image realism. |
GeoRC: A Benchmark for Geolocation Reasoning Chains (2026.acl-long)
Copied to clipboard
Mohit Talreja, Joshua Diao, Jim James, Radu Casapu, Tejas Santanam, Ethan Mendes, Alan Ritter, Wei Xu, James Hays
| Challenge: | Vision Language Models (VLMs) are good at recognizing the global location of a photograph but are startlingly bad at explaining which image evidence led to their location prediction. |
| Approach: | They propose a benchmark for geolocation reasoning chains based on the global location prediction task in the popular GeoGuessr game. |
| Outcome: | The proposed benchmark compares LLM-as-a-judge and VLM-As-jumble strategies against human scoring. |
MPTc-Bench: Measuring Cross-market Generative Ability of Vision-Language Models via Movie Poster Transcreation (2026.findings-acl)
Copied to clipboard
| Challenge: | Recent work adapts textual transcreation to image editing and formulates image transcreations to better match a target audience while preserving meaning. |
| Approach: | They propose a two-stage planner-editor pipeline in which an VLM planner specifies executable edits and an image editor renders them. |
| Outcome: | The proposed model can transcreate a visual asset for a different market while preserving its identity while matching market-specific design preferences and multilingual typography. |
Mechanisms of Prompt-Induced Hallucination in Vision–Language Models (2026.acl-long)
Copied to clipboard
William Rudman, Michal Golovanevsky, Dana Arad, Yonatan Belinkov, Carsten Eickhoff, Ritambhara Singh, Kyle Mahowald
| Challenge: | Large vision–language models (VLMs) often hallucinate by favoring textual prompts over visual evidence. |
| Approach: | They study the failure mode of large vision–language models by focusing on textual prompts over visual evidence. |
| Outcome: | The proposed model overestimates the number of objects in an image . it hallucinates additional waterlilies when asked to describe a mismatched number of items . the model ablation reduces prompt-induced hallucinosities by at least 40% without additional training . |
RealChart2Code: Bridging the Gap in Real-World Chart-to-Code Generation via Multi-Task Evaluation (2026.acl-long)
Copied to clipboard
Jiajun Zhang, Yuying Li, Zhixun Li, Xingyu Guo, Jingzhuo Wu, Leqi Zheng, Yiran Yang, Jianke Zhang, Qingbin Li, Shannan Yan, Changguo Jia, Junfei Wu, Zilei Wang, Qiang Liu, Liang Wang
| Challenge: | Vision-Language Models (VLMs) have demonstrated impressive capabilities in code generation across various domains, but their ability to replicate complex, multi-panel visualizations remains largely unassessed. |
| Approach: | They propose a large-scale benchmark to evaluate chart generation from large- scale raw data and assess iterative code refinement in a multi-turn conversational setting. |
| Outcome: | The new benchmark evaluates 14 leading VLMs on real-world data and shows they struggle with complex plot structures and authentic data. |
VLMGuard-R1: Proactive Safety Alignment for VLMs via Reasoning-Driven Prompt Optimization (2026.findings-acl)
Copied to clipboard
| Challenge: | integrating vision and language models with safety standards is essential to mitigate multimodal complexity . integrating visual inputs with vision and text unveils subtle threats beyond the reach of conventional safeguards . |
| Approach: | They propose a framework that combines vision and language to provide a multimodal reasoning-driven prompt rewriting. |
| Outcome: | The proposed framework outperforms baseline models on five benchmarks with six VLMs. |
Through the Magnifying Glass: Adaptive Perception Magnification for Hallucination-Free VLM Decoding (2026.acl-long)
Copied to clipboard
| Challenge: | Existing vision-language models suffer from visual hallucination, where the generated responses contain inaccuracies that are not grounded in the visual input. |
| Approach: | They propose a visual decoding method that iteratively isolates relevant visual tokens based on attention and magnifies the corresponding regions. |
| Outcome: | The proposed method reduces language biases and amplifies weights of visual embedding during decoding, while still preserving strong reasoning capabilities. |
Do VLMs Have a Moral Backbone? A Study on the Fragile Morality of Vision-Language Models (2026.findings-acl)
Copied to clipboard
Zhining Liu, Tianyi Wang, Xiao Lin, Penghao Ouyang, Gaotang Li, Ze Yang, Hui Liu, Sumit Keswani, Vishwa Pardeshi, Huijun Zhao, Wei Fan, Hanghang Tong
| Challenge: | Vision-Language Models (VLMs) have advanced multimodal learning, driving progress in cross-modal reasoning. |
| Approach: | They propose to examine moral robustness of vision-language models by analyzing their moral stances under multimodal perturbations. |
| Outcome: | The proposed model-agnostic multimodal perturbations expose VLMs to a variety of moral vulnerabilities, including a sycophancy trade-off where stronger instruction-following models are more susceptible to persuasion. |
Grounded Concreteness: Human-Like Concreteness Sensitivity in Vision–Language Models (2026.findings-acl)
Copied to clipboard
| Challenge: | a long tradition in cognitive science treats concreteness as a graded dimension of conceptual representation . concrete words benefit from richer sensory codes and exhibit robust behavioral advantages over abstract words . |
| Approach: | They compare vision-language models with text-only large language models to test their concreteness . they find that VLMs show more human-like sensitivity to concreteness than LLMs . |
| Outcome: | The proposed model-based training improves on the Llama text backbones and Llma Vision counterparts. |
MMErroR: A Benchmark for Erroneous Reasoning in Vision-Language Models (2026.acl-long)
Copied to clipboard
Yang Shi, Yifeng Xie, Minzhe Guo, Liangsi Lu, Mingxuan Huang, Jingchao Wang, Zhihong Zhu, Boyan Xu, Zhiqi Huang
| Challenge: | Recent advances in vision-language models have improved performance in multi-modal learning. |
| Approach: | They propose a multi-modal benchmark that embeds a single coherent reasoning error in 1997 samples. |
| Outcome: | The proposed benchmark is based on a set of 1997 samples embedding a single coherent reasoning error. |
Does RLVR Extend Reasoning Boundaries? Investigating Capability Expansion in Vision-Language Models (2026.acl-long)
Copied to clipboard
| Challenge: | Recent studies suggest that RLVR amplifies behaviors inherent to the pre-training distribution rather than inducing new capabilities. |
| Approach: | They propose a framework for RLVR that extends the spatial reasoning boundary . they use a mapping framework where the difficulty is precisely regulated by path length and number of turns . |
| Outcome: | The proposed framework extends the spatial reasoning boundary on two real-world navigation benchmarks. |
SynthRL: Scaling Visual Reasoning with Verifiable Data Synthesis (2026.findings-acl)
Copied to clipboard
| Challenge: | SynthRL synthesizes over 3.3K additional verifiable, challenging questions from approximately 8K seed samples. |
| Approach: | They propose a scalable and guaranteed pipeline for automatic data scaling in reasoning-oriented RL training. |
| Outcome: | The proposed pipeline synthesizes over 3.3K additional verifiable, challenging questions from approximately 8K seed samples. |
Long Story Short: Disentangling Compositionality and Long-Caption Understanding in Contrastive VLMs (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing benchmarks for vision-language models treat compositionality and long-caption understanding in isolation. |
| Approach: | They analyze when compositional reasoning and long-caption understanding transfer across tasks and when this relationship fails. |
| Outcome: | The proposed model can generalize on poorly grounded captions and with strong visual grounding, while architectural choices can limit compositional learning. |
CEBC: Conformal Evidence-Bounded Control for Low-Hallucination Vision–Language Generation (2026.acl-long)
Copied to clipboard
| Challenge: | Existing mitigation approaches reduce hallucinated object mentions at the cost of degraded generation quality or require expensive retraining and task-specific supervision. |
| Approach: | They propose a lightweight framework for low-hallucination vision–language generation . it uses evidence-bounded minimal editing to revise or suppress unsupported referenced entities . |
| Outcome: | The proposed framework reduces hallucinations while maintaining or improving quality metrics. |
DREAM-S: Speculative Decoding with Searchable Drafting and Target-Aware Refinement for Multimodal Generation (2026.acl-long)
Copied to clipboard
| Challenge: | Speculative decoding (SD) has proven to be effective for autoregressive generation in large language models (LLMs), however its application to vision-language models (VLMs) remains relatively unexplored. |
| Approach: | They propose a Speculative Decoding framework for vision-language models that integrates a neural architecture search framework and target-aware supernet training to identify optimal interaction strategies. |
| Outcome: | DREAM-S achieves 3.85 speedup compared to baselines on well-established vision-language models. |
MathSight: A Benchmark Exploring Have Vision-Language Models Really Seen in University-Level Mathematical Reasoning? (2026.acl-long)
Copied to clipboard
| Challenge: | Existing benchmarks rarely isolate how much visual information contributes to reasoning . a growing collection of benchmarks has catalyzed rapid progress in multimodal reasoning - but how much it contributes remains unclear . |
| Approach: | They propose a university-level multimodal mathematical reasoning benchmark to quantify the effect of visual input. |
| Outcome: | The proposed benchmark disentangles and quantifies the effect of visual input on multimodal reasoning models. |