LLAVA

Protein2Text: Resampling Mechanism to Translate Protein Sequences into Human-Interpretable Text (2025.naacl-industry)

Challenge:	Existing multimodal large language models are limited to general-purpose multimodal tasks like question-answering on natural images.
Approach:	They propose to use cross-modal projection networks and a large language model to model domain-specific visual attributes of MLLMs.
Outcome:	The proposed models gain domain-specific visual capabilities when the projection is fine-tuned, but the updates do not extract relevant domain-specific visual attributes.

Copied to clipboard

Ala Jararweh, Oladimeji Macaulay, David Arredondo, Yue Hu, Luis E Tafoya, Kushal Virupakshappa, Avinash Sahu

Explainability & Interpretability in NLP

Challenge:	Proteins play critical roles in biological systems, yet 99.7% of 227 million known protein sequences remain uncharacterized due to the limitations of experimental methods.
Approach:	They propose a multimodal large language model that interprets protein sequences and generates informative text to address open-ended questions about protein functions and attributes.
Outcome:	The proposed model outperforms existing models in open-ended question-answering tasks.

A Multi-modal Large Language Model with Graph-of-Thought for Effective Recommendation (2025.naacl-long)

Copied to clipboard

Zixuan Yi, Iadh Ounis

IndiFoodVQA: Advancing Visual Question Answering and Reasoning with a Knowledge-Infused Synthetic Data Generation Pipeline (2024.findings-eacl)

Challenge:	Large Language Models (LLMs) have demonstrated a remarkable capability in language understanding and text generation in various realworld scenarios.
Approach:	They propose a Graph-of-Thought prompting technique in a Multi-modal LLM to leverage the complex structure of interaction graphs.
Outcome:	The proposed model outperforms 12 existing state-of-the-art models on 6 benchmark datasets.

Copied to clipboard

Pulkit Agarwal, Settaluri Sravanthi, Pushpak Bhattacharyya

LVLM-Compress-Bench: Benchmarking the Broader Impact of Large Vision-Language Model Compression (2025.findings-naacl)

Challenge:	Large Vision Language Models lack domain-specific data for reasoning on complex problems.
Approach:	They propose to use explicit knowledge-infused questions, answers, and reasons to answer and reason upon the questions.
Outcome:	The proposed model improves by 25% over the baseline model.

Copied to clipboard

Souvik Kundu, Anahita Bhiwandiwalla, Sungduk Yu, Phillip Howard, Tiep Le, Sharath Nittur Sridhar, David Cobbley, Hao Kang, Vasudev Lal

Not All Directions Matter: Towards Structured and Task-Aware Low-Rank Model Adaptation (2026.acl-long)

Challenge:	LVLMs have been shown to perform well on simple uni-modal benchmarks, but their detailed study on multi-modal models is still lacking.
Approach:	They propose a framework to analyze the impact of compression on LVLMs on multi-modal input driven tasks.
Outcome:	The proposed framework analyzes the impact of compression on generative performance of large vision language models on multi-modal input driven tasks.

Copied to clipboard

Xi Xiao, Chenrui Ma, Yunbei Zhang, Chen Liu, Zhuxuanzi Wang, Yanshu Li, Lin Zhao, Guosheng Hu, Tianyang Wang, Hao Xu

UniRAG: Universal Retrieval Augmentation for Large Vision Language Models (2025.findings-naacl)

Challenge:	Low-Rank Adaptation (LoRA) is a key parameter-efficient fine-tuning method . however, its effectiveness is hampered by semantic drift and structural incoherence .
Approach:	They propose a low-rank Adaptation framework that tackles semantic drift and structural incoherence by pruning task-irrelevant directions.
Outcome:	Experiments on large language models, vision models, and vision models show that the proposed framework outperforms LoRA and advanced dynamic rank allocation and sparsity-based methods.

Copied to clipboard

Sahel Sharifymoghaddam, Shivani Upadhyay, Wenhu Chen, Jimmy Lin

Semantic Text Processing

Challenge:	Large Vision Language Models (LVLMs) have unlocked many complex use cases that require Multi-Modal (MM) understanding and MM generation.
Approach:	They propose a plug-and-play technique that adds relevant retrieved information to prompts as few-shot examples during inference.
Outcome:	The proposed method significantly improves the output quality of large vision language models when input prompts are augmented with relevant information retrieved by Vision-Language retrievers like UniRAG.

PuzzleGPT: Emulating Human Puzzle-Solving Ability for Time and Location Prediction (2025.findings-naacl)

Copied to clipboard

Hammad Ayyubi, Xuande Feng, Junzhang Liu, Xudong Lin, Zhecan Wang, Shih-Fu Chang

MEGAVERSE: Benchmarking Large Language Models Across Languages, Modalities, Models and Tasks (2024.naacl-long)

Challenge:	PuzzleGPT is a modular and iterative puzzlesolving method for predicting time and location from images.
Approach:	They propose to formalize this ability into core skills and implement it using different modules in an expert pipeline called PuzzleGPT.
Outcome:	The proposed method outperforms large VLMs and finetuned models on TARA and WikiTilo and rivals or surpasses finetuned models.

Copied to clipboard

Sanchit Ahuja, Divyanshu Aggarwal, Varun Gumma, Ishaan Watts, Ashutosh Sathe, Millicent Ochieng, Rishav Hada, Prachi Jain, Mohamed Ahmed, Kalika Bali, Sunayana Sitaram

Safe Inputs but Unsafe Output: Benchmarking Cross-modality Safety Alignment of Large Vision-Language Models (2025.findings-naacl)

Challenge:	Several new LLMs have been introduced necessitating their evaluation on non-English languages.
Approach:	They perform a thorough evaluation of the non-English capabilities of SoTA LLMs by comparing them on the same set of multilingual datasets.
Outcome:	The proposed model outperforms models on multilingual datasets on 22 languages including low-resource African languages.

Copied to clipboard

Siyin Wang, Xingsong Ye, Qinyuan Cheng, Junwen Duan, Shimin Li, Jinlan Fu, Xipeng Qiu, Xuanjing Huang

Your Vision-Language Model Itself Is a Strong Filter: Towards High-Quality Instruction Tuning with Data Selection (2024.findings-acl)

Challenge:	Recent studies focus on single-modality threats, but this approach fails to address cross-modal safety alignment.
Approach:	They propose a safety alignment challenge to evaluate cross-modality safety alignment . they propose 'Safe Inputs but Unsafe Output' to consider safety of single modalities .
Outcome:	The proposed safety alignment challenge examines cases where modalities are safe independently but could lead to unsafe outputs when combined.

Copied to clipboard

Ruibo Chen, Yihan Wu, Lichang Chen, Guodong Liu, Qi He, Tianyi Xiong, Chenxi Liu, Junfeng Guo, Heng Huang

Crafting Adversarial Inputs for Large Vision-Language Models Using Black-Box Optimization (2026.findings-eacl)

Challenge:	Existing data selection methods for instruction-following large language models rely on unreliable scores or use downstream tasks for selection.
Approach:	They propose a method that utilizes the VLM itself as a filter to select high-quality instruction-tuning data.
Outcome:	The proposed method can reach better results compared to full data settings with merely about 15% samples and can achieve superior performance against competitive baselines.

Copied to clipboard

Jiwei Guan, Haibo Jin, Haohan Wang

Robustness in NLP

Challenge:	Existing white-box jailbreak methods require full model accessibility and require computational costs.
Approach:	They propose a black-box jailbreak attack using Zeroth-Order optimization using ZO-SPSA.
Outcome:	The proposed method achieves highest jailbreak success rate on three LVLMs, including InstructBLIP, LLaVA and MiniGPT-4.

DeepInsert: Early Layer Bypass for Efficient and Performant Multimodal Understanding (2026.eacl-long)

Copied to clipboard

Moulik Choraria, Xinbo Wu, Akhil Bhimaraju, Nitesh Sekhar, Yue Wu, Xu Zhang, Prateek Singhal, Lav R. Varshney

Hope ‘The Paragraph Guy’ explains the rest : Introducing MeSum, the Meme Summarizer (2024.findings-emnlp)

Challenge:	Recent work shows that hyperscaling of data and parameter count in LLMs is yielding diminishing improvement when weighed against training costs.
Approach:	They propose to insert multimodal tokens directly into the middle of the model to bypass the early layers.
Outcome:	The proposed method reduces training and inference costs while preserving performance.

Copied to clipboard

Anas Khan, Tanik Saikh, Arpan Phukan, Asif Ekbal

Advancement in Graph Understanding: A Multimodal Benchmark and Fine-Tuning of Vision-Language Models (2024.acl-long)

Challenge:	a lack of large datasets for supervised learning and resource-intensive vision language models have hindered the development of meme comprehension.
Approach:	They propose a framework to bridge the gap between meme comprehension and vision language models by using a multimodal dataset.
Outcome:	The proposed framework outperforms existing methods in the meme comprehension test.

Copied to clipboard

Qihang Ai, Jiafan Li, Jincheng Dai, Jianwu Zhou, Lemao Liu, Haiyun Jiang, Shuming Shi

VLSBench: Unveiling Visual Leakage in Multimodal Safety (2025.acl-long)

Challenge:	Graph data organizes complex relationships and interactions between objects . Graph neural networks (GNNs) are becoming more popular in graph learning .
Approach:	They propose a new paradigm for interactive and instructional graph data understanding and reasoning . they first evaluate the capabilities of public VLMs in graph learning from multiple aspects .
Outcome:	The proposed model achieves an accuracy increase of 5%-15% compared to baseline models . the best-performing model achieve scores comparable to Gemini in GPT-asissted Evaluation .

Copied to clipboard

Xuhao Hu, Dongrui Liu, Hao Li, Xuanjing Huang, Jing Shao

AbsVis – Benchmarking How Humans and Vision-Language Models “See” Abstract Concepts in Images (2025.emnlp-main)

Challenge:	Existing studies show that textual unlearning does not achieve comparable safety performance with image-text alignment.
Approach:	They propose to use textual unlearning to align MLLMs with image-text pairs to explain this problem . they construct a visual leakless safety bench with 2.2k image- text pairs to test this problem.
Outcome:	The proposed model can refuse image-text pairs according to textual queries, leading to unreliable safety evaluations.

Copied to clipboard

Tarun Tater, Diego Frassinelli, Sabine Schulte im Walde

Explainability & Interpretability in NLP

Challenge:	Abstract concepts like mercy and peace lack clear visual grounding, and therefore challenge humans and models to provide suitable image representations.
Approach:	They propose a dataset of 675 images annotated with 14,175 concept–explanation attributions from humans and two Vision-Language Models where each concept is accompanied by a textual explanation.
Outcome:	The proposed dataset compares human and VLM attributions in terms of diversity, abstractness, and alignment, and shows that overlapping concepts are most preferred.

What Makes for Good Visual Instructions? Synthesizing Complex Visual Reasoning Instructions for Visual Instruction Tuning (2025.coling-main)

Copied to clipboard

Yifan Du, Hangyu Guo, Kun Zhou, Wayne Xin Zhao, Jinpeng Wang, Chuyuan Wang, Mingchen Cai, Ruihua Song, Ji-Rong Wen

What Do VLMs NOTICE? A Mechanistic Interpretability Pipeline for Gaussian-Noise-free Text-Image Corruption and Evaluation (2025.naacl-long)

Challenge:	Experimental results show that visual instruction tuning improves performance of Multi-modal Large Language Models (MLLMs) to extend the application scope of Large Language Modells, a surge of work augments LLMs with vision encoders to endow the ability of multi-modal cognition and reasoning.
Approach:	They propose a systematic approach to create high-quality visual reasoning instructions using a synthesize-complicate-reformulate paradigm.
Outcome:	The proposed method improves performance of MLLMs by 27.86% and 27.60% on MME-Perception and MME Cognition.

Copied to clipboard

Michal Golovanevsky, William Rudman, Vedant Palit, Carsten Eickhoff, Ritambhara Singh

InferAligner: Inference-Time Alignment for Harmlessness through Cross-Model Guidance (2024.emnlp-main)

Challenge:	Vision-Language Models (VLMs) have gained prominence due to their success in solving complex cross-modal tasks.
Approach:	They propose a Gaussian-Noise-free pipeline for mechanistic interpretability in VLMs that introduces Semantic Image Pairs corruption, the first visual counterpart to Symmetric Token Replacement for text.
Outcome:	The proposed pipeline identifies a set of “universal attention heads” in BLIP and LLaVA that consistently contribute across different tasks and modalities.

Copied to clipboard

Pengyu Wang, Dong Zhang, Linyang Li, Chenkun Tan, Xinghao Wang, Mozhi Zhang, Ke Ren, Botian Jiang, Xipeng Qiu

Making Every Step Effective: Jailbreaking Large Vision-Language Models Through Hierarchical KV Equalization (2025.findings-emnlp)

Challenge:	Existing methods for enhancing harmlessness and helpfulness of large language models (LLMs) involve complex and resource-intensive training processes.
Approach:	They propose a method that decouples harmlessness from helpfulness during inference phase.
Outcome:	The proposed method significantly reduces the attack success rate (ASR) of harmful instructions and jailbreak instructions while maintaining almost unchanged performance in downstream tasks.

Copied to clipboard

Shuyang Hao, Yiwei Wang, Bryan Hooi, Jun Liu, Muhao Chen, Zi Huang, Yujun Cai

Robustness in NLP

Challenge:	HKVE selectively accepts gradient optimization results based on the distribution of attention scores across different layers, ensuring that every optimization step positively contributes to the attack.
Approach:	They propose a framework that selectively accepts gradient optimization results based on the distribution of attention scores across different layers and selectively takes them into account when calculating the attack success rate.
Outcome:	The proposed framework outperforms existing methods by achieving success rates of 75.08% on MiniGPT4, 85.84% on LLaVA and 81.00% on Qwen-VL.

Not (yet) the whole story: Evaluating Visual Storytelling Requires More than Measuring Coherence, Grounding, and Repetition (2024.findings-emnlp)

Copied to clipboard

Aditya Kaushik Surikuchi, Raquel Fernández, Sandro Pezzelle

LLaVA Steering: Visual Instruction Tuning with 500x Fewer Parameters through Modality Linear Representation-Steering (2025.acl-long)

Challenge:	Visual storytelling is a task of generating a story for a sequence of several temporally-ordered images or video frames.
Approach:	They propose a method that measures story quality in terms of human likeness regarding three key aspects highlighted in previous work: visual grounding, coherence, and repetitiveness.
Outcome:	The proposed method improves on the foundation model LLaVA but only slightly compared to TAPM, a 50-times smaller visual storytelling model.

Copied to clipboard

Jinhe Bi, Yujun Wang, Haokun Chen, Xun Xiao, Artur Hecker, Volker Tresp, Yunpu Ma

Semantic Text Processing

Challenge:	Multimodal Large Language Models (MLLMs) enhance visual tasks by integrating visual representations into large language models.
Approach:	They propose a method to re-balance modalities by steering visual representations . they propose LLaVA Steering, a platform that enables rapid customization of MLLMs a component-based architecture .
Outcome:	The proposed model re-balances the modalities of visual representations in large language models . the model requires 500 times fewer trainable parameters than LoRA while maintaining comparable performance .

Multi-modal Preference Alignment Remedies Degradation of Visual Instruction Tuning on Language Models (2024.acl-long)

Copied to clipboard

Shengzhi Li, Rongyu Lin, Shichao Pei

Why do LLaVA Vision-Language Models Reply to Images in English? (2024.findings-emnlp)

Challenge:	Multi-modal large language models (MLLMs) are expected to support multi-turn queries of interchanging image and text modalities in production.
Approach:	They propose to use visual-question-answering (VQA) datasets to annotate a 5k-sample VQA preference dataset and to investigate the degradation of VQA datasets.
Outcome:	The proposed model surpasses the instruction-following capabilities of the language model with DPO and SteerLM.

Copied to clipboard

Musashi Hinck, Carolin Holtermann, Matthew Olson, Florian Schneider, Sungduk Yu, Anahita Bhiwandiwalla, Anne Lauscher, Shao-Yen Tseng, Vasudev Lal

From Inheritance to Saturation: Disentangling the Evolution of Visual Redundancy for Architecture-Aware MLLM Inference Acceleration (2026.acl-long)

Challenge:	Including an image in a multimodal query significantly increases the likelihood of the model returning an English response regardless of the language of the query.
Approach:	They propose a two-pronged approach that combines extensive ablation of the design space with a mechanistic analysis of the models’ internal representations of image and text inputs.
Outcome:	The proposed approach reduces the multilingual error by switching the language backbone for a bilingual language model.

Copied to clipboard

Jiaqi Shi, Xulong Zhang, Yuechan Li, Xiaoyang Qu, Jianzong Wang

Task-Aware Resolution Optimization for Visual Large Language Models (2025.emnlp-main)

Challenge:	Existing acceleration strategies suffer from severe "backbone dependency" Existing strategies such as token pruning or layer sparsity suffer from this .
Approach:	They propose a framework that decouples visual redundancy into IVR and architecture-dependent secondary saturation redundancies.
Outcome:	The proposed framework outperforms existing frameworks on Qwen25-VL and Qwa25-LL.

Copied to clipboard

Weiqing Luo, Zhen Tan, Yifan Li, Xinyu Zhao, Kwonjoon Lee, Behzad Dariush, Tianlong Chen

ViPE: Visual Perception in Parameter Space for Efficient Video-Language Understanding (2025.emnlp-main)

Challenge:	Existing visual large language models pre-assume a fixed resolution for downstream tasks, leading to sub-optimal performance.
Approach:	They propose a formula to determine the optimal resolution for a given vision-language task . they then propose 'parameter-efficient' fine-tuning technique to extend the visual input resolution .
Outcome:	The proposed method is based on rigorous experiments on vision-language tasks.

Copied to clipboard

Shichen Lu, Tongtian Yue, Longteng Guo, Handong Li, Xingjian He, Si Liu, Jing Liu

CounterCurate: Enhancing Physical and Semantic Visio-Linguistic Compositional Reasoning via Counterfactual Examples (2024.findings-acl)

Challenge:	Existing video-language models rely on concatenating visual tokens with textual inputs for joint modeling, but this method suffers from significant inefficiency when scaling to long videos with dense visual inputs.
Approach:	They propose a video-to-parameter efficiency paradigm called ViPE that transforms video content into visual perceptual weights, which are directly injected into the LLM’s parameters.
Outcome:	The proposed model reduces FLOPs by 85% and inference time by up to 65% while reducing FLOP and FLOP inference times by up-to-65%.

Copied to clipboard

Jianrui Zhang, Mu Cai, Tengyang Xie, Yong Jae Lee

Worse than Random? An Embarrassingly Simple Probing Evaluation of Large Multimodal Models in Medical VQA (2025.findings-acl)

Challenge:	Using high-performing text and image generation models, we can improve compositional reasoning capabilities for large multimodal models.
Approach:	They propose a framework to improve visio-linguistic compositional reasoning capability for multimodal models by leveraging physical grounding and using image-based models for semantic counterfactual fine-tuning.
Outcome:	The proposed framework outperforms GPT-4V and DALLE-3 on a new dataset and checkpoints.

Copied to clipboard

Qianqi Yan, Xuehai He, Xiang Yue, Xin Eric Wang

Token Preference Optimization with Self-Calibrated Visual-Anchored Rewards for Hallucination Mitigation (2025.findings-emnlp)

Challenge:	Large Multimodal Models (LMMs) have demonstrated impressive performance on existing medical visual question answering benchmarks.
Approach:	They evaluate large multimodal models that perform worse than random guessing on medical questions . authors suggest more robust evaluation methods to ensure reliability of LMMs .
Outcome:	a new study shows that large multimodal models perform worse than random guessing on medical visual question answering benchmarks.

Copied to clipboard

Jihao Gu, Yingyao Wang, Meng Cao, Pi Bu, Jun Song, Bo Zheng, Yancheng He, Shilong Li

Text-Guided Multi-Scale Frequency Representation Adaptation (2026.acl-long)

Challenge:	Existing methods for predicting hallucinations suffer from two drawbacks: Lack of scalable token-level rewards and Neglect of visual-anchored tokens.
Approach:	They propose a Token Preference Optimization model with self-calibrated rewards . they propose based on visual-anchored tokens and visual-aware training objective .
Outcome:	The proposed model improves hallucination performance by focusing on visual-anchored tokens without fine-grained annotations.

Copied to clipboard

Weicai Yan, Xinhua Ma, Wang Lin, Tao Jin

Semantic Text Processing

Challenge:	Existing methods for fine-tuning visual signals are limited by their size and complexity.
Approach:	They propose a multi-scale frequency-based fine-tuning method that integrates textual information and performs multi-level fine- tuning of visual signals in the frequency domain.
Outcome:	Extensive experiments on multimodal models, including CLIP and LLaVA, demonstrate that the proposed method significantly improves performance and efficiency with minimal cost and fast convergence within one epoch.

Pruning Unsafe Tickets: A Resource-Efficient Framework for Safer and More Robust LLMs (2026.acl-long)

Copied to clipboard

Wai Man Si, Mingjie Li, Michael Backes, Yang Zhang

Responsible & Trustworthy NLP

Challenge:	Empirical evaluations on ML models show substantial reductions in unsafe generations and improved robustness against jailbreak attacks.
Approach:	They propose a resource-efficient pruning framework that directly identifies unsafe behaviors while preserving model utility.
Outcome:	The proposed framework reduces unsafe generations and improves robustness against jailbreak attacks with minimal utility loss.

Concept-pedia: a Wide-coverage Semantically-annotated Multimodal Dataset (2025.emnlp-main)

Copied to clipboard

Karim Ghonim, Andrei Stefan Bejgu, Alberte Fernández-Castro, Roberto Navigli

S2H-DPO: Hardness-Aware Preference Optimization for Vision–Language Models (2026.findings-acl)

Challenge:	Current evaluations for Vision-language Models remain heavily anchored to ImageNet .
Approach:	They propose a large-scale semantically-annotated multimodal resource that extends the range of visual concepts, including diverse abstract categories.
Outcome:	The proposed model expands the range of visual concepts, including diverse abstract categories.

Copied to clipboard

Nitish Shukla, Surgan Jandial, Arun Ross

LOTUS: Evolving Multimodal Unlearning via Hyperbolic Entailment and Lorentz Transport (2026.acl-long)

Challenge:	Existing methods focus on localized reasoning with pre-specified image indices, bypassing the skills of global visual search and autonomous cross-image comparison.
Approach:	They propose a learning framework that constructs multi-image preference data across three hierarchical reasoning levels requiring an increasing level of capabilities.
Outcome:	The proposed approach maintains strong single-image reasoning performance while strengthening multi-image understanding capabilities.

Copied to clipboard

Zekun Wang, Jingjie Zeng, Yingxu Li, Hongfei Lin, Liang Yang