| Challenge: | Existing studies show that textual unlearning does not achieve comparable safety performance with image-text alignment. |
| Approach: | They propose to use textual unlearning to align MLLMs with image-text pairs to explain this problem . they construct a visual leakless safety bench with 2.2k image- text pairs to test this problem. |
| Outcome: | The proposed model can refuse image-text pairs according to textual queries, leading to unreliable safety evaluations. |
Similar Papers
AlignMMBench: Evaluating Chinese Multimodal Alignment in Large Vision-Language Models (2025.acl-long)
Copied to clipboard
| Challenge: | Existing benchmarks focus on basic abilities using nonverbal methods, such as yes-no and multiple-choice questions. |
| Approach: | They propose a benchmark that provides more nuanced evaluations of alignment capabilities for large Vision-Language Models (VLMs) they use a rule-calibrated evaluator that exceeds GPT-4's evaluation ability and a “alignment score” to assess the robustness and stability of models across diverse prompts. |
| Outcome: | The proposed benchmark covers 13 tasks across three categories and includes both single-turn and multi-turn dialogue scenarios. |
MVTamperBench: Evaluating Robustness of Vision-Language Models (2025.findings-acl)
Copied to clipboard
Amit Agarwal, Srikant Panda, Angeline Charles, Hitesh Laxmichand Patel, Bhargava Kumar, Priyaranjan Pattnayak, Taki Hasan Rafi, Tejaswini Kumar, Hansa Meghwani, Karan Gupta, Dong-Kyu Chae
| Challenge: | Multimodal Large Language Models (MLLMs) have been a key advance in video understanding but their vulnerability to adversarial tampering remains underexplored. |
| Approach: | They evaluate MLLMs against five prevalent tampering techniques to assess their robustness . they use a tampered video format to examine the vulnerability of ML models . |
| Outcome: | The benchmark evaluates MLLMs against five prevalent tampering techniques based on 19 video manipulation tasks. |
Can’t See the Forest for the Trees: Benchmarking Multimodal Safety Awareness for Multimodal LLMs (2025.acl-long)
Copied to clipboard
Wenxuan Wang, Xiaoyuan Liu, Kuiyi Gao, Jen-tse Huang, Youliang Yuan, Pinjia He, Shuai Wang, Zhaopeng Tu
| Challenge: | Multimodal Large Language Models (MLLMs) have expanded the capabilities of traditional language models by enabling interaction through both text and images. |
| Approach: | They propose a multimodal safety awareness benchmark to evaluate MLLMs across 29 safety scenarios with 1,500 carefully curated image-prompt pairs. |
| Outcome: | The proposed model is able to identify unsafe content and avoid over-sensitivity that can hinder helpfulness. |
USB: A COMPREHENSIVE AND UNIFIED SAFETY EVALUATION BENCHMARK FOR MULTIMODAL LARGE LANGUAGE MODELS (2026.acl-long)
Copied to clipboard
Baolin Zheng, Guanlin Chen, Qingyang Teng, Hongqiong Zhong, Yingshui Tan, Zhendong Liu, Weixun Wang, Jiaheng Liu, Jian Yang, Huiyun Jing, Jincheng Wei, Wenbo Su, Xiaoyong Zhu, Bo Zheng, Kaifu Zhang
| Challenge: | Existing safety benchmarks fail to provide reliable assessments due to limited risk coverage, insufficient scale and the oversight of complex modality combinations. |
| Approach: | They propose a framework that covers 61 risk categories across four modality interactions to address this gap. |
| Outcome: | The proposed framework covers 61 risk categories across four distinct modality interactions. |
MIBench: Evaluating Multimodal Large Language Models over Multiple Images (2024.emnlp-main)
Copied to clipboard
Haowei Liu, Xi Zhang, Haiyang Xu, Yaya Shi, Chaoya Jiang, Ming Yan, Ji Zhang, Fei Huang, Chunfeng Yuan, Bing Li, Weiming Hu
| Challenge: | Existing benchmarks and MLLMs focus on single-image input scenarios, leaving performance of ML models when handling multiple images underexplored. |
| Approach: | They propose a benchmark to evaluate fine-grained abilities of multimodal large language models in multi-image scenarios. |
| Outcome: | The proposed benchmark categorizes the multi-image abilities into three scenarios: MII, MKS and MIC. |
SEA: Low-Resource Safety Alignment for Multimodal Large Language Models via Synthetic Embeddings (2025.acl-long)
Copied to clipboard
| Challenge: | Existing low-resource security alignment methods struggle with the security risks posed by additional modalities. |
| Approach: | They propose to use multimodal datasets to enhance safety alignment but it is costly to construct these datasets. |
| Outcome: | Experiments on image, video, and audio-based MLLMs show that the proposed method can synthesize a high-quality embedding on a single RTX3090 GPU within 24 seconds. |
Can Textual Unlearning Solve Cross-Modality Safety Alignment? (2024.findings-emnlp)
Copied to clipboard
Trishna Chakraborty, Erfan Shayegani, Zikui Cai, Nael Abu-Ghazaleh, M. Salman Asif, Yue Dong, Amit Roy-Chowdhury, Chengyu Song
| Challenge: | integrating new modalities into large language models creates new attack surface . existing safety training techniques like SFT and RLHF are not feasible in multi-modal settings . |
| Approach: | They explore whether unlearning in the textual domain can be effective for cross-modality safety alignment. |
| Outcome: | The proposed approach reduces the Attack Success Rate (ASR) to less than 8% and preserves the utility. |
MLLM-Protector: Ensuring MLLM’s Safety without Hurting Performance (2024.emnlp-main)
Copied to clipboard
Renjie Pi, Tianyang Han, Jianshu Zhang, Yueqi Xie, Rui Pan, Qing Lian, Hanze Dong, Jipeng Zhang, Tong Zhang
| Challenge: | MLLMs are deployed on limited image-text pairs, which makes them more vulnerable to catastrophic forgetting of their original abilities during safety fine-tuning. |
| Approach: | They propose a plug-and-play strategy that detects harmful visual inputs and transforms harmful ones into harmless ones. |
| Outcome: | The proposed approach mitigates the risks posed by malicious visual inputs without compromising the original performance of MLLMs. |
MCiteBench: A Multimodal Benchmark for Generating Text with Citations (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing work focuses on generating citations for text-only content . experimental results reveal MLLMs struggle to ground outputs reliably when handling multimodal input . |
| Approach: | They propose a benchmark to assess the ability of MLLMs to generate text with citations in multimodal contexts. |
| Outcome: | The proposed benchmark assesses the ability of MLLMs to generate text with citations in multimodal contexts. |
MANBench: Is Your Multimodal Model Smarter than Human? (2025.findings-acl)
Copied to clipboard
| Challenge: | Multimodal Large Language Models (MLLMs) have been gaining popularity in multimodal tasks . a bilingual benchmark is available for MLLM users to evaluate their multimodal capabilities . |
| Approach: | They propose a bilingual multimodal ability norms benchmark that measures multimodality across nine tasks. |
| Outcome: | The proposed benchmark compared human performance against state-of-the-art MLLMs. |