| Challenge: | Recent advances in Multimodal Large Language Models (MLLMs) have unlocked powerful cross-modal reasoning abilities, but also raised new safety concerns, especially when faced with adversarial multimodal inputs. |
| Approach: | They propose a modular and adaptive inference-time intervention technology, AutoSteer, that integrates a safety awareness score, an adaptive safety prober, and a lightweight Refusal Head to modulate generation when safety risks are detected. |
| Outcome: | Experiments on LLaVA-OV and Chameleon show that AutoSteer significantly reduces the Attack Success Rate (ASR) for textual, visual, and cross-modal threats while maintaining general abilities. |
Similar Papers
SafeSteer: A Decoding-level Defense Mechanism for Multimodal Large Language Models (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing defense methods rely on fine-tuning or inefficient post-hoc interventions, limiting their ability to address novel attacks. |
| Approach: | They propose a decoding-level defense mechanism that employs a lightweight discriminator to iteratively steer the decoding process toward safety. |
| Outcome: | The proposed method improves safety performance by up to 33.40% without fine-tuning on multiple MLLMs. |
SURE: Safety Understanding and Reasoning Enhancement for Multimodal Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing multimodal large language models incorporate visual and textual information, but introduces new and complex safety risks. |
| Approach: | They propose a safety reasoning framework that integrates visual modalities into multimodal models to help them resist jailbreak attacks. |
| Outcome: | The proposed framework improves model safety while avoiding over-defense . it is based on a large-scale safety reasoning dataset . |
SafeEraser: Enhancing Safety in Multimodal Large Language Models through Multimodal Machine Unlearning (2025.findings-acl)
Copied to clipboard
Junkai Chen, Zhijie Deng, Kening Zheng, Yibo Yan, Shuliang Liu, PeiJun Wu, Peijie Jiang, Jia Liu, Xuming Hu
| Challenge: | Existing methods for MU forget quality and model utility are not fully explored for safety in MLLMs. |
| Approach: | They propose a safety unlearning benchmark for MLLMs to measure over-forgetting . they propose MU methods to forget quality and model utility . |
| Outcome: | The proposed method reduces over-forgetting by 79.5% while maintaining forget quality and model utility. |
When Helpers Become Hazards: A Benchmark for Analyzing Multimodal LLM-Powered Safety in Daily Life (2026.findings-acl)
Copied to clipboard
Xinyue Lou, Xu Jinan, Jingyi Yin, Xiaolong Wang, Zhaolu Kang, null Liaoyouwei, Yixuan Wang, Xiangyu Shi, Fengran Mo, SU Yao, Kaiyu Huang
| Challenge: | Safety impact of Multimodal Large Language Models (MLLMs) on human behavior is evaluated in this study. |
| Approach: | They propose a safety-warning-based evaluation framework that encourages models to provide clear and informative safety warnings, rather than generic refusals. |
| Outcome: | The proposed safety-warning-based evaluation framework encourages models to provide clear and informative safety warnings, rather than generic refusals. |
A Simple Yet Effective Method for Non-Refusing Context Relevant Fine-grained Safety Steering in LLMs (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for fine-tuning large language models to meet safety policies are costly and impractical. |
| Approach: | They propose a method to fine-tune large language models to meet evolving safety policies by applying a gradient-free, unsupervised approach. |
| Outcome: | The proposed method provides precise control, avoids blanket refusals, and directs models to generate safe, relevant content. |
Can’t See the Forest for the Trees: Benchmarking Multimodal Safety Awareness for Multimodal LLMs (2025.acl-long)
Copied to clipboard
Wenxuan Wang, Xiaoyuan Liu, Kuiyi Gao, Jen-tse Huang, Youliang Yuan, Pinjia He, Shuai Wang, Zhaopeng Tu
| Challenge: | Multimodal Large Language Models (MLLMs) have expanded the capabilities of traditional language models by enabling interaction through both text and images. |
| Approach: | They propose a multimodal safety awareness benchmark to evaluate MLLMs across 29 safety scenarios with 1,500 carefully curated image-prompt pairs. |
| Outcome: | The proposed model is able to identify unsafe content and avoid over-sensitivity that can hinder helpfulness. |
MLLM-Protector: Ensuring MLLM’s Safety without Hurting Performance (2024.emnlp-main)
Copied to clipboard
Renjie Pi, Tianyang Han, Jianshu Zhang, Yueqi Xie, Rui Pan, Qing Lian, Hanze Dong, Jipeng Zhang, Tong Zhang
| Challenge: | MLLMs are deployed on limited image-text pairs, which makes them more vulnerable to catastrophic forgetting of their original abilities during safety fine-tuning. |
| Approach: | They propose a plug-and-play strategy that detects harmful visual inputs and transforms harmful ones into harmless ones. |
| Outcome: | The proposed approach mitigates the risks posed by malicious visual inputs without compromising the original performance of MLLMs. |
USB: A COMPREHENSIVE AND UNIFIED SAFETY EVALUATION BENCHMARK FOR MULTIMODAL LARGE LANGUAGE MODELS (2026.acl-long)
Copied to clipboard
Baolin Zheng, Guanlin Chen, Qingyang Teng, Hongqiong Zhong, Yingshui Tan, Zhendong Liu, Weixun Wang, Jiaheng Liu, Jian Yang, Huiyun Jing, Jincheng Wei, Wenbo Su, Xiaoyong Zhu, Bo Zheng, Kaifu Zhang
| Challenge: | Existing safety benchmarks fail to provide reliable assessments due to limited risk coverage, insufficient scale and the oversight of complex modality combinations. |
| Approach: | They propose a framework that covers 61 risk categories across four modality interactions to address this gap. |
| Outcome: | The proposed framework covers 61 risk categories across four distinct modality interactions. |
DREAM: Disentangling Risks to Enhance Safety Alignment in Multimodal Large Language Models (2025.naacl-long)
Copied to clipboard
Jianyu Liu, Hangyu Guo, Ranjie Duan, Xingyuan Bu, Yancheng He, Shilong Li, Hui Huang, Jiaheng Liu, Yucheng Wang, Chenchen Jing, Xingwei Qu, Xiao Zhang, Pei Wang, Yanan Wu, Jihao Gu, Yangguang Li, Jianke Zhu
| Challenge: | Multimodal Large Language Models (MLLMs) pose unique safety challenges due to their integration of visual and textual data. |
| Approach: | They propose a method to disentangle risks through step-by-step reasoning within multimodal inputs. |
| Outcome: | The proposed approach improves safety alignment in MLLMs by fine-tuning and iterative Reinforcement Learning from AI feedback. |
ErrorRadar: Benchmarking Complex Mathematical Reasoning of Multimodal Large Language Models Via Error Detection (2026.findings-acl)
Copied to clipboard
Yibo Yan, Shen Wang, Jiahao Huo, Hang Li, Boyan Li, Jiamin Su, Xiong Gao, YiFan Zhang, Tianlong Xu, Zhendong Chu, Aoxiao Zhong, Kun Wang, Hui Xiong, Philip S. Yu, Xuming Hu, Qingsong Wen
| Challenge: | Current mathematical benchmarks focus on evaluating MLLMs’ problem-solving ability, yet there is a crucial gap in addressing more complex scenarios such as error detection. |
| Approach: | They propose to evaluate multimodal error detection by evaluating two sub-tasks error step identification and error categorization. |
| Outcome: | The proposed task evaluates MLLMs' ability to handle multimodal questions compared to text-only models. |