Can Textual Unlearning Solve Cross-Modality Safety Alignment? (2024.findings-emnlp)
Copied to clipboard
Trishna Chakraborty, Erfan Shayegani, Zikui Cai, Nael Abu-Ghazaleh, M. Salman Asif, Yue Dong, Amit Roy-Chowdhury, Chengyu Song
| Challenge: | integrating new modalities into large language models creates new attack surface . existing safety training techniques like SFT and RLHF are not feasible in multi-modal settings . |
| Approach: | They explore whether unlearning in the textual domain can be effective for cross-modality safety alignment. |
| Outcome: | The proposed approach reduces the Attack Success Rate (ASR) to less than 8% and preserves the utility. |
Similar Papers
Unraveling and Mitigating Safety Alignment Degradation of Vision-Language Models (2025.findings-acl)
Copied to clipboard
Qin Liu, Chao Shang, Ling Liu, Nikolaos Pappas, Jie Ma, Neha Anna John, Srikanth Doss, Lluis Marquez, Miguel Ballesteros, Yassine Benajiba
| Challenge: | LLaVA-7B demonstrated a decline in safety alignment ability on multi-modal inputs compared to its LLM backbone. |
| Approach: | They propose a method to recover alignment ability from LLM backbone while preserving functional capabilities of VLMs. |
| Outcome: | The proposed framework recovers alignment ability that is inherent in the LLM backbone with minimal impact on fluency and linguistic capabilities of pre-trained VLMs. |
Multimodal Unlearning Across Vision, Language, Video, and Audio: Survey of Methods, Datasets, and Benchmarks (2026.findings-acl)
Copied to clipboard
| Challenge: | Survey aims to identify challenges of multimodal unlearning for vision, language, audio and video . retraining after deletion requests or policy updates is often impractical, survey finds . |
| Approach: | They propose to enable selective removal across modalities while retaining overall utility. |
| Outcome: | This study compares models with existing models to identify weaknesses and improves performance. |
The Language Barrier: Dissecting Safety Challenges of LLMs in Multilingual Contexts (2024.findings-acl)
Copied to clipboard
Lingfeng Shen, Weiting Tan, Sihao Chen, Yunmo Chen, Jingyu Zhang, Haoran Xu, Boyuan Zheng, Philipp Koehn, Daniel Khashabi
| Challenge: | Recent studies show that malicious prompt instructions could solicit objectionable content from LLMs. |
| Approach: | They compare how state-of-the-art LLMs respond to malicious prompts in different languages . they find that LLM's generate unsafe responses more often when a prompt is written in a lower-resource language . |
| Outcome: | The proposed model can generate unsafe responses more often when a malicious prompt is written in a lower-resource language, and less irrelevant responses when written in lower-source languages. |
Text-centric Alignment for Bridging Test-time Unseen Modality (2025.findings-emnlp)
Copied to clipboard
| Challenge: | a text-centric alignment method is used to handle unseen modalities and dynamic modality combinations at test time. |
| Approach: | They propose a text-centric alignment method that unifies different input modalities into a single semantic text representation by leveraging in-context learning with Large Language Models and uni-modal foundation models. |
| Outcome: | The proposed method unifies input modalities into a single semantic representation . it significantly improves the ability to manage unseen, diverse, and unpredictable modality combinations . |
Towards Understanding the Fragility of Multilingual LLMs against Fine-Tuning Attacks (2025.findings-naacl)
Copied to clipboard
| Challenge: | Recent advances in Large Language Models have sparked concerns about their safety. |
| Approach: | They propose a method to identify safety-related information in the model parameter space . they propose to use a few adversarially chosen examples to fine-tune LLMs . |
| Outcome: | The proposed method can break safety alignment in multilingual LLMs using a few examples . it also shows that the proposed method jailbreaks LLM models adapted to new languages . |
A Survey on Training-free Alignment of Large Language Models (2025.findings-emnlp)
Copied to clipboard
Birong Pan, Yongqi Li, Weiyu Zhang, Wenpeng Lu, Mayi Xu, Shen Zhou, Yuanyuan Zhu, Ming Zhong, Tieyun Qian
| Challenge: | a survey of large language models (LLMs) aims to ensure outputs adhere to human values, ethical standards, and legal norms. |
| Approach: | They present the first systematic review of TF alignment methods . they categorize them by stages of pre-decoding, in-decoder and post-decoration . |
| Outcome: | The proposed methods are based on training-free (TF) alignment techniques . they are able to be used in open-source and closed-source environments without retraining . |
CLASP: Cross-modal Alignment Using Pre-trained Unimodal Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent advances in speech-text pretraining rely on parallel speech- text data . however, data accessibility is a challenge due to the limited data available. |
| Approach: | They propose a framework for jointly performing speech and text processing without parallel corpora during pre-training but only downstream. |
| Outcome: | The proposed framework extracts distinct representations for speech and text, aligning them effectively in a newly defined space using a multi-level contrastive learning mechanism. |
Safety Alignment via Constrained Knowledge Unlearning (2025.acl-long)
Copied to clipboard
Zesheng Shi, Yucheng Zhou, Jing Li, Yuxin Jin, Yu Li, Daojing He, Fangming Liu, Saleh Alharbi, Jun Yu, Min Zhang
| Challenge: | Existing defense mechanisms have not fully deleted harmful knowledge in large language models (LLMs) Existing methods to address safety alignment have not completely deleted harmful information in LLMs. |
| Approach: | They propose a safety alignment strategy that uses scoring neurons to identify useful knowledge in LLMs and pruning the gradients of neurons in U to preserve beneficial information. |
| Outcome: | The proposed method significantly improves model safety while maintaining utility compared to existing methods. |
Multi-modal Preference Alignment Remedies Degradation of Visual Instruction Tuning on Language Models (2024.acl-long)
Copied to clipboard
| Challenge: | Multi-modal large language models (MLLMs) are expected to support multi-turn queries of interchanging image and text modalities in production. |
| Approach: | They propose to use visual-question-answering (VQA) datasets to annotate a 5k-sample VQA preference dataset and to investigate the degradation of VQA datasets. |
| Outcome: | The proposed model surpasses the instruction-following capabilities of the language model with DPO and SteerLM. |
Thesis Proposal: Targeted and Unified Cross-Lingual Unlearning from Multilingual Language Models (2026.acl-srw)
Copied to clipboard
| Challenge: | Large language models trained on corpora scraped from the web can reproduce sensitive and copyright-protected data. |
| Approach: | They propose to extend existing benchmarks to multilingual data by compiling parallel translations of question-answer pairs consisting of real-world facts and synthetic personally identifiable information. |
| Outcome: | The proposed dataset will include translations of question-answer pairs consisting of real-world facts and synthetic personally identifiable information. |