Protecting Privacy in Multimodal Large Language Models with MLLMU-Bench

Challenge:	Large Language Models (LLMs) and Multimodal Large Language models (MLLMs) trained on vast web corpora can memorize and disclose individuals’ confidential and private data, raising legal and ethical concerns.
Approach:	They propose a benchmark to assess unlearning algorithms from multiple perspectives and provide a baseline for existing generative models.
Outcome:	The proposed benchmark consists of 500 fictitious profiles and 153 profiles for public celebrities, evaluated from both multimodal (image+text) and unimodal (text) perspectives.

Similar Papers

Overview

Challenge

Approach

Outcome

SafeEraser: Enhancing Safety in Multimodal Large Language Models through Multimodal Machine Unlearning (2025.findings-acl)

Copied to clipboard

Junkai Chen, Zhijie Deng, Kening Zheng, Yibo Yan, Shuliang Liu, PeiJun Wu, Peijie Jiang, Jia Liu, Xuming Hu

Language Models

Challenge:	Existing methods for MU forget quality and model utility are not fully explored for safety in MLLMs.
Approach:	They propose a safety unlearning benchmark for MLLMs to measure over-forgetting . they propose MU methods to forget quality and model utility .
Outcome:	The proposed method reduces over-forgetting by 79.5% while maintaining forget quality and model utility.

OFFSIDE: Benchmarking Unlearning Misinformation in Multimodal Large Language Models (2026.findings-acl)

Copied to clipboard

Hao Zheng, Zirui Pang, Ling Li, Zhijie Deng, Yuhan Pu, Zhaowei Zhu, Xiaobo Xia, Jiaheng Wei

Responsible & Trustworthy NLP

Challenge:	Existing benchmarks for MU are limited by a lack of image diversity and coarse-grained unlearning targets.
Approach:	They propose a benchmark to evaluate misinformation unlearning in MLLMs . OFFSIDE supports advanced unlearning targets such as fine-grained unlearning and visual rumor removal.
Outcome:	OFFSIDE supports advanced unlearning targets, such as fine-grained unlearning and visual rumor removal.

MLLM-Bench: Evaluating Multimodal LLMs with Per-sample Criteria (2025.naacl-long)

Copied to clipboard

Wentao Ge, Shunian Chen, Hardy Chen, Nuo Chen, Junying Chen, Zhihong Chen, Wenya Xie, Shuo Yan, ChenghaoZhu ChenghaoZhu, Ziyue Lin, Dingjie Song, Xidong Wang, Anningzhe Gao, Zhang Zhiyi, Jianquan Li, Xiang Wan, Benyou Wang

Multimodality

Challenge:	Existing evaluation methodologies for multimodal large language models are limited in evaluating objective queries without considering real-world user experiences.
Approach:	They propose to evaluate multimodal large language models with per-sample criteria using potent MLLM as the judge.
Outcome:	The proposed evaluation paradigm shows that it can be used to evaluate multimodal large language models with per-sample criteria.

Both Text and Images Leaked! A Systematic Analysis of Data Contamination in Multimodal LLM (2025.findings-emnlp)

Copied to clipboard

Dingjie Song, Sicheng Lai, Mingxuan Wang, Shunian Chen, Lichao Sun, Benyou Wang

Multimodality

Challenge:	Existing methods for unimodal large language models are inadequate for MLLMs due to multimodal data complexity and multi-phase training.
Approach:	MM-DETECT analyzes data contamination using a framework that defines two contamination categories - unimodal and cross-modal .
Outcome:	The proposed framework quantifies contamination severity across multiple-choice and caption-based Visual Question Answering tasks.

From LLMs to MLLMs: Exploring the Landscape of Multimodal Jailbreaking (2024.emnlp-main)

Copied to clipboard

Siyuan Wang, Zhuohan Long, Zhihao Fan, Zhongyu Wei

Semantic Text Processing

Challenge:	Recent advances in Large Language Models (LLMs) have demonstrated remarkable performance across various tasks, effectively following instructions to meet diverse user needs.
Approach:	They propose a framework for evaluation benchmarks and attack techniques for LLMs and MLLMs to enhance their security.
Outcome:	The proposed frameworks have been exploited to exploit the weaknesses of LLMs and MLLMs.

MM-SOC: Benchmarking Multimodal Large Language Models in Social Media Platforms (2024.findings-acl)

Copied to clipboard

Yiqiao Jin, Minje Choi, Gaurav Verma, Jindong Wang, Srijan Kumar

Language Models

Challenge:	Social media platforms are hubs for multimodal information exchange, encompassing text, images, and videos, making it challenging for machines to comprehend the information or emotions associated with interactions in online spaces.
Approach:	They propose a benchmark to evaluate MLLMs' understanding of multimodal social media content and a large-scale YouTube tagging dataset to evaluate their performance.
Outcome:	The proposed model performs better in a zero-shot setting, suggesting potential improvements.

PrivLM-Bench: A Multi-level Privacy Evaluation Benchmark for Language Models (2024.acl-long)

Copied to clipboard

Haoran Li, Dadi Guo, Donghao Li, Wei Fan, Qi Hu, Xin Liu, Chunkit Chan, Duanyi Yao, Yuan Yao, Yangqiu Song

Responsible & Trustworthy NLP

Challenge:	generative large language models (LLMs) exhibit surprising capability and integrate previous tasks into a unified text generation formulation.
Approach:	They propose a privacy evaluation benchmark to quantify the privacy leakage of language models.
Outcome:	The proposed benchmark compares PPLMs with different privacy implementations to find out how privacy leakage is handled.

Unveiling Uncertainty: A Deep Dive into Calibration and Performance of Multimodal Large Language Models (2025.coling-main)

Copied to clipboard

Zijun Chen, Wenbo Hu, Guande He, Zhijie Deng, ZHeng ZHang, Richang Hong

Multimodality

Challenge:	Multimodal large language models combine visual and textual data for tasks like image captioning and visual question answering.
Approach:	They propose temperature scaling and iterative prompt optimization to calibrate MLLMs and enhance model reliability.
Outcome:	The proposed techniques improve MLLMs and improve model reliability.

SUA: Stealthy Multimodal Large Language Model Unlearning Attack (2025.emnlp-main)

Copied to clipboard

Xianren Zhang, Hui Liu, Delvin Ce Zhang, Xianfeng Tang, Qi He, Dongwon Lee, Suhang Wang

Semantic Text Processing

Challenge:	Multimodal Large Language Models (MLLMs) trained on massive data may memorize sensitive personal information and photos, posing privacy and copyright concerns.
Approach:	They propose a framework that learns a universal noise pattern to recover unlearned information from MLLMs.
Outcome:	The proposed framework learns a universal noise pattern and can reveal unlearned content when applied to images.

Towards Unified Multimodal Large Language Models: A survey (2026.findings-acl)

Copied to clipboard

Xu Ma, Yitian Zhang, Yun Fu

Semantic Text Processing

Challenge:	unified multimodal large language models (MLLMs) are emerging but lack a systematic framework to connect them and situate current trends within a broader landscape.
Approach:	They present a systematic review of unified Multimodal Large Language Models . they outline the foundational concepts and prerequisites for understanding them .
Outcome:	The present review provides a systematic and systematic overview of unified MLLMs . it discusses persistent challenges and identify promising directions for future research .