Papers by Tianyang Han
MLLM-Protector: Ensuring MLLM’s Safety without Hurting Performance (2024.emnlp-main)
Copied to clipboard
Renjie Pi, Tianyang Han, Jianshu Zhang, Yueqi Xie, Rui Pan, Qing Lian, Hanze Dong, Jipeng Zhang, Tong Zhang
| Challenge: | MLLMs are deployed on limited image-text pairs, which makes them more vulnerable to catastrophic forgetting of their original abilities during safety fine-tuning. |
| Approach: | They propose a plug-and-play strategy that detects harmful visual inputs and transforms harmful ones into harmless ones. |
| Outcome: | The proposed approach mitigates the risks posed by malicious visual inputs without compromising the original performance of MLLMs. |
The Instinctive Bias: Spurious Images lead to Illusion in MLLMs (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing multi-modal large language models (MLLMs) are able to process visual inputs by converting them into visual tokens that share the same latent space as language tokens in LLMs. |
| Approach: | They propose a benchmark that assesses the visual illusion level given spurious images and a pipeline that converts visual inputs into visual tokens. |
| Outcome: | The proposed benchmark shows that MLLMs suffer from an instinctive bias to varying degrees when presented with spurious images. |
Failure makes the agent stronger: Enhancing Accuracy through Structured Reflection for Reliable Tool Interactions (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to self-reflection rely on heuristic prompting or unidirectional reasoning traces. |
| Approach: | They propose a structured reflection method that transforms the "from error to repair" process into a first-class, controllable, and trainable action. |
| Outcome: | The proposed method improves multi-turn tool-call success rates and error recovery while reducing redundant calls. |