Papers by Qing Lian
MLLM-Protector: Ensuring MLLM’s Safety without Hurting Performance (2024.emnlp-main)
Copied to clipboard
Renjie Pi, Tianyang Han, Jianshu Zhang, Yueqi Xie, Rui Pan, Qing Lian, Hanze Dong, Jipeng Zhang, Tong Zhang
| Challenge: | MLLMs are deployed on limited image-text pairs, which makes them more vulnerable to catastrophic forgetting of their original abilities during safety fine-tuning. |
| Approach: | They propose a plug-and-play strategy that detects harmful visual inputs and transforms harmful ones into harmless ones. |
| Outcome: | The proposed approach mitigates the risks posed by malicious visual inputs without compromising the original performance of MLLMs. |
The Instinctive Bias: Spurious Images lead to Illusion in MLLMs (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing multi-modal large language models (MLLMs) are able to process visual inputs by converting them into visual tokens that share the same latent space as language tokens in LLMs. |
| Approach: | They propose a benchmark that assesses the visual illusion level given spurious images and a pipeline that converts visual inputs into visual tokens. |
| Outcome: | The proposed benchmark shows that MLLMs suffer from an instinctive bias to varying degrees when presented with spurious images. |
R-Tuning: Instructing Large Language Models to Say ‘I Don’t Know’ (2024.naacl-long)
Copied to clipboard
Hanning Zhang, Shizhe Diao, Yong Lin, Yi Fung, Qing Lian, Xingyao Wang, Yangyi Chen, Heng Ji, Tong Zhang
| Challenge: | Existing methods for instruction tuning force the model to complete a sentence no matter whether it knows the knowledge or not. |
| Approach: | They propose a new approach to tuning large language models to refrain from answering questions beyond its parametric knowledge by identifying the disparity in parametric and parametric information. |
| Outcome: | The proposed approach improves a model’s ability to answer known questions and refrain from answering unknown questions. |
IPL: Leveraging Multimodal Large Language Models for Intelligent Product Listing (2024.emnlp-industry)
Copied to clipboard
Kang Chen, Qing Zhang, Chengbao Lian, Yixin Ji, Xuwei Liu, Shuguang Han, Guoqiang Wu, Fei Huang, Jufeng Chen
| Challenge: | Unlike professional Business-to-Consumer (B2C) e-commerce platforms, consumer-to consumer (C2C), is mainly targeting individual sellers. |
| Approach: | They develop an intelligent product listing tool that generates product descriptions using various product attributes such as category, brand, color, condition, etc. |
| Outcome: | The proposed tool outperforms the base model in domain-specific tasks while producing less hallucination. |