Papers by Chaowei Xiao
Test-time Backdoor Mitigation for Black-Box Large Language Models with Defensive Demonstrations (2025.findings-naacl)
Copied to clipboard
| Challenge: | Existing studies on backdoor defense have focused on training phase, overlooking critical aspect of testing time defense. |
| Approach: | They propose to use demonstrations as a defense mechanism against backdoor attacks in black-box LLMs. |
| Outcome: | The proposed method outperforms existing defense baselines across most evaluation scenarios. |
AGrail: A Lifelong Agent Guardrail with Effective and Adaptive Safety Detection (2025.acl-long)
Copied to clipboard
| Challenge: | Existing defense agencies fail to adaptively and effectively mitigate these risks. |
| Approach: | They propose a lifelong agent guardrail that enhances LLM agent safety by enabling adaptive safety check generation, effective safety check optimization, and tool compatibility & flexibility. |
| Outcome: | The proposed agent guardrail achieves strong performance against task-specific and systemic risks and is transferable across different LLM agents’ tasks. |
Re-ViLM: Retrieval-Augmented Visual Language Model for Zero and Few-Shot Image Captioning (2023.findings-emnlp)
Copied to clipboard
Zhuolin Yang, Wei Ping, Zihan Liu, Vijay Korthikanti, Weili Nie, De-An Huang, Linxi Fan, Zhiding Yu, Shiyi Lan, Bo Li, Mohammad Shoeybi, Ming-Yu Liu, Yuke Zhu, Bryan Catanzaro, Chaowei Xiao, Anima Anandkumar
| Challenge: | Existing methods for image-to-text generation store all knowledge within parameters, thus requiring computational-expensive fine-tuning. |
| Approach: | They propose a Retrieval-augmented Visual Language Model that stores all the knowledge within parameters and can be used to retrieve it from the external database. |
| Outcome: | The proposed model significantly boosts performance for image-to-text generation tasks with 4x less parameters compared with baseline methods. |
Defending against Insertion-based Textual Backdoor Attacks via Attribution (2023.findings-acl)
Copied to clipboard
| Challenge: | Textual backdoor attacks are vulnerable to backdoors and can be used to infect models trained on poisoned data. |
| Approach: | They propose an efficient attribution-based pipeline to defend against two insertion-based poisoning attacks, BadNL and InSent. |
| Outcome: | The proposed method can generalize sufficiently well in two common attack scenarios, which consistently improves previous methods. |
Cognitive Overload: Jailbreaking Large Language Models with Overloaded Logical Thinking (2024.findings-naacl)
Copied to clipboard
| Challenge: | Large language models (LLMs) have demonstrated increasing power, but they also have vulnerabilities. |
| Approach: | They propose a black-box attack that targets the cognitive structure and processes of large language models (LLMs) they propose defending cognitive overload attacks from three perspectives. |
| Outcome: | The proposed attack is a black-box attack with no need for knowledge of model architecture or access to model weights. |
Instructions as Backdoors: Backdoor Vulnerabilities of Instruction Tuning for Large Language Models (2024.naacl-long)
Copied to clipboard
| Challenge: | et al., 2021) show that instruction models can be trained on crowdsourced datasets with task instructions to achieve superior performance. |
| Approach: | They examine security concerns of emergent instruction tuning paradigm that models are trained on crowdsourced datasets with task instructions to achieve superior performance. |
| Outcome: | The proposed model can achieve 90% success rate across four commonly used datasets. |
From Shortcuts to Triggers: Backdoor Defense with Denoised PoE (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing backdoor defense methods focus on specific triggers, leaving a universal defense unexplored. |
| Approach: | They propose an ensemble-based backdoor defense framework that denies backdoor attacks by capturing backdoor shortcuts and preventing learning them. |
| Outcome: | The proposed framework significantly improves defense performance against backdoor attacks . it is also effective under a more challenging but practical setting . |
HiCL: Hierarchical Contrastive Learning of Unsupervised Sentence Embeddings (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods that encode a sequence in its entirety for contrast with others often neglect local representation learning. |
| Approach: | They propose a hierarchical contrastive learning framework, HiCL, which considers local segment-level and global sequence-level relationships to improve training efficiency and effectiveness. |
| Outcome: | The proposed framework improves training efficiency and effectiveness by dividing a sequence into several segments and using local and global contrastive learning to model relationships. |
RLHFPoison: Reward Poisoning Attack for Reinforcement Learning with Human Feedback in Large Language Models (2024.acl-long)
Copied to clipboard
| Challenge: | Recent advances in Large Language Models (LLMs) have significantly enhanced the capabilities in natural language processing. |
| Approach: | They propose a method to poison large language models by using annotators to rank a set of collected responses to generate longer tokens. |
| Outcome: | The proposed method can generate longer tokens without harming the original safety alignment performance. |
Shall We Pretrain Autoregressive Language Models with Retrieval? A Comprehensive Study (2023.emnlp-main)
Copied to clipboard
Boxin Wang, Wei Ping, Peng Xu, Lawrence McAfee, Zihan Liu, Mohammad Shoeybi, Yi Dong, Oleksii Kuchaiev, Bo Li, Chaowei Xiao, Anima Anandkumar, Bryan Catanzaro
| Challenge: | a recent study shows that retrieval-augmented LMs can improve text generation quality and accuracy. |
| Approach: | They propose a model that reproduces RETRO parameters while retrieving a text corpus . they find RETRO outperforms GPT on text generation with less repetition . |
| Outcome: | The proposed model outperforms standard retrieval-augmented GPT and retrieval augmented GTP on text generation and accuracy tasks. |
Copyright Detective: A Forensic System to Evidence LLMs Flickering Copyright Leakage Risks (2026.acl-demo)
Copied to clipboard
Guangwei Zhang, Jianing Zhu, Cheng Qian, Neil Zhenqiang Gong, Rada Mihalcea, Zhaozhuo Xu, Jingrui He, Jiaqi W. Ma, Chaowei Xiao, Bo Li, Ahmed Abbasi, Dongwon Lee, Heng Ji, Denghui Zhang
| Challenge: | **Copyright Detective** is the first interactive forensic system for detecting, analyzing, and visualizing potential copyright risks in LLM outputs. |
| Approach: | They propose a system that detects copyright infringements and visualizes them . they use content recall testing, paraphrase-level similarity analysis and persuasive jailbreak probing . |
| Outcome: | The proposed system detects, analyzes, and visualizes potential copyright risks in LLM outputs. |
Instructional Fingerprinting of Large Language Models (2024.naacl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) require considerable cost to train from scratch . fingerprinting is essential to protect intellectual property and to ensure downstream users and developers adhere to their license terms. |
| Approach: | They propose a method for fingerprinting Large language models that implants a private key into the model to generate specific text when the key is present. |
| Outcome: | The proposed method prevents publisher overclaim and maintains robustness against fingerprint guessing and parameter-efficient training. |
PIGuard: Prompt Injection Guardrail via Mitigating Overdefense for Free (2025.acl-long)
Copied to clipboard
| Challenge: | Prompt injection attacks pose a critical threat to large language models, enabling goal hijacking and data leakage. |
| Approach: | They propose a prompt guard model that incorporates a new training strategy to mitigate over-defense for free . PIGuard significantly reduces the bias on trigger words, enabling fine-grained evaluation . |
| Outcome: | The proposed model outperforms the existing model on diverse benchmarks by 30.4%. |
RePD: Defending Jailbreak Attack through a Retrieval-based Prompt Decomposition Process (2025.findings-naacl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are still susceptible to jailbreak exploits despite rigorous pre-training and fine-tuning focused on ethical alignment . |
| Approach: | They propose a Retrieval-based attack Retriest-based Prompt Decomposition framework that decomposes harmful queries embedded within user prompts into a one-shot learning example to effectively teach the LLM to discern and separate malicious components. |
| Outcome: | The proposed framework is capable of decomposing harmful queries from the original query and enhancing the resilience of large language models against jailbreak attacks without compromising their performance. |
ChatGPT as an Attack Tool: Stealthy Textual Backdoor Attack via Blackbox Generative Model Trigger (2024.naacl-long)
Copied to clipboard
| Challenge: | Textual backdoor attacks are increasingly challenging to detect due to the use of advanced generative models such as GPT-4. |
| Approach: | They propose a framework that harnesses advanced generative models to execute stealthier backdoor attacks on text classifiers. |
| Outcome: | The proposed framework achieves state-of-the-art attack success rate of 97.35% over four sentiment classification tasks and four human cognition stealthiness tests. |
Defenses Against Prompt Attacks Learn Surface Heuristics (2026.acl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) are increasingly deployed in security-sensitive applications . recent defenses rely on supervised fine-tuning with benign and malicious labels . position bias arises when benign content placed later in a prompt is rejected at much higher rates . |
| Approach: | They analyze three recurring shortcut behaviors induced by supervised fine-tuning . position bias arises when benign content placed later in a prompt is rejected . token trigger bias occurs when strings common in attack data raise rejection probability . |
| Outcome: | The proposed model overrides intended logic when adversarial instructions appear . the proposed model has low rejection rates but narrow correlations in defense data . |
CVE-Bench: Benchmarking LLM-based Software Engineering Agent’s Ability to Repair Real-World CVE Vulnerabilities (2025.naacl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) and LLM agents have demonstrated significant potential in this domain by understanding descriptions in natural language and generating corresponding formal code. |
| Approach: | They propose an evaluation framework that provides LLM agents with a test environment that simulates the real-world vulnerability repair process. |
| Outcome: | The proposed framework can repair 21% of vulnerabilities at its best, but lacks expert knowledge . the evaluation framework can only repair 29% of vulnerabilities, but it can be used in real-world scenarios . |
SudoLM: Learning Access Control of Parametric Knowledge with Authorization Alignment (2025.acl-long)
Copied to clipboard
| Challenge: | Existing preference alignment is a one-size-fits-all alignment mechanism, where the part of the large language model parametric knowledge with non-preferred features is uniformly blocked to all the users. |
| Approach: | They propose a framework that lets LLMs learn access control over parametric knowledge for users with different credentials via authorization alignment. |
| Outcome: | Experiments on two application scenarios show that the proposed framework effectively controls the user’s access to parametric knowledge and maintains its general utility. |
Combating Security and Privacy Issues in the Era of Large Language Models (2024.naacl-tutorials)
Copied to clipboard
| Challenge: | a tutorial aims to provide a summary of risks and vulnerabilities in large language models . a number of studies have focused on security, privacy and copyright aspects of LLMs . |
| Approach: | This tutorial seeks to provide a systematic summary of risks and vulnerabilities in large language models . authors will discuss security, privacy and copyright aspects of LLMs . |
| Outcome: | This tutorial aims to provide a systematic summary of risks and vulnerabilities in large language models . it will also outline emerging challenges in security, privacy and reliability of LLMs . |