| Challenge: | In-context learning (ICL) has emerged as a capability of large language models (LLMs) but there is limited understanding of its vulnerability against data poisoning attacks. |
| Approach: | They propose an attack method that exploits ICL’s unique learning mechanisms by identifying discrete text perturbations that influence LLM hidden states. |
| Outcome: | The proposed attack method exploits ICL’s learning mechanisms by identifying discrete text perturbations that influence LLM hidden states. |
Similar Papers
Universal Vulnerabilities in Large Language Models: Backdoor Attacks for In-context Learning (2024.emnlp-main)
Copied to clipboard
| Challenge: | In-context learning has shown high efficacy in several NLP tasks, especially in few-shot settings. |
| Approach: | They propose a backdoor attack method that poisons demonstration examples and poisons the demonstration context, preserving the model's generality. |
| Outcome: | The proposed method can make models behave in alignment with predefined intentions without fine-tuning the model. |
The Mystery of In-Context Learning: A Comprehensive Survey on Interpretation and Analysis (2024.emnlp-main)
Copied to clipboard
| Challenge: | In-context learning (ICL) is a capability that enables large language models to excel in proficiency through demonstration examples. |
| Approach: | They present a survey on the interpretation and analysis of in-context learning . they focus on theoretical and empirical perspectives on the concept . |
| Outcome: | The proposed model can perform tasks with minimal examples without re-training and has demonstrated proficiency across various tasks with a minimal set of task-oriented examples. |
A Survey on In-context Learning (2024.emnlp-main)
Copied to clipboard
Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Jingyuan Ma, Rui Li, Heming Xia, Jingjing Xu, Zhiyong Wu, Baobao Chang, Xu Sun, Lei Li, Zhifang Sui
| Challenge: | In-context learning (ICL) is a new paradigm for natural language processing . large language models (LLMs) demonstrate the ability to learn from a few examples . |
| Approach: | They propose to explore ICL to evaluate and extrapolate the ability of large language models. |
| Outcome: | The proposed methods can be used to evaluate and extrapolate the ability of large language models. |
Unveiling the Implicit Toxicity in Large Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies focus on probing toxic outputs that can be easily detected with existing toxicity classifiers, but LLMs can generate diverse implicit toxic output that are difficult to detect via simply zero-shot prompting. |
| Approach: | They propose a reinforcement learning based attacking method to induce the implicit toxic outputs in large language models by fine-tuning toxicity classifiers. |
| Outcome: | The proposed method generates implicit toxic outputs that are difficult to detect via zero-shot prompting on five widely-adopted toxicity classifiers. |
Vulnerabilities of Large Language Models to Adversarial Attacks (2024.acl-tutorials)
Copied to clipboard
| Challenge: | This tutorial focuses on the vulnerabilities of Large Language Models to adversarial attacks . the tutorial lays the foundation by explaining safety-aligned models and concepts in cybersecurity . |
| Approach: | This tutorial lays the foundation by explaining safety-aligned LLMs and concepts in cybersecurity. |
| Outcome: | The tutorial lays the foundation by explaining safety-aligned models and concepts in cybersecurity. |
Defending Jailbreak Prompts via In-Context Adversarial Game (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) demonstrate remarkable capabilities across diverse applications, but concerns regarding their security persist. |
| Approach: | They propose an adversarial game that leverages agent learning to extend knowledge to defend against jailbreaks. |
| Outcome: | The proposed game shows that LLMs safeguarded by ICAG exhibit significantly reduced jailbreak success rates across various attack scenarios. |
Public Data Assisted Differentially Private In-Context Learning (2025.findings-emnlp)
Copied to clipboard
| Challenge: | In-context learning has shown remarkable performance across tasks without fine-tuning . however, recent studies have highlighted the risk of private data leakage through the prompt in ICL . |
| Approach: | They propose a private in-context learning algorithm that effectively balances privacy protection and model utility. |
| Outcome: | The proposed algorithm is robust against membership inference attacks and is robust to membership infertility attacks. |
Attacks against Abstractive Text Summarization Models through Lead Bias and Influence Functions (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are vulnerable to adversarial perturbations and data poisoning attacks, especially in tasks like text classification and translation. |
| Approach: | They propose a novel approach exploiting the inherent lead bias in large language models to perform adversarial perturbations and an innovative application of influence functions to execute data poisoning attacks. |
| Outcome: | The proposed approach shows that the models under attack tend to generate extractive summaries rather than abstractive summarizations. |
In-Context Learning Creates Task Vectors (2023.findings-emnlp)
Copied to clipboard
| Challenge: | In-context learning (ICL) is a powerful new learning paradigm for Large Language Models (LLMs). |
| Approach: | They propose to use a model with a prompt and a query to learn a mapping based on two examples to produce the output. |
| Outcome: | The proposed model can learn functions from a simple structure based on a training set and a single task vector calculated from the training set. |
PoisonedParrot: Subtle Data Poisoning Attacks to Elicit Copyright-Infringing Content from Large Language Models (2025.naacl-long)
Copied to clipboard
Michael-Andrei Panaitescu-Liess, Pankayaraj Pathmanathan, Yigitcan Kaya, Zora Che, Bang An, Sicheng Zhu, Aakriti Agrawal, Furong Huang
| Challenge: | PoisonedParrot is the first stealthy data poisoning attack that induces an LLM to generate copyrighted content even when the model has not been directly trained on the copyright material. |
| Approach: | They propose a stealthy data poisoning attack that induces an LLM to generate copyrighted content even when it has not been directly trained on the copyright material. |
| Outcome: | The proposed model induces an LLM to generate copyrighted content with no discernible side effects and is surprisingly effective at priming the model to generate content with little side effects. |