Agent vs. Agent: Automated Data Generation and Red-Teaming for Custom Agentic Workflows (2025.emnlp-industry)
Copied to clipboard
| Challenge: | Existing red-teaming frameworks like AgentHarm use static prompts and hardcoded toolsets . |
| Approach: | They propose a red-teaming framework that generates adversarial tasks and evaluation functions tailored to arbitrary toolsets and uses iterative prompt refinement with self-reflection to develop more effective attacks. |
| Outcome: | The proposed approach achieves 162% increase in attack success rate on o4-mini and 86% success on gemini 2.5 Pro. |
Similar Papers
Attack Prompt Generation for Red Teaming and Defending Large Language Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies construct attack prompts via manual or automatic methods, but these methods have limitations on cost and quality. |
| Approach: | They propose an attack framework to instruct LLMs to mimic human-generated prompts through in-context learning and a defense framework that fine-tunes victim LLM's through iterative interactions with the attack framework. |
| Outcome: | The proposed approach is based on experiments on different LLMs to evaluate their effectiveness against red teaming attacks. |
RedCoder: Automated Multi-Turn Red Teaming for Code LLMs (2026.acl-long)
Copied to clipboard
Wenjie Jacky Mo, Qin Liu, Xiaofei Wen, Dongwon Jung, Hadi Askari, Wenxuan Zhou, Zhe Zhao, Muhao Chen
| Challenge: | Existing red-teaming approaches for code generation rely on extensive human effort and are prone to generating malicious code under adversarial environments. |
| Approach: | They propose a red-teaming agent that engages victim models in multi-turn conversations to elicit vulnerable code. |
| Outcome: | Experiments show that RedCoder outperforms red-teaming methods in inducing vulnerabilities in code generation. |
Automated Progressive Red Teaming (2025.coling-main)
Copied to clipboard
| Challenge: | Automated red teaming (ART) is effective but time-consuming, costly and lacks scalability. |
| Approach: | They propose an automated red teaming framework that generates adversarial prompts to expose LLM vulnerabilities. |
| Outcome: | The proposed framework explores and exploits LLM vulnerabilities through multi-round interactions. |
AART: AI-Assisted Red-Teaming with Diverse Data Generation for New LLM-powered Applications (2023.emnlp-industry)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are rapidly becoming more and more popular, but dealing with the potential harms associated with their deployment in real-world scenarios is still an open research question. |
| Approach: | They propose an automated approach for automated generation of adversarial evaluation datasets to test the safety of LLM generations on new downstream applications. |
| Outcome: | AART generates evaluation datasets with high diversity of content characteristics critical for effective adversarial testing. |
SafeAgent: Safeguarding LLM Agents via an Automated Risk Simulator (2026.acl-long)
Copied to clipboard
Xueyang Zhou, Weidong Wang, Lin Lu, Jiawen Shi, Guiyao Tie, Xu Yongtian, Lixing Chen, Pan Zhou, Neil Zhenqiang Gong, Lichao Sun
| Challenge: | SafeAgent improves agent safety through fully automated synthetic data generation. |
| Approach: | They propose a framework that improves agent safety through fully automated synthetic data generation. |
| Outcome: | The proposed framework outperforms closed-source models on two safety benchmarks and one real-world task. |
SIRAJ: Diverse and Efficient Red-Teaming for LLM Agents via Distilled Structured Reasoning (2026.findings-eacl)
Copied to clipboard
| Challenge: | Existing red-teaming frameworks do not cover all the risks associated with arbitrary black-box LLMs. |
| Approach: | They propose a generic red-teaming framework for arbitrary black-box LLM agents that iteratively constructs and refines model-based adversarial attacks based on the execution trajectories of former attempts. |
| Outcome: | The proposed model improves attack success rate by 100%, surpassing the 671B Deepseek-R1 model. |
Effective Red-Teaming of Policy-Adherent Agents (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Model (LLM)-based agents are increasingly used in domains with strict policies, such as refund eligibility or cancellation rules. |
| Approach: | They propose a multi-agent red-teaming system that leverages policy-aware persuasive strategies to undermine a policy-adherence agent in a customer-service scenario. |
| Outcome: | The proposed model outperforms jailbreak methods and tau-break to assess agent's robustness against manipulative user behavior. |
TAMAS: Benchmarking Adversarial Risks in Multi-Agent LLM Systems (2026.acl-long)
Copied to clipboard
| Challenge: | Existing benchmarks and datasets focus on single-agent settings, failing to capture the unique vulnerabilities of multi-agend LLM dynamics and co-ordination. |
| Approach: | They propose a benchmark to evaluate the robustness and safety of multi-agent LLM systems. |
| Outcome: | The proposed benchmark evaluates the robustness and safety of multi-agent LLM systems. |
Holistic Automated Red Teaming for Large Language Models through Top-Down Test Case Generation and Multi-turn Interaction (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches focus on improving attack success rates while overlooking the need for comprehensive test case coverage. |
| Approach: | They propose a top-down approach to automated red teaming that scales up the diversity of test cases using an extensible, fine-grained risk taxonomy. |
| Outcome: | The proposed approach scales up the diversity of test cases using a top-down approach based on an extensible, fine-grained risk taxonomy and leverages reinforcement learning techniques to facilitate multi-turn adversarial probing in a human-like manner. |
Gradient-Based Language Model Red Teaming (2024.eacl-long)
Copied to clipboard
| Challenge: | generative language models generate unsafe responses by producing adversarial prompts . red teaming is labor-intensive and difficult to scale when done by humans. |
| Approach: | They propose a red teaming method that generates diverse prompts that are likely to cause an LM to generate unsafe responses. |
| Outcome: | The proposed method is more effective at finding prompts that trigger an LM to generate unsafe responses than a strong reinforcement learning-based red teaming approach. |