AEGIS2.0: A Diverse AI Safety Dataset and Risks Taxonomy for Alignment of LLM Guardrails (2025.naacl-long)
Copied to clipboard
Shaona Ghosh, Prasoon Varshney, Makesh Narsimhan Sreedhar, Aishwarya Padmakumar, Traian Rebedea, Jibin Rajan Varghese, Christopher Parisien
| Challenge: | Existing safety-related content safety models are not well-suited for commercial use. |
| Approach: | They propose a taxonomy that can be used to categorize safety risks . it combines human annotations with a multi-LLM "jury" system to assess safety . they plan to open-source Aegis2.0 data and models to aid in safety guardrailing . |
| Outcome: | The proposed taxonomy can be used to assess the safety of human-LLM interactions . it can be trained on large, non-commercial datasets and is open-source . |
Similar Papers
OmniCompliance-100K: A Multi-Domain, Rule-Grounded, Real-World Safety Compliance Dataset (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing LLM safety datasets rely on ad-hoc taxonomies and lack rule-grounded, real-world cases. |
| Approach: | They construct a rule-grounded, real-world case dataset OmniCompliance-100K from a compliance perspective using a powerful web-searching agent. |
| Outcome: | The proposed dataset spans 74 regulations and policies across a wide range of domains including security and privacy regulations, content safety and user data privacy policies, financial security requirements, medical device risk management standards, educational integrity guidelines, and protections of fundamental human rights. |
LS-Guard: Adaptive Safety Guardrails Tailored to Individual LLMs (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing security guardrails built from static datasets ignore each model’s unique safety profile and often force trade-offs between safety and utility. |
| Approach: | They propose a framework for learning model-specific guardrails tailored to each LLM’s vulnerabilities. |
| Outcome: | The proposed framework significantly outperforms baseline guardrails on multiple real-world LLMs, achieving superior robustness, adaptability, and generalization. |
A Chinese Dataset for Evaluating the Safeguards in Large Language Models (2024.findings-acl)
Copied to clipboard
Yuxia Wang, Zenan Zhai, Haonan Li, Xudong Han, Shom Lin, Zhenxuan Zhang, Angela Zhao, Preslav Nakov, Timothy Baldwin
| Challenge: | a recent study has shown that large language models can produce harmful responses, exposing users to unexpected risks. |
| Approach: | They propose a dataset for the safety evaluation of Chinese LLMs in Mandarin Chinese . they extend the dataset to better identify false negative and false positive examples . |
| Outcome: | The proposed dataset is for the safety evaluation of Chinese LLMs, and is based on a Chinese dataset. |
Interpretation Meets Safety: A Survey on Interpretation Methods and Tools for Improving LLM Safety (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing surveys focus on interpretation or safety, but safety and understanding are core motivations for interpretation research. |
| Approach: | They propose a framework that connects interpretation methods, enhancements they inform, and tools that operationalize them. |
| Outcome: | The proposed framework summarizes nearly 70 studies at their intersections and concludes with open challenges and future directions. |
Guardrails and Security for LLMs: Safe, Secure and Controllable Steering of LLM Applications (2025.acl-tutorials)
Copied to clipboard
Traian Rebedea, Leon Derczynski, Shaona Ghosh, Makesh Narsimhan Sreedhar, Faeze Brahman, Liwei Jiang, Bo Li, Yulia Tsvetkov, Christopher Parisien, Yejin Choi
| Challenge: | Pretrained generative models provide novel ways for users to interact with computers. |
| Approach: | This tutorial provides an overview of key guardrail mechanisms developed for LLMs along with evaluation methodologies and a detailed security assessment protocol. |
| Outcome: | This tutorial provides an overview of key guardrail mechanisms developed for LLMs, along with evaluation methodologies and a detailed security assessment protocol. |
Do-Not-Answer: Evaluating Safeguards in LLMs (2024.findings-eacl)
Copied to clipboard
| Challenge: | a dataset evaluating harmful capabilities in large language models is available at https://github.com/Libr-AI/do-not-answer. |
| Approach: | They collect an open-source dataset to evaluate the safeguards in large language models . they find that simple BERT-style classifiers can achieve results comparable to GPT-4 . |
| Outcome: | The proposed dataset compares the safety of six popular LLMs to GPT-4 on automatic safety evaluation. |
Safety of Large Language Models Beyond English: A Systematic Literature Review of Risks, Biases, and Safeguards (2026.eacl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) have a growing number of applications that generate harmful, biased, or unsafe content. |
| Approach: | They synthesize findings from recent studies that evaluate their robustness across languages . they highlight gaps in multilingual safety research and recommend future work . |
| Outcome: | The systematic review examines the multilingual safety of large language models in English . it identifies challenges such as dataset availability and evaluation biases . |
Multitask-Bench: Unveiling and Mitigating Safety Gaps in LLMs Fine-tuning (2025.coling-main)
Copied to clipboard
| Challenge: | Recent advances in Large Language Models (LLMs) have led to their adoption across a wide range of tasks, ranging from code generation to machine translation and sentiment analysis. |
| Approach: | They propose to fine-tune LLMs on benign (non-harmful) data to ensure safe outputs. |
| Outcome: | The proposed model reduces attack success rates across a range of tasks without compromising its usefulness. |
SLM as Guardian: Pioneering AI Safety with Small Language Model (2024.emnlp-industry)
Copied to clipboard
Ohjoon Kwon, Donghyeon Jeon, Nayoung Choi, Gyu-Hwung Cho, Hwiyeol Jo, Changbong Kim, Hyunwoo Lee, Inho Kang, Sun Kim, Taiwoo Park
| Challenge: | Prior safety research on large language models focused on aligning them to safety requirements, but internalizing such safeguard features into larger models brought challenges of higher training cost and unintended degradation of helpfulness. |
| Approach: | They propose a multi-task learning mechanism that integrates harmful query detection and safeguard response into a single model. |
| Outcome: | The proposed approach outperforms the publicly available LLMs in harmful query detection and safeguard response generation. |
TRIDENT: Enhancing Large Language Model Safety with Tri-Dimensional Diversified Red-Teaming Data Synthesis (2025.acl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) excel in natural language processing tasks but are vulnerable to harmful content and being exploited for malicious purposes. |
| Approach: | They propose a framework to measure the risk coverage of alignment datasets across three dimensions: Lexical Diversity, Malicious Intent, and Jailbreak Tactics. |
| Outcome: | The proposed framework measures risk coverage across Lexical Diversity, Malicious Intent, and Jailbreak Tactics. |