Course-Correction: Safety Alignment Using Synthetic Preferences (2024.emnlp-industry)
Copied to clipboard
Rongwu Xu, Yishuo Cai, Zhenhong Zhou, Renjie Gu, Haiqin Weng, Liu Yan, Tianwei Zhang, Wei Xu, Han Qiu
| Challenge: | Recent studies show that large language models generate harmful content, but the potential for generating harmful content is an escalating concern. |
| Approach: | They propose to fine-tune LLMs with preference learning to emphasize the preference for timely course-correction by using an automated pipeline. |
| Outcome: | The proposed model improves course-correction skills without affecting general performance and resists jailbreak attacks. |
Similar Papers
TRIDENT: Enhancing Large Language Model Safety with Tri-Dimensional Diversified Red-Teaming Data Synthesis (2025.acl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) excel in natural language processing tasks but are vulnerable to harmful content and being exploited for malicious purposes. |
| Approach: | They propose a framework to measure the risk coverage of alignment datasets across three dimensions: Lexical Diversity, Malicious Intent, and Jailbreak Tactics. |
| Outcome: | The proposed framework measures risk coverage across Lexical Diversity, Malicious Intent, and Jailbreak Tactics. |
PURE: Aligning LLM via Pluggable Query Reformulation for Enhanced Helpfulness (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models (LLMs) depend on vast amounts of text data sourced from the Internet for their training. |
| Approach: | They propose a new alignment paradigm that reformulates risky queries into highly relevant yet harmless ones before feeding them into LLMs. |
| Outcome: | The proposed approach eliminates the high costs of training base LLMs and achieves a promising balance of harmlessness and helpfulness. |
On the Vulnerability of Safety Alignment in Open-Access LLMs (2024.findings-acl)
Copied to clipboard
Jingwei Yi, Rui Ye, Qisi Chen, Bin Zhu, Siheng Chen, Defu Lian, Guangzhong Sun, Xing Xie, Fangzhao Wu
| Challenge: | Large language models (LLMs) are susceptible to malicious exploitation, but are often rejected and limited harmfulness is limited. |
| Approach: | They propose two types of reverse alignment techniques: reverse supervised fine-tuning (RSFT) and reverse preference optimization (RPO). |
| Outcome: | The proposed methods can significantly enhance the success rate and harmfulness of jailbreak attacks, but they face high rejection rates and limited harmfulness. |
CodecLM: Aligning Language Models with Tailored Synthetic Data (2024.findings-naacl)
Copied to clipboard
Zifeng Wang, Chun-Liang Li, Vincent Perot, Long Le, Jin Miao, Zizhao Zhang, Chen-Yu Lee, Tomas Pfister
| Challenge: | Recent work on generating diverse instructions and applying LLM to increase instruction complexity neglects downstream use cases. |
| Approach: | They propose a framework for generating high-quality synthetic data for LLM alignment with different downstream instruction distributions and LLMs. |
| Outcome: | Experiments on four open-domain instruction using the proposed framework validate the effectiveness of CodecLM over the current state-of-the-art. |
SAFER: Advancing Safety Alignment via Efficient Ex-Ante Reasoning (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing alignment methods struggle to cover diverse safety scenarios and remain vulnerable to adversarial attacks. |
| Approach: | They propose a framework for 'S**afety' alignment via e**F**ficient' E**x-Ante-R**easoning that instantiates structured Ex-Ance reasoning and embeds predefined safety rules. |
| Outcome: | The proposed framework enhances safety performance while maintaining usefulness and efficiency. |
SABER: Uncovering Vulnerabilities in Safety Alignment via Cross-Layer Residual Connection (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) with safe-alignment training are vulnerable to jailbreak attacks, causing malicious users to generate harmful outputs. |
| Approach: | They propose a safe-alignment jailbreak method that bypasses the middle-to-late layers of large language models by a residual connection. |
| Outcome: | The proposed method improves by 51% over the best performing baseline GCG on HarmBench test set. |
SeaPO: Strategic Error Amplification for Robust Preference Optimization of Large Language Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for preference optimization of large language models use pairs of positive and negative samples, but the quality of positive samples may become similar during training, complicating preference learning. |
| Approach: | SeaPO introduces error types commonly occurring in large language models to improve preference learning. |
| Outcome: | SeaPO introduces error types into model Preference Optimization to improve model performance . negative samples are more erroneous than positive samples, and preference-based training mitigates errors . |
Multitask-Bench: Unveiling and Mitigating Safety Gaps in LLMs Fine-tuning (2025.coling-main)
Copied to clipboard
| Challenge: | Recent advances in Large Language Models (LLMs) have led to their adoption across a wide range of tasks, ranging from code generation to machine translation and sentiment analysis. |
| Approach: | They propose to fine-tune LLMs on benign (non-harmful) data to ensure safe outputs. |
| Outcome: | The proposed model reduces attack success rates across a range of tasks without compromising its usefulness. |
Improving Preference Alignment of LLM with Inference-Free Self-Refinement (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models (LLMs) develop in-context learning capability through pretraining and instruction tuning. |
| Approach: | Large language models (LLMs) develop in-context learning capability through pretraining and instruction tuning. |
| Outcome: | Experiments show that incorporating IFSR into preference alignment yields performance improvement over 10%. |
Safeguard Fine-Tuned LLMs Through Pre- and Post-Tuning Model Merging (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Fine-tuning large language models for downstream tasks often leads to catastrophic forgetting, notably degrading the safety of original alignments. |
| Approach: | They propose to merge the weights of pre- and post-fine-tuned models to improve safety while enhancing performance. |
| Outcome: | Experiments across different downstream tasks and models validate the method’s practicality and effectiveness. |