PURE: Aligning LLM via Pluggable Query Reformulation for Enhanced Helpfulness (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models (LLMs) depend on vast amounts of text data sourced from the Internet for their training. |
| Approach: | They propose a new alignment paradigm that reformulates risky queries into highly relevant yet harmless ones before feeding them into LLMs. |
| Outcome: | The proposed approach eliminates the high costs of training base LLMs and achieves a promising balance of harmlessness and helpfulness. |
Similar Papers
Adaptive Helpfulness–Harmlessness Alignment with Preference Vectors (2026.eacl-long)
Copied to clipboard
Ren-Wei Liang, Chin Ting Hsu, Chan-Hung Yu, Saransh Agrawal, Shih-Cheng Huang, Chieh-Yen Lin, Shang-Tse Chen, Kuan-Hao Huang, Shao-Hua Sun
| Challenge: | Existing approaches to balancing helpfulness and harmlessness suffer from performance conflicts, limited controllability, and poor extendability. |
| Approach: | They propose a framework that allows users to control their own preferences and dynamically merge them at test time. |
| Outcome: | The proposed framework improves helpfulness without conservatism and smooth control over preference trade-offs. |
You Only Need One Single Token to Refine Safety Alignment (2026.findings-acl)
Copied to clipboard
| Challenge: | Excessive safety can lead to over-refusal, where models reject harmful-looking yet benign queries, severely limiting utility. |
| Approach: | They propose a lightweight training-based approach that reshapes the distributions of harmful and benign samples within the model’s decision space by using a single-token prefix. |
| Outcome: | The proposed approach can distinguish between harmful and benign samples while keeping the model frozen. |
InferAligner: Inference-Time Alignment for Harmlessness through Cross-Model Guidance (2024.emnlp-main)
Copied to clipboard
Pengyu Wang, Dong Zhang, Linyang Li, Chenkun Tan, Xinghao Wang, Mozhi Zhang, Ke Ren, Botian Jiang, Xipeng Qiu
| Challenge: | Existing methods for enhancing harmlessness and helpfulness of large language models (LLMs) involve complex and resource-intensive training processes. |
| Approach: | They propose a method that decouples harmlessness from helpfulness during inference phase. |
| Outcome: | The proposed method significantly reduces the attack success rate (ASR) of harmful instructions and jailbreak instructions while maintaining almost unchanged performance in downstream tasks. |
Re-Align: Aligning Vision Language Models via Retrieval-Augmented Direct Preference Optimization (2025.emnlp-main)
Copied to clipboard
Shuo Xing, Peiran Li, Yuping Wang, Ruizheng Bai, Yueqi Wang, Chan-Wei Hu, Chengxuan Qian, Huaxiu Yao, Zhengzhong Tu
| Challenge: | emergence of large Vision Language Models (VLMs) has broadened the capabilities of single-modal Large Language Model (LLM) but VLMs are prone to significant hallucinations, especially in the form of cross-modal inconsistencies. |
| Approach: | They propose a new alignment framework that leverages image retrieval to integrate both textual and visual preference signals. |
| Outcome: | The proposed framework mitigates hallucinations more effectively than previous methods . it maintains robustness and scalability across a wide range of VLM sizes and architectures . |
A Survey on Training-free Alignment of Large Language Models (2025.findings-emnlp)
Copied to clipboard
Birong Pan, Yongqi Li, Weiyu Zhang, Wenpeng Lu, Mayi Xu, Shen Zhou, Yuanyuan Zhu, Ming Zhong, Tieyun Qian
| Challenge: | a survey of large language models (LLMs) aims to ensure outputs adhere to human values, ethical standards, and legal norms. |
| Approach: | They present the first systematic review of TF alignment methods . they categorize them by stages of pre-decoding, in-decoder and post-decoration . |
| Outcome: | The proposed methods are based on training-free (TF) alignment techniques . they are able to be used in open-source and closed-source environments without retraining . |
Constructing Your Model’s Value Distinction: Towards LLM Alignment with Anchor Words Tuning (2025.findings-emnlp)
Copied to clipboard
| Challenge: | a study of large language models (LLMs) shows that they can generate outputs that are honest, positive, harmless, etc. |
| Approach: | They propose a method that amplifies logits difference between positive and negative tokens . they propose to use the logits gap to generate positive and positive tokens after alignment . |
| Outcome: | The proposed method achieves effective alignment, but requires fewer computational resources compared to training-time alignment methods. |
DeAL: Decoding-time Alignment for Large Language Models (2025.acl-long)
Copied to clipboard
James Y. Huang, Sailik Sengupta, Daniele Bonadiman, Yi-An Lai, Arshit Gupta, Nikolaos Pappas, Saab Mansour, Katrin Kirchhoff, Dan Roth
| Challenge: | Large Language Models (LLMs) are expected to generate content aligned with human preferences. |
| Approach: | They propose a framework that allows the user to customize reward functions and enables Decoding-time Alignment of LLMs (DeAL). |
| Outcome: | The proposed framework allows the user to customize reward functions and enables Decoding-time Alignment of LLMs. |
Beyond Online Sampling: Bridging Offline-to-Online Alignment via Dynamic Data Transformation for LLMs (2025.emnlp-main)
Copied to clipboard
| Challenge: | Direct Preference Optimization (DPO) eliminates complex reward modeling in aligning large language models with human preferences, but its online variant faces significant efficiency bottlenecks due to costly real-time preference sampling and the reward model annotation. |
| Approach: | They propose a framework that transforms static datasets into dynamically adaptive equivalents without the need for an explicit reward model. |
| Outcome: | The proposed approach matches or exceeds the performance of a fully online DPO. |
Aligning Large Language Models via Fully Self-Synthetic Data (2026.acl-long)
Copied to clipboard
| Challenge: | Existing approaches to reinforcement learning from human feedback (RLHF) require expensive human-annotated datasets and proprietary models like GPT-4 to annotate preference pairs. |
| Approach: | They propose a self-synthetic framework for LLM alignment where all training data, including prompts (i.e., user queries), responses, and preferences, are generated by the model itself. |
| Outcome: | The proposed framework enhances the model’s chat capabilities on standard benchmarks like AlpacaEval 2.0 while maintaining strong performance on downstream objective tasks. |
PKU-SafeRLHF: Towards Multi-Level Safety Alignment for LLMs with Human Preference (2025.acl-long)
Copied to clipboard
Jiaming Ji, Donghai Hong, Borong Zhang, Boyuan Chen, Josef Dai, Boren Zheng, Tianyi Alex Qiu, Jiayi Zhou, Kaile Wang, Boxun Li, Sirui Han, Yike Guo, Yaodong Yang
| Challenge: | Using large-scale annotation data, large language models can generate noise, errors and biases, leading to unexpected behaviours. |
| Approach: | They propose a dataset to promote safety alignment in large language models . they separate helpfulness and harmlessness annotations for question-answering pairs . |
| Outcome: | The proposed dataset provides 44.6k prompts and 265k question-answer pairs with safety meta-labels for 19 harm categories and three severity levels, with answers generated by Llama-family models. |