Dynamic Rewarding with Prompt Optimization Enables Tuning-free Self-Alignment of Language Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | Empirical evaluations on eight recent LLMs reveal that DRPO significantly enhances alignment performance, enabling base models to outperform their SFT/RLHF-tuned counterparts. |
| Approach: | They propose a tuning-free approach to self-alignment called Dynamic Rewarding with Prompt Optimization (DRPO) it leverages a dynamic rewarding mechanism to identify and rectify alignment weaknesses . |
| Outcome: | The proposed approach outperforms existing methods and is highly adaptable to various alignment challenges. |
Similar Papers
Reinforcement Learning with Supervised Alignment (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Supervised fine-tuning (SFT) is a widely used method for adapting Large Language Models to specific tasks. |
| Approach: | They propose a method that uses supervised fine-tuning to train a reward model for reinforcement learning. |
| Outcome: | The proposed method outperforms existing methods on in-domain benchmarks but surpasses them 50 times on out-of-domain and cross-task evaluations. |
Topology-Enhanced Alignment for Large Language Models: Trajectory Topology Loss and Topological Preference Optimization (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing algorithms for supervised fine-tuning and reinforcement learning from human feedback (RLHF) do not constrain how hidden states move from a user prompt to an answer. |
| Approach: | They propose a topology-enhanced alignment framework that regularizes these trajectories using 0-dimensional persistent homology. |
| Outcome: | The proposed framework regularizes semantic trajectory in hidden space using 0-dimensional persistent homology. |
Direct Large Language Model Alignment Through Self-Rewarding Contrastive Prompt Distillation (2024.acl-long)
Copied to clipboard
| Challenge: | Existing methods to evaluate preference data without human annotations are difficult . et al., 2022b) is effective for aligning large language models with human expectations . |
| Approach: | They propose a method to evaluate the response preference using output probabilities under contrastive prompts. |
| Outcome: | The proposed method could surpass the RLHF method without human-annotated preference data. |
RS-DPO: A Hybrid Rejection Sampling and Direct Preference Optimization Method for Alignment of Large Language Models (2024.findings-naacl)
Copied to clipboard
| Challenge: | Reinforcement learning with human feedback (RLHF) is widely employed to align large language models with user intent. |
| Approach: | They propose to combine rejection sampling and direct preference optimization to improve alignment with user intent by identifying pairs of contrastive samples from human annotator and alternative LLMs. |
| Outcome: | The proposed method outperforms existing methods including RS, PPO, and DPO in a limited resource environment. |
Aligning Large Language Models via Fully Self-Synthetic Data (2026.acl-long)
Copied to clipboard
| Challenge: | Existing approaches to reinforcement learning from human feedback (RLHF) require expensive human-annotated datasets and proprietary models like GPT-4 to annotate preference pairs. |
| Approach: | They propose a self-synthetic framework for LLM alignment where all training data, including prompts (i.e., user queries), responses, and preferences, are generated by the model itself. |
| Outcome: | The proposed framework enhances the model’s chat capabilities on standard benchmarks like AlpacaEval 2.0 while maintaining strong performance on downstream objective tasks. |
Beyond Online Sampling: Bridging Offline-to-Online Alignment via Dynamic Data Transformation for LLMs (2025.emnlp-main)
Copied to clipboard
| Challenge: | Direct Preference Optimization (DPO) eliminates complex reward modeling in aligning large language models with human preferences, but its online variant faces significant efficiency bottlenecks due to costly real-time preference sampling and the reward model annotation. |
| Approach: | They propose a framework that transforms static datasets into dynamically adaptive equivalents without the need for an explicit reward model. |
| Outcome: | The proposed approach matches or exceeds the performance of a fully online DPO. |
AlignDistil: Token-Level Language Model Alignment as Adaptive Policy Distillation (2025.acl-long)
Copied to clipboard
| Challenge: | Existing methods for LLM alignment optimize tokens using a sparse, response-level reward or preference annotation. |
| Approach: | They propose an RLHF-equivalent distillation method for token-level reward optimization that incorporates the reward learned by DPO into the RLHG objective and builds a token-based teacher distribution. |
| Outcome: | The proposed method bridges the accuracy gap between the reward from the DPO model and the pure reward model by building a contrastive DPO reward with a normal and a reverse DPO. |
Black-Box Prompt Optimization: Aligning Large Language Models without Model Training (2024.acl-long)
Copied to clipboard
| Challenge: | Large language models are often not well aligned with human intents, which requires additional training. |
| Approach: | They propose to use Black-Box Prompt Optimization (BPO) to perform alignments on large language models that are not well aligned with human intents. |
| Outcome: | The proposed model outperforms existing models and is model-agnostic. |
Reward Alignment Optimization: A Direct Point-wise Alignment Approach (2026.acl-long)
Copied to clipboard
| Challenge: | Existing Direct Alignment Algorithms (DAAs) are limiting in generalizaiton to implicit rewards. |
| Approach: | They propose a point-wise direct alignment method that uses an explicit reward model to specify exact target generation probabilities and align the policy offline towards them. |
| Outcome: | The proposed method outperforms existing direct alignment algorithms while enabling controllable target probability distributions. |
Beyond One-Preference-Fits-All Alignment: Multi-Objective Direct Preference Optimization (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent approaches to language model alignment assume homogeneous human preferences, but actual human preferences vary widely and are hard to satisfy with a single language model. |
| Approach: | They propose an RL-free extension of Direct Preference Optimization (DPO) that folds language modeling directly into reward modeling and trains language models as collective reward models that combine all objectives with specific weights. |
| Outcome: | The proposed method matches or outperforms existing methods in safety alignment and long-form question answering. |