Reward-Shifted Speculative Sampling Is An Efficient Test-Time Weak-to-Strong Aligner (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent research has focused on test-time alignment, where additional compute is allocated during inference to enhance LLM safety and reasoning capabilities. |
| Approach: | They propose a reward-shifted speculative sampling algorithm that aligns a draft model with human preferences while the target model remains unchanged. |
| Outcome: | The proposed algorithm achieves superior gold reward scores at a significantly reduced inference cost in test-time weak-to-strong alignment experiments. |
Similar Papers
Reward-Guided Tree Search for Inference Time Alignment of Large Language Models (2025.naacl-long)
Copied to clipboard
| Challenge: | Inference-time computation methods enhance performance of Large Language Models by leveraging additional computational resources. |
| Approach: | They propose an inference-time alignment method that leverages a reward model to achieve alignment through reward-guided tree search. |
| Outcome: | The proposed method outperforms other inference-time alignment methods on two benchmarks . it achieves comparable performance to preference-tuned models on both benchmarks, authors show . |
TokenTiming: A Dynamic Alignment Method for Universal Speculative Decoding Model Pairs (2026.acl-long)
Copied to clipboard
| Challenge: | Speculative decoding (SD) is a useful tool for accelerating large language models . but its utility is limited by a fundamental constraint: draft and target models must share the same vocabulary . |
| Approach: | They propose an algorithm that uses a draft token sequence to get a new target token sequence and then uses DTW to build a mapping to transfer probability distributions. |
| Outcome: | The proposed method shows 1.57x speedup on various tasks. |
Efficient Safety Alignment of Large Language Models via Preference Re-ranking and Representation-based Reward Modeling (2025.acl-long)
Copied to clipboard
| Challenge: | Existing safety alignment methods for Large Language Models (LLMs) face the distribution shift issue, which requires significant computational resources. |
| Approach: | They propose a framework that leverages the model’s intrinsic safety judgment capability to extract reward signals, which are then used to calculate label confidence for preference reordering. |
| Outcome: | The proposed framework improves safety performance while avoiding 300x computational overheads. |
Alignment-Augmented Speculative Decoding with Alignment Sampling and Conditional Verification (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to accelerate autoregressive generation of large language models require training costs. |
| Approach: | They propose a training-free alignment-augmented speculative decoding algorithm . it leverages the output distribution obtained in the prefilling phase to provide more aligned draft candidates . |
| Outcome: | The proposed method increases the average generation score by 3.3 points for the LLaMA3 model. |
Distributional Alignment for Large Language Models under Domain Shift (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing distributional alignment models are unstable and degrade under cultural and domain shifts. |
| Approach: | They propose a distributional alignment technique that improves distribution prediction under cultural and domain shift. |
| Outcome: | The proposed method improves fidelity and robustness of LLM distribution estimation under domain and cultural shift. |
Beyond Online Sampling: Bridging Offline-to-Online Alignment via Dynamic Data Transformation for LLMs (2025.emnlp-main)
Copied to clipboard
| Challenge: | Direct Preference Optimization (DPO) eliminates complex reward modeling in aligning large language models with human preferences, but its online variant faces significant efficiency bottlenecks due to costly real-time preference sampling and the reward model annotation. |
| Approach: | They propose a framework that transforms static datasets into dynamically adaptive equivalents without the need for an explicit reward model. |
| Outcome: | The proposed approach matches or exceeds the performance of a fully online DPO. |
Don’t Forget Your Reward Values: Language Model Alignment via Value-based Calibration (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for generating large language models have been criticized for their complexity and instability. |
| Approach: | They propose a value-based calibration method to better align Large Language Models with human preferences. |
| Outcome: | The proposed method surpasses existing methods on AI assistant and summarization datasets, providing impressive generalizability, robustness, and diversity in different settings. |
EMS-SD: Efficient Multi-sample Speculative Decoding for Accelerating Large Language Models (2025.naacl-long)
Copied to clipboard
| Challenge: | Speculative decoding is a key technique for enhancing the inference speed of Large Language Models. |
| Approach: | They propose a method that adds padding tokens to ensure that the number of new tokens remains consistent across samples. |
| Outcome: | The proposed method can handle the issue of inconsistent prediction tokens without adding padding tokens. |
DeAL: Decoding-time Alignment for Large Language Models (2025.acl-long)
Copied to clipboard
James Y. Huang, Sailik Sengupta, Daniele Bonadiman, Yi-An Lai, Arshit Gupta, Nikolaos Pappas, Saab Mansour, Katrin Kirchhoff, Dan Roth
| Challenge: | Large Language Models (LLMs) are expected to generate content aligned with human preferences. |
| Approach: | They propose a framework that allows the user to customize reward functions and enables Decoding-time Alignment of LLMs (DeAL). |
| Outcome: | The proposed framework allows the user to customize reward functions and enables Decoding-time Alignment of LLMs. |
On the Rejection Criterion for Proxy-based Test-time Alignment (2026.acl-short)
Copied to clipboard
| Challenge: | Recent work suggests that test-time alignment methods rely on a small aligned model as a proxy that guides the generation of a larger base model. |
| Approach: | They propose a rejection criterion based on a conservative confidence bet for test-time alignment methods that use a small aligned model as a proxy to guide the generation of a larger base model. |
| Outcome: | The proposed approach outperforms previous work on several datasets. |