V-RoLoRA: RLVR-Driven MoE Routing for Steerable Pluralistic Alignment (2026.findings-acl)
Copied to clipboard
| Challenge: | Current methods for steering large language models rely on prompt engineering or reasoning-time guidance. |
| Approach: | They propose a value-controllable pluralistic alignment framework enhanced with conditioned gating that dynamically directs the flow among multiple experts based on an input value or moral vector. |
| Outcome: | The proposed method outperforms prompt-based steering and multi-task PEFT benchmarks on two 8-billion-parameter backbones. |
Similar Papers
SteerLM: Attribute Conditioned SFT as an (User-Steerable) Alternative to RLHF (2023.findings-emnlp)
Copied to clipboard
| Challenge: | supervised fine-tuning and reinforcement learning from human feedback (RLHF) are not effective in generating useful and high-quality responses. |
| Approach: | They propose a supervised fine-tuning method that empowers end-users to control responses during inference. |
| Outcome: | Experiments show that supervised fine-tuning and reinforcement learning from human feedback (RLHF) can generate helpful and high-quality responses while maintaining customizability. |
Exploring Chain-of-Thought Reasoning for Steerable Pluralistic Alignment (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are typically trained to reflect a relatively uniform set of values, which limits their applicability to tasks that require understanding of nuanced human perspectives. |
| Approach: | They propose to use Chain-of-Thought reasoning techniques to build steerable pluralistic models by fine-tuning on human-authored CoT and synthetic explanations. |
| Outcome: | The proposed methods outperform others and demonstrate strong sample efficiency. |
When the Model Said ‘No Comment’, We Knew Helpfulness Was Dead, Honesty Was Alive, and Safety Was Terrified (2026.eacl-long)
Copied to clipboard
| Challenge: | Existing work uses SFT and MoE to align Large Language Models, but these work face challenges in multi-objective settings. |
| Approach: | They propose a framework that uses prompt-injected fine-tuning to extract axis-specific task features . it deploys a MoCaE module that calibrates expert routing using fractal and natural geometry . |
| Outcome: | The proposed framework achieves significant gains on Alpaca, BeaverTails, TruthfulQA and TruthfulQ with +171.5% win rate and +110.1% truthfulness-informativeness. |
Don’t Forget Your Reward Values: Language Model Alignment via Value-based Calibration (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for generating large language models have been criticized for their complexity and instability. |
| Approach: | They propose a value-based calibration method to better align Large Language Models with human preferences. |
| Outcome: | The proposed method surpasses existing methods on AI assistant and summarization datasets, providing impressive generalizability, robustness, and diversity in different settings. |
Too Helpful, Too Harmless, Too Honest or Just Right? (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods optimize for individual alignment dimensions in isolation, leading to trade-offs and inconsistent behavior. |
| Approach: | They propose a modular alignment framework that integrates a Mixture of Calibrated Experts (MoCaE) within the Transformer architecture. |
| Outcome: | The proposed framework outperforms baselines on three alignment benchmarks, achieving 32.5% win rate, 33.9% safety score, and 28.4% truthfulness. |
Arithmetic Control of LLMs for Diverse User Preferences: Directional Preference Alignment with Multi-Objective Rewards (2024.acl-long)
Copied to clipboard
| Challenge: | Reinforcement Learning from Human Feedback (RLHF) relies on scalar rewards to capture user preferences. |
| Approach: | They propose a framework that integrates multi-objective reward modeling to represent diverse preference profiles. |
| Outcome: | The proposed method improves performance across reward objectives and targets. |
Verifying the Subjective: Structured Multilingual Rewards for Low-Resource Alignment (2026.findings-acl)
Copied to clipboard
| Challenge: | Structured Multilingual Reward Modeling Framework extends Reinforcement Learning with Verifiable Rewards (RLVR) to subjective and open-ended tasks. |
| Approach: | They propose a framework that extends Reinforcement Learning with Verifiable Rewards to subjective and open-ended tasks. |
| Outcome: | The proposed framework improves reasoning capability and response quality on 7 tasks across 50 low-resource languages. |
TinyJudge: Unverifiable Constraint Alignment via Lightweight Specialist Ensembles (2026.acl-long)
Copied to clipboard
Yirong Zeng, Yufei Liu, Xiao Ding, Yutai Hou, Yuxian Wang, Wu Ning, Haonan Song, Dandan Tu, Qixun Zhang, Yuxiang He, Bibo Cai, Ting Liu
| Challenge: | Instruction Following (IF) is a core capability of LLMs, requiring strict adherence to diverse constraints. |
| Approach: | They propose a framework that uses tiny language models to evaluate instruction following . they propose to use a set of specialized tiny language model to provide rewards for soft constraints. |
| Outcome: | The proposed framework outperforms baseline models by 12% and speeds up training time by 3. |
Are the Values of LLMs Structurally Aligned with Humans? A Causal Perspective (2025.findings-acl)
Copied to clipboard
Yipeng Kang, Junqi Wang, Yexin Li, Mengmeng Wang, Wenming Tu, Quansen Wang, Hengli Li, Tingjun Wu, Xue Feng, Fangwei Zhong, Zilong Zheng
| Challenge: | Current approaches to value alignment focus on a few core values, such as helpfulness, harmlessness, and honesty. |
| Approach: | They propose to use latent causal value graphs to guide two lightweight value-steering methods . role-based prompting and sparse autoencoder (SAE) steering are also used . |
| Outcome: | Experiments on Gemma-2B-IT and Llama3-8B- IT show that the proposed methods are effective and controllable. |
PASs-MoE: Mitigating Misaligned Co-drift among Router and Experts via Pathway Activation Subspaces for Continual Learning (2026.acl-long)
Copied to clipboard
| Challenge: | Existing LoRA-based Mixture-of-Experts (MoE) methods often jointly update the router and experts in an indiscriminate way, causing the router’s preferences to co-drift with experts’ adaptation pathways and exacerbate forgetting. |
| Approach: | They propose a LoRA-induced subspace that reflects which low-rank pathway directions an input activates in each expert, providing a capability-aligned coordinate system for routing and preservation. |
| Outcome: | The proposed method outperforms conventional continual learning baselines and MoE–LoRA variants in accuracy and resistance to forgetting, without increasing model parameters. |