Exploring Domain Robust Lightweight Reward Models based on Router Mechanism (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent advances in large language models have relied on the large reward model for fine-tuning, but the use of a single reward model across domains may not always be optimal. |
| Approach: | They propose to use router mechanisms to train small language models in a domain-specific manner . they use internal routers, external routers and router adapters to create a single reward model . |
| Outcome: | The proposed approach reduces parameter size while minimizing parameter size. |
Similar Papers
A Comprehensive Survey on Learning from Rewards for Large Language Models: Reward Models and Learning Strategies (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Recent developments in Large Language Models have shifted from pre-training to post-training and test-time scaling. |
| Approach: | They present a comprehensive overview of learning from rewards from the perspective of reward models and learning strategies across training, inference, and post-inference stages. |
| Outcome: | The proposed paradigm enables the transition from passive learning from static data to active learning from dynamic feedback. |
Routing to the Expert: Efficient Reward-guided Ensemble of Large Language Models (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing ensemble methods for Large Language Models focus on reward model ranking of outputs, leading to significant computation overhead. |
| Approach: | They propose a reward-guided routing method distilling rewards on training queries to train a routing function. |
| Outcome: | The proposed method outperforms the best single model and ranks first on 44% of tasks. |
Prototypical Reward Network for Data-Efficient Model Alignment (2024.acl-long)
Copied to clipboard
| Challenge: | Reinforcement Learning from Human Feedback (RLHF) is a reward model that fine-tunes Large Language Models (LLMs) by utilizing Prototypical Networks. |
| Approach: | They propose a framework utilizing Prototypical Networks to enhance reward models under limited human feedback, enabling more stable and reliable structural learning from fewer samples. |
| Outcome: | The proposed framework improves reward models under limited human feedback, surpassing traditional methods, especially in data-limited scenarios. |
Inverse Reinforcement Learning Meets Large Language Model Alignment (2025.acl-tutorials)
Copied to clipboard
| Challenge: | This tutorial will provide a comprehensive review of recent advances in LLM alignment . it will highlight the necessity of constructing neural reward models from human data . |
| Approach: | This tutorial will provide a comprehensive review of recent advances in LLM alignment through the lens of inverse reinforcement learning. |
| Outcome: | This tutorial will provide a comprehensive review of recent advances in LLM alignment through the lens of inverse reinforcement learning (IRL). |
Reinforcement Learning for Large Language Models via Group Preference Reward Shaping (2025.emnlp-main)
Copied to clipboard
Huaisheng Zhu, Siyuan Xu, Hangfan Zhang, Teng Xiao, Zhimeng Guo, Shijie Zhou, Shuyue Hu, Vasant G. Honavar
| Challenge: | Existing methods for fine-tuning Large Language Models (LLMs) are expensive and sensitive to reward model quality. |
| Approach: | They propose a method that leverages preference-based comparisons rather than precise numerical rewards. |
| Outcome: | Experiments show that GPRS outperforms critic-model-free RL algorithms on RLHF and reasoning tasks. |
DogeRM: Equipping Reward Models with Domain Knowledge through Model Merging (2024.emnlp-main)
Copied to clipboard
| Challenge: | Modern large language models (LLMs) showcase impressive capabilities across various tasks with aligning their behavior with human preferences. |
| Approach: | They propose a framework that integrates domain-specific knowledge into a general reward model by model merging. |
| Outcome: | The proposed framework improves performance across different benchmarks and provides detailed analysis showing the effects of model merging. |
Fundamental Capabilities of Large Language Models and their Applications in Domain Scenarios: A Survey (2024.acl-long)
Copied to clipboard
Jiawei Li, Yizhe Yang, Yu Bai, Xiaofeng Zhou, Yinghao Li, Huashan Sun, Yuhang Liu, Xingpeng Si, Yuhao Ye, Yixiao Wu, 林一冠 林一冠, Bin Xu, Ren Bowen, Chong Feng, Yang Gao, Heyan Huang
| Challenge: | Existing benchmark-based evaluations cannot accurately reflect the performance of real-world applications. |
| Approach: | They propose a reliable strategy for domains to choose more robust LLMs for real-world applications. |
| Outcome: | The proposed strategy addresses the challenges faced by domains to choose more robust LLMs for real-world applications. |
Navigating Noisy Feedback: Enhancing Reinforcement Learning with Error-Prone Language Models (2024.findings-emnlp)
Copied to clipboard
Muhan Lin, Shuyang Shi, Yue Guo, Behdad Chalaki, Vaishnav Tadiparthi, Ehsan Moradi Pari, Simon Stepputtis, Joseph Campbell, Katia Sycara
| Challenge: | Reward hacking is a problem in reinforcement learning where the ability to specify the desired behavior of a reward function is difficult. |
| Approach: | They propose to use feedback as a potential-based shaping function to solicit and apply feedback from large language models to improve convergence speed and policy returns. |
| Outcome: | The proposed method improves convergence speed and policy returns over baselines even with significant ranking errors and eliminates the need for complex post-processing of reward functions. |
APLOT: Robust Reward Modeling via Adaptive Preference Learning with Optimal Transport (2025.emnlp-main)
Copied to clipboard
| Challenge: | Experimental results show that RLHF improves performance of Large Language Models . BT-based RMs struggle to distinguish between similar preference responses . |
| Approach: | They propose to enhance BT-based reward models by using an adaptive margin mechanism . they use semantic similarity and reward-predicted reward differences to adjust focus . |
| Outcome: | Experimental results show that the proposed method outperforms existing methods in both in-distribution and OOD settings. |
Can Small Language Models Help Large Language Models Reason Better?: LM-Guided Chain-of-Thought (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing frameworks for guiding a language model in reasoning tasks are limited by their tendency to generate low-quality rationales that are repetitive and vacuous. |
| Approach: | They propose a framework that leverages a lightweight language model for guiding a black-box large LM in reasoning tasks. |
| Outcome: | The proposed framework outperforms baselines in answer prediction accuracy. |