Cross-lingual Transfer of Reward Models in Multilingual Alignment (2025.naacl-short)
Copied to clipboard
| Challenge: | Recent studies in reward modeling schemes are skewed towards English, limiting the applicability of RLHF in multilingual alignments. |
| Approach: | They investigate cross-lingual transfer of English RMs by representation shifts . they also analyze cross-linguistic transfer of RM through the representation shift . |
| Outcome: | The results show that English RMs can be transferred across languages by 34% . |
Similar Papers
M-RewardBench: Evaluating Reward Models in Multilingual Settings (2025.acl-long)
Copied to clipboard
Srishti Gureja, Lester James Validad Miranda, Shayekh Bin Islam, Rishabh Maheshwary, Drishti Sharma, Gusti Triandi Winata, Nathan Lambert, Sebastian Ruder, Sara Hooker, Marzieh Fadaee
| Challenge: | Reward models (RMs) are primarily trained and evaluated in English and their capabilities in multilingual settings remain understudied. |
| Approach: | They construct a multilingual RM evaluation benchmark that tests the chat, safety, reasoning, and translation capabilities of RMs in 23 languages. |
| Outcome: | The proposed model performs better for high-resource languages and improves with translation quality. |
Reuse Your Rewards: Reward Model Transfer for Zero-Shot Cross-Lingual Alignment (2024.emnlp-main)
Copied to clipboard
| Challenge: | Multilingual human preference data are difficult to obtain at scale, making it challenging to extend this framework to diverse languages. |
| Approach: | They propose a method where a reward model is trained on preference data in one source language and applied to other target languages. |
| Outcome: | The proposed approach is effective under comprehensive evaluation settings, including human evaluation. |
The Accuracy Paradox in RLHF: When Better Reward Models Don’t Yield Better Language Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | Reinforcement Learning from Human Feedback (RLHF) significantly enhances Natural Language Processing by aligning language models with human expectations. |
| Approach: | They propose to integrate feedback from humans into RLHF to improve language models by capturing human-like preferences. |
| Outcome: | The proposed model outperforms models trained with moderately accurate reward models on relevance, factuality, and completeness tasks. |
RewardBench: Evaluating Reward Models for Language Modeling (2025.findings-naacl)
Copied to clipboard
Nathan Lambert, Valentina Pyatkin, Jacob Morrison, Lester James Validad Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, Noah A. Smith, Hannaneh Hajishirzi
| Challenge: | Evaluating reward models presents an opportunity to understand the opaque technologies used for alignment of language models. |
| Approach: | They present a benchmark dataset and code-base for evaluation of reward models . they use prompt-chosen-rejected trios to benchmark how they perform on queries . |
| Outcome: | The proposed dataset compares RMs with other models on a set of questions. |
RED: Unleashing Token-Level Rewards from Holistic Feedback via Reward Redistribution (2025.emnlp-main)
Copied to clipboard
| Challenge: | Experimental results demonstrate the superiority of our approach to aligning large language models with human preferences. |
| Approach: | They propose a method that evaluates and assigns specific credit to each token using an off-the-shelf reward model. |
| Outcome: | The proposed method evaluates and assigns specific credit to each token using an off-the-shelf reward model. |
RLHF Can Speak Many Languages: Unlocking Multilingual Preference Optimization for LLMs (2024.emnlp-main)
Copied to clipboard
| Challenge: | Preference optimization is a widely adopted post-training technique to align large language models with human preferences. |
| Approach: | They propose a method for generating multilingual feedback data to balance data coverage. |
| Outcome: | The proposed method achieves 54.4% win-rate against current state-of-the-art multilingual LLM in its parameter class and 69.5% win- rate or higher against widely used models like Gemma, Mistral and Llama 3. |
Demystifying Multilingual Reasoning in Process Reward Modeling (2025.findings-emnlp)
Copied to clipboard
| Challenge: | a recent study focuses on the use of large language models to solve multi-step reasoning tasks. |
| Approach: | They propose to extend large language models to multilingual settings by extending process reward models to English . they train multilingual PRMs on a dataset spanning seven languages, which is translated from english . |
| Outcome: | The proposed model improves accuracy and reduces early-stage reasoning errors. |
Best-of-L: Cross-Lingual Reward Modeling for Mathematical Reasoning (2026.findings-eacl)
Copied to clipboard
| Challenge: | Recent studies have focused on improving reasoning ability in English models, with multilingual models receiving comparatively little attention. |
| Approach: | They propose a framework that ranks candidate reasoning traces across languages rather than within a single language. |
| Outcome: | The proposed framework improves accuracy by up to 10 points in English compared to using reward modeling within a single language. |
Not All Voices Are Rewarded Equally: Probing and Repairing Reward Models across Human Diversity (2025.findings-emnlp)
Copied to clipboard
Zihao Li, Feihao Fang, Xitong Zhang, Jiaru Zou, Zhining Liu, Wei Xiong, Ziwei Wu, Baoyu Jing, Jingrui He
| Challenge: | Using real-world datasets, we conduct the most comprehensive study to date, auditing various state-of-the-art reward models across nine sensitive attributes, including age, gender, ethnicity, etc. |
| Approach: | They propose a method to mitigate group disparities in reward modeling by using real-world data. |
| Outcome: | The proposed method is based on a population-based dataset with nine demographic attributes, including gender, ethnicity, age, gender, and ethnicity. |
Towards Pareto-Efficient RLHF: Paying Attention to a Few High-Reward Samples with Reward Dropout (2024.findings-emnlp)
Copied to clipboard
| Challenge: | RLHF is a bi-objective problem that has the nature of a Pareto optimization . reward dropout is generalizable and most effective with non-pretrained target models . |
| Approach: | They propose a method that guarantees a Pareto improvement by leveraging reinforcement learning to fine-tune language models. |
| Outcome: | The proposed method guarantees a Pareto improvement on two benchmark datasets . it is generalizable and most effective with non-pretrained target models, saving the effort of pretraining. |