ConsistRM: Improving Generative Reward Models via Consistency-Aware Self-Training

Challenge:	ConsistRM is a self-training framework that enables effective and stable GRM training without human annotations.
Approach:	They propose a self-training framework that enables effective and stable GRM training without human annotations.
Outcome:	The proposed framework outperforms vanilla Reinforcement Fine-Tuning (RFT) by 1.5% on five benchmark datasets.

Similar Papers

Overview

Challenge

Approach

Outcome

ReflectRM: Boosting Generative Reward Models via Self-Reflection within a Unified Judgment Framework (2026.acl-long)

Copied to clipboard

Kai Qin, Liangxin Liu, Yu Liang, Longzheng Wang, null Wangyan, Zhang Yueyang, Long Xia, Zhiyuan Sun, Houde Liu, Daiting Shi

Challenge:	Existing methods for generating reward models focus on outcome-level supervision, neglecting analytical process quality, which constrains their potential.
Approach:	They propose a novel reward model that leverages self-reflection to assess analytical quality and enhance preference modeling.
Outcome:	The proposed model improves performance on four benchmarks and significantly mitigates positional bias.

From Outcomes to Processes: Guiding PRM Learning from ORM for Inference-Time Alignment (2025.acl-long)

Copied to clipboard

Bin Xie, Bingbing Xu, Yige Yuan, Shengmao Zhu, Huawei Shen

Summarization

Challenge:	Existing approaches to align large language models with human preferences suffer from inconsistent scoring and suboptimal alignment.
Approach:	They propose a dual-consistency framework that aligns partial sequences with human preferences.
Outcome:	The proposed framework significantly reduces granularity discrepancies and improves GPT-4 evaluation scores.

Aligning Large Language Models via Fully Self-Synthetic Data (2026.acl-long)

Copied to clipboard

Shangjian Yin, Zhepei Wei, Xinyu Zhu, Wei-Lin Chen, Yu Meng

Language Models

Challenge:	Existing approaches to reinforcement learning from human feedback (RLHF) require expensive human-annotated datasets and proprietary models like GPT-4 to annotate preference pairs.
Approach:	They propose a self-synthetic framework for LLM alignment where all training data, including prompts (i.e., user queries), responses, and preferences, are generated by the model itself.
Outcome:	The proposed framework enhances the model’s chat capabilities on standard benchmarks like AlpacaEval 2.0 while maintaining strong performance on downstream objective tasks.

Aligning Large Language Models through Synthetic Feedback (2023.emnlp-main)

Copied to clipboard

Sungdong Kim, Sanghwan Bae, Jamin Shin, Soyoung Kang, Donghyun Kwak, Kang Yoo, Minjoon Seo

Semantic Text Processing

Challenge:	Currently, alignment learning requires significant human demonstrations and feedback from proprietary LLMs such as ChatGPT.
Approach:	They propose a framework that uses synthetic feedback to align large language models to human values without extensive human annotations and proprietary LLMs.
Outcome:	The proposed model outperforms open-source models on human-annotated demonstrations in alignment benchmarks.

Outcome Accuracy is Not Enough: Aligning the Reasoning Process of Reward Models (2026.acl-long)

Copied to clipboard

Binghai Wang, Yantao Liu, Yuxuan Liu, Tianyi Tang, Shenzhi Wang, Chang Gao, Chujie Zheng, Yichang Zhang, Le Yu, Shixuan Liu, Tao Gui, Qi Zhang, Xuanjing Huang, Bowen Yu, Fei Huang, Junyang Lin

Reasoning

Challenge:	Recent studies observe a phenomenon where reward models achieve high accuracy on static datasets but fail to generalize effectively during RLHF.
Approach:	They propose a method that combines rationale consistency with outcome accuracy to improve performance on RM-Bench and JudgeBench.
Outcome:	The proposed method surpasses baselines on RM-Bench and JudgeBench by an average of 5% and improves creative writing tasks by 7%.

Self-Generated Critiques Boost Reward Modeling for Language Models (2025.naacl-long)

Copied to clipboard

Yue Yu, Zhengxing Chen, Aston Zhang, Liang Tan, Chenguang Zhu, Richard Yuanzhe Pang, Yundi Qian, Xuewei Wang, Suchin Gururangan, Chao Zhang, Melanie Kambadur, Dhruv Mahajan, Rui Hou

Language Models

Challenge:	Existing reward models produce scalar scores and struggle to incorporate critiques in a natural language format.
Approach:	They propose a framework that predicts critiques and rewards using self-generated critiques without extra supervision.
Outcome:	The proposed framework improves reward modeling accuracy by 3.7%-7.3% compared to standard reward models and LLM judges.

OpenRubrics: Towards Scalable Synthetic Rubric Generation for Reward Modeling and LLM Alignment (2026.acl-long)

Copied to clipboard

Tianci Liu, Ran Xu, Tony Yu, Ilgee Hong, Carl Yang, Tuo Zhao, Haoyu Wang

Challenge:	Existing reward models rely on scalar or pairwise judgments that fail to capture multifaceted nature of human preferences.
Approach:	They propose a rubric-based reward model that uses a large collection of prompt, rubric pairs to generate a scalar score or preference label for each response.
Outcome:	The proposed model surpasses strong size-matched baselines by 8.4% across multiple benchmarks.

Prototypical Reward Network for Data-Efficient Model Alignment (2024.acl-long)

Copied to clipboard

Jinghan Zhang, Xiting Wang, Yiqiao Jin, Changyu Chen, Xinhao Zhang, Kunpeng Liu

Green & Sustainable NLP

Challenge:	Reinforcement Learning from Human Feedback (RLHF) is a reward model that fine-tunes Large Language Models (LLMs) by utilizing Prototypical Networks.
Approach:	They propose a framework utilizing Prototypical Networks to enhance reward models under limited human feedback, enabling more stable and reliable structural learning from fewer samples.
Outcome:	The proposed framework improves reward models under limited human feedback, surpassing traditional methods, especially in data-limited scenarios.

Structural Reward Model: Enhancing Interpretability, Efficiency, and Scalability in Reward Modeling (2025.emnlp-industry)

Copied to clipboard

Xiaoyu Liu, Di Liang, Hongyu Shan, Peiyang Liu, Yonghao Liu, Muling Wu, Yuntao Li, Xianjie Wu, Li Miao, Jiangrong Shen, Minlong Peng

Responsible & Trustworthy NLP

Challenge:	Generative RMs (GRMs) lack contextual and background information during inference, leading to incomplete evaluations.
Approach:	They propose a modular and interpretable framework that integrates side-branch models as auxiliary feature generators.
Outcome:	The proposed framework outperforms scalar and saline reward models in robustness and alignment with human preferences.

HAF-RM: A Hybrid Alignment Framework for Reward Model Training (2025.acl-long)

Copied to clipboard

Shujun Liu, Xiaoyu Shen, Yuhang Lai, Siyuan Wang, Shengbin Yue, Zengfeng Huang, Xuanjing Huang, Zhongyu Wei

Language Models

Challenge:	Recent studies have focused on enhancing reward models through data improvements, following the conventional training framework for reward models that directly optimizes the predicted rewards.
Approach:	They propose a hybrid alignment framework HAF-RM that incorporates additional constraint on token-level policy probabilities in addition to the reward score.
Outcome:	The proposed framework can supervise the internal preference model at the token level and optimize the mapping layer of the reward model at sequence level.