Papers with RLAIF
Neuro-Symbolic Agentic Reinforcement Learning for Long-Term Original Character Companionship and Interaction (2026.acl-short)
Copied to clipboard
| Challenge: | Existing LLM-based agents that are optimized by prompting or supervised fine-tuning exhibit a generalization gap in long-horizon, socially rich interactions. |
| Approach: | They propose a framework that formalizes OC companion agents’ interactions as a POMDP and decomposes the agent into three sub-policies optimized via closed-loop RL from AI feedback with verifiable rewards in a graph-constrained action space. |
| Outcome: | The proposed framework formalizes OC companion agents’ interactions as a POMDP and decomposes the agent into three sub-policies (Router, Memory, and Persona) with verifiable rewards in a graph-constrained action space. |
Tuning Large Multimodal Models for Videos using Reinforcement Learning from AI Feedback (2024.acl-long)
Copied to clipboard
| Challenge: | Recent advances in large language models have influenced the development of video large multimodal models (VLMMs). |
| Approach: | They propose a method that integrates video descriptions as context into a multimodal AI system to enrich the understanding of video content. |
| Outcome: | Empirical evaluations show that the proposed approach outperforms existing approaches for video large multimodal models (VLMMs) |
Self-Renewal Prompt Optimizing with Implicit Reasoning (2024.findings-emnlp)
Copied to clipboard
Zihan Liang, Ben Chen, Zhuoran Ran, Zihan Wang, Huangyu Dai, Yufei Ma, Dehong Gao, Xiaoyan Cai, Libin Yang
| Challenge: | Recent advances in NLP have been driven by the development of Large Language Models (LLMs). |
| Approach: | They propose a self-renewal approach to optimize LLM outputs to better align with human preferences without supervised fine-tuning. |
| Outcome: | The proposed approach improves outputs to better align with human preferences across LLMs and tasks without supervised fine-tuning. |
ARES: Alternating Reinforcement Learning and Supervised Fine-Tuning for Enhanced Multi-Modal Chain-of-Thought Reasoning Through Diverse AI Feedback (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large Multimodal Models excel at comprehending human instructions and demonstrate remarkable results across a broad spectrum of tasks. |
| Approach: | They propose an algorithm that alters REinforcement Learning and Supervised Fine-Tuning to refine large multimodal models with specific preferences. |
| Outcome: | The proposed algorithm achieves 70% win rate compared to baseline models judged by GPT-4o. |
RLKGF: Reinforcement Learning from Knowledge Graph Feedback Without Human Annotations (2025.findings-acl)
Copied to clipboard
| Challenge: | Lack of human preference labels remains a significant bottleneck when applying RLHF to a downstream domain. |
| Approach: | They propose a method that leverages human priors encoded in Knowledge Graphs (KGs) to derive RL rewards in the absence of manual annotations. |
| Outcome: | Experiments on three public and one private medical dialogue datasets show that the proposed method outperforms the competitive RLAIF in improving LLM diagnostic accuracy. |
Reinforcement Learning with Supervised Alignment (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Supervised fine-tuning (SFT) is a widely used method for adapting Large Language Models to specific tasks. |
| Approach: | They propose a method that uses supervised fine-tuning to train a reward model for reinforcement learning. |
| Outcome: | The proposed method outperforms existing methods on in-domain benchmarks but surpasses them 50 times on out-of-domain and cross-task evaluations. |
AceGPT, Localizing Large Language Models in Arabic (2024.naacl-long)
Copied to clipboard
Huang Huang, Fei Yu, Jianqing Zhu, Xuening Sun, Hao Cheng, Song Dingjie, Zhihong Chen, Mosen Alharthi, Bang An, Juncai He, Ziche Liu, Junying Chen, Jianquan Li, Benyou Wang, Lian Zhang, Ruoyu Sun, Xiang Wan, Haizhou Li, Jinchao Xu
| Challenge: | Significant concerns emerge when addressing cultural sensitivity and local values. |
| Approach: | They propose a localized Large Language Model (LLM) specifically for Arabic, a language imbued with unique cultural characteristics inadequately addressed by current mainstream models. |
| Outcome: | The proposed model sets the state-of-the-art standard for open Arabic LLMs across various benchmarks. |
Direct Large Language Model Alignment Through Self-Rewarding Contrastive Prompt Distillation (2024.acl-long)
Copied to clipboard
| Challenge: | Existing methods to evaluate preference data without human annotations are difficult . et al., 2022b) is effective for aligning large language models with human expectations . |
| Approach: | They propose a method to evaluate the response preference using output probabilities under contrastive prompts. |
| Outcome: | The proposed method could surpass the RLHF method without human-annotated preference data. |
One fish, two fish, but not the whole sea: Alignment reduces language models’ conceptual diversity (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing studies suggest large language models can capture certain behavioral patterns, but there are ongoing debates as to whether they are valid replacements for human subjects. |
| Approach: | They propose to use large language models as replacements for humans in behavioral research by relating the internal variability of simulated individuals to the population-level variability. |
| Outcome: | The proposed model can capture human-like conceptual diversity, but it is unclear whether post-training alignment affects models’ internal diversity. |
Optimizing Language Models with Fair and Stable Reward Composition in Reinforcement Learning (2024.emnlp-main)
Copied to clipboard
| Challenge: | Recent research has developed algorithms for reinforcement learning from human feedback and AI-generated feedback. |
| Approach: | They propose a method for reinforcement learning from human feedback and AI-generated feedback that incorporates weighting, ranking, and constraining to handle disparate rewards. |
| Outcome: | The proposed method reduces disparity and enhances stability among rewards . empirical results show that the proposed method is efficient and straightforward . |
DREAM: Disentangling Risks to Enhance Safety Alignment in Multimodal Large Language Models (2025.naacl-long)
Copied to clipboard
Jianyu Liu, Hangyu Guo, Ranjie Duan, Xingyuan Bu, Yancheng He, Shilong Li, Hui Huang, Jiaheng Liu, Yucheng Wang, Chenchen Jing, Xingwei Qu, Xiao Zhang, Pei Wang, Yanan Wu, Jihao Gu, Yangguang Li, Jianke Zhu
| Challenge: | Multimodal Large Language Models (MLLMs) pose unique safety challenges due to their integration of visual and textual data. |
| Approach: | They propose a method to disentangle risks through step-by-step reasoning within multimodal inputs. |
| Outcome: | The proposed approach improves safety alignment in MLLMs by fine-tuning and iterative Reinforcement Learning from AI feedback. |
SWE-QA-Pro: A Representative Benchmark and Scalable Training Recipe for Repository-Level Code Understanding (2026.findings-acl)
Copied to clipboard
Songcheng Cai, Zhiheng Lyu, Yuansheng Ni, Xiangchao Chen, Baichuan Zhou, Shenzhe Zhu, Yi Lu, Haozhe Wang, Chi Ruan, Benjamin Schneider, Weixu Zhang, Xiang Li, Andy Zheng, Yuyu Zhang, Ping Nie, Wenhu Chen
| Challenge: | Existing benchmarks for agentic repository-level code understanding overlook long tail topics and rely on memorized knowledge. |
| Approach: | They propose a repository-level agentic code understanding benchmark that uses long-tail repositories with executable environments to enforce topical balance. |
| Outcome: | Empirically, a Qwen3-8B model trained with the proposed benchmark outperforms GPT-4o by 2.3 points. |
Igniting Creative Writing in Small Language Models: LLM-as-a-Judge versus Multi-Agent Refined Rewards (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for enhancing Large Language Models (LLMs) struggle with novelty and Reinforcement Learning from human feedback (RLHF) is costly. |
| Approach: | They propose to use a Reward Model (RM) and a principle-guided LLM-as-a-Judge to enhance creative output over baselines. |
| Outcome: | The proposed approach significantly enhances creative output over baselines, but the principle-guided LLM-as-a-Judge yields superior generation quality. |
A Comprehensive Survey on Learning from Rewards for Large Language Models: Reward Models and Learning Strategies (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Recent developments in Large Language Models have shifted from pre-training to post-training and test-time scaling. |
| Approach: | They present a comprehensive overview of learning from rewards from the perspective of reward models and learning strategies across training, inference, and post-inference stages. |
| Outcome: | The proposed paradigm enables the transition from passive learning from static data to active learning from dynamic feedback. |
PopAlign: Diversifying Contrasting Patterns for a More Comprehensive Alignment (2025.acl-long)
Copied to clipboard
Zekun Moore Wang, Shenzhi Wang, King Zhu, Jiaheng Liu, Ke Xu, Jie Fu, Wangchunshu Zhou, Wenhao Huang
| Challenge: | Typical approaches to training large language models rely on limited contrasting patterns . contrasting data is limited and models are susceptible to harmful response tendencies . |
| Approach: | They propose a framework that integrates contrasting patterns across the prompt, model, and pipeline levels. |
| Outcome: | The proposed framework outperforms existing methods in the comparison of RQ1 and RQ2 . the proposed framework significantly outperformed existing methods, leading to more comprehensive alignment. |
Towards Better Value Principles for Large Language Model Alignment: A Systematic Evaluation and Enhancement (2025.acl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) show remarkable performance across tasks . alignment with human values is critical for their responsible development. |
| Approach: | They propose a framework that evaluates value principles along three desirable properties . they propose supervised fine-tuning, reinforcement learning-based approaches . |
| Outcome: | The proposed framework improves value principles along the three desirable properties of LLMs. |
Token-level Proximal Policy Optimization for Query Generation (2025.emnlp-main)
Copied to clipboard
Yichen Ouyang, Lu Wang, Fangkai Yang, Pu Zhao, Chenghua Huang, Jianfeng Liu, Bochen Pang, Yaming Yang, Yuefeng Zhan, Hao Sun, Qingwei Lin, Saravan Rajmohan, Weiwei Deng, Dongmei Zhang, Feng Sun
| Challenge: | Large Language Models (LLMs) have improved search engines and recommendation systems through their text understanding capabilities. |
| Approach: | They propose a token-level proximal policy optimization approach to empower LLMs to perform better in query generation through fine-tuning. |
| Outcome: | The proposed approach outperforms existing LLMs on an open-source and industrial dataset. |
Aligning Large Language Models via Fully Self-Synthetic Data (2026.acl-long)
Copied to clipboard
| Challenge: | Existing approaches to reinforcement learning from human feedback (RLHF) require expensive human-annotated datasets and proprietary models like GPT-4 to annotate preference pairs. |
| Approach: | They propose a self-synthetic framework for LLM alignment where all training data, including prompts (i.e., user queries), responses, and preferences, are generated by the model itself. |
| Outcome: | The proposed framework enhances the model’s chat capabilities on standard benchmarks like AlpacaEval 2.0 while maintaining strong performance on downstream objective tasks. |
Curriculum-RLAIF: Curriculum Alignment with Reinforcement Learning from AI Feedback (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to align large language models with human preferences are limited in generalizability due to distribution shift, preference label noise, and mismatch of challenging samples with model capacity. |
| Approach: | They propose a framework that constructs preference pairs with varying difficulty levels and then produces a specific curriculum for reward model training. |
| Outcome: | The proposed framework improves generalizability of reward models by a significant margin without incurring additional inference costs compared to existing non-curriculum baselines. |