Papers with PPO
Rethinking the Evaluation of Alignment Methods: Insights into Diversity, Generalisation, and Safety (2026.eacl-srw)
Copied to clipboard
Denis Janiak, Julia Moska, Dawid Motyka, Karolina Seweryn, Paweł Walkowiak, Bartosz Żuk, Arkadiusz Janz
| Challenge: | Existing studies focus on individual techniques or specific dimensions, lacking a holistic assessment of the inherent trade-offs. |
| Approach: | They propose a framework that compares LLM alignment methods across five axes . they use a validated LLM-as-judge prompt to compare the results . |
| Outcome: | The proposed framework compares LLM alignment methods across factuality, safety, conciseness, proactivity, diversity and safety axes . it provides insights into trade-offs of common alignment methods, guiding the development of more balanced and reliable LLMs. |
GAPO: Learning Preferential Prompt through Generative Adversarial Policy Optimization (2025.acl-long)
Copied to clipboard
Zhouhong Gu, Xingzhou Chen, Xiaoran Shi, Tao Wang, Suhang Zheng, Tianyu Li, Hongwei Feng, Yanghua Xiao
| Challenge: | Existing methods for achieving this require a limited understanding of constraints and can be hallucinating or brittle. |
| Approach: | They propose a framework that combines adversarial training dynamics with an encoder-only reward model to progressively learn and adapt to increasingly complex constraints. |
| Outcome: | Extensive experiments show that GAPO significantly outperforms existing methods like PPO, DPO, and KTO in fine-grained constraints. |
P-TA: Using Proximal Policy Optimization to Enhance Tabular Data Augmentation via Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Contemporary approaches to generate tabular data are limited due to the lack of external knowledge. |
| Approach: | They propose to use proximal policy optimization to apply GANs and fine-tune Large Language Models to enhance the probability distribution of tabular features. |
| Outcome: | The proposed method improves accuracy of GANs and LLMs over state-of-the-art over three real-world datasets. |
Automatic Unit Test Data Generation and Actor-Critic Reinforcement Learning for Code Synthesis (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Large pre-trained language models have shown remarkable performance on various benchmarks, treating the problem of Code Generation in a fashion similar to Natural Language Generation, trained with a Language Modelling (LM) objective. |
| Approach: | They propose a method to automatically obtain function signatures and associated Unit Tests, suitable for RL training of Code Synthesis models. |
| Outcome: | The proposed model improves on a pre-trained model on various benchmarks and is more efficient than previous models. |
RLHF Algorithms Ranked: An Extensive Evaluation Across Diverse Tasks, Rewards, and Hyperparameters (2025.emnlp-industry)
Copied to clipboard
Lucas Spangher, Rama Kumar Pasumarthi, Nick Masiewicki, William F. Arnold, Aditi Kaushal, Dale Johnson, Peter Grabowski, Eugene Ie
| Challenge: | Proximal Policy Optimization (PPO) has fallen out of favor for Large Language Models (LLMs), but its complexity and inefficiency have spurred the investigation of simpler alternatives. |
| Approach: | They evaluate 17 RLHF algorithms on two benchmarks, OpenAI’s TL;DR Summarization and Anthropic’s Helpfulness / Harmlessness. |
| Outcome: | The proposed methods are based on OpenAI’s TL;DR Summarization and Anthropic’s Helpfulness / Harmlessness benchmarks with two different reward models and a Rules based reward model. |
Aligning Large Language Models via Fine-grained Supervision (2024.acl-short)
Copied to clipboard
| Challenge: | Pre-trained large-scale language models often generate biased or toxic text, misaligning with human intentions. |
| Approach: | They propose to use human feedback to improve LLM alignment by fine-grained token supervision . they ask annotators to edit less preferred responses to make them more favorable . |
| Outcome: | The proposed method improves LLM alignment by up to 5.1% in terms of win rate compared with the traditional model. |
SeRTS: Self-Rewarding Tree Search for Biomedical Retrieval-Augmented Generation (2024.findings-emnlp)
Copied to clipboard
Minda Hu, Licheng Zong, Hongru Wang, Jingyan Zhou, Jingjing Li, Yichen Gao, Kam-Fai Wong, Yu Li, Irwin King
| Challenge: | Existing retrieval-augmented approaches to large language models face performance limitations due to the lack of publicly available training data. |
| Approach: | They propose a plug-and-play LLM-based retrieval method called Self-Rewarding Tree Search based on Monte Carlo Tree Search and a self-rewarding paradigm to address these limitations. |
| Outcome: | The proposed method improves the performance of the BM25 retriever and surpasses the baseline of self-reflection in both efficiency and scalability. |
BV-Blend: Uncertainty-Weighted Historical Baselines for Stable Critic-Free RL with Verifiable Rewards (2026.findings-acl)
Copied to clipboard
| Challenge: | Critic-free reinforcement learning with verifiable rewards (RLVR) is a practical paradigm for aligning Large Language Models. |
| Approach: | They propose a framework that stabilizes advantage estimation by combining prompt-local on-policy statistics with semantic-cluster-conditioned historical moments. |
| Outcome: | Experiments show that RLVR improves training stability and performance compared to critic-based methods . compared with other approaches, RL VR improves in cold-start regimes with binary verifiers . |
MPPO: Multi Pair-wise Preference Optimization for LLMs with Arbitrary Negative Samples (2025.coling-main)
Copied to clipboard
Shuo Xie, Fangzhi Zhu, Jiahui Wang, Lulu Wen, Wei Dai, Xiaowei Chen, Junxiong Zhu, Kai Zhou, Bo Zheng
| Challenge: | Existing preference optimization methods such as DPO and KTO are inherently derived from PPO, requiring a reference model that adds GPU memory resources and relies heavily on abundant preference data. |
| Approach: | They propose an algorithm that leverages the average likelihood of model responses to fit the reward function and maximizes the utilization of preference data. |
| Outcome: | The proposed algorithm outperforms DPO, ORPO, and SimPO on MT-Bench and Arena-Hard. |
RS-DPO: A Hybrid Rejection Sampling and Direct Preference Optimization Method for Alignment of Large Language Models (2024.findings-naacl)
Copied to clipboard
| Challenge: | Reinforcement learning with human feedback (RLHF) is widely employed to align large language models with user intent. |
| Approach: | They propose to combine rejection sampling and direct preference optimization to improve alignment with user intent by identifying pairs of contrastive samples from human annotator and alternative LLMs. |
| Outcome: | The proposed method outperforms existing methods including RS, PPO, and DPO in a limited resource environment. |
Unsupervised Text Style Transfer for Controllable Intensity (2026.findings-eacl)
Copied to clipboard
| Challenge: | Unsupervised Text Style Transfer (UTST) aims to transfer the stylistic properties of a given text without parallel text pairs. |
| Approach: | They propose a SFT-then-PPO paradigm to fine-tune an LLM with parallel data and reward functions for distinguishing stylistic intensity in hierarchical levels. |
| Outcome: | The proposed system can transfer stylistic properties without parallel text pairs even for adjacent levels of intensity. |
Black-Box Prompt Optimization: Aligning Large Language Models without Model Training (2024.acl-long)
Copied to clipboard
| Challenge: | Large language models are often not well aligned with human intents, which requires additional training. |
| Approach: | They propose to use Black-Box Prompt Optimization (BPO) to perform alignments on large language models that are not well aligned with human intents. |
| Outcome: | The proposed model outperforms existing models and is model-agnostic. |
Learning to Ask: Multi-Decoder Fine-Tuning for Multi-Hop Visual Question Generation with External Knowledge (2026.findings-eacl)
Copied to clipboard
| Challenge: | Traditional supervised QG methods rely on tokenlevel alignment with fixed gold labels struggle to capture diverse valid question formulations. |
| Approach: | They propose a model-agnostic framework that integrates multimodal inputs with a multi-decoder architecture to optimize for multiple labels per sample. |
| Outcome: | The proposed framework improves fluency, reasoning depth, and relevance of visual questions. |
Guided Dialogue Policy Learning without Adversarial Learning in the Loop (2020.findings-emnlp)
Copied to clipboard
Ziming Li, Sungjin Lee, Baolin Peng, Jinchao Li, Julia Kiseleva, Maarten de Rijke, Shahin Shayandeh, Jianfeng Gao
| Challenge: | Reinforcement learning methods suffer from sparse and unstable reward signals . alternating training of dialogue agent and reward model can get stuck in local optima . |
| Approach: | They propose to decompose adversarial training into two steps to improve dialogue policy learning. |
| Outcome: | The proposed method achieves remarkable task success rate using both on-policy and off-poly reinforcement learning methods. |
Probabilistic Robustness for Data Filtering (2023.eacl-main)
Copied to clipboard
| Challenge: | Modern machine learning works with massive amounts of data on a range of tasks like language modeling, object detection, and data mining. |
| Approach: | They propose a probabilistic robustness rewarded data optimization approach to enhance the model's generalization power by selecting training data that optimizes probabilistic metrics. |
| Outcome: | The proposed approach achieves +17.2% increase of accuracy and -28.05 decrease of perplexity on unknown-domain test sets. |
Clipping Loops for Sample-Efficient Dialogue Policy Optimisation (2021.naacl-main)
Copied to clipboard
| Challenge: | In previous work, a large number of human dialogues are required to train dialogue agents. |
| Approach: | They propose loop-clipping policy optimisation to eliminate useless responses by clipping loops from dialogue history and clipping advantage to distinguish useless actions from others. |
| Outcome: | The proposed method achieves 80% success rate on a Cambridge restaurant dialogue system using 260 training dialogues compared to baseline of 2160 dialogues. |
UAlign: Leveraging Uncertainty Estimations for Factuality Alignment on Large Language Models (2025.acl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) often struggle to accurately express factual knowledge, especially in cases where the knowledge boundaries are ambiguous. |
| Approach: | They propose a framework that leverages Uncertainty estimations to represent knowledge boundaries and incorporates these representations into prompts for LLMs to Align with factual knowledge. |
| Outcome: | The proposed framework significantly improves the LLMs’ capacities to confidently answer known questions and refuse unknown questions on both in-domain and out-of-domain tasks. |
SPO: Self Preference Optimization with Self Regularization (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing reference-free preference optimization methods exhibit higher training efficiency but are prone to overoptimization, leading to performance degradation. |
| Approach: | They propose a reference-free preference optimization method that replaces the logsigmoid loss function with a SiLU function to improve the model's performance. |
| Outcome: | The proposed method achieves 7% improvement over SimPO on AlpacaEval 2 and MT-Bench. |
Turn-PPO: Turn-Level Advantage Estimation with PPO for Improved Multi-Turn RL in Agentic LLMs (2026.findings-eacl)
Copied to clipboard
| Challenge: | Reinforcement learning (RL) has re-emerged as a natural approach for training interactive LLM agents in real-world environments. |
| Approach: | They propose a variant that operates on a turn-level MDP formulation, instead of the commonly used token-level one. |
| Outcome: | The proposed method is more robust than the widely used GRPO algorithm and more efficient than token-level MDPs. |
InternLM-XComposer2.5-Reward: A Simple Yet Effective Multi-Modal Reward Model (2025.findings-acl)
Copied to clipboard
Yuhang Zang, Xiaoyi Dong, Pan Zhang, Yuhang Cao, Ziyu Liu, Shengyuan Ding, Shenxi Wu, Yubo Ma, Haodong Duan, Wenwei Zhang, Kai Chen, Dahua Lin, Jiaqi Wang
| Challenge: | Despite the promising performance of Large Vision Language Models, they sometimes generate incorrect outputs. |
| Approach: | They propose a multi-modal reward model that aligns LVLMs with human preferences. |
| Outcome: | The proposed model achieves excellent results on the latest multi-modal reward model benchmark and shows competitive performance on text-only reward model. |
Chatbot To Help Patients Understand Their Health (2025.findings-emnlp)
Copied to clipboard
Won Seok Jang, Hieu Tran, Manav Shaileshkumar Mistry, Sai Kiran Gandluri, Yifan Zhang, Sharmin Sultana, Sunjae Kwon, Yuan Zhang, Zonghai Yao, Hong Yu
| Challenge: | NoteAid-Chatbot is a conversational AI designed to help patients better understand their health . |
| Approach: | They propose a new learning paradigm that leverages a multi-agent large language model and reinforcement learning framework without relying on costly human-generated training data. |
| Outcome: | The proposed framework surpasses non-expert human training methods. |
SPPO: Sequence-Level PPO for Long-Horizon Reasoning Tasks (2026.acl-long)
Copied to clipboard
Tianyi Wang, Yixia Li, Long Li, Yibiao Chen, Shaohan Huang, Yun Chen, Peng Li, Yang Liu, Guanhua Chen
| Challenge: | Proximal Policy Optimization (PPO) is central to aligning Large Language Models with verifiable rewards. |
| Approach: | They propose a scalable algorithm that harmonizes sample efficiency with stability of outcome-based updates. |
| Outcome: | The proposed algorithm outperforms standard PPO and matches the performance of computation-heavy group-based methods. |
NAT: Enhancing Agent Tuning with Negative Samples (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing methods for fine-tuning and reinforcement learning use only positive examples, limiting their efficiency in low-resource scenarios. |
| Approach: | They propose a method that leverages both successful and failed trajectories for fine-tuning, maximizing the utility of limited resources. |
| Outcome: | The proposed method surpasses existing methods, including SFT, DPO, and PPO, across various tasks. |
ReFT: Reasoning with Reinforced Fine-Tuning (2024.acl-long)
Copied to clipboard
| Challenge: | Existing approaches to improve the generalization of large language models are using Supervised Fine-Tuning (SFT) this approach does not show sufficient generalization ability because it only relies on the given CoT data. |
| Approach: | They propose to use Chain-of-Thought annotations to train Large Language Models using supervised fine-tuning to improve generalization. |
| Outcome: | The proposed approach outperforms SFT on GSM8K, MathQA, and SVAMP datasets and shows a superior generalization ability. |
Teach a Reward Model to Correct Itself: Reward Guided Adversarial Failure Discovery for Robust Reward Modeling (2026.acl-long)
Copied to clipboard
| Challenge: | Existing failure discovery methods rely on prior knowledge of preference attributes . Existing methods do not scale to new models or data. |
| Approach: | They propose a preference distribution agnostic procedure that uses the reward model itself to guide controlled decoding toward mis specified responses while preserving the underlying preference class. |
| Outcome: | The proposed procedure improves robustness without degrading reward quality across models. |
ARM: Alignment with Residual Energy-Based Model (2024.naacl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) acquire a wide range of abilities and abilities, but their behavior does not align with human preferences. |
| Approach: | They propose to minimize a forward Kullback–Leibler divergence from a target policy to a parameteric policy instead of a reverse KL as in RLHF methods. |
| Outcome: | The proposed method can learn an aligned policy by minimizing a forward Kullback–Leibler divergence from a target policy to a parameteric policy instead of a reverse KL as in RLHF methods. |
Inverse-Q*: Token Level Reinforcement Learning for Aligning Large Language Models Without Preference Data (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Reinforcement Learning from Human Feedback (RLHF) relies on complex methodologies like Proximal Policy Optimization (PPO) that require extensive hyper-parameter tuning and pose challenges in sample efficiency and stability. |
| Approach: | They propose an innovative framework that leverages direct preference optimization techniques but extends them by estimating the conditionally optimal policy directly from the model’s responses. |
| Outcome: | The proposed framework matches and exceeds the effectiveness of Proximal Policy Optimization (PPO) in terms of convergence speed and alignment of model responses with human preferences. |
DialCoT Meets PPO: Decomposing and Exploring Reasoning Paths in Smaller Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Chain-of-Thought prompting has improved the reasoning capabilities of Large Language Models (LLMs) but it is ineffective or detrimental to the performance on reasoning tasks in Smaller Language Model (SLMs) with less than 10 billion parameters. |
| Approach: | They propose a Dialogue-guided Chain-of-Thought method to improve the reasoning capabilities of Large Language Models (LLMs) by generating intermediate reasoning steps in a dialogue format to guide the model to the final answer. |
| Outcome: | The proposed method can achieve significant performance gains over state-of-the-art competitors on four arithmetic reasoning datasets. |
Efficient Integration of External Knowledge to LLM-based World Models via Retrieval-Augmented Generation and Reinforcement Learning (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing attempts to enhance LLM-based world models through prompting or fine-tuning approaches are either requiring human knowledge or computationally extensive. |
| Approach: | They propose a framework that leverages retrieval-augmented generation to integrate external knowledge to LLM-based world models. |
| Outcome: | The proposed framework outperforms baseline models and exhibits strong generalizability. |
Exploiting Tree Structure for Credit Assignment in Reinforcement Learning with Large Language Models (2026.findings-acl)
Copied to clipboard
| Challenge: | Reinforcement learning has shown strong promise for strengthening reasoning ability of large language models, but sparse, delayed rewards make token-level credit assignment a central challenge. |
| Approach: | They propose a critic-free algorithm that rewards tokens that change the solution. |
| Outcome: | The proposed algorithm improves on in-distribution benchmarks and out-of-disttribution settings. |
trlX: A Framework for Large Scale Reinforcement Learning from Human Feedback (2023.emnlp-main)
Copied to clipboard
Alexander Havrilla, Maksym Zhuravinskyi, Duy Phung, Aman Tiwari, Jonathan Tow, Stella Biderman, Quentin Anthony, Louis Castricato
| Challenge: | Current RLHF paradigms rely on Proximal Policy Optimization (PPO), which quickly becomes a challenge to implement and scale up to large architectures. |
| Approach: | They propose an open-source framework for reinforcement learning from human feedback . it allows for offline fine-tuning of large language models . |
| Outcome: | The framework can be used to fine-tune models up to and exceeding 70 billion parameters. |
MAPO: Advancing Multilingual Reasoning through Multilingual-Alignment-as-Preference Optimization (2024.acl-long)
Copied to clipboard
| Challenge: | Existing models exhibit inconsistent reasoning abilities across different languages . existing models lack consistency across languages due to imbalance of training data . |
| Approach: | They propose a multilingual alignment-as-preference optimization framework to align reasoning processes in other languages with the dominant language. |
| Outcome: | The proposed framework improves multilingual reasoning across languages on three benchmarks. |
Improving NMT Models by Retrofitting Quality Estimators into Trainable Energy Loss (2025.coling-main)
Copied to clipboard
| Challenge: | Reinforcement learning has shown great promise in aligning language models with human preferences in a variety of text generation tasks, including machine translation. |
| Approach: | They propose a method that employs quality estimators as trainable loss networks to backpropagate to the NMT model. |
| Outcome: | The proposed method outperforms strong baselines and proximal policy optimizations on English-to-Mongolian translation. |
EROS:Entity-Driven Controlled Policy Document Summarization (2024.lrec-main)
Copied to clipboard
| Challenge: | a privacy policy is a crucial component of any organization that allows it to legally collect, process, store, and/or distribute personal data. |
| Approach: | They propose to use a policy-document summarization dataset to enforce the summaries to include critical privacy-related entities and organization’s rationale in collecting those entities. |
| Outcome: | The proposed model improves over baselines and qualitatively evaluates the proposed model on human and qualitative data. |
StablePrompt : Automatic Prompt Tuning using Reinforcement Learning for Large Language Model (2024.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in large language models have made it difficult to find appropriate prompts for tasks with multiple input-output formats. |
| Approach: | They propose a prompt tuning method based on reinforcement learning (RL) they propose an anchor model and an extension for generating input-dependent prompts. |
| Outcome: | The proposed method outperforms existing methods on a variety of tasks and achieves State-of-the-art performance across diverse types and sizes of LLMs. |
RLHS: Mitigating Misalignment in RLHF with Hindsight Simulation (2026.findings-acl)
Copied to clipboard
| Challenge: | Reinforcement Learning from Hindsight Simulation (RLHF) can cause severe misalignment in generative AI, but it is not a universal method for fine-tuning large language models. |
| Approach: | They propose a method that uses evaluator feedback to decouple alignment signal from potentially compromised predictions. |
| Outcome: | The proposed method significantly outperforms RLHF in comparisons with baselines and human evaluations. |
SoRFT: Issue Resolving with Subtask-oriented Reinforced Fine-Tuning (2025.acl-long)
Copied to clipboard
| Challenge: | Existing issue-resolving frameworks rely on commercial models, leading to high costs and privacy concerns. |
| Approach: | They propose a training approach to enhance issue resolving capability of LLMs by decomposing issue reasolving into subtasks. |
| Outcome: | The proposed approach improves issue-resolving performance and generalizes model . it is cost-effective and provides a cost-efficient alternative to commercial models . |
Retrieval-based Knowledge Transfer: An Effective Approach for Extreme Large Language Model Compression (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Large-scale pre-trained language models (LLMs) have demonstrated exceptional performance in various natural language processing (NLP) tasks. |
| Approach: | They propose a new compression paradigm that extracts knowledge from pre-trained language models to construct a knowledge store from which the model can leverage it for effective inference. |
| Outcome: | The proposed model extracts knowledge from LLMs to construct a knowledge store, which the model can leverage for effective inference. |
Memory-R1: Enhancing Large Language Model Agents to Manage and Utilize Memories via Reinforcement Learning (2026.acl-long)
Copied to clipboard
Sikuan Yan, Xiufeng Yang, Zuchao Huang, Ercong Nie, Zifeng Ding, Zonggen Li, Xiaowen Ma, Jinhe Bi, Kristian Kersting, Jeff Z. Pan, Hinrich Schuetze, Volker Tresp, Yunpu Ma
| Challenge: | Large Language Models (LLMs) are stateless and limited by a finite context window, preventing them from maintaining knowledge across long conversations or evolving tasks. |
| Approach: | They propose a reinforcement learning framework that empowers LLMs to actively manage external memory through two specialized agents. |
| Outcome: | The proposed framework outperforms baselines and benchmarks across diverse question types, three benchmarks, and multiple model scales. |
Android Coach: Improve Online Agentic Training Efficiency with Single State Multiple Actions (2026.acl-long)
Copied to clipboard
| Challenge: | Existing reinforcement learning methods are expensive due to high latency and sample inefficiency . Currently, RL is limited to one-to-one state-action pairs . |
| Approach: | They propose a framework that shifts the training paradigm to Single State Multiple Actions and introduce a group-wise advantage estimator based on the averaged critic outputs. |
| Outcome: | The proposed framework achieves 7.5% and 8.3% success rate improvements on AndroidLab and AndroidWorld over UI-TARS-1.5-7B and attains 1.4x higher training efficiency than existing methods. |
Back to Basics: Revisiting REINFORCE-Style Optimization for Learning from Human Feedback in LLMs (2024.acl-long)
Copied to clipboard
Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, Sara Hooker
| Challenge: | Proximal Policy Optimization (PPO) is used for RLHF but requires high computational cost and sensitive hyperparameter tuning. |
| Approach: | They propose to use Proximal Policy Optimization to align large language models to human preferences. |
| Outcome: | The proposed method preserves and even increases performance while preserving the motivational principles that led to the development of PPO. |
FAITH: Factuality Alignment through Integrating Trustworthiness and Honestness (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to correct factually inaccurate outputs are lacking the semantic richness needed to properly understand its internal states of trustworthiness and honesty. |
| Approach: | They propose a framework for factuality alignment that integrates natural-language uncertainty signals with external knowledge and computes confidence scores and semantic entropy from LLM outputs. |
| Outcome: | Extensive experiments on four knowledge-intensive benchmarks show that FAITH improves the factual accuracy and truthfulness of Large Language Models (LLMs). |
SeqPO-SiMT: Sequential Policy Optimization for Simultaneous Machine Translation (2025.findings-acl)
Copied to clipboard
| Challenge: | SeqPO-SiMT is a new policy optimization framework for simultaneous machine translation that combines a tailored reward with a single step task. |
| Approach: | They propose a new policy optimization framework that defines the simultaneous machine translation task as a sequential decision making problem with a tailored reward. |
| Outcome: | The proposed framework outperforms the supervised fine-tuning model by 1.13 points while reducing the Average Lagging by 6.17 in the NEWSTEST2021 En Zh dataset. |
Can Post-Training Transform LLMs into Causal Reasoners? (2026.findings-acl)
Copied to clipboard
| Challenge: | Causal inference is a core component of human cognition and requires decision-makers to distinguish between causation and association. |
| Approach: | They propose a dataset comprising seven core causal tasks for training and five diverse test sets and evaluate five different post-training approaches. |
| Outcome: | The proposed model achieves 93.5% accuracy on the CaLM benchmark, compared to 55.4% by OpenAI o3. |
CycleAlign: Iterative Distillation from Black-box LLM to White-box Models for Better Human Alignment (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing language models that generate harmful responses are constrained by their inherent capability. |
| Approach: | They propose to align large language models with human preferences from AI feedback. |
| Outcome: | The proposed framework improves the alignment of large language models with human preferences from AI feedback. |
DEFT: Distribution-guided Efficient Fine-Tuning for Human Alignment (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Experimental results show that the methods enhanced by DEFT outperform the original methods in both alignment capability and generalization ability, with significantly reduced training time. |
| Approach: | They propose a distribution-based alignment framework that integrates data filtering and distributional guidance to improve alignment efficiency and generalization ability. |
| Outcome: | The proposed framework outperforms existing methods in alignment capability and generalization ability with significantly reduced training time. |
Med-VRAgent: A Framework for Medical Visual Reasoning-Enhanced Agents (2025.emnlp-main)
Copied to clipboard
| Challenge: | Visual Language Models (VLMs) have shown strong performance in tasks like radiology report generation but struggle with hallucinations, vague descriptions, Inconsistent logic and poor localization. |
| Approach: | They propose a framework for medical visual reasoning based on Visual Guidance and Self-Reward paradigms and Monte Carlo Tree Search to improve the model's visual reasoning capabilities. |
| Outcome: | The proposed framework outperforms existing models on multiple medical VQA benchmarks. |
TRIP NEGOTIATOR: A Travel Persona-aware Reinforced Dialogue Generation Model for Personalized Integrative Negotiation in Tourism (2024.findings-emnlp)
Copied to clipboard
Priyanshu Priya, Desai Yasheshbhai, Ratnesh Joshi, Roshni Ramnani, Anutosh Maitra, Shubhashis Sengupta, Asif Ekbal
| Challenge: | Incorporating traveler preferences, constraints, and expectations allows for customizing negotiation strategies, resulting in a more personalized and integrative experience. |
| Approach: | They propose a novel travel persona-aware Reinforced dIalogue generation model for personalized integrative negotiation in the tourism domain. |
| Outcome: | The proposed system generates coherent and diverse responses consistent with the traveler's personality. |
Don’t Forget Your Reward Values: Language Model Alignment via Value-based Calibration (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for generating large language models have been criticized for their complexity and instability. |
| Approach: | They propose a value-based calibration method to better align Large Language Models with human preferences. |
| Outcome: | The proposed method surpasses existing methods on AI assistant and summarization datasets, providing impressive generalizability, robustness, and diversity in different settings. |
Reinforcement Learning for Large Language Models via Group Preference Reward Shaping (2025.emnlp-main)
Copied to clipboard
Huaisheng Zhu, Siyuan Xu, Hangfan Zhang, Teng Xiao, Zhimeng Guo, Shijie Zhou, Shuyue Hu, Vasant G. Honavar
| Challenge: | Existing methods for fine-tuning Large Language Models (LLMs) are expensive and sensitive to reward model quality. |
| Approach: | They propose a method that leverages preference-based comparisons rather than precise numerical rewards. |
| Outcome: | Experiments show that GPRS outperforms critic-model-free RL algorithms on RLHF and reasoning tasks. |
VRPO: Rethinking Value Modeling for Robust RL under Noisy Supervision in LLM Post-Training (2026.acl-long)
Copied to clipboard
Dingwei Zhu, Shihan Dou, Zhiheng Xi, Senjie Jin, Guoqiang Zhang, Jiazheng Zhang, Junjie Ye, Mingxu Chai, Enyu Zhou, Ming Zhang, Yuhui Wang, Caishuang Huang, Chenhao Huang, Yunke Zhang, Yuran Wang, Tao Gui, Qi Zhang, Xipeng Qiu, Xuanjing Huang
| Challenge: | Reinforcement Learning (RL) in real-world environments often suffers from ambiguous or incomplete supervision. |
| Approach: | They propose a framework that enhances value modeling for robust RL in LLM post-training by integrating auxiliary losses guided by entropy and perplexity from a frozen language model and variational information bottleneck. |
| Outcome: | The proposed framework outperforms baselines on multi-turn dialogue, math reasoning, and science QA with rule-based and model-based rewards. |
PopALM: Popularity-Aligned Language Models for Social Media Trendy Response Prediction (2024.lrec-main)
Copied to clipboard
| Challenge: | Recent work focuses on generic human responses without considering popularity factors in the social contexts. |
| Approach: | They propose Popularity-Aligned Language Models to distinguish responses liked by a larger audience through reinforcement learning. |
| Outcome: | The proposed model can distinguish responses liked by a larger audience through reinforcement learning. |
Think Less, Know More: State-Aware Reasoning Compression with Knowledge Guidance for Efficient Reasoning (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing CoT compression methods struggle to balance accuracy and efficiency . long CoT reasoning also introduces an overthinking phenomenon, authors say . |
| Approach: | They propose a framework that performs step-wise CoT compression by modeling stage-specific redundancy sources and integrating with a retrieval-augmented guidance. |
| Outcome: | The proposed framework reduces average response length by 59.9% while improving accuracy by 4.8 points over existing methods. |
Towards Aligning Language Models with Textual Feedback (2024.emnlp-main)
Copied to clipboard
| Challenge: | Using textual feedback, language models can be trained to learn from textual inputs. |
| Approach: | They propose an approach that aligns language models with user preferences expressed in text. |
| Outcome: | The proposed approach outperforms PPO on toxicity reduction, summarization, and dialog response tasks while achieving the same performance with only 20% of the samples. |
AAPO: Enhancing the Reasoning Capabilities of LLMs with Advantage Margin (2026.acl-long)
Copied to clipboard
| Challenge: | Reinforcement learning (RL) has emerged as an effective approach for enhancing the reasoning capabilities of large language models. |
| Approach: | They propose an algorithm that optimizes cross-entropy loss using advantages enhanced through a margin-based estimation scheme. |
| Outcome: | Experimental results show that AAPO improves group relative advantage estimation compared to other methods. |
Sharing the Cost of Success: A Game for Evaluating and Learning Collaborative Multi-Agent Instruction Giving and Following Policies (2024.lrec-main)
Copied to clipboard
| Challenge: | Recent advances in natural language processing have led to language model-based systems that do a good job at creating natural dialogue behaviour but are often verbose and brittle. |
| Approach: | They propose a game that requires two players to coordinate on vision and language observations. |
| Outcome: | The proposed game achieves high success rates when bootstrapped with heuristic partner behaviors that implement insights from the analysis of human-human interactions. |
Improving the Language Understanding Capabilities of Large Language Models Using Reinforcement Learning (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Instruction-fine-tuned large language models (LLMs) under 14B parameters underperform on NLU tasks . we explore a framework to improve the NLU capabilities of LLMs . |
| Approach: | They propose to use Proximal Policy Optimization to improve NLU capabilities . they frame NLU as a reinforcement learning environment and optimize for reward signals . |
| Outcome: | The proposed framework outperforms supervised fine-tuning on GLUE and superGLUE tasks. |
MDCure: A Scalable Pipeline for Multi-Document Instruction-Following (2025.acl-long)
Copied to clipboard
| Challenge: | Multi-document (MD) processing is crucial for LLMs to handle real-world tasks such as summarization and question-answering across large sets of documents. |
| Approach: | They propose a framework that generates high-quality synthetic MD instruction data over sets of articles via targeted prompts. |
| Outcome: | MDCure generates high-quality synthetic MD instruction data over sets of articles . evaluations show it improves over pre-trained models by up to 75.1% . |
RAG-on-a-Diet: A Reinforcement Learning-Based Dynamic Resource Optimization Framework for RAG (2026.acl-long)
Copied to clipboard
| Challenge: | Existing frameworks for knowledge-intensive multi-hop question answering do not adapt to how a trajectory unfolds. |
| Approach: | They propose a lightweight reinforcement-learning agent that treats each reasoning hop as an independent decision and selects the smallest model sufficient for it. |
| Outcome: | The proposed agent cuts Monetary Inference Cost by 60.07% against IRCoT with only a 3.7% F1 drop and matches Adaptive-RAG’s F1 at 37.30% lower cost. |
NL2Lean: Translating Natural Language into Lean 4 through Multi-Aspect Reinforcement Learning (2025.emnlp-main)
Copied to clipboard
Yue Fang, Shaohan Huang, Xin Yu, Haizhen Huang, Zihan Zhang, Weiwei Deng, Furu Wei, Feng Sun, Qi Zhang, Zhi Jin
| Challenge: | Existing formal proof assistants rely on instruction tuning and lack fine-grained structural and semantic alignment. |
| Approach: | They propose a reinforcement learning framework that enables LLMs to translate natural language into formal language such as Lean 4 . they use a model with basic translation ability to refine the model's reinforcement learning . |
| Outcome: | The proposed method outperforms baseline models on NL-to-Lean 4 tasks. |
ARF-RLHF: Adaptive Reward-Following for RLHF through Emotion-Driven Self-Supervision and Trace-Biased Dynamic Optimization (2026.acl-long)
Copied to clipboard
| Challenge: | prevailing RLHF methods such as PPO and DPO depend on large-scale binary preference annotations. |
| Approach: | They propose a method which converts natural feedback into continuous preference trajectories and optimizes them using the novel TraceBias algorithm. |
| Outcome: | The proposed approach outperforms PPO and DPO in a variety of domains and improves alignment by up to 7.6% across diverse LLMs and preference domains. |
CE-GPPO: Coordinating Entropy via Gradient-Preserving Clipping Policy Optimization in Reinforcement Learning (2026.acl-long)
Copied to clipboard
| Challenge: | Existing methods for proximal policy optimization discard valuable gradient signals from low-probability tokens due to the clipping mechanism. |
| Approach: | They propose an algorithm that reintroduces gradients from clipped tokens in native PPO in a gentle and bounded manner. |
| Outcome: | The proposed algorithm outperforms strong baselines on reasoning benchmarks on different model scales. |
BiasGRPO: Stabilizing Bias Mitigation in High-Variance Reward Landscapes via Group-Relative Policy Optimization (2026.findings-acl)
Copied to clipboard
| Challenge: | Recent preference-based fine-tuning methods have limited exploration in offline training . previous methods have been limited by the lack of exploration inherent in offline learning . |
| Approach: | They propose a method that normalizes rewards across a group of completed tasks to mitigate social bias in Large Language Models. |
| Outcome: | The proposed approach outperforms DPO and PPO in multiple benchmarks . it can overcome limitations of previous preference-based methods . |
Influence-based Online Experience Selection for Effective RLHF (2026.acl-long)
Copied to clipboard
| Challenge: | Existing methods for RL fail to establish an interpretable connection between data and optimization objectives. |
| Approach: | They propose a data selection method that dynamically estimates the influence of individual training samples on policy optimization. |
| Outcome: | The proposed method significantly improves training effectiveness with fewer optimization steps. |