Papers with MDP
Intuitive Fine-Tuning: Towards Simplifying Alignment into a Single Process (2025.acl-long)
Copied to clipboard
| Challenge: | Supervised Fine-Tuning (SFT) and Preference Optimization (PO) are key processes for aligning Language Models with human preferences post pre-training. |
| Approach: | They propose to combine Supervised Fine-Tuning and Preference Optimization (PO) with two sub-processes defined at token level within the Markov Decision Process (MDP) |
| Outcome: | The proposed process performs comparably or even superiorly to SFT and some typical PO methods across several tasks, particularly those requires generation, reasoning, and fact-following abilities. |
LLM-Enhanced Self-Evolving Reinforcement Learning for Multi-Step E-Commerce Payment Fraud Risk Detection (2025.acl-industry)
Copied to clipboard
| Challenge: | e-commerce payment fraud detection is a new area for reinforcement learning (RL) and Large Language Models (LLMs). |
| Approach: | They propose to integrate reinforcement learning (RL) with Large Language Models (LLMs) by framing transaction risk as a multi-step Markov Decision Process (MDP), RL optimizes risk detection across multiple payment stages. |
| Outcome: | The proposed approach improves fraud detection accuracy and demonstrates zero-shot capability. |
SPPD: Self-training with Process Preference Learning Using Dynamic Value Margin (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to improve numerical and logical reasoning of Large Language Models are limited . existing approaches rely on prompt engineering and pretrained knowledge to ensure correctness . |
| Approach: | They propose to train LLMs with process-based reasoning using a dynamic value margin . they use the Bellman optimality equation to derive a value margin for step-level preference optimization . |
| Outcome: | The proposed method is equivalent to on-policy policy gradient methods under constrained reward functions. |
DecEx-RAG: Boosting Agentic Retrieval-Augmented Generation with Decision and Execution Optimization via Process Supervision (2025.emnlp-industry)
Copied to clipboard
Yongqi Leng, Yikun Lei, Xikai Liu, Meizhi Zhong, Bojian Xiong, Yurong Zhang, Yan Gao, null Yiwu, Yao Hu, Deyi Xiong
| Challenge: | Recent advances in outcome-supervised reinforcement learning (RL) have shown strong performance, but this approach still suffers from inefficient exploration, sparse reward signals, and ambiguous global reward feedback. |
| Approach: | They propose a model that models RAG as a Markov Decision Process (MDP) and introduces an efficient pruning strategy to optimize data expansion. |
| Outcome: | The proposed model outperforms existing methods and achieves an average performance improvement of 6.2% across six datasets. |
Iterative Document-level Information Extraction via Imitation Learning (2023.eacl-main)
Copied to clipboard
| Challenge: | Documents may feature zero or more instances of a template of any given type, and the task of template extraction entails identifying the templates in a document and extracting each template’s slot values. |
| Approach: | They propose to use iterative extraction to extract complex relations, i.e., N-tuples representing a mapping from named slots to spans of text within a document. |
| Outcome: | The proposed model leads to state-of-the-art results on two established benchmarks and a strong baseline on the new BETTER Granular task. |
Reward Mixology: Crafting Hybrid Signals for Reinforcement Learning Driven In-Context Learning (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for in-context learning (ICL) performance rely on quality and ordering of demonstrations. |
| Approach: | They propose a method that models iterative demonstration selection as a Markov Decision Process and craft hybrid reward signals. |
| Outcome: | The proposed method combines outcome-based accuracy signals with process-oriented signals like stepwise influence and label entropy improvement. |
Reinforcing Compositional Retrieval: Retrieving Step-by-Step for Composing Informative Contexts (2025.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have demonstrated remarkable capabilities across numerous tasks, yet they often rely on external context to handle complex tasks. |
| Approach: | They propose a tri-encoder sequential retriever that models a Markov Decision Process (MDP) this method decomposes the probability of retrieving a set of elements into a sequence of conditional probabilities and allows each retrieval step to be conditioned on previously selected examples. |
| Outcome: | The proposed method outperforms baselines and shows that it can handle multiple pieces of evidence or examples. |
CATCH: A Novel Data Synthesis Framework for High Therapy Fidelity and Memory-Driven Planning Chain of Thought in AI Counseling (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies employ a one-time generation approach to synthesize multi-turn dialogue samples, resulting in low therapy fidelity and failing to capture decision-making rationale behind each response. |
| Approach: | They propose a data synthesis framework that synthesizes multi-turn dialogue samples and incrementally generates stage-aligned counseling dialogues. |
| Outcome: | The proposed framework significantly improves therapy fidelity and logical coherence in AI counseling. |
RLAE: Reinforcement Learning-Assisted Ensemble for LLMs (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing ensemble methods for ensembling large language models rely on fixed weighting strategies that fail to adapt to dynamic, context-dependent characteristics of LLMs. |
| Approach: | They propose a framework that reformulates LLM ensemble through a Markov Decision Process. |
| Outcome: | The proposed framework outperforms existing methods by 3.3% on a diverse set of tasks while achieving lower time latency. |
Q-PRM: Adaptive Query Rewriting for Retrieval-Augmented Generation via Step-level Process Supervision (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to rewriting queries often lack supervision signals for intermediate steps . existing approaches rely on outcome-supervised training or heuristic rules to guide the rewrite process . |
| Approach: | They propose a query rewriting framework that generates process-level supervision signals for intermediate steps. |
| Outcome: | a new query rewriting framework outperforms existing approaches on open-domain QA benchmarks. |
MTSQL-R1: Towards Long-Horizon Multi-Turn Text-to-SQL via Agentic Training (2026.acl-long)
Copied to clipboard
Taicheng Guo, Hai Wang, Chaochun Liu, Mohsen Golalikhani, Xin Chen, Xiangliang Zhang, Chandan K. Reddy
| Challenge: | Existing systems for multi-turn Text-to-SQL are limited to a short-horizon paradigm, generating a query per turn without execution, explicit verification, and refinement, which leads to non-executable or incoherent outputs. |
| Approach: | They propose to train an agentic training framework for long-horizon multi-turn Text-to-SQL that uses a Markov Decision Process to generate a query per turn without execution, explicit verification, and refinement. |
| Outcome: | Experiments on CoSQL and SParC show that MTSQL-R1 consistently outperforms strong baselines, highlighting the importance of environment-driven verification and memory-guided refinement for conversational semantic parsing. |
NaviMaster: Learning a Unified Policy for GUI and Embodied Navigation Tasks (2026.acl-long)
Copied to clipboard
| Challenge: | Recent advances in Graphical User Interface (GUI) and embodied navigation have driven progress, yet these domains have largely evolved in isolation, with disparate datasets and training paradigms. |
| Approach: | They propose a visual-target trajectory collection pipeline that generates trajectories for GUI and embodied tasks using a single formulation. |
| Outcome: | The proposed agent outperforms state-of-the-art agents in GUI navigation, spatial affordance prediction, and embodied navigation. |