LLaMA-Berry: Pairwise Optimization for Olympiad-level Mathematical Reasoning via O1-like Monte Carlo Tree Search (2025.naacl-long)
Copied to clipboard
Di Zhang, Jianbo Wu, Jingdi Lei, Tong Che, Jiatong Li, Tong Xie, Xiaoshui Huang, Shufei Zhang, Marco Pavone, Yuqiang Li, Wanli Ouyang, Dongzhan Zhou
| Challenge: | LLaMA-Berry is an advanced mathematical reasoning framework to enhance the problem-solving ability of large language models (LLMs). |
| Approach: | They propose a Monte Carlo Tree Search and Self-Refine framework to optimize reasoning paths and a pairwise reward model to evaluate different paths globally. |
| Outcome: | The proposed framework overcomes inefficiencies and limitations of step-wise and greedy search algorithms, enabling more efficient exploration of solution spaces. |
Similar Papers
Reinforced Efficient Reasoning via Semantically Diverse Exploration (2026.acl-long)
Copied to clipboard
Ziqi Zhao, Zhaochun Ren, Jiahong Zou, Liu Yang, Zhiwei Xu, Xuri Ge, Zhumin Chen, Xinyu Ma, Daiting Shi, Shuaiqiang Wang, Dawei Yin, Xin Xin
| Challenge: | Existing methods for reinforcement learning with verifiable rewards suffer from limited exploration diversity and inefficient reasoning. |
| Approach: | They propose a method that rewards concise and correct reasoning while penalizing unnecessarily long reasoning chains. |
| Outcome: | Extensive experiments on Qwen and Llama models validate the effectiveness and efficiency of ROSE. |
Boosting Policy and Process Reward Models with Monte Carlo Tree Search in Open-Domain QA (2025.findings-acl)
Copied to clipboard
Chi-Min Chan, Chunpu Xu, Junqi Zhu, Jiaming Ji, Donghai Hong, Pengcheng Wen, Chunyang Jiang, Zhen Ye, Yaodong Yang, Wei Xue, Sirui Han, Yike Guo
| Challenge: | Experimental results show that our approach can effectively improve the performance of both the policy model and the reward model. |
| Approach: | They propose to use Monte Carlo Tree Search for both policy model improvement and reward model improvement to bridge it to more subtle open-domain question answering. |
| Outcome: | The proposed approach surpasses existing methods for annotation and training data with fewer data points and achieves better performance in test-time scaling strategies. |
MPPO: Multi Pair-wise Preference Optimization for LLMs with Arbitrary Negative Samples (2025.coling-main)
Copied to clipboard
Shuo Xie, Fangzhi Zhu, Jiahui Wang, Lulu Wen, Wei Dai, Xiaowei Chen, Junxiong Zhu, Kai Zhou, Bo Zheng
| Challenge: | Existing preference optimization methods such as DPO and KTO are inherently derived from PPO, requiring a reference model that adds GPU memory resources and relies heavily on abundant preference data. |
| Approach: | They propose an algorithm that leverages the average likelihood of model responses to fit the reward function and maximizes the utilization of preference data. |
| Outcome: | The proposed algorithm outperforms DPO, ORPO, and SimPO on MT-Bench and Arena-Hard. |
I-MCTS: Enhancing Agentic AutoML via Introspective Monte Carlo Tree Search (2026.findings-eacl)
Copied to clipboard
| Challenge: | Existing LLM-based agents struggle with low diversity and suboptimal code generation. |
| Approach: | They propose an approach that iteratively expands tree nodes through an introspective process that meticulously analyzes solutions and results from parent and sibling nodes. |
| Outcome: | The proposed approach shows a 4% improvement in performance compared to the strong open-source AutoML agents. |
Self-Explore: Enhancing Mathematical Reasoning in Language Models with Fine-grained Rewards (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Recent studies have shown that large language models can solve complex reasoning tasks with Chain-of-Thought Prompting. |
| Approach: | They propose a training method where the LLM is tasked to explore the first wrong step within the rationale and use such signals as fine-grained rewards for further improvement. |
| Outcome: | The proposed model improves on the GSM8K and MATH test sets by 11.57% and 2.89% on average compared to supervised fine-tuning (SFT). |
Probability-Consistent Preference Optimization for Enhanced LLM Reasoning (2025.findings-acl)
Copied to clipboard
Yunqiao Yang, Houxing Ren, Zimu Lu, Ke Wang, Weikang Shi, Aojun Zhou, Junting Pan, Mingjie Zhan, Hongsheng Li
| Challenge: | Recent advances in preference optimization have demonstrated significant potential for improving mathematical reasoning capabilities in large language models. |
| Approach: | They propose a framework that establishes two quantitative metrics for preference selection: surface-level answer correctness and intrinsic token-level probability consistency. |
| Outcome: | The proposed framework outperforms existing outcome-only criterion approaches across a diverse range of LLMs and benchmarks. |
DSG-MCTS: A Dynamic Strategy-Guided Monte Carlo Tree Search for Diversified Reasoning in Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) have shown strong potential in complex reasoning tasks, but their performance often degrades, resulting in hallucinations, errors, and logical inconsistencies. |
| Approach: | They propose a framework that integrates multiple reasoning strategies to expand the reasoning space and a dynamic strategy selection mechanism that adapts to the task context. |
| Outcome: | The proposed framework outperforms existing state-of-the-art methods on a set of reasoning benchmarks. |
Reasoning in a Combinatorial and Constrained World: Benchmarking LLMs on Natural-Language Combinatorial Optimization (2026.findings-acl)
Copied to clipboard
| Challenge: | NLCO evaluates large language models for combinatorial optimization (CO) . existing evaluations emphasize relatively simple reasoning competencies . |
| Approach: | They propose a combinatorial optimization benchmark that evaluates large language models on CO reasoning. |
| Outcome: | The proposed model can handle combinatorial optimization without writing code or calling external solvers. |
RASPberry: Retrieval-Augmented Monte Carlo Tree Self-Play with Reasoning Consistency for Multi-Hop Question Answering (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for generating and analyzing multiple document knowledge are not effective for multi-hop question answering. |
| Approach: | They propose a Monte Carlo tree-based approach to inference-time scaling using RASPberry. |
| Outcome: | Experimental results show that the proposed method achieves better inference-time scaling on smaller LLMs. |
MASTER: A Multi-Agent System with LLM Specialized MCTS (2025.naacl-long)
Copied to clipboard
Bingzheng Gan, Yufan Zhao, Tianyi Zhang, Jing Huang, Li Yusu, Shu Xian Teo, Changwang Zhang, Wei Shi
| Challenge: | Large Language Models (LLMs) are increasingly being explored for problem-solving tasks . their strategic planning capability is often viewed with skepticism due to their limited planning capabilities. |
| Approach: | They propose a framework that coordinates agent recruitment and communication through LLM specialized MCTS. |
| Outcome: | The proposed framework achieves 76% accuracy on HotpotQA and 80% on WebShop . it relies on extensive sampling simulations to approximate the true reward distribution . |