CiPO: Counterfactual Unlearning for Large Reasoning Models through Iterative Preference Optimization (2026.acl-long)
Copied to clipboard
| Challenge: | Existing methods to unlearning large reasoning models do not remove unwanted knowledge from CoT traces or interfere with the reasoning process. |
| Approach: | They propose a framework that targets the CoT reasoning in Large Reasoning Models by generating a valid counterfactual reasoning trace for preference tuning. |
| Outcome: | Experiments on large LRMs show that CiPO completely removes knowledge from the intermediate CoT steps and the final answer while preserving the reasoning abilities of LRM. |
Similar Papers
R-TOFU: Unlearning in Large Reasoning Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Reasoning Models embed private or copyrighted information in their final answers and throughout multi-step chain-of-thought (CoT) traces. |
| Approach: | They propose a benchmark for Large Reasoning Models that augments existing unlearning tasks with realistic CoT annotations and step-wise metrics that expose residual knowledge invisible to answer-level checks. |
| Outcome: | The proposed benchmark shows that answer-only objectives leave substantial forget traces in reasoning. |
Alternate Preference Optimization for Unlearning Factual Knowledge in Large Language Models (2025.coling-main)
Copied to clipboard
Anmol Reddy Mekala, Vineeth Dorna, Shreya Dubey, Abhishek Lalwani, David Koleczek, Mukund Rungta, Sadid A. Hasan, Elita A.A Lobo
| Challenge: | Existing methods for large language models rely on negative feedback to suppress responses related to the forget set, which often results in nonsensical or inconsistent outputs, diminishing model utility and posing potential privacy risks. |
| Approach: | They propose an approach which combines negative feedback with in-domain positive feedback on the forget set and introduces new evaluation metrics to assess the quality of responses related to the forget sets. |
| Outcome: | The proposed approach avoids undesirable model behaviors while maintaining overall model performance. |
Reasoning Model Unlearning: Forgetting Traces, Not Just Answers, While Preserving Reasoning Skills (2025.emnlp-main)
Copied to clipboard
Changsheng Wang, Chongyu Fan, Yihua Zhang, Jinghan Jia, Dennis Wei, Parikshit Ram, Nathalie Baracaldo, Sijia Liu
| Challenge: | Existing methods for LRM unlearning overlook critical information leakage in reasoning traces, even when final answers are successfully removed. |
| Approach: | They propose a method that suppresses reasoning traces while preserving the model's general reasoning ability. |
| Outcome: | The proposed method significantly reduces reasoning trace leakage and achieves strong performance across reasoning and safety benchmarks, including WMDP, StrongReject, JBB-Behaviors and WildJailbreak. |
Efficiently Learning To Reason or Not to Reason: Root-token Policy Optimization for Adaptive Thinking (2026.acl-long)
Copied to clipboard
| Challenge: | Large reasoning models (LRMs) externalize explicit reasoning traces before producing the answer, yet suffer from overthinking challenge. |
| Approach: | They propose a framework that enables large reasoning models to self-determine when to reason by training only the initial root token via group relative reward and group-wise advantages. |
| Outcome: | The proposed framework reduces training overhead and VRAM usage by focusing on the root token . it learns difficulty-aware adaptive thinking at just 2% of the training compute of prior methods. |
Uncertainty-Aware Iterative Preference Optimization for Enhanced LLM Reasoning (2025.acl-long)
Copied to clipboard
| Challenge: | Existing methods for enhancing the performance of large language models require expensive manual annotations. |
| Approach: | They propose an offline direct preference optimization method that collects preference pairs through iterative sampling and execution feedback to improve model confidence. |
| Outcome: | The proposed method improves performance on three reasoning tasks and shows a 3.6% improvement over the standard method. |
Full-Step-DPO: Self-Supervised Preference Optimization with Step-wise Rewards for Mathematical Reasoning (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to improve long-chain mathematical reasoning focus on the first erroneous step, but ignore all other steps and rely heavily on external signals. |
| Approach: | They propose a DPO framework that leverages step-wise rewards from the entire reasoning chain instead of optimizing only the first erroneous step. |
| Outcome: | The proposed framework improves on in-domain and out-of-domain mathematical reasoning benchmarks. |
LEPO: Latent Reasoning Policy Optimization for Large Language Models (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing latent reasoning methods that use chain of thought (CoT) are limited to selecting one discrete token at each reasoning step, which potentially induces information loss. |
| Approach: | They propose a framework that injects controllable stochasticity into latent reasoning via Gumbel-Softmax, restoring LLMs' exploratory capacity and enhancing their compatibility with Reinforcement Learning (RL). |
| Outcome: | The proposed framework preserves richer information for more comprehensive reasoning and is compatible with Reinforcement Learning (RL). |
InfoPO: On Mutual Information Maximization for Large Language Model Alignment (2025.naacl-long)
Copied to clipboard
Teng Xiao, Zhen Ge, Sujay Sanghavi, Tian Wang, Julian Katz-Samuels, Marc Versage, Qingjun Cui, Trishul Chilimbi
| Challenge: | Recent studies have shown that direct preference optimization and its variants can be useful for fine-tuning large language models with human preferences data. |
| Approach: | They propose a preference fine-tuning algorithm that effectively and efficiently aligns large language models using preference data. |
| Outcome: | Extensive experiments show that the proposed algorithm outperforms established baselines on reasoning tasks. |
Correct, Concise and Complete: Multi-stage Training For Adaptive Reasoning (2026.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) increase test-time computation, often in the form of chain-of-thought (CoT) however, reasoning traces can become unnecessarily long, increasing computation costs without improving accuracy and sometimes even degrading performance. |
| Approach: | They propose a multi-stage efficient reasoning method that combines supervised fine-tuning with reinforcement learning using an adaptive length penalty. |
| Outcome: | The proposed method reduces response length by an average of 28% for 8B models and 40% for 32B models while incurring only minor performance drops of 1.6 and 2.5 points, respectively. |
Think Less, Know More: State-Aware Reasoning Compression with Knowledge Guidance for Efficient Reasoning (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing CoT compression methods struggle to balance accuracy and efficiency . long CoT reasoning also introduces an overthinking phenomenon, authors say . |
| Approach: | They propose a framework that performs step-wise CoT compression by modeling stage-specific redundancy sources and integrating with a retrieval-augmented guidance. |
| Outcome: | The proposed framework reduces average response length by 59.9% while improving accuracy by 4.8 points over existing methods. |