Papers with ORPO
Rethinking the Evaluation of Alignment Methods: Insights into Diversity, Generalisation, and Safety (2026.eacl-srw)
Copied to clipboard
Denis Janiak, Julia Moska, Dawid Motyka, Karolina Seweryn, Paweł Walkowiak, Bartosz Żuk, Arkadiusz Janz
| Challenge: | Existing studies focus on individual techniques or specific dimensions, lacking a holistic assessment of the inherent trade-offs. |
| Approach: | They propose a framework that compares LLM alignment methods across five axes . they use a validated LLM-as-judge prompt to compare the results . |
| Outcome: | The proposed framework compares LLM alignment methods across factuality, safety, conciseness, proactivity, diversity and safety axes . it provides insights into trade-offs of common alignment methods, guiding the development of more balanced and reliable LLMs. |
Controlling Language Confusion in Multilingual LLMs (2025.acl-srw)
Copied to clipboard
| Challenge: | Large language models suffer from language confusion, a phenomenon in which responses are partially or entirely generated in unintended languages. |
| Approach: | They propose a supervised fine-tuning methodology which optimizes the likelihood of correct tokens without explicitly penalizing undesired outputs such as cross-lingual mixing. |
| Outcome: | The proposed model suppresses language-confused generation while maintaining strong language consistency even under high decoding temperatures while preserving general QA performance. |
MPPO: Multi Pair-wise Preference Optimization for LLMs with Arbitrary Negative Samples (2025.coling-main)
Copied to clipboard
Shuo Xie, Fangzhi Zhu, Jiahui Wang, Lulu Wen, Wei Dai, Xiaowei Chen, Junxiong Zhu, Kai Zhou, Bo Zheng
| Challenge: | Existing preference optimization methods such as DPO and KTO are inherently derived from PPO, requiring a reference model that adds GPU memory resources and relies heavily on abundant preference data. |
| Approach: | They propose an algorithm that leverages the average likelihood of model responses to fit the reward function and maximizes the utilization of preference data. |
| Outcome: | The proposed algorithm outperforms DPO, ORPO, and SimPO on MT-Bench and Arena-Hard. |
The Art of (Mis)alignment: How Fine-Tuning Methods Effectively Misalign and Realign LLMs in Post-Training (2026.findings-acl)
Copied to clipboard
Rui Zhang, Hongwei Li, Yun Shen, Xinyue Shen, Wenbo Jiang, Guowen Xu, Yang Liu, Michael Backes, Yang Zhang
| Challenge: | Misaligned large language models can magnify harm by exploiting them to undermine safety . et al., 2022b; Bai e.t., 2023): misalignment, realignment and model-specific resistance are important . |
| Approach: | They evaluate four methods to identify a mechanism asymmetry between attack and defense . they find that ORPO is most effective for misalignment, but DPO excels in realignment . |
| Outcome: | The proposed methods show a mechanism asymmetry between attack and defense . the proposed methods excel in realignment, but at the expense of model utility . |
Learning to Reason via Self-Iterative Process Feedback for Small Language Models (2025.coling-main)
Copied to clipboard
| Challenge: | Existing methods for enhancing SLMs’ reasoning depend on costly external signals, resulting in SLM overly confident with limited supervision signals. |
| Approach: | They propose to fine-tune and align SLMs using positive and negative feedback signals and introduce process supervision for rewards in preference alignment by sampling-based inference simulation and process reward models. |
| Outcome: | The proposed method improves Gemma-2B's performance on GSM8K and MBPP, and out-of-domain generalization capabilities on MMLU_Math and HumanEval. |
Aligning to What? Limits to RLHF Based Alignment (2025.findings-naacl)
Copied to clipboard
| Challenge: | Existing studies on RLHF and covert and overt biases in large language models are unclear . et al. analyzed off-the-shelf language models to evaluate their overt and cover racial biase . |
| Approach: | They evaluate the relationship between reinforcement learning from human feedback and biases in large language models. |
| Outcome: | The proposed approach can be used to mitigat covert biases, the authors show . they found that the RLHF approach calcifies model biase . |
ORPO: Monolithic Preference Optimization without Reference Model (2024.emnlp-main)
Copied to clipboard
| Challenge: | Pre-trained language models with vast training corpora have shown remarkable abilities in diverse natural language processing tasks. |
| Approach: | They propose a model-free monolithic odds ratio preference optimization algorithm, ORPO, to improve preference alignment. |
| Outcome: | The proposed algorithm outperforms state-of-the-art language models with more than 7B and 13B parameters on the ultrafeedback alone. |
HalluGuard: Evidence-Grounded Small Reasoning Models to Mitigate Hallucinations in Retrieval-Augmented Generation (2026.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models excel at NLP tasks but remain prone to hallucinations . small language models can achieve competitive results in specific tasks . |
| Approach: | They propose a 4B-parameter Small Reasoning Model (SRM) that can be used to classify document-claim pairs as grounded or hallucinated in closed-book, document-grounded settings. |
| Outcome: | The proposed model achieves 84.4% balanced accuracy on the RAGTruth subset of the LLM-AggreFact benchmark, surpassing specialized models, MiniCheck (7B; 84.0%) and Granite Guardian 3.3 (82.2%) Across the benchmark, it reaches 77.1% BAcc, surpasses larger general-purpose LLMs such as GPT-4o (75.9%). |
TLPO: Token-Level Policy Optimization for Mitigating Language Confusion in Large Language Models (2026.acl-long)
Copied to clipboard
| Challenge: | Prior mitigation approaches that optimize entire responses operate at the level of entire responses and can lead to unintended degradation of general model capabilities. |
| Approach: | They propose a fine-tuning framework to mitigate erroneous outputs by localizing and updating the policy at a granular level. |
| Outcome: | The proposed framework outperforms baselines on multiple multilingual LLMs across diverse languages while preserving task accuracy. |