InfoPO: On Mutual Information Maximization for Large Language Model Alignment (2025.naacl-long)
Copied to clipboard
Teng Xiao, Zhen Ge, Sujay Sanghavi, Tian Wang, Julian Katz-Samuels, Marc Versage, Qingjun Cui, Trishul Chilimbi
| Challenge: | Recent studies have shown that direct preference optimization and its variants can be useful for fine-tuning large language models with human preferences data. |
| Approach: | They propose a preference fine-tuning algorithm that effectively and efficiently aligns large language models using preference data. |
| Outcome: | Extensive experiments show that the proposed algorithm outperforms established baselines on reasoning tasks. |
Similar Papers
Insights into Alignment: Evaluating DPO and its Variants Across Multiple Tasks (2025.acl-srw)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) excel in math reasoning problemsolving, text generation, summarization, creative writing, among other tasks. |
| Approach: | They evaluate Direct Preference Optimization and its variants for aligning Large Language Models with human preferences. |
| Outcome: | The proposed alignment methods achieve near-optimal performance even with smaller subsets of training data. |
K-order Ranking Preference Optimization for Large Language Models (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing list-wise methods focus on optimizing list ranking consistency for LLMs to improve ranking abilities. |
| Approach: | They propose to extend the Plackett-Luce model to accommodate top-K ranking by extending the DPO’s Plact-Lucer model to dynamically determine appropriate K for different samples. |
| Outcome: | The proposed model can be extended to accommodate top-K ranking and improve training efficiency. |
Expectation Preference Optimization: Reliable Preference Estimation for Improving the Reasoning Capability of Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Pairwise preference optimization is used to improve supervised fine-tuning performance of large language models. |
| Approach: | They propose an algorithm that takes pairs of sample groups instead of single samples for preference learning. |
| Outcome: | The proposed algorithm outperforms baseline methods on reasoning benchmarks. |
IOPO: Empowering LLMs with Complex Instruction Following via Input-Output Preference Optimization (2025.acl-long)
Copied to clipboard
| Challenge: | Existing algorithms to improve the ability of LLMs to follow complex instructions are lacking. |
| Approach: | They propose a benchmark to improve the ability to follow complex instructions by using a IOPO alignment method to take input and output preference into consideration. |
| Outcome: | The proposed algorithm shows 8.15%, 2.18% improvements on in-domain data and 5.91%, 2.83% on out-of-domain datasets compared to SFT and DPO respectively. |
ADELIE: Aligning Large Language Models on Information Extraction (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) struggle to follow complex instructions of IE tasks due to not being aligned with humans. |
| Approach: | They propose an aligned large language moDEL that effectively solves various IE tasks including closed IE, open IE and on-demand IE. |
| Outcome: | The proposed model achieves state-of-the-art (SoTA) performance among open-source models. |
sDPO: Don’t Use Your Data All at Once (2025.coling-industry)
Copied to clipboard
| Challenge: | Large language models (LLMs) are increasingly requiring precision and accuracy in alignment tuning. |
| Approach: | They propose a stepwise DPO technique that partitions available preference datasets incrementally rather than utilizing entire dataset simultaneously. |
| Outcome: | The proposed technique improves the accuracy of reference models and the overall performance of the final model. |
DPO Kernels: A Semantically-Aware, Kernel-Enhanced, and Divergence-Rich Paradigm for Direct Preference Optimization (2025.findings-acl)
Copied to clipboard
Amitava Das, Suranjana Trivedy, Danush Khanna, Yaswanth Narsupalli, Basab Ghosh, Rajarshi Roy, Gurpreet Singh, Vinija Jain, Vasu Sharma, Aishwarya Naresh Reganti, Aman Chadha
| Challenge: | Direct Preference Optimization (DPO) is a cornerstone for preference alignment but is constrained by fixed divergence measures and limited feature transformations. |
| Approach: | They propose a new enhancement of Direct Preference Optimization that integrates kernel methods to overcome these challenges. |
| Outcome: | The proposed model improves divergence measures and features by using kernels . the proposed model achieves state-of-the-art generalization in factuality, safety, reasoning, and instruction following . |
LIONs: An Empirically Optimized Approach to Align Language Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies have focused on aligning large language models with pre-trained datasets. |
| Approach: | They conduct a rigorous analysis of a three-stage training pipeline using sequence packing, loss masking and increasing the preference dataset size in DPO to improve the performance of language models. |
| Outcome: | The proposed models outperform the official instruct models tuned with closed-source data and algorithms. |
Causal Direct Preference Optimization for Language Model Alignment (2026.findings-eacl)
Copied to clipboard
| Challenge: | Empirical evaluations show that CDPO surpasses DPO-based baselines by achieving unbiased fine-tuning through causal reasoning. |
| Approach: | They propose a framework that incorporates causal inference principles to mitigate the influence of confounders and sharpen the signal of genuine human preferences. |
| Outcome: | The proposed framework preserves the tractability of direct optimization while enhancing robustness to spurious correlations and annotation biases. |
Comparing Bad Apples to Good Oranges Aligning Large Language Models via Joint Preference Optimization (2025.findings-acl)
Copied to clipboard
| Challenge: | Recent studies have shown that acquiring human preferences by comparing generations is not effective for large language models. |
| Approach: | They propose a preference optimization objective that elicits preferences jointly over the instruction-response pairs. |
| Outcome: | The proposed approach outperforms prior preference optimizations by 5.2% and 3.3% in summarization and open-ended dialogue datasets. |