MT-BENCH

MPPO: Multi Pair-wise Preference Optimization for LLMs with Arbitrary Negative Samples (2025.coling-main)

Challenge:	a recent study shows that human and Large Language Model preferences are important for model fine-tuning and evaluation.
Approach:	They dissect the preferences of human and 32 different Large Language Models to understand their quantitative composition.
Outcome:	The proposed model is compared with 32 different large language models using real-world user-model conversations.

Copied to clipboard

Shuo Xie, Fangzhi Zhu, Jiahui Wang, Lulu Wen, Wei Dai, Xiaowei Chen, Junxiong Zhu, Kai Zhou, Bo Zheng

Challenge:	Existing preference optimization methods such as DPO and KTO are inherently derived from PPO, requiring a reference model that adds GPU memory resources and relies heavily on abundant preference data.
Approach:	They propose an algorithm that leverages the average likelihood of model responses to fit the reward function and maximizes the utilization of preference data.
Outcome:	The proposed algorithm outperforms DPO, ORPO, and SimPO on MT-Bench and Arena-Hard.

DCRM: A Heuristic to Measure Response Pair Quality in Preference Optimization (2025.findings-emnlp)

Copied to clipboard

Chengyu Huang, Tanya Goyal

Self-Evolution Fine-Tuning for Policy Optimization (2024.findings-emnlp)

Challenge:	Recent research has attempted to associate preference optimization (PO) performance with the underlying preference datasets.
Approach:	They propose a best-of-N2 pairing method that selects response pairs with the highest DCRM to measure the quality of a response pair for PO.
Outcome:	The proposed method produces training datasets that can further improve models’ performance on AlpacaEval, MT-Bench, and Arena-Hard over the existing training sets.

Copied to clipboard

Ruijun Chen, Jiehao Liang, Shiping Gao, Fanqi Wan, Xiaojun Quan

One-Shot Learning as Instruction Data Prospector for Large Language Models (2024.acl-long)

Challenge:	Recent years have showcased the remarkable capabilities and performance of large language models (LLMs) across a broad range of tasks.
Approach:	They propose supervised fine-tuning (SEFT) for LLM alignment to eliminate the need for annotated samples while retaining the stability and efficiency of SFT.
Outcome:	The proposed method eliminates the need for annotated samples while maintaining the stability and efficiency of SFT.

Copied to clipboard

Yunshui Li, Binyuan Hui, Xiaobo Xia, Jiaxi Yang, Min Yang, Lei Zhang, Shuzheng Si, Ling-Hao Chen, Junhao Liu, Tongliang Liu, Fei Huang, Yongbin Li

CoEvol: Constructing Better Responses for Instruction Finetuning through Multi-Agent Cooperation (2024.emnlp-main)

Challenge:	Contemporary practices in instruction tuning often hinge on enlarging data scaling without a clear strategy for ensuring data quality.
Approach:	They propose a method that leverages one-shot learning to discern and select high-quality instruction data from extensive datasets.
Outcome:	Nuggets outperforms existing methods on MT-Bench and Alpaca-Eval benchmarks.

Copied to clipboard

Renhao Li, Minghuan Tan, Derek Wong, Min Yang

Language Models can Evaluate Themselves via Probability Discrepancy (2024.findings-acl)

Challenge:	Recent studies have focused on constructing substantial quantities of IFT data with minimal human effort.
Approach:	They propose a multi-agent cooperation framework for the improvement of IFT responses for large language models using a debate-advise-edit-judge paradigm.
Outcome:	The proposed framework outperforms baseline models on unseen tasks and shows that it can improve instruction-following capabilities on large language models.

Copied to clipboard

Tingyu Xia, Bowen Yu, Yuan Wu, Yi Chang, Chang Zhou

SPO: Self Preference Optimization with Self Regularization (2025.findings-emnlp)

Challenge:	Existing evaluation frameworks focus on superficial text differences and fail to align with human judgment.
Approach:	They propose a new method to evaluate the performance of Large Language Models (LLMs) by calculating probability discrepancies between original response generation and revised versions of LLMs.
Outcome:	The proposed method eliminates the need for training an additional evaluation model or relying on external proprietary models such as GPT-4 as a judger.

Copied to clipboard

Yuhao Sun, Yifan Zhang, Quandong Wang, Qinzhuo Wu, Wei Liu, Jian Luan

SafeDecoding: Defending against Jailbreak Attacks via Safety-Aware Decoding (2024.acl-long)

Challenge:	Existing reference-free preference optimization methods exhibit higher training efficiency but are prone to overoptimization, leading to performance degradation.
Approach:	They propose a reference-free preference optimization method that replaces the logsigmoid loss function with a SiLU function to improve the model's performance.
Outcome:	The proposed method achieves 7% improvement over SimPO on AlpacaEval 2 and MT-Bench.

Copied to clipboard

Zhangchen Xu, Fengqing Jiang, Luyao Niu, Jinyuan Jia, Bill Yuchen Lin, Radha Poovendran

Distilling Instruction-following Abilities of Large Language Models with Task-aware Curriculum Planning (2024.findings-emnlp)

Challenge:	Despite advances in large language models, they face substantial challenges in terms of safety.
Approach:	They develop a safety-aware decoding strategy for large language models to defend against jailbreak attacks.
Outcome:	The proposed strategy outperforms six defense methods against jailbreak attacks on five LLMs.

Copied to clipboard

Yuanhao Yue, Chengyu Wang, Jun Huang, Peng Wang

Debate, Deliberate, Decide (D3): A Cost-Aware Adversarial Framework for Reliable and Interpretable LLM Evaluation (2026.eacl-long)

Challenge:	Instruction tuning aims to align large language models (LLMs) with open-domain instructions and human-preferred responses.
Approach:	They propose a multi-round distillation framework that uses an oracle LLM to select instructions that are difficult for a student LLM.
Outcome:	The proposed framework outperforms large language models and user-tuned models on several widely recognized benchmarks and multiple student LLMs.

Copied to clipboard

Abir Harrasse, Chaithanya Bandi, Hari Bandi

Automatic Instruction Evolving for Large Language Models (2024.emnlp-main)

Challenge:	Existing evaluation tools for Large Language Models (LLMs) are inconsistency, bias, and lack of transparent decision criteria.
Approach:	They propose a cost-aware, adversarial multi-agent framework that orchestrates structured debate among role-specialized agents to produce reliable and interpretable evaluations.
Outcome:	The proposed framework orchestrates structured debate among role-specialized agents to produce reliable and interpretable evaluations.

Copied to clipboard

Weihao Zeng, Can Xu, Yingxiu Zhao, Jian-Guang Lou, Weizhu Chen

Large Language Models are Superpositions of All Characters: Attaining Arbitrary Role-play via Self-Alignment (2024.acl-long)

Challenge:	Evol-Instruct is an end-to-end framework that evolves instruction datasets without human effort.
Approach:	They propose an end-to-end framework that evolves instruction datasets without human effort by analyzing and analyzing evolutionary strategies for the given instruction data.
Outcome:	The proposed method outperforms human-designed methods on various benchmarks including MT-Bench, AlpacaEval, GSM8K, and HumanEval.

Copied to clipboard

Keming Lu, Bowen Yu, Chang Zhou, Jingren Zhou

PlatoLM: Teaching LLMs in Multi-Round Dialogue via a User Simulator (2024.acl-long)

Challenge:	Existing work cheaply emulates LLMs, allowing users to create profiles for their preferred characters.
Approach:	They propose a self-alignment method that encourages an instruction-following LLM to simulate role-play dialogues as a variant of reading comprehension.
Outcome:	The proposed model outperforms open-source role-play benchmarks and the roleplay subset of MT-Bench in multiple parameters.

Copied to clipboard

Chuyi Kong, Yaxin Fan, Xiang Wan, Feng Jiang, Benyou Wang

Natural Language Interfaces

Challenge:	Recent efforts to democratize ChatGPT have focused on leveraging real user and ChatGPP dialogues, but the most direct human needs are often ignored.
Approach:	They propose a method to simulate human behavior better by using real human-like questions extracted from real human conversations as a learning goal and a user simulator called ‘Socratic’.
Outcome:	The proposed model achieves SoTA performance among LLaMA-based 7B models in MT-Bench.

Amphista: Bi-directional Multi-head Decoding for Accelerating LLM Inference (2025.naacl-long)

Copied to clipboard

Zeping Li, Xinlong Yang, Ziheng Gao, Ji Liu, Guanchen Li, Zhuang Liu, Dong Li, Jinzhang Peng, Lu Tian, Emad Barsoum

Ambiguity Awareness Optimization: Towards Semantic Disambiguation for Direct Preference Optimization (2025.emnlp-main)

Challenge:	Existing methods such as Medusa lack adequate information interaction between different drafting heads.
Approach:	They propose an enhanced speculative decoding framework that builds upon Medusa and integrates a drafting block capable of parallel inference.
Outcome:	The proposed framework outperforms Medusa in terms of head accuracy and latency.

Copied to clipboard

Jian Li, Shenglin Yin, Yujia Zhang, Alan Zhao, Xi Chen, Xiaohui Zhou, Pengfei Xu

Hierarchical Speculative Decoding with Dynamic Window (2025.findings-naacl)

Challenge:	Direct Preference Optimization (DPO) is a widely used reinforcement learning from human feedback (RLHF) method across various domains.
Approach:	They propose an approach that automatically re-weights ambiguous content to reduce ambiguities by calculating semantic similarity from preference pairs.
Outcome:	The proposed approach outperforms state-of-the-art approaches in performance across multiple model scales and widely adopted benchmark datasets.

Copied to clipboard

Shensian Syu, Hung-yi Lee

Challenge:	Speculative decoding (SD) uses an efficient draft model to generate multiple tokens . previous methods depend on simple heuristics to select K or dynamically adjust the window size .
Approach:	They propose a framework that allows a draft model to generate multiple tokens . they propose HSDDW, which allows the draft model autonomously decide when to stop generating tokens.
Outcome:	The proposed framework outperforms existing state-of-the-art methods on four datasets.

KNN-Instruct: Automatic Instruction Construction with K Nearest Neighbor Deduction (2024.emnlp-main)

Copied to clipboard

Jianshang Kou, Benfeng Xu, Chiwei Zhu, Zhendong Mao

Reasoning

Challenge:	Existing methods for generating synthetic instructions for large language models suffer from stale distribution and scalability.
Approach:	They propose a method which incorporates KNN deduction to produce meaningful new instructions by summarizing and learning from existing ones.
Outcome:	The proposed method outperforms all 7B models on the LMSYS leaderboard.

Reward-Guided Tree Search for Inference Time Alignment of Large Language Models (2025.naacl-long)

Copied to clipboard

Chia-Yu Hung, Navonil Majumder, Ambuj Mehrish, Soujanya Poria

ORPO: Monolithic Preference Optimization without Reference Model (2024.emnlp-main)

Challenge:	Inference-time computation methods enhance performance of Large Language Models by leveraging additional computational resources.
Approach:	They propose an inference-time alignment method that leverages a reward model to achieve alignment through reward-guided tree search.
Outcome:	The proposed method outperforms other inference-time alignment methods on two benchmarks . it achieves comparable performance to preference-tuned models on both benchmarks, authors show .

Copied to clipboard

Jiwoo Hong, Noah Lee, James Thorne

“Vorbești Românește?” A Recipe to Train Powerful Romanian LLMs with English Instructions (2024.findings-emnlp)

Challenge:	Pre-trained language models with vast training corpora have shown remarkable abilities in diverse natural language processing tasks.
Approach:	They propose a model-free monolithic odds ratio preference optimization algorithm, ORPO, to improve preference alignment.
Outcome:	The proposed algorithm outperforms state-of-the-art language models with more than 7B and 13B parameters on the ultrafeedback alone.

Copied to clipboard

Mihai Masala, Denis Ilie-Ablachim, Alexandru Dima, Dragos Georgian Corlatescu, Miruna-Andreea Zavelca, Ovio Olaru, Simina-Maria Terian, Andrei Terian, Marius Leordeanu, Horia Velicu, Marius Popescu, Mihai Dascalu, Traian Rebedea

Multilinguality

Challenge:	Large Language Models (LLMs) have achieved almost human-like performance on various tasks.
Approach:	They are the first to collect and translate a large collection of texts, instructions, and benchmarks and train, evaluate and release open-source LLMs tailored for Romanian.
Outcome:	The proposed model trains, evaluates and releases open-source models tailored for Romanian.

Multi-modal Preference Alignment Remedies Degradation of Visual Instruction Tuning on Language Models (2024.acl-long)

Copied to clipboard

Shengzhi Li, Rongyu Lin, Shichao Pei

What Do LLMs Learn First? Asymmetric Learning Dynamics of Input Complexity and Output Ambiguity in Preference Alignment (2026.acl-long)

Challenge:	Multi-modal large language models (MLLMs) are expected to support multi-turn queries of interchanging image and text modalities in production.
Approach:	They propose to use visual-question-answering (VQA) datasets to annotate a 5k-sample VQA preference dataset and to investigate the degradation of VQA datasets.
Outcome:	The proposed model surpasses the instruction-following capabilities of the language model with DPO and SteerLM.

Copied to clipboard

Mengyang Li, Jingwen Wang, Pinlong Zhao

Challenge:	Existing methods treat all preference pairs uniformly during training.
Approach:	They propose a training framework that maintains separate, adaptive pacing schedules for each dimension.
Outcome:	The proposed training framework outperforms curriculum baselines by 2.1% and 0.21 points . it achieves 42.3% length-controlled win rate on AlpacaEval 2.0 and 7.66 on MT-Bench .

Capturing Nuanced Preferences: Preference-Aligned Distillation for Small Language Models (2025.findings-acl)

Copied to clipboard

Yanggan Gu, Junzhuo Li, Sirui Huang, Xin Zou, Zhenghua Li, Xuming Hu

Draft Model Knows When to Stop: Self-Verification Speculative Decoding for Long-Form Generation (2025.emnlp-main)

Challenge:	Existing methods for aligning small language models with human values model preference knowledge from large language models (LLMs) however, this limitation hinders student SLMs from capturing nuanced preferences for multiple responses.
Approach:	They propose a framework which models teacher's preference knowledge as a probability distribution over all potential preferences, thereby providing more nuanced supervisory signals.
Outcome:	The proposed framework outperforms existing methods on four benchmark tasks and achieves 20% improvement on AlpacaEval 2 and Arena-Hard.

Copied to clipboard

Ziyin Zhang, Jiahao Xu, Tian Liang, Xingyu Chen, Zhiwei He, Rui Wang, Zhaopeng Tu

Reasoning

Challenge:	Conventional speculative decoding methods use a predefined length policy for proposing drafts, but the reality deviates from this assumption.
Approach:	They propose a self-verification length policy that adaptively determines the lengths of draft sequences by referring to the draft entropy.
Outcome:	The proposed method achieves 17% speedup on MT-Bench and 22% speedup in long-form reasoning.

Review-Instruct: A Review-Driven Multi-Turn Conversations Generation Method for Large Language Models (2025.findings-acl)

Copied to clipboard

Jiangxu Wu, Cong Wang, Tianhuang Su, Lin Haozhi, JunYang JunYang, Zhangchao Zhangchao, Binqiang Pan, SongpanYang SongpanYang, Mingpeng Mingpeng, Kai Shi, Zixian Li

Dialogue Systems & Conversational Agents

Challenge:	Existing methods for generating multi-turn dialogue data struggle to ensure both diversity and quality in instructions.
Approach:	They propose a framework that synthesizes multi-turn conversations through an iterative "Ask-Respond-Review" process involving three agent roles: a Candidate, multiple Reviewers, and a Chairman.
Outcome:	The proposed framework synthesizes multi-turn conversations through an iterative "Ask-Respond-Review" process involving three agent roles: a Candidate, multiple Reviewers, and a Chairman.

FFN-SkipLLM: A Hidden Gem for Autoregressive Decoding with Adaptive Feed Forward Skipping (2024.emnlp-main)

Copied to clipboard

Ajay Jaiswal, Bodun Hu, Lu Yin, Yeonju Ro, Tianlong Chen, Shiwei Liu, Aditya Akella

Challenge:	Autoregressive Large Language Models (LLMs) are omnipresent but typically come with a substantial model size.
Approach:	They propose a novel fine-grained skip strategy for autoregressive large language models . they observe the saturation of computationally expensive feed-forward blocks of LLMs .
Outcome:	The proposed method can skip 25-30% of FFN blocks with marginal change in performance on knowledge-intensive generation tasks.

Outlier-weighed Layerwise Sampling for LLM Fine-tuning (2025.findings-acl)

Copied to clipboard

Pengxiang Li, Lu Yin, Xiaowei Gao, Shiwei Liu

Debiasing Reward Models via Causally Motivated Inference-Time Intervention (2026.acl-long)

Challenge:	Large Language Models (LLMs) are a powerful tool for processing complex natural language processing tasks.
Approach:	They propose an approach to fine-tune LLMs with outliers and a gradient low-rank projection to increase the number of fine-sampled layers without a proportional increase in memory costs.
Outcome:	The proposed approach outperforms baseline approaches while being more memory efficient.

Copied to clipboard

Kazutoshi Shinoda, Kosuke Nishida, Kyosuke Nishida

Robustness in NLP

Challenge:	Existing approaches for mitigating spurious features in RMs focus on response length . Existing methods focus on RM activation, resulting in performance trade-offs .
Approach:	They propose a method that uses neurons to suppress spurious features in RMs at inference time.
Outcome:	The proposed method reduces sensitivity to spurious features without inducing performance trade-offs on RM benchmarks.

FuseChat: Knowledge Fusion of Chat Models (2025.emnlp-main)

Copied to clipboard

Fanqi Wan, Longguang Zhong, Ziyi Yang, Ruijun Chen, Xiaojun Quan

Adversarial Preference Learning for Robust LLM Alignment (2025.findings-acl)

Challenge:	Large language models (LLMs) are costly and require significant computational resources and time.
Approach:	They propose a fuse-and-merge framework for the knowledge fusion of chat LLMs . they conduct pairwise knowledge fusing on source chat LRMs to create multiple target LLM .
Outcome:	The proposed framework is superior to baselines of various sizes.

Copied to clipboard

Yuanfu Wang, Pengyu Wang, Chenyang Xi, Bo Tang, Junyi Zhu, Wenqiang Wei, Chen Chen, Chao Yang, Jingfeng Zhang, Chaochao Lu, Yijun Niu, Keming Mao, Zhiyu Li, Feiyu Xiong, Jie Hu, Mingchuan Yang

C2PO: Diagnosing and Disentangling Bias Shortcuts in LLMs (2026.findings-acl)

Challenge:	Modern language models rely on Reinforcement Learning from Human Feedback (RLHF) to encourage safe behaviors, but they remain vulnerable to adversarial attacks due to three key limitations: (1) the inefficiency and high cost of human annotation; (2) the vast diversity of potential adversarials; and (3) the risk of feedback bias and reward hacking.
Approach:	They propose an iterative adversarial training method that incorporates three key innovations to address these challenges.
Outcome:	Experiments on Mistral-7B-Instruct-v0.3 show that the proposed method significantly enhances robustness and reduces harmful outputs from 5.88% to 0.43%.

Copied to clipboard

Xuan Feng, Bo An, Tianlong Gu, Liang Chang, Fengrui Hao, Peipeng Yu, Shuai Zhao

Weights-Rotated Preference Optimization for Large Language Models (2025.emnlp-main)

Challenge:	Existing approaches to solve large language models address stereotypical and structural biases in isolation . however, prior paradigms address these in isolation, often at the expense of exacerbating the other .
Approach:	They propose a framework to tackle latent spurious feature correlations within input that drive erroneous reasoning shortcuts.
Outcome:	The proposed framework mitigates stereotypical and structural biases while preserving robust general reasoning capabilities.

Copied to clipboard

Chenxu Yang, Ruipeng Jia, Mingyu Zheng, Naibin Gu, Zheng Lin, Siyuan Chen, Weichong Yin, Hua Wu, Weiping Wang

CopySpec: Accelerating LLMs with Speculative Copy-and-Paste (2025.emnlp-main)

Challenge:	Existing methods to align large language models with high reward hacking are limited by the complexity of the parameter space and the complexity.
Approach:	They propose a weights-rotated preference optimization algorithm that constrains the output layer logits with the KL divergence inherited from DPO and fine-tunes the intermediate hidden states.
Outcome:	The proposed algorithm achieves a 3.27-point improvement on AlpacaEval 2 and surpasses the best baseline by 6.2 to 7.5 points on MT-Bench with merely 0.015% of the trainable parameters.

Copied to clipboard

Razvan-Gabriel Dumitru, Minglai Yang, Vikas Yadav, Mihai Surdeanu

GeLoRA: Geometric Adaptive Ranks For Efficient LoRA Fine-tuning (2025.findings-emnlp)

Challenge:	Large Language Models (LLMs) suffer from slower inference as context size grows, but CopySpec leverages larger contexts to accelerate inference.
Approach:	They propose a technique that speculates that the same tokens will follow repeated sequences in the model’s chat history or context and enables seamless copying without compromising output quality.
Outcome:	The proposed technique can generate responses that closely resemble previous outputs or responses that can be verbatim extracted from context without compromising output quality and without requiring additional GPU memory.

Copied to clipboard

Abdessalam Ed-dib, Zhanibek Datbayev, Amine M. Aboussalah