Evolutionary Contrastive Distillation for Language Model Alignment (2024.findings-emnlp)
Copied to clipboard
Julian Katz-Samuels, Zheng Li, Hyokun Yun, Priyanka Nigam, Yi Xu, Vaclav Petricek, Bing Yin, Trishul Chilimbi
| Challenge: | Existing studies indicate that large language models struggle with challenging instructions. |
| Approach: | They propose a method for generating high-quality synthetic preference data to enhance the complex instruction-following capability of language models. |
| Outcome: | The proposed method exceeds the performance of current SOTA 7B models and is competitive even with open-source 70B models. |
Similar Papers
CodecLM: Aligning Language Models with Tailored Synthetic Data (2024.findings-naacl)
Copied to clipboard
Zifeng Wang, Chun-Liang Li, Vincent Perot, Long Le, Jin Miao, Zizhao Zhang, Chen-Yu Lee, Tomas Pfister
| Challenge: | Recent work on generating diverse instructions and applying LLM to increase instruction complexity neglects downstream use cases. |
| Approach: | They propose a framework for generating high-quality synthetic data for LLM alignment with different downstream instruction distributions and LLMs. |
| Outcome: | Experiments on four open-domain instruction using the proposed framework validate the effectiveness of CodecLM over the current state-of-the-art. |
Better Alignment with Instruction Back-and-Forth Translation (2024.findings-emnlp)
Copied to clipboard
| Challenge: | et al., 2023) proposes a method to improve instruction-tuning data . e.g., we generate synthetic instructions using the backtranslation approach . |
| Approach: | They propose a method to improve instruction-tuning data using web-based inputs . they generate synthetic instructions using the backtranslation approach and filter the generated data . |
| Outcome: | The proposed method improves the quality of instruction-tuning data based on preprocessed texts . it yields better AlpacaEval win rates than direct distillation . |
A Systematic Examination of Preference Learning through the Lens of Instruction-Following (2025.naacl-long)
Copied to clipboard
Joongwon Kim, Anirudh Goyal, Aston Zhang, Bo Xiong, Rui Hou, Melanie Kambadur, Dhruv Mahajan, Hannaneh Hajishirzi, Liang Tan
| Challenge: | a recent study has found that preference learning is a key tool for enhancing LLM training and alignment. |
| Approach: | They use a synthetic data generation pipeline to generate 48,000 unique instruction-following prompts with 23 verifiable constraints to obtain preference pairs. |
| Outcome: | The proposed pipeline generates 48,000 unique instruction-following prompts with 23 verifiable constraints that enable fine-grained and automated quality assessments of model responses. |
MuSC: Improving Complex Instruction Following with Multi-granularity Self-Contrastive Training (2025.acl-long)
Copied to clipboard
| Challenge: | Existing methods for complex instruction-following with elaborate constraints rely on a weaker model, especially GPT-4, limiting their application. |
| Approach: | They propose a Multi-granularity Self-Contrastive Training framework to improve instruction alignment without relying on a stronger model. |
| Outcome: | The proposed framework improves instruction-following with elaborate constraints without external supervision on coarse and fine granularity. |
IOPO: Empowering LLMs with Complex Instruction Following via Input-Output Preference Optimization (2025.acl-long)
Copied to clipboard
| Challenge: | Existing algorithms to improve the ability of LLMs to follow complex instructions are lacking. |
| Approach: | They propose a benchmark to improve the ability to follow complex instructions by using a IOPO alignment method to take input and output preference into consideration. |
| Outcome: | The proposed algorithm shows 8.15%, 2.18% improvements on in-domain data and 5.91%, 2.83% on out-of-domain datasets compared to SFT and DPO respectively. |
Comparing Bad Apples to Good Oranges Aligning Large Language Models via Joint Preference Optimization (2025.findings-acl)
Copied to clipboard
| Challenge: | Recent studies have shown that acquiring human preferences by comparing generations is not effective for large language models. |
| Approach: | They propose a preference optimization objective that elicits preferences jointly over the instruction-response pairs. |
| Outcome: | The proposed approach outperforms prior preference optimizations by 5.2% and 3.3% in summarization and open-ended dialogue datasets. |
Direct Large Language Model Alignment Through Self-Rewarding Contrastive Prompt Distillation (2024.acl-long)
Copied to clipboard
| Challenge: | Existing methods to evaluate preference data without human annotations are difficult . et al., 2022b) is effective for aligning large language models with human expectations . |
| Approach: | They propose a method to evaluate the response preference using output probabilities under contrastive prompts. |
| Outcome: | The proposed method could surpass the RLHF method without human-annotated preference data. |
AMR-Evol: Adaptive Modular Response Evolution Elicits Better Knowledge Distillation for Large Language Models in Code Generation (2024.emnlp-main)
Copied to clipboard
| Challenge: | proprietary large language models (LLMs) have demonstrated impressive code generation performance. |
| Approach: | They propose an adaptive module-based model that refines the direct response distillation process by modular decomposition and adaptive response evolution. |
| Outcome: | The proposed framework outperforms baseline model and code generation methods on three popular benchmarks. |
PopAlign: Diversifying Contrasting Patterns for a More Comprehensive Alignment (2025.acl-long)
Copied to clipboard
Zekun Moore Wang, Shenzhi Wang, King Zhu, Jiaheng Liu, Ke Xu, Jie Fu, Wangchunshu Zhou, Wenhao Huang
| Challenge: | Typical approaches to training large language models rely on limited contrasting patterns . contrasting data is limited and models are susceptible to harmful response tendencies . |
| Approach: | They propose a framework that integrates contrasting patterns across the prompt, model, and pipeline levels. |
| Outcome: | The proposed framework outperforms existing methods in the comparison of RQ1 and RQ2 . the proposed framework significantly outperformed existing methods, leading to more comprehensive alignment. |
LaMini-LM: A Diverse Herd of Distilled Models from Large-Scale Instructions (2024.eacl-long)
Copied to clipboard
| Challenge: | Large language models with instruction tuning are resource-intensive . a recent study suggests that the performance of LLMs scales proportionally with the size of the model. |
| Approach: | They propose to distill knowledge from instruction-tuned LLMs into much smaller ones . they develop a large set of 2.58M instructions based on existing and newly-generated instructions . |
| Outcome: | The proposed models are comparable to strong baselines while being much smaller in size. |