ROSE: A Reward-Oriented Data Selection Framework for LLM Task-Specific Instruction Tuning (2025.findings-emnlp)
Copied to clipboard
Yang Wu, Huayi Zhang, Yizheng Jiao, Lin Ma, Xiaozhong Liu, Jinhong Yu, Dongyu Zhang, Dezhi Yu, Wei Xu
| Challenge: | Prevailing methods for task-specific instruction tuning use similarity metrics to select training data . but instruction tuning loss often fails to exhibit a monotonic relationship with actual task performance . |
| Approach: | They propose a task-specific instruction tuning method that leverages pairwise preference loss as a reward signal. |
| Outcome: | The proposed method surpasses state-of-the-art methods for task-specific instruction tuning. |
Similar Papers
Your Vision-Language Model Itself Is a Strong Filter: Towards High-Quality Instruction Tuning with Data Selection (2024.findings-acl)
Copied to clipboard
Ruibo Chen, Yihan Wu, Lichang Chen, Guodong Liu, Qi He, Tianyi Xiong, Chenxi Liu, Junfeng Guo, Heng Huang
| Challenge: | Existing data selection methods for instruction-following large language models rely on unreliable scores or use downstream tasks for selection. |
| Approach: | They propose a method that utilizes the VLM itself as a filter to select high-quality instruction-tuning data. |
| Outcome: | The proposed method can reach better results compared to full data settings with merely about 15% samples and can achieve superior performance against competitive baselines. |
From Quantity to Quality: Boosting LLM Performance with Self-Guided Data Selection for Instruction Tuning (2024.naacl-long)
Copied to clipboard
Ming Li, Yong Zhang, Zhitao Li, Jiuhai Chen, Lichang Chen, Ning Cheng, Jianzong Wang, Tianyi Zhou, Jing Xiao
| Challenge: | Large Language Models (LLMs) have revolutionized the landscape of artificial intelligence. |
| Approach: | They propose a self-guided method to identify and select cherry samples from open-source datasets, minimizing manual curation and potential cost for instruction tuning an LLM. |
| Outcome: | The proposed method enables LLMs to identify discrepancies between expected responses and intrinsic generation capability, and a marked uptick in model training efficiency. |
Stratified Selective Sampling for Instruction Tuning with Dedicated Scoring Strategy (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Recent work shows that post-training datasets can be substantially downsampled without noticeably deteriorating performance. |
| Approach: | They propose a method that efficiently bins data into groups and scores difficulty using specialized models. |
| Outcome: | The proposed method can be efficient and universally applied to post-training datasets. |
Improving Influence-based Instruction Tuning Data Selection for Balanced Learning of Diverse Capabilities (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Influence-based methods show promise in achieving (1), but often struggle with (2) . data selection is often biased towards high-influence tasks, harming performance on them . |
| Approach: | They propose a Balanced and Influential Data Selection algorithm that normalizes influence scores of training data and iteratively chooses the training example with the highest influence on the most underrepresented task. |
| Outcome: | The proposed model outperforms both state-of-the-art influence-based methods and non-influence-based frameworks on seven benchmarks spanning five diverse capabilities. |
Selective Reflection-Tuning: Student-Selected Data Recycling for LLM Instruction-Tuning (2024.findings-acl)
Copied to clipboard
| Challenge: | Instruction tuning is critical to large language models but its success heavily relies on the training data quality. |
| Approach: | They propose a paradigm that synergizes a teacher LLM’s reflection and introspection with the data selection capability of the student LLM to automatically refine existing instruction-tuning data. |
| Outcome: | The proposed method achieves much stronger and top-tier 7B and 13B LLMs without collecting brand-new data. |
PIRA: Preference-Oriented Instruction-Tuned Reward Models with Dual Aggregation (2026.findings-eacl)
Copied to clipboard
| Challenge: | Existing approaches to align large language models with human preferences are limited by their large-scale annotation and prone to reward overoptimization. |
| Approach: | They propose a training paradigm that integrates three complementary strategies to address these challenges by reformulating question–answer pairs into preference-task instructions, averaging the rewards aggregated from diverse preference- task instructions for each sample, and a balancing outputs from the value head under different dropout rates. |
| Outcome: | Experiments on public datasets show that PIRA improves performance considerably, enhances generalization, and effectively mitigates reward overoptimization. |
ROSE Doesn’t Do That: Boosting the Safety of Instruction-Tuned Large Language Models with Reverse Prompt Contrastive Decoding (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for aligning LLMs output with expected safety require substantial training efforts and expensive computational resources. |
| Approach: | They propose a method to directly boost the safety of existing instruction-tuned large language models without additional training. |
| Outcome: | The proposed method improves safety of instruction-tuned large language models without training and requires expensive computational resources. |
Data Whisperer: Efficient Data Selection for Task-Specific LLM Fine-Tuning via Few-Shot In-Context Learning (2025.acl-long)
Copied to clipboard
Shaobo Wang, Xiangqi Jin, Ziming Wang, Jize Wang, Jiajun Zhang, Kaixin Li, Zichen Wen, Zhong Li, Conghui He, Xuming Hu, Linfeng Zhang
| Challenge: | Using fine-tuning on task-specific data is essential for large language models to be effective in specialized tasks. |
| Approach: | They propose a method that leverages few-shot in-context learning with the model to be fine-tuned. |
| Outcome: | The proposed method outperforms existing methods with a 3.1-point improvement and a 7.4 speedup on the Llama-3-8B-Instruct model using just 10% of the dataset. |
Priority on High-Quality: Selecting Instruction Data via Consistency Verification of Noise Injection (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for instruction selection rely on external models or rules, overlooking the intrinsic association between pre-trained model and instruction data. |
| Approach: | They propose a method that utilizes noise injection to identify the quality of instruction data without relying on external models. |
| Outcome: | The proposed method outperforms the model trained on the entire dataset and established baselines. |
A Systematic Examination of Preference Learning through the Lens of Instruction-Following (2025.naacl-long)
Copied to clipboard
Joongwon Kim, Anirudh Goyal, Aston Zhang, Bo Xiong, Rui Hou, Melanie Kambadur, Dhruv Mahajan, Hannaneh Hajishirzi, Liang Tan
| Challenge: | a recent study has found that preference learning is a key tool for enhancing LLM training and alignment. |
| Approach: | They use a synthetic data generation pipeline to generate 48,000 unique instruction-following prompts with 23 verifiable constraints to obtain preference pairs. |
| Outcome: | The proposed pipeline generates 48,000 unique instruction-following prompts with 23 verifiable constraints that enable fine-grained and automated quality assessments of model responses. |