Fine-Grained Data Ordering Improves Fine-Tuning for Large Language Models

Challenge:	Prior work focused on data preprocessing, focusing on filtering and cleaning data . a study aimed to improve fine-grained scheduling of data order in epochs .
Approach:	They propose a fine-grained scheduling method of data order in epochs to fill this gap . they define data difficulty based on relevance between data and model .
Outcome:	The proposed method improves on pre-training and small-scale fine-tuning experiments 2.4% over baselines.

Similar Papers

Overview

Challenge

Approach

Outcome

Fine-tuning Large Language Models with Limited Data: A Survey and Practical Guide (2026.tacl-1)

Copied to clipboard

Marton Szep, Daniel Rueckert, Rüdiger von Eisenhart-Rothe, Florian Hinterwimmer

Language Models

Challenge:	Pre-trained language models provide strong foundations, but effective adaptation under data scarcity requires efficient and efficient fine-tuning techniques.
Approach:	They propose to review parameter-efficient fine-tuning techniques that lower training and deployment costs and domain and cross-lingual adaptation methods for both encoder and decoder models.
Outcome:	The proposed techniques lower training and deployment costs, domain and cross-lingual adaptation methods, and model specialization strategies.

Massive Supervised Fine-tuning Experiments Reveal How Data, Layer, and Training Factors Shape LLM Alignment Quality (2025.emnlp-main)

Copied to clipboard

Yuto Harada, Yusuke Yamauchi, Yusuke Oda, Yohei Oseki, Yusuke Miyao, Yu Takagi

Language Models

Challenge:	Recent advances in large language models (LLMs) have greatly improved natural language understanding and generation.
Approach:	They train a wide range of base models on a variety of datasets including code generation, mathematical reasoning, and general-domain tasks.
Outcome:	The results show that training–task synergies persist across all models while others vary substantially, emphasizing the importance of model-specific strategies.

Demystifying Instruction Mixing for Fine-tuning Large Language Models (2024.acl-srw)

Copied to clipboard

Renxi Wang, Haonan Li, Minghao Wu, Yuxia Wang, Xudong Han, Chiyu Zhang, Timothy Baldwin

Semantic Text Processing

Challenge:	Instruction tuning is effective for aligning large language models with human instructions, but the procedure to optimizing the mixing of instruction datasets is still unclear.
Approach:	They categorize instructions into three primary types: NLP downstream tasks, coding, and general chat.
Outcome:	The proposed method improves performance of large language models (LLMs) but it is difficult to combine different instruction datasets to optimize overall performance.

Take the essence and discard the dross: A Rethinking on Data Selection for Fine-Tuning Large Language Models (2025.naacl-long)

Copied to clipboard

Ziche Liu, Rui Ke, Yajiao Liu, Feng Jiang, Haizhou Li

Semantic Text Processing

Challenge:	Existing studies focus on data selection but lack a clear, unified framework . variability in experimental settings complicates systematic comparisons .
Approach:	They propose a three-stage scheme to standardize data selection for fine-tuning large language models . they propose unified comparison approach that incorporates ratio-based efficiency and ranking-based feasibility metrics to address inconsistencies across experiments.
Outcome:	The proposed scheme outperforms existing methods in a dozen key studies and identifies key challenges.

Mixture-of-Skills: Learning to Optimize Data Usage for Fine-Tuning Large Language Models (2024.emnlp-main)

Copied to clipboard

Minghao Wu, Thuy-Trang Vu, Lizhen Qu, Reza Haf

Language Models

Challenge:	Large language models are fine-tuned on diverse datasets to develop a range of skills . each skill has unique characteristics, and datasets are heterogeneous and imbalanced . a general, model-agnostic, reinforcement learning framework is proposed to optimize data usage .
Approach:	They propose a general, model-agnostic, reinforcement learning framework that optimizes data usage automatically during the fine-tuning process.
Outcome:	The proposed framework optimizes data usage automatically during the fine-tuning process.

Noise-Robust Fine-Tuning of Pretrained Language Models via External Guidance (2023.findings-emnlp)

Copied to clipboard

Song Wang, Zhen Tan, Ruocheng Guo, Jundong Li

Robustness in NLP

Challenge:	Pretrained Language Models (PLMs) are advanced but data labels are noisy due to the complex annotation process.
Approach:	They propose a framework for fine-tuning PLMs using noisy labels that incorporates guidance from Large Language Models like ChatGPT.
Outcome:	Experiments on synthetic and real-world noisy datasets show that the proposed framework outperforms the state-of-the-art framework.

Improving the OOD Performance of Closed-Source LLMs on NLI Through Strategic Data Selection (2026.findings-eacl)

Copied to clipboard

Joe Stacey, Lisa Alazraki, Aran Ubhi, Beyza Ermis, Aaron Mueller, Marek Rei

Robustness in NLP

Challenge:	Existing methods to improve robustness require changing the fine-tuning process or large-scale data augmentation, which are infeasible or cost prohibitive for closed-source models.
Approach:	They propose to prioritize more complex examples or replace existing training examples with LLM-generated data to improve performance on OOD NLI datasets.
Outcome:	The proposed methods improve performance on difficult OOD datasets while training with synthetic data leads to substantial improvements on easier OOD data.

Demystifying Data Organization for Enhanced LLM Training (2026.acl-long)

Copied to clipboard

Yalun Dai, Yangyu Huang, Tongshen Yang, Yonghan Wang, Xin Zhang, Wenshan Wu, Qihao Zhao, Hao Li, Yuanyuan Gao, Kim-Hui Yap, Scarlett Li

Responsible & Trustworthy NLP

Challenge:	Large Language Models (LLMs) have revolutionized various fields, yet their training efficiency is heavily reliant on effective data curation.
Approach:	They propose to reuse pre-computed sample-level scores originally generated for data efficiency and introduce two new data ordering methods to improve LLM training.
Outcome:	The proposed methods improve the stability and performance of LLM training.

Dynamics of Instruction Fine-Tuning for Chinese Large Language Models (2025.coling-main)

Copied to clipboard

Chiyu Song, Zhanchao Zhou, Jianhao Yan, Yuejiao Fei, Zhenzhong Lan, Yue Zhang

Language Models

Challenge:	Instruction tuning is a burgeoning method to elicit the general intelligence of Large Language Models.
Approach:	They investigate the effects of data quantity, model size, and data construction methods on instruction tuning for Chinese LLMs.
Outcome:	The proposed model includes over 40,000 high-quality instruction instances covering ten underlying abilities.

Extreme Fine-tuning: A Novel and Fast Fine-tuning Approach for Text Classification (2024.eacl-short)

Copied to clipboard

Boonnithi Jiaramaneepinit, Thodsaporn Chay-intr, Kotaro Funakoshi, Manabu Okumura

Information Extraction & Text Mining

Challenge:	Existing methods for fine-tuning pre-trained models require massive computational resources and time.
Approach:	They propose a novel approach for fine-tuning a pre-trained model using backpropagation and an iterative extreme learning machine for training a classifier.
Outcome:	The proposed approach outperforms state-of-the-art approaches in training-time measurement and performance with comparable model performance.