Yalun Dai, Yangyu Huang, Tongshen Yang, Yonghan Wang, Xin Zhang, Wenshan Wu, Qihao Zhao, Hao Li, Yuanyuan Gao, Kim-Hui Yap, Scarlett Li
| Challenge: | Large Language Models (LLMs) have revolutionized various fields, yet their training efficiency is heavily reliant on effective data curation. |
| Approach: | They propose to reuse pre-computed sample-level scores originally generated for data efficiency and introduce two new data ordering methods to improve LLM training. |
| Outcome: | The proposed methods improve the stability and performance of LLM training. |
Similar Papers
Beyond Random Sampling: Efficient Language Model Pretraining via Curriculum Learning (2026.eacl-long)
Copied to clipboard
| Challenge: | Curriculum learning has improved efficiency across machine learning domains, but remains underexplored for language model pretraining. |
| Approach: | They present a systematic investigation of curriculum learning in LLM pretraining . they use vanilla curriculum learning, pacing-based sampling, and interleaved curricula . |
| Outcome: | The proposed framework accelerates convergence in early and mid-training phases, reducing training steps by 18-45% to reach baseline performance. |
Fine-Grained Data Ordering Improves Fine-Tuning for Large Language Models (2026.findings-acl)
Copied to clipboard
Xiaomeng Hu, Yixuan Tang, Haoze Li, Hao Chen, Qi Zhang, Zhanming Shen, Yiming Zhang, Haobo Wang, Junbo Zhao
| Challenge: | Prior work focused on data preprocessing, focusing on filtering and cleaning data . a study aimed to improve fine-grained scheduling of data order in epochs . |
| Approach: | They propose a fine-grained scheduling method of data order in epochs to fill this gap . they define data difficulty based on relevance between data and model . |
| Outcome: | The proposed method improves on pre-training and small-scale fine-tuning experiments 2.4% over baselines. |
A Survey on Efficient Large Language Model Training: From Data-centric Perspectives (2025.acl-long)
Copied to clipboard
Junyu Luo, Bohan Wu, Xiao Luo, Zhiping Xiao, Yiqiao Jin, Rong-Cheng Tu, Nan Yin, Yifan Wang, Jingyang Yuan, Wei Ju, Ming Zhang
| Challenge: | achieving data-efficient post-training of Large Language Models is a key research question. |
| Approach: | They propose a taxonomy of data-efficient LLM post-training methods from a data-centric perspective. |
| Outcome: | The proposed methods cover data selection, data quality enhancement, synthetic data generation, data distillation and compression, and self-evolving data ecosystems. |
Deciphering the Impact of Pretraining Data on Large Language Models through Machine Unlearning (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing studies have suggested that the composition of the pretraining corpus exerts a significant impact upon the performance of LLMs. |
| Approach: | They analyze the impact of 48 datasets from 5 major categories of pretraining data of Large Language Models and measure their impacts on LLMs using benchmarks about nine major categories. |
| Outcome: | The proposed analysis provides insights into the organization of data to support more efficient pretraining of Large Language Models. |
The Data Frontier for Large Language Models: Selection, Synthesis, and Tools (2026.acl-tutorials)
Copied to clipboard
| Challenge: | acquiring and curating high-quality training data remains a significant bottleneck . acquiring such high-quality data is a key challenge for researchers and practitioners . |
| Approach: | This tutorial provides a comprehensive and practical guide to the state-of-the-art in data research directions for LLMs. |
| Outcome: | The tutorial covers methods for curating the most valuable information from vast, noisy datasets and the synthetic data revolution. |
LLM2LLM: Boosting LLMs with Novel Iterative Data Enhancement (2024.findings-acl)
Copied to clipboard
Nicholas Lee, Thanakul Wattanawong, Sehoon Kim, Karttikeya Mangalam, Sheng Shen, Gopala Anumanchipalli, Michael Mahoney, Kurt Keutzer, Amir Gholami
| Challenge: | Pretrained large language models are currently state-of-the-art for solving most tasks . however, many of them are in the low-data regime, making fine-tuning challenging . a new data augmentation strategy uses a teacher LLM to augment a small seed dataset . |
| Approach: | They propose a targeted and iterative data augmentation strategy that augments a teacher LLM to fine-tune a small seed dataset by adding additional data. |
| Outcome: | The proposed approach outperforms fine-tuning and other data augmentation strategies on a small seed dataset. |
CodecLM: Aligning Language Models with Tailored Synthetic Data (2024.findings-naacl)
Copied to clipboard
Zifeng Wang, Chun-Liang Li, Vincent Perot, Long Le, Jin Miao, Zizhao Zhang, Chen-Yu Lee, Tomas Pfister
| Challenge: | Recent work on generating diverse instructions and applying LLM to increase instruction complexity neglects downstream use cases. |
| Approach: | They propose a framework for generating high-quality synthetic data for LLM alignment with different downstream instruction distributions and LLMs. |
| Outcome: | Experiments on four open-domain instruction using the proposed framework validate the effectiveness of CodecLM over the current state-of-the-art. |
From Complex to Simple: Enhancing Multi-Constraint Complex Instruction Following Ability of Large Language Models (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models (LLMs) follow instructions with elaborate requirements, yet it remains under-explored how to enhance their ability to follow complex instructions with multiple constraints. |
| Approach: | They propose a method to obtain and utilize effective training data to enhance LLMs' ability to follow complex instructions with multiple constraints. |
| Outcome: | The proposed framework improves models' ability to follow instructions generally and generalize effectively across out-of-domain, in domain, and adversarial settings while maintaining general capabilities. |
From Parameters to Performance: A Data-Driven Study on LLM Structure and Development (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models have revolutionized a wide range of domains, driving significant advancements in both technology and real-world applications. |
| Approach: | They present a large-scale dataset encompassing diverse open-source LLM structures and their performance across multiple benchmarks. |
| Outcome: | The proposed model validates the relationship between structural configurations and performance across multiple benchmarks and further corroborates the findings using mechanistic interpretability techniques. |
Structure Trumps Size: Rethinking Data Quality for LLM Reasoning (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for fine-tuning Large Language Models rely on heuristic strategies and lack systematic, quantitative frameworks for evaluating data quality. |
| Approach: | They propose a multi-dimensional quantitative framework for reasoning data management . they rigorously evaluate and optimize datasets along six orthogonal dimensions . |
| Outcome: | The proposed framework rigorously evaluates and optimizes datasets along six orthogonal dimensions. |