Tracing the Roots: A Multi-Agent Framework for Uncovering Data Lineage in Post-Training LLMs (2026.acl-long)
Copied to clipboard
Yu Li, Xiaoran Shang, Qizhi Pei, Yun Zhu, Xin Gao, Honglin Lin, Zhanping Zhong, Zhuoshi Pan, Zheng Liu, Xiaoyang Wang, Conghui He, Dahua Lin, Feng Zhao, Lijun Wu
| Challenge: | High-quality post-training data is the primary engine driving LLM capabilities . datasets are often treated as isolated artifacts, overlooking their true developmental context . |
| Approach: | They propose a framework to reconstruct the evolutionary graph of dataset development using data lineage. |
| Outcome: | The proposed framework characterizes domain-specific structural patterns in Math-oriented datasets and general-domain corpora. |
Similar Papers
A Survey on Efficient Large Language Model Training: From Data-centric Perspectives (2025.acl-long)
Copied to clipboard
Junyu Luo, Bohan Wu, Xiao Luo, Zhiping Xiao, Yiqiao Jin, Rong-Cheng Tu, Nan Yin, Yifan Wang, Jingyang Yuan, Wei Ju, Ming Zhang
| Challenge: | achieving data-efficient post-training of Large Language Models is a key research question. |
| Approach: | They propose a taxonomy of data-efficient LLM post-training methods from a data-centric perspective. |
| Outcome: | The proposed methods cover data selection, data quality enhancement, synthetic data generation, data distillation and compression, and self-evolving data ecosystems. |
Low-Perplexity LLM-Generated Sequences and Where To Find Them (2025.acl-srw)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are increasingly applied across various domains, but the ways they leverage their training data during inference remains only partially understood. |
| Approach: | They propose a systematic approach that analyzes low-perplexity sequences and traces them back to their sources in the training data. |
| Outcome: | The proposed pipeline extracts low-perplexity sequences across diverse topics while avoiding degeneration, then trace them back to their sources in the training data. |
Crosscoding Through Time: Tracking Emergence & Consolidation Of Linguistic Representations Throughout LLM Pretraining (2026.acl-long)
Copied to clipboard
| Challenge: | Large language models learn non-trivial abstractions during pretraining, but it is not well understood when and how these specific linguistic abilities emerge. |
| Approach: | They propose a method to track the evolution of linguistic features during pretraining by using sparse crosscoders to discover and align features across model checkpoints. |
| Outcome: | The proposed approach can detect features emergence, maintenance, and discontinuation during training stages. |
The Data Frontier for Large Language Models: Selection, Synthesis, and Tools (2026.acl-tutorials)
Copied to clipboard
| Challenge: | acquiring and curating high-quality training data remains a significant bottleneck . acquiring such high-quality data is a key challenge for researchers and practitioners . |
| Approach: | This tutorial provides a comprehensive and practical guide to the state-of-the-art in data research directions for LLMs. |
| Outcome: | The tutorial covers methods for curating the most valuable information from vast, noisy datasets and the synthetic data revolution. |
A Survey of Post-Training Scaling in Large Language Models (2025.acl-long)
Copied to clipboard
Hanyu Lai, Xiao Liu, Junjie Gao, Jiale Cheng, Zehan Qi, Yifan Xu, Shuntian Yao, Dan Zhang, Jinhua Du, Zhenyu Hou, Xin Lv, Minlie Huang, Yuxiao Dong, Jie Tang
| Challenge: | Large language models (LLMs) have demonstrated proficiency in understanding and generating human natural languages. |
| Approach: | They propose a framework for scaling large language models using supervised fine-tuning, RLxF and test-time compute methodologies. |
| Outcome: | The proposed model can be used to understand and generate human natural languages. |
On LLMs-Driven Synthetic Data Generation, Curation, and Evaluation: A Survey (2024.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) provide a data-centric solution to alleviate limitations of real-world data with synthetic data generation. |
| Approach: | They propose a generic workflow for LLM-driven synthetic data generation. |
| Outcome: | The proposed workflows highlight gaps in existing research and outline avenues for future studies. |
A Data-Efficient Path to Multilingual LLMs: Language Expansion via Post-training PARAM𝛥 Integration into Upcycled MoE (2026.acl-long)
Copied to clipboard
Hao Zhou, Tianhao Li, Zhijun Wang, Shuaijie She, Linjuan Wu, Hao-Ran Wei, Baosong Yang, Jiajun Chen, Shujian Huang
| Challenge: | Large Language Models (LLMs) are expensive and require extensive Continued Pre-Training and data-intensive alignment to expand. |
| Approach: | They propose a method which upcycles a dense model into a Mixture-of-Experts architecture, allocating different experts to different languages. |
| Outcome: | Experiments show that the proposed model upcycles a dense model into a Mixture-of-Experts(MoE) architecture, allocating different experts to different languages. |
Demystifying Data Organization for Enhanced LLM Training (2026.acl-long)
Copied to clipboard
Yalun Dai, Yangyu Huang, Tongshen Yang, Yonghan Wang, Xin Zhang, Wenshan Wu, Qihao Zhao, Hao Li, Yuanyuan Gao, Kim-Hui Yap, Scarlett Li
| Challenge: | Large Language Models (LLMs) have revolutionized various fields, yet their training efficiency is heavily reliant on effective data curation. |
| Approach: | They propose to reuse pre-computed sample-level scores originally generated for data efficiency and introduce two new data ordering methods to improve LLM training. |
| Outcome: | The proposed methods improve the stability and performance of LLM training. |
LLMSurgeon: Diagnosing Data Mixture of Large Language Models (2026.acl-long)
Copied to clipboard
Yaxin Luo, Jiacheng Cui, Xiaohan Zhao, Xinyi Shang, Jiacheng Liu, Xinyue Bi, Zhaoyi Li, Zhiqiang Shen
| Challenge: | a lack of transparency in large language models makes auditing their "digital DNA" difficult. |
| Approach: | They propose a framework that casts DMS as an inverse problem under label-shift assumption . they propose LLMScan, a recipe-verifiable evaluation suite built from open-source LLMs . |
| Outcome: | The proposed framework casts DMS as an inverse problem under label-shift assumption . compared with existing frameworks, it recovers domain mixtures with high fidelity . |
Synthesizing Post-Training Data for LLMs through Multi-Agent Simulation (2025.acl-long)
Copied to clipboard
Shuo Tang, Xianghe Pang, Zexi Liu, Bohan Tang, Rui Ye, Tian Jin, Xiaowen Dong, Yanfeng Wang, Siheng Chen
| Challenge: | Extensive experiments demonstrate that our framework effectively generates both general and domain-specific data. |
| Approach: | They propose a multi-agent simulator that automatically generates diverse text-based scenarios, capturing a wide range of real-world human needs. |
| Outcome: | Experiments show that the proposed model outperforms Meta’s Llama-3-8B-Instruct model on AlpacaEval 2 and Arena-Hard benchmarks with just 20K instruction-response pairs. |