SoftDedup: an Efficient Data Reweighting Method for Speeding Up Language Model Pre-training (2024.acl-long)
Copied to clipboard
| Challenge: | Current methods focus on detecting and removing duplicates, which risks the loss of valuable information and neglects the varying degrees of duplication. |
| Approach: | They propose a method that maintains dataset integrity while selectively reducing the sampling weight of data with high commonness. |
| Outcome: | The proposed method significantly improves training efficiency on deduplicated datasets and improves downstream accuracy by 1.77%. |
Similar Papers
Deduplicating Training Data Makes Language Models Better (2022.acl-long)
Copied to clipboard
Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, Nicholas Carlini
| Challenge: | Existing language modeling datasets contain near-duplicate examples and long repetitive substrings. |
| Approach: | They develop tools that allow us to deduplicate existing language modeling datasets . they found that over 1% of the unprompted output of language models is copied verbatim . |
| Outcome: | The proposed tools reduce train-test overlap, which affects over 4% of validation sets, and improve model accuracy. |
Low-Perplexity LLM-Generated Sequences and Where To Find Them (2025.acl-srw)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are increasingly applied across various domains, but the ways they leverage their training data during inference remains only partially understood. |
| Approach: | They propose a systematic approach that analyzes low-perplexity sequences and traces them back to their sources in the training data. |
| Outcome: | The proposed pipeline extracts low-perplexity sequences across diverse topics while avoiding degeneration, then trace them back to their sources in the training data. |
DRPruning: Efficient Large Language Model Pruning through Distributionally Robust Optimization (2025.acl-long)
Copied to clipboard
| Challenge: | Structured pruning reduces model size but often causes uneven degradation across domains, leading to biased performance. |
| Approach: | They propose a method that dynamically adjusts the data distribution during training to restore balanced performance across heterogeneous and multi-tasking data. |
| Outcome: | Experiments in monolingual and multilingual settings show that the proposed method surpasses similarly sized models in pruning and continued pretraining over perplexity, downstream tasks, and instruction tuning. |
Virtual Data Augmentation: A Robust and General Framework for Fine-tuning Pre-trained Models (2021.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies have shown that powerful pre-trained language models can be fooled by small perturbations or intentional attacks. |
| Approach: | They propose a framework for fine-tuning PLMs using a masked language model and Gaussian noise to augment semantically relevant examples with sufficient diversity. |
| Outcome: | The proposed framework improves the robustness of pre-trained language models and alleviates performance degradation under adversarial attacks. |
DEFT-UCS: Data Efficient Fine-Tuning for Pre-Trained Language Models via Unsupervised Core-Set Selection for Text-Editing (2024.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in language modelling have led to the availability of many pre-trained language models (PLMs); however, how much data is needed to fine-tune PLMs for downstream tasks? |
| Approach: | They propose a data-efficient fine-tuning framework that leverages unsupervised core-set selection to identify a smaller, representative dataset to fine- tune PLMs for text-editing tasks. |
| Outcome: | The proposed framework is as accurate as CoEDIT across eight different datasets consisting of six different editing tasks, while finetuning on 70% less data. |
Data Weighted Training Strategies for Grammatical Error Correction (2020.tacl-1)
Copied to clipboard
| Challenge: | Recent advances in the task of Grammatical Error Correction (GEC) have been driven by addressing data sparsity, both through new methods for generating large and noisy pretraining data and through the publication of small and higher-quality finetuning data in the BEA-2019 shared task. |
| Approach: | They propose to incorporate delta-log-perplexity, a type of example scoring, into a training schedule for Grammatical Error Correction (GEC) they perform experiments that shed light on the function and applicability of delta- log-perplicity. |
| Outcome: | The proposed methods incorporate delta-log-perplexity, a type of example scoring, into a training schedule for the task. |
CoLA: Compute-Efficient Pre-Training of LLMs via Low-Rank Activation (2025.emnlp-main)
Copied to clipboard
Ziyue Liu, Ruijie Zhang, Zhengyang Wang, Mingsong Yan, Zi Yang, Paul D. Hovland, Bogdan Nicolae, Franck Cappello, Sui Tang, Zheng Zhang
| Challenge: | Large foundation models have become huge, but they consume computational resources in pretraining. |
| Approach: | They propose to replace full-size layers with compute-efficient auto-encoders that enforce low-rank activations throughout training. |
| Outcome: | The proposed method reduces the computing cost by 2pmbtimes and improves training throughput by 1.86pmtime. |
YuLan-Mini: Pushing the Limits of Open Data-efficient Language Model (2025.acl-long)
Copied to clipboard
Hu Yiwen, Huatong Song, Jie Chen, Jia Deng, Jiapeng Wang, Kun Zhou, Yutao Zhu, Jinhao Jiang, Zican Dong, Yang Lu, Xu Miao, Xin Zhao, Ji-Rong Wen
| Challenge: | prevailing pre-training approaches for large language models involve several complexities. |
| Approach: | They propose a low-cost training recipe and a robust optimization approach to mitigate training instability . they also propose synthesis, curriculum, and data selection pipelines to integrate data . |
| Outcome: | The proposed model achieves top-tier performance among models with similar parameter scale . it is comparable to industry-leading models that require significantly more data . |
Accelerating Dense LLMs via L0-regularized Mixture-of-Experts (2025.acl-short)
Copied to clipboard
| Challenge: | Existing methods for accelerating large language models (LLMs) suffer from slow and costly inference. |
| Approach: | They propose a lightweight MoE approach using cluster confusion matrix and dynamic batching to accelerate dense LLMs. |
| Outcome: | The proposed method achieves 2.5x speedup over dense models while maintaining competitive performance. |
A Data Cartography based MixUp for Pre-trained Language Models (2022.naacl-main)
Copied to clipboard
| Challenge: | Using MixUp, additional samples are generated during training by combining random pairs of training samples and their labels. |
| Approach: | They propose a new MixUp strategy that leverages Training Dynamics and allows more informative samples to be combined for generating new data samples. |
| Outcome: | The proposed method achieves competitive performance using a smaller subset of training data compared with strong baselines and yields lower expected calibration error on the pre-trained language model, BERT, on both in-domain and out-of-domain settings. |