Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, Nicholas Carlini
| Challenge: | Existing language modeling datasets contain near-duplicate examples and long repetitive substrings. |
| Approach: | They develop tools that allow us to deduplicate existing language modeling datasets . they found that over 1% of the unprompted output of language models is copied verbatim . |
| Outcome: | The proposed tools reduce train-test overlap, which affects over 4% of validation sets, and improve model accuracy. |
Similar Papers
SoftDedup: an Efficient Data Reweighting Method for Speeding Up Language Model Pre-training (2024.acl-long)
Copied to clipboard
| Challenge: | Current methods focus on detecting and removing duplicates, which risks the loss of valuable information and neglects the varying degrees of duplication. |
| Approach: | They propose a method that maintains dataset integrity while selectively reducing the sampling weight of data with high commonness. |
| Outcome: | The proposed method significantly improves training efficiency on deduplicated datasets and improves downstream accuracy by 1.77%. |
Low-Perplexity LLM-Generated Sequences and Where To Find Them (2025.acl-srw)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are increasingly applied across various domains, but the ways they leverage their training data during inference remains only partially understood. |
| Approach: | They propose a systematic approach that analyzes low-perplexity sequences and traces them back to their sources in the training data. |
| Outcome: | The proposed pipeline extracts low-perplexity sequences across diverse topics while avoiding degeneration, then trace them back to their sources in the training data. |
How Much Do Language Models Copy From Their Training Data? Evaluating Linguistic Novelty in Text Generation Using RAVEN (2023.tacl-1)
Copied to clipboard
| Challenge: | Current language models generate high-quality text, but are they copying it or have they learned generalizable linguistic abstractions? |
| Approach: | They propose a suite of analyses for assessing the novelty of generated text . they focus on sequential structure (n-grams) and syntactic structure (syntactical structure). |
| Outcome: | The proposed model-generated text is as novel as the baseline human-generated model- generated text, but it is copied substantially, the authors show . |
Language Models for German Text Simplification: Overcoming Parallel Data Scarcity through Style-specific Pre-training (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to train automatic text simplification systems for languages other than English are limited by the lack of parallel data. |
| Approach: | They propose to use German Easy Language as a corpus of automatic text simplification systems to fine-tune language models to the style characteristics of the language. |
| Outcome: | The proposed language models adapt to the style characteristics of Easy Language and output more accessible texts. |
How much pretraining data do language models need to learn syntax? (2021.emnlp-main)
Copied to clipboard
| Challenge: | Pretraining methods are convenient, but expensive in terms of time and resources. |
| Approach: | They investigate the impact of pretraining data size on the syntactic capabilities of RoBERTa by using syntaktic structural probes to determine whether models pretrained on more data encode a higher amount of syntastic information. |
| Outcome: | The proposed models perform better on part-of-speech tagging, dependency parsing and paraphrase identification. |
Unlearning Traces the Influential Training Data of Language Models (2024.acl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) exhibit remarkable abilities without being explicitly trained for such tasks. |
| Approach: | They propose a method that unlearns a test dataset and evaluates the unlearned model on training datasets. |
| Outcome: | The proposed method resembles UnTrac, while being efficient for massive training datasets. |
Analysing The Impact of Sequence Composition on Language Model Pre-Training (2024.acl-long)
Copied to clipboard
Yu Zhao, Yuanbin Qu, Konrad Staniszewski, Szymon Tworkowski, Wei Liu, Piotr Miłoś, Yuxiang Wu, Pasquale Minervini
| Challenge: | Existing studies show that pretraining sequence composition strategy can lead to distracting information from previous documents. |
| Approach: | They propose to use a sequence construction method to concatenate documents into fixed-length sequences to compute the likelihood of each token given its context. |
| Outcome: | The proposed method can improve in-context learning, knowledge memorisation and context utilisation without sacrificing efficiency. |
Investigating How Pre-training Data Leakage Affects Models’ Reproduction and Detection Capabilities (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies do not examine how leaked instances in training datasets influence LLMs’ output and detection capabilities. |
| Approach: | They conduct an experimental survey to examine the relationship between data leakage in training datasets and its effects on the generation and detection by Large Language Models (LLMs). |
| Outcome: | The results show that enhancing leakage detection through few-shot learning can help mitigate the impact of the leakage rate in the training data on detection performance. |
Deciphering the Impact of Pretraining Data on Large Language Models through Machine Unlearning (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing studies have suggested that the composition of the pretraining corpus exerts a significant impact upon the performance of LLMs. |
| Approach: | They analyze the impact of 48 datasets from 5 major categories of pretraining data of Large Language Models and measure their impacts on LLMs using benchmarks about nine major categories. |
| Outcome: | The proposed analysis provides insights into the organization of data to support more efficient pretraining of Large Language Models. |
Language Contamination Helps Explains the Cross-lingual Capabilities of English Pretrained Models (2022.emnlp-main)
Copied to clipboard
| Challenge: | a large number of pretraining corpora are not publicly available, and it is unclear how much foreign language data exists in monolingual models. |
| Approach: | They propose to use English pretraining corpora to analyze their language composition . they find that even when less than 1% of data is not English, it facilitates cross-lingual transfer . |
| Outcome: | The proposed model is not truly monolingual when pretrained at scale, the authors show . they show that even when less than 1% of data is not English, it facilitates cross-lingual transfer . |