PDR: A Plug-and-Play Positional Decay Framework for LLM Pre-training Data Detection (2026.acl-long)
Copied to clipboard
| Challenge: | Existing likelihood-based methods for detecting pretraining data are limited in black-box, zero-shot settings. |
| Approach: | They propose a training-free and plug-and-play framework that reweights token-level scores to amplify distinct signals from early positions while suppressing noise from later ones. |
| Outcome: | The proposed framework amplifys signals from early positions while suppressing noise from later positions. |
Similar Papers
Con-ReCall: Detecting Pre-training Data in LLMs via Contrastive Decoding (2025.coling-main)
Copied to clipboard
| Challenge: | Existing methods analyze training data with member and non-member contexts, overlooking potential insights from both member and not-member. |
| Approach: | They propose a method that leverages asymmetric distributional shifts induced by member and non-member contexts through contrastive decoding to enhance membership inference. |
| Outcome: | The proposed approach outperforms the current state-of-the-art on the WikiMIA benchmark and is robust against various text manipulation techniques. |
Identifying Pre-training Data in LLMs: A Neuron Activation-Based Detection Framework (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for detecting pre-training data in large language models rely on superficial features like prediction confidence and loss, resulting in mediocre performance. |
| Approach: | They propose a new algorithm to analyze neuron activation patterns between training and non-training data in large language models to improve their performance. |
| Outcome: | The proposed algorithm outperforms existing methods across three benchmarks and multiple LLMs. |
Mitigating Posterior Salience Attenuation in Long-Context LLMs with Positional Contrastive Decoding (2025.acl-short)
Copied to clipboard
| Challenge: | Current solutions incur prohibitive training costs, leaving statistical behaviors and cost-effective approaches underexplored. |
| Approach: | They propose a positional contrast decoding technique that contrasts long-aware attention with designed local-awn attention. |
| Outcome: | The proposed model achieves state-of-the-art performance on long-context benchmarks. |
Pretraining Data Detection for Large Language Models: A Divergence-based Calibration Method (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to detect text in training corpus are limited due to their low token probabilities. |
| Approach: | They propose a method to calibrate token probabilities for pretraining data detection by using a divergence-based calibration method. |
| Outcome: | The proposed method significantly outperforms existing methods on Chinese text on English-language benchmarks and patents. |
Tag&Tab: Pretraining Data Detection in Large Language Models Using Keyword-Based Membership Inference Attack (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Recent studies on detecting pretraining data in large language models have focused on sentence-level membership inference attacks (MIAs) but these methods often exhibit poor accuracy, failing to account for the semantic importance of textual content and word significance. |
| Approach: | They propose a method that leverages established natural language processing techniques to tag keywords in input text and then uses them to obtain probabilities and calculate their average log-likelihood to determine input text membership. |
| Outcome: | The proposed method exploits established natural language processing techniques to tag keywords in input text and calculate their average log-likelihood to determine input text membership. |
Position-Aware Depth Decay Decoding (D3): Boosting Large Language Model Inference Efficiency (2025.findings-acl)
Copied to clipboard
| Challenge: | Recent dynamic computation methods show that not all components are required for inference, enabling a training-free pipeline. |
| Approach: | They propose a token-position aware layer skipping framework to save 1.5x times operations efficiently while maintaining performance. |
| Outcome: | The proposed algorithm achieves 1.5x speedup on large language models with no retraining and with comparable performance on the GSM8K and BBH benchmarks. |
CoLA: Compute-Efficient Pre-Training of LLMs via Low-Rank Activation (2025.emnlp-main)
Copied to clipboard
Ziyue Liu, Ruijie Zhang, Zhengyang Wang, Mingsong Yan, Zi Yang, Paul D. Hovland, Bogdan Nicolae, Franck Cappello, Sui Tang, Zheng Zhang
| Challenge: | Large foundation models have become huge, but they consume computational resources in pretraining. |
| Approach: | They propose to replace full-size layers with compute-efficient auto-encoders that enforce low-rank activations throughout training. |
| Outcome: | The proposed method reduces the computing cost by 2pmbtimes and improves training throughput by 1.86pmtime. |
DPDLLM: A Black-box Framework for Detecting Pre-training Data from Large Language Models (2024.findings-acl)
Copied to clipboard
Baohang Zhou, Zezhong Wang, Lingzhi Wang, Hongru Wang, Ying Zhang, Kehui Song, Xuhui Sui, Kam-Fai Wong
| Challenge: | Existing methods to detect pretraining data from large language models are unrealistic to them. |
| Approach: | They propose to detect pre-training data from LLM in a black-box way by using GPT-2 as reference model and feed it with sequence probabilities to detect whether it was used to train it. |
| Outcome: | The proposed framework outperforms existing methods on the benchmark datasets and shows that it is effective on different popular LLMs. |
Retracing the Past: LLMs Emit Training Data When They Get Lost (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for extracting training data from large language models exhibit limited success . existing methods offer limited insight into the fundamental drivers of memorization leakage . |
| Approach: | They propose a framework for extracting memorized data by maximizing model uncertainty . they propose mismatched fine-tuning to weaken alignment and induce confusion . |
| Outcome: | The proposed attacks outperform baselines on unaligned and aligned LLMs . the proposed attacks exploit the model uncertainty of the input snippets induced by the model entropy spike . |
Fine-tuning LLMs with Cross-Attention-based Weight Decay for Bias Mitigation (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) excel in natural language processing tasks but often propagate societal biases from their training data, leading to discriminatory outputs. |
| Approach: | They propose a method that modifies the LLM architecture to mitigate bias by adjusting the attention weights of sensitive tokens. |
| Outcome: | The proposed method can handle multiple sensitive attributes and does not require full knowledge of sensitive tokens presented in the dataset. |