Controllable Contamination Detection for Reliable LLM Evaluation with Statistical Guarantees

Challenge:	Existing training data detectors fail to detect clean samples from contaminated test sets . existing methods fail to identify clean samples due to black-box nature of LLMs .
Approach:	They propose a framework that detects and filters contaminated evaluation data . they propose 'failure detection' to reduce the proportion of contaminated samples mistakenly retained .
Outcome:	The proposed framework reduces false discovery rate (FDR) under valid FDR control while maintaining evaluation consistency.

Similar Papers

Overview

Challenge

Approach

Outcome

Generalization or Memorization: Data Contamination and Trustworthy Evaluation for Large Language Models (2024.findings-acl)

Copied to clipboard

Yihong Dong, Xue Jiang, Huanyu Liu, Zhi Jin, Bin Gu, Mengfei Yang, Ge Li

Semantic Text Processing

Challenge:	Considering the vast size and wide-ranging sources of LLMs’ training data, it could explicitly or implicitly include test data.
Approach:	They propose a Contamination Detection via output Distribution (CDD) which detects data contamination only by identifying the peakedness of LLM's output distribution.
Outcome:	The proposed method improves performance by 21.8%-30.2% on humanEval and TED: trustworthy evaluation via output distribution.

Does Data Contamination Detection Work (Well) for LLMs? A Survey and Evaluation on Detection Assumptions (2025.findings-naacl)

Copied to clipboard

Yujuan Fu, Ozlem Uzuner, Meliha Yetisgen, Fei Xia

Challenge:	Large language models have demonstrated great performance across various benchmarks, but data contamination is a concern in their evaluation.
Approach:	They analyze 50 papers on data contamination detection and test three of them as case studies to identify the possibility of data contamination.
Outcome:	The proposed methods can detect membership inference attacks on instance-level data, and can perform similar to random guessing on LLM pretraining datasets.

DCR: Quantifying Data Contamination in LLMs Evaluation (2025.emnlp-main)

Copied to clipboard

Cheng Xu, Nan Yan, Shuhao Guan, Changhong Jin, Yuke Mei, Yibing Guo, Tahar Kechadi

Challenge:	Large language models (LLMs) memorize evaluation data during training, inflating performance metrics and undermining genuine generalization assessment.
Approach:	They propose a framework to detect and quantify benchmark data contamination (BDC) by synthesizing contamination scores via a fuzzy inference system.
Outcome:	The proposed framework detects and quantifies BDC risk across semantic, informational, data, and label levels.

Towards Data Contamination Detection for Modern Large Language Models: Limitations, Inconsistencies, and Oracle Challenges (2025.coling-main)

Copied to clipboard

Vinay Samuel, Yue Zhou, Henry Peng Zou

Robustness in NLP

Challenge:	Existing methods for detecting data contamination in large language models have limitations and limitations . data contamination occurs when test or evaluation data is exposed to the model during its training phases .
Approach:	They evaluate five different methods for detecting data contamination in large language models . they find that current methods have non-trivial limitations in their assumptions and practical applications .
Outcome:	The proposed methods have non-trivial limitations and difficulties in detecting contamination . the authors highlight the complexity of contamination detection in advanced LLMs .

Data Contamination Can Cross Language Barriers (2024.emnlp-main)

Copied to clipboard

Feng Yao, Yufan Zhuang, Zihao Sun, Sunan Xu, Animesh Kumar, Jingbo Shang

Cross-Lingual Transfer

Challenge:	Existing methods to detect contamination of public benchmarks are too superficial to reflect deeper forms of contamination.
Approach:	They propose generalization-based approaches to unmask a cross-lingual form of contamination that inflates LLMs’ performance while evading current detection methods.
Outcome:	The proposed model outperforms existing detection methods while avoiding contamination of public benchmarks in the pre-training data.

Data Contamination Calibration for Black-box LLMs (2024.findings-acl)

Copied to clipboard

Wentao Ye, Jiaqi Hu, Liyao Li, Haobo Wang, Gang Chen, Junbo Zhao

Challenge:	Despite the rapid advancements of Large Language Models, the unchecked ultra-large-scale training sets introduce a series of potential risks like data contamination.
Approach:	They propose a method to detect contaminated training data and diminish the contamination effect by using a to-be-released dataset.
Outcome:	The proposed method outperforms existing methods by at least 4.5% on more 4 dataset formats, with more than 10 base LLMs.

Unveiling the Spectrum of Data Contamination in Language Model: A Survey from Detection to Remediation (2024.findings-acl)

Copied to clipboard

Chunyuan Deng, Yilun Zhao, Yuzhao Heng, Yitong Li, Jiannan Cao, Xiangru Tang, Arman Cohan

Language Models

Challenge:	Data contamination is a problem in Large language models due to the reliance on extensive internet-derived training corpora.
Approach:	They present a survey on the topic of data contamination in large language models.
Outcome:	The results of the first survey on data contamination in large language models provide a comprehensive guide for NLP researchers seeking a systematic understanding of the issue.

When Benchmarks Leak: Inference-Time Decontamination for LLMs (2026.acl-long)

Copied to clipboard

Jianzhe Chai, YU Zhe, Jun Sakuma

Challenge:	a large number of large language models (LLMs) are being evaluated for their performance, but their reliability is threatened by test set contamination.
Approach:	They propose a framework that decontaminates large language models by applying small perturbations to the input embedding space.
Outcome:	The proposed framework achieves strong decontamination effectiveness while incurring minimal degradation in benign utility.

LNE-Blocking: An Efficient Framework for Contamination Mitigation Evaluation on Large Language Models (2025.findings-emnlp)

Copied to clipboard

Ruijie Hou, Yueyang Jiao, Hanxu Hu, Yingming Li, Wai Lam, Huajian Zhang, Hongyuan Lu

Language Models

Challenge:	a problem of data contamination is now almost inevitable during the development of large language models, with the training data often integrating evaluation benchmarks even unintentionally.
Approach:	They propose a framework to restore model performance prior to data contamination on potentially leaked datasets by using contamination detection and disruption operation.
Outcome:	The proposed framework restores model performance prior to contamination on potentially leaked datasets.

SSA: Semantic Contamination of LLM-Driven Fake News Detection (2025.emnlp-main)

Copied to clipboard

Cheng Xu, Nan Yan, Shuhao Guan, Yuke Mei, Tahar Kechadi

Responsible & Trustworthy NLP

Challenge:	Evaluating 45 variants of nine LLMs, we find LIAR2 accuracy climbs monotonically with injected contamination, while the SSA Factor escalates in near-perfect lock-step.
Approach:	They propose a framework that detects BDC risks across semantic to label level via entity shift perturbation and an interpretable metric, the SSA Factor.
Outcome:	The proposed framework detects BDC risks across semantic to label level via entity shift perturbation and interpretable metric, the SSA Factor.