| Challenge: | a library for evaluating membership inference attacks against large language models (LLMs) has emerged as a crucial technique for auditing privacy risks and copyright infringement in LLMs. |
| Approach: | They propose a Python library for efficiently evaluating membership inference attacks against large language models (LLMs) they use a high-throughput batch inference via vLLM and a cross-method caching architecture that computes intermediate results once and shares them across methods. |
| Outcome: | The proposed library performs a 5 speedup in inference and a cross-method caching architecture that computes intermediate results once and shares them across methods. |
Similar Papers
Scaling Up Membership Inference: When and How Attacks Succeed on Large Language Models (2025.findings-naacl)
Copied to clipboard
| Challenge: | Membership inference attacks (MIAs) attempt to verify the membership of a data sample in the training set for a model. |
| Approach: | They propose to use membership inference attacks to verify the membership of a given data sample in a model training set. |
| Outcome: | The proposed methods are based on a new benchmark that measures the performance of membership inference attacks on large language models at a continuous scale. |
CheckMIABench: Firm Foundations For Membership Inference Attacks on Language Models (2026.acl-short)
Copied to clipboard
| Challenge: | Membership inference attacks are a canonical way to assess a machine learning model’s privacy properties. |
| Approach: | They propose a framework for principled evaluation of membership inference attacks against large language models by leveraging the insight that training data before and after a fixed point during training are drawn from the same distribution. |
| Outcome: | The proposed framework can be used to evaluate membership inference attacks against large language models. |
Robust Membership Inference for Large Language Models under Adversarial Generative Corruption (2026.acl-long)
Copied to clipboard
Yuanhong Huang, Huili Wang, Xueying Bai, Jinrui Wang, Jiajun Liu, Ziqin Wang, Wanchun Ni, Shangguang Wang, Tao Qi
| Challenge: | Membership inference attacks are a promising tool for auditing training data of LLMs . existing methods rely on the assumption that LLM's assign higher confidence scores to training samples than to non-training ones. |
| Approach: | They propose a membership inference framework that can be robust against adversarial MIAs. |
| Outcome: | The proposed framework can be robust against adversarial MIA methods and AIGT detectors while maintaining the performance of baselines. |
Order of Magnitude Speedups for LLM Membership Inference (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are complex and require fine-tuning on proprietary datasets to improve performance and relevance. |
| Approach: | They propose a low-cost membership inference attack that leverages an ensemble of small quantile regression models to determine if a document belongs to the model’s training set. |
| Outcome: | The proposed approach achieves comparable or improved accuracy on fine-tuned LLMs of varying families and across multiple datasets. |
A Statistical and Multi-Perspective Revisiting of the Membership Inference Attack in Large Language Models (2025.acl-long)
Copied to clipboard
| Challenge: | Membership Inference Attack (MIA) is a method that differentiates trained (member) and untrained (non-member) data. |
| Approach: | They used thousands of experiments to examine membership inference attacks from different settings and then revisited them with thousands of different methods. |
| Outcome: | The proposed methods outperform baselines in the study and improve with model size and varies with domains. |
Context-Aware Membership Inference Attacks against Pre-trained Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Prior Membership Inference Attacks on pre-trained Large Language Models fail at LLMs due to ignoring the generative nature of LLM data. |
| Approach: | They propose a method that adapts MIA statistical tests to the perplexity dynamics of subsequences within a data point. |
| Outcome: | The proposed method significantly outperforms prior approaches, revealing context-dependent memorization patterns in pre-trained LLMs. |
Neural Breadcrumbs: Membership Inference Attacks on LLMs Through Hidden State and Attention Pattern Analysis (2026.eacl-long)
Copied to clipboard
| Challenge: | Membership inference attacks (MIAs) reveal whether specific data was used to train machine learning models, serving as important tools for privacy auditing and compliance assessment. |
| Approach: | They propose to examine LLMs’ internal representations rather than just their outputs to gain additional insights into potential membership inference signals. |
| Outcome: | The proposed framework yields strong membership detection across several model families achieving average AUC scores of 0.85 on popular MIA benchmarks. |
Powerful Training-Free Membership Inference Against Fine-Tuned Autoregressive Language Models (2026.acl-long)
Copied to clipboard
| Challenge: | Existing methods for auditing fine-tuned language models have limited detection rates . membership inference attacks aim to determine if a specific record was in a model's training set . |
| Approach: | They propose a membership inference attack that exploits memorization at error positions . EZ-MIA achieves 3.8 higher detection than previous state-of-the-art . |
| Outcome: | The proposed attack achieves 3.8 higher detection than previous state-of-the-art models . EZ-MIA achieves 8 higher detectability than prior work, requiring no model training . |
Detecting Training Data of Large Language Models via Expectation Maximization (2026.eacl-long)
Copied to clipboard
| Challenge: | Membership inference attacks aim to determine whether a specific example was used to train a given language model. |
| Approach: | They propose a membership inference approach that iteratively refines prefix effectiveness and membership scores using an expectation-maximization strategy without requiring labeled non-member examples. |
| Outcome: | The proposed approach outperforms baselines under systematically varied distributional overlap and difficulty. |
FiMMIA: scaling semantic perturbation-based membership inference across modalities (2026.eacl-demo)
Copied to clipboard
| Challenge: | Membership Inference attacks aim to determine whether a specific data point was included in the training set of a target model. |
| Approach: | They propose to train a neural network to analyze the target model’s behavior on perturbed inputs, capturing interactions between semantic domains and loss values on members and non-members in the local neighborhood of each sample. |
| Outcome: | The proposed methods can detect distribution shifts in existing datasets and release a baseline pipeline to detect them. |