Exploiting the Shadows: Unveiling Privacy Leaks through Lower-Ranked Tokens in Large Language Models (2025.acl-long)
Copied to clipboard
| Challenge: | Large language models face vulnerabilities related to the extraction of sensitive information. |
| Approach: | They propose a method to exploit the model's lower-ranked output tokens to extract private information from retrieved documents or training knowledge. |
| Outcome: | The proposed method is effective in both the agentic application privacy extraction setting and the direct training data extraction. |
Similar Papers
R.R.: Unveiling LLM Training Privacy through Recollection and Ranking (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing privacy attacks focus on membership inference or data extraction, but reconstructing specific personally identifiable information (PII) in training data remains challenging. |
| Approach: | They propose a two-step privacy stealing attack that enables attackers to reconstruct PII entities from scrubbed training data where the PI I entities have been masked. |
| Outcome: | The proposed attack can reconstruct PII entities from scrubbed training data where the PI I entities have been masked. |
LLMs are Privacy Erasable (2025.findings-emnlp)
Copied to clipboard
| Challenge: | a new study examines the privacy of large language models and their capabilities . the study aims to address the balance between the convenience of LLMs and user privacy concerns . |
| Approach: | They propose a strategy that safeguards user prompt while accessing LLM cloud services . they evaluate the efficacy of their method across prominent LLM benchmarks . |
| Outcome: | The proposed method thwarts reconstruction attacks and improves model performance . it also surpasses the results reported in official model cards . |
Adaptive Backtracking for Privacy Protection in Large Language Models (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing privacy protection methods are prone to privacy leakage, but they are not effective in ensuring the privacy of users. |
| Approach: | They propose to capture latent leakage tendency of large language models during generation process and to construct a new benchmark for personal information. |
| Outcome: | The proposed method improves privacy by up to 14% over strong baselines against adversarial attacks, avoiding the degradation of response utility. |
Combating Security and Privacy Issues in the Era of Large Language Models (2024.naacl-tutorials)
Copied to clipboard
| Challenge: | a tutorial aims to provide a summary of risks and vulnerabilities in large language models . a number of studies have focused on security, privacy and copyright aspects of LLMs . |
| Approach: | This tutorial seeks to provide a systematic summary of risks and vulnerabilities in large language models . authors will discuss security, privacy and copyright aspects of LLMs . |
| Outcome: | This tutorial aims to provide a systematic summary of risks and vulnerabilities in large language models . it will also outline emerging challenges in security, privacy and reliability of LLMs . |
Tokens for Learning, Tokens for Unlearning: Mitigating Membership Inference Attacks in Large Language Models via Dual-Purpose Training (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing defenses for large language models do not account for the sequential nature of text data. |
| Approach: | They propose a lightweight yet effective empirical privacy defense that leverages token-specific characteristics to protect training data of large language models. |
| Outcome: | The proposed approach provides strong protection against membership inference attacks and improves language modeling performance by 10% across different LLM architectures and datasets compared to baselines. |
SecureSQL: Evaluating Data Leakage of Large Language Models as Natural Language Interfaces to Databases (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies on the vulnerability of large language models to SQL injection have been limited. |
| Approach: | They propose to evaluate the potential of language models to leak sensitive data when generating SQL queries. |
| Outcome: | The proposed model with the best performance has an accuracy of 61.7%, compared to humans who achieve 94% accuracy. |
Unintended Memorization of Sensitive Information in Fine-Tuned Language Models (2026.eacl-long)
Copied to clipboard
Marton Szep, Jorge Marin Ruiz, Georgios Kaissis, Paulina Seidl, Rüdiger von Eisenhart-Rothe, Florian Hinterwimmer, Daniel Rueckert
| Challenge: | Large Language Models (LLMs) on sensitive datasets carry a substantial risk of unintended memorization and leakage of Personally Identifiable Information (PII) prior studies have analyzed memorizing dynamics in LLMs during pre-training and fine-tuning. |
| Approach: | They investigate the vulnerability of PII that appears only in model inputs, not in training targets. |
| Outcome: | The proposed methods show that post-training methods provide more consistent privacy-utility trade-offs . |
Unveiling Privacy Risks in LLM Agent Memory (2025.acl-long)
Copied to clipboard
| Challenge: | Large Language Model (LLM) agents store private user-agent interactions in memory for demonstrations, introducing new privacy risks for LLM agents. |
| Approach: | They propose an attack that extracts private information from memory under a black-box setting and propose a method that can be used to attack the agent. |
| Outcome: | The proposed attack is effective under a black-box setting and it is demonstrated on two representative agents. |
REVS: Unlearning Sensitive Information in Language Models via Rank Editing in the Vocabulary Space (2025.findings-acl)
Copied to clipboard
| Challenge: | Current approaches to address this issue involve costly dataset scrubbing or model filtering through unlearning and model editing. |
| Approach: | They propose a method for unlearning sensitive information from language models . they curate email and URL datasets and a social security number dataset . |
| Outcome: | The proposed method shows superior performance and robustness to extraction attacks on real-world datasets. |
Controlling the Extraction of Memorized Data from Large Language Models via Prompt-Tuning (2023.acl-short)
Copied to clipboard
Mustafa Ozdayi, Charith Peris, Jack FitzGerald, Christophe Dupuy, Jimit Majmudar, Haidar Khan, Rahil Parikh, Rahul Gupta
| Challenge: | Large Language Models memorize significant portions of training data, which poses privacy risk. |
| Approach: | They propose a prompt-tuning approach to control the extraction rates of memorized content in large language models. |
| Outcome: | The proposed techniques yield 9.3% increase in extraction rate compared to baseline model . the proposed defense achieves 97.7% reduction with a perplexity increase of 16.9% . |