Beat LLMs at Their Own Game: Zero-Shot LLM-Generated Text Detection via Querying ChatGPT (2023.emnlp-main)
Copied to clipboard
Biru Zhu, Lifan Yuan, Ganqu Cui, Yangyi Chen, Chong Fu, Bingxiang He, Yangdong Deng, Zhiyuan Liu, Maosong Sun, Ming Gu
| Challenge: | Large language models (LLMs) are capable of performing tasks but are likely to be misused. |
| Approach: | They propose a zero-shot black-box method to detect LLM-generated texts . they revise the text to be detected using the ChatGPT model . |
| Outcome: | The proposed method can detect LLM-generated texts with a zero-shot black-box model . it is based on intuition that the model will make fewer revisions to LLMs than to human-written texts . |
Similar Papers
A Survey on Detection of LLMs-Generated Content (2024.findings-emnlp)
Copied to clipboard
Xianjun Yang, Liangming Pan, Xuandong Zhao, Haifeng Chen, Linda Petzold, William Yang Wang, Wei Cheng
| Challenge: | Recent advances in large language models have led to an increase in synthetic content generation . the ability to detect LLMs-generated content has become of paramount importance . |
| Approach: | They propose to provide a detailed overview of existing detection strategies and benchmarks, scrutinizing their differences and advocating for more adaptable and robust models to enhance detection accuracy. |
| Outcome: | The proposed model will be able to detect human-written content in real time. |
Who Wrote This? The Key to Zero-Shot LLM-Generated Text Detection Is GECScore (2025.coling-main)
Copied to clipboard
| Challenge: | Existing methods for detecting LLM-generated text require no training data. |
| Approach: | They propose a black-box zero-shot detection approach that calculates the Grammar Error Correction Score for a given text to differentiate between human-written and LLM-generated texts. |
| Outcome: | The proposed method outperforms current state-of-the-art zero-shot and supervised methods, achieving an average AUROC of 98.62% across XSum and Writing Prompts datasets. |
Efficient Detection of LLM-generated Texts with a Bayesian Surrogate Model (2024.findings-acl)
Copied to clipboard
| Challenge: | Large language models can be used to produce text that is coherent, well-written, and persuasive . some individuals have misused LLMs for nefarious purposes, such as creating fake news articles or engaging in cheating . |
| Approach: | They propose to incorporate a Bayesian surrogate model to improve query efficiency . they propose to select typical samples based on Bayes' uncertainty and interpolate scores . |
| Outcome: | The proposed method significantly outperforms existing approaches under a low query budget. |
DetectLLM: Leveraging Log Rank Information for Zero-Shot Detection of Machine-Generated Text (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models generate huge amounts of text, making it impractical to manually distinguish whether a text is machine-generated. |
| Approach: | They propose two methods to detect machine-generated text by leveraging Log-Rank information and propose a faster method that uses less perturbations to achieve the same level of performance. |
| Outcome: | The proposed methods improve over the state of the art by 3.9 and 1.75 AUROC points absolute and require less perturbations to achieve the same level of performance. |
Zero-Shot Detection of LLM-Generated Text using Token Cohesiveness (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing zero-shot detection paradigms that use token cohesiveness are not available for large language models. |
| Approach: | They propose a generic dual-channel detection paradigm that uses token cohesiveness as a plug-and-play module to improve existing zero-shot detectors. |
| Outcome: | The proposed model is able to detect human-like text in black-box environments. |
Zero-Shot Detection of LLM-Generated Text using Temperature Sensitivity (2026.acl-long)
Copied to clipboard
| Challenge: | Existing methods for detecting LLM-generated text rely on statistical features that are insufficient for reliable detection. |
| Approach: | They propose a temperature-sensitive detector that modulates decoding temperature and monitors how probability distributions respond to temperature. |
| Outcome: | The proposed method is based on a temperature sensitivity feature and a simple zero-shot detector built upon normalized temperature sensitivity. |
Text Fluoroscopy: Detecting LLM-Generated Text through Intrinsic Features (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) have revolutionized the field of natural language processing because of their excellent performance on various tasks. |
| Approach: | They propose a black-box method with better generalizability for detecting LLM-generated text by mining the intrinsic features of the text to be detected. |
| Outcome: | The proposed method achieves 7.36% and 2.84% improvement in detection performance compared to baselines in detecting texts from different domains generated by GPT-4 and Claude3 respectively. |
On the Generalization of Training-based ChatGPT Detection Methods (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies show that training-based methods are ineffective to detect LLM generated texts from unseen tasks or topics which are not collected during training. |
| Approach: | They propose to train classification models to distinguish LLMs from human texts by a distribution shift caused by prompts, text lengths, topics, and language tasks. |
| Outcome: | The proposed methods can detect LLMs from black-box models, but they suffer from distribution shifts due to a wide range of factors, including prompts, text lengths, topics, and language tasks. |
On the Zero-Shot Generalization of Machine-Generated Text Detectors (2023.findings-emnlp)
Copied to clipboard
| Challenge: | rampant proliferation of large language models generates text indistinguishable from human-written language. |
| Approach: | They train neural detectors on outputs of a new generator and test their performance on held-out generators. |
| Outcome: | The proposed detectors can be built on training data from medium-sized models. |
People who frequently use ChatGPT for writing tasks are accurate and robust detectors of AI-generated text (2025.acl-long)
Copied to clipboard
| Challenge: | Qualitative analysis of experts’ free-form explanations shows that while they rely heavily on specific lexical clues (‘AI vocabulary’), they also pick up on more complex phenomena within the text (e.g., formality, originality, clarity). |
| Approach: | They hire annotators to read 300 non-fiction English articles, label them as either human-written or AI-generated, and provide paragraph-length explanations for their decisions. |
| Outcome: | The annotators who frequently use LLMs for writing tasks outperform commercial and open-source detectors even without evasion tactics like paraphrasing and humanization. |