Can LLMs Find a Needle in a Haystack? A Look at Anomaly Detection Language Modeling (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Anomaly detection (AD) is a problem in machine learning, but it is not always competitive on certain datasets. |
| Approach: | They propose a new approach to Anomaly detection based on large pre-trained language models in three modalities. |
| Outcome: | The proposed model beats baselines on anomaly detection when presented as imbalanced classification problem regardless of the concentration of anomalous samples. |
Similar Papers
AD-LLM: Benchmarking Large Language Models for Anomaly Detection (2025.findings-acl)
Copied to clipboard
Tiankai Yang, Yi Nian, Li Li, Ruiyao Xu, Yuangang Li, Jiaqi Li, Zhuo Xiao, Xiyang Hu, Ryan A. Rossi, Kaize Ding, Xia Hu, Yue Zhao
| Challenge: | Anomaly detection (AD) is an important machine learning task with many real-world uses, including fraud detection, medical diagnosis, and industrial monitoring. |
| Approach: | They propose a benchmark that evaluates how large language models (LLMs) can help with NLP anomaly detection. |
| Outcome: | The proposed model can perform zero-shot detection without tasks-specific training, data augmentation and model selection, and it can suggest unsupervised AD models. |
Large Language Models for Anomaly and Out-of-Distribution Detection: A Survey (2025.findings-naacl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have demonstrated their effectiveness in natural language processing but also in broader applications due to their advanced comprehension and generative capabilities. |
| Approach: | They propose a taxonomy to categorize existing approaches into two classes based on the role played by LLMs. |
| Outcome: | The proposed taxonomy categorizes existing approaches into two classes based on the role played by LLMs. |
AD-NLP: A Benchmark for Anomaly Detection in Natural Language Processing (2023.emnlp-main)
Copied to clipboard
| Challenge: | Methods for Anomaly Detection in text have shown strong empirical results on ad-hoc anomaly setups that are usually made by downsampling some classes of a labeled dataset. |
| Approach: | They propose a unified benchmark for detecting various types of anomalies . they evaluate two strong shallow baselines and two current state-of-the-art neural approaches . |
| Outcome: | The proposed benchmarks provide insights into the knowledge the neural models are learning when performing the task. |
Oddballness: universal anomaly detection with language models (2025.coling-main)
Copied to clipboard
| Challenge: | a new method to detect anomalies in texts uses a metric called oddballness . the method considers probabilities generated by a language model but not low-likelihood tokens . |
| Approach: | They propose a method to detect anomalies in texts using unsupervised language models . they define oddballness as a function that measures how strange a given token is . |
| Outcome: | The proposed method is better than state-of-the-art models for grammatical error detection tasks. |
Does Data Contamination Detection Work (Well) for LLMs? A Survey and Evaluation on Detection Assumptions (2025.findings-naacl)
Copied to clipboard
| Challenge: | Large language models have demonstrated great performance across various benchmarks, but data contamination is a concern in their evaluation. |
| Approach: | They analyze 50 papers on data contamination detection and test three of them as case studies to identify the possibility of data contamination. |
| Outcome: | The proposed methods can detect membership inference attacks on instance-level data, and can perform similar to random guessing on LLM pretraining datasets. |
How Good Are LLMs at Out-of-Distribution Detection? (2024.lrec-main)
Copied to clipboard
| Challenge: | Out-of-distribution (OOD) detection is crucial for ensuring AI safety . large language models (LLMs) are becoming more prevalent due to their scale, pre-training objectives, and paradigms used for inference. |
| Approach: | They propose to use large language models to investigate out-of-distribution (OOD) detection in machine learning. |
| Outcome: | The proposed method outperforms other OOD detectors in zero-grad and fine-tuning scenarios. |
Knowing What LLMs DO NOT Know: A Simple Yet Effective Self-Detection Method (2024.naacl-long)
Copied to clipboard
Yukun Zhao, Lingyong Yan, Weiwei Sun, Guoliang Xing, Chong Meng, Shuaiqiang Wang, Zhicong Cheng, Zhaochun Ren, Dawei Yin
| Challenge: | Recent literature reveals that Large Language Models (LLMs) hallucinate intermittently, which impedes their reliability for further utilization. |
| Approach: | They propose a self-detection method to detect which questions an LLM does not know by combining the two components to identify whether the model generates a non-factual response to the question. |
| Outcome: | The proposed method can detect which questions an LLM does not know across factoid question-answering, arithmetic reasoning, and commonsense reasoning tasks. |
AutoAnoEval: Semantic-Aware Model Selection via Tree-Guided LLM Reasoning for Tabular Anomaly Detection (2026.findings-eacl)
Copied to clipboard
Suhee Yoon, Sanghyu Yoon, Ye Seul Sim, Seungdong Yoa, Dongmin Kim, Soonyoung Lee, Hankook Lee, Woohyung Lim
| Challenge: | Existing approaches to tabular anomaly detection fail to reflect domain specific nature of real-world anomalies. |
| Approach: | They propose a framework that constructs pseudo-evaluation sets with semantically grounded synthetic anomalies. |
| Outcome: | The proposed framework generates pseudo-evaluation sets with semantically grounded synthetic anomalies. |
Too Consistent to Detect: A Study of Self-Consistent Errors in LLMs (2025.emnlp-main)
Copied to clipboard
Hexiang Tan, Fei Sun, Sha Liu, Du Su, Qi Cao, Xin Chen, Jingang Wang, Xunliang Cai, Yuanzhuo Wang, Huawei Shen, Xueqi Cheng
| Challenge: | Existing detection methods fail to account for **self-consistent error** . study identifies self-consistency errors and evaluates them . |
| Approach: | They propose a method that fuses hidden state evidence from an external verifier LLM to detect self-consistent errors. |
| Outcome: | The proposed method significantly enhances performance on self-consistent errors across three LLM families. |
NLP-ADBench: NLP Anomaly Detection Benchmark (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Anomaly detection (AD) is an important machine learning task, but its effectiveness in detecting harmful content, phishing attempts, and spam reviews is limited. |
| Approach: | They introduce NLP-ADBench, the most comprehensive NLP anomaly detection benchmark to date . it includes eight curated datasets and 19 state-of-the-art algorithms . |
| Outcome: | The NLP-ADBench benchmark includes 19 state-of-the-art methods and 8 curated datasets . no single model dominates across all datasets, indicating need for automated model selection . |