Lost in Transcription, Found in Distribution Shift: Demystifying Hallucination in Speech Foundation Models (2025.findings-acl)
Copied to clipboard
| Challenge: | Automatic speech recognition systems have seen remarkable improvements in recent years, but evaluation of performance remains dependent on word and character error rate (WER/CER). |
| Approach: | They investigate how distribution shifts, model size and model architecture influence hallucination error rate (HER) HER is a metric used to quantify hallucinosity in automatic speech recognition systems. |
| Outcome: | The proposed model can be used to measure hallucination errors in high-stakes domains such as healthcare, legal, and aviation. |
Similar Papers
Lost in Transcription: Identifying and Quantifying the Accuracy Biases of Automatic Speech Recognition Systems Against Disfluent Speech (2024.naacl-long)
Copied to clipboard
| Challenge: | Automatic speech recognition systems fail to accurately interpret speech patterns deviating from typical fluency, leading to critical usability issues and misinterpretations. |
| Approach: | They evaluate six leading automatic speech recognition systems based on a real-world dataset and a synthetic dataset derived from the widely-used LibriSpeech benchmark. |
| Outcome: | The six leading speech recognition systems were evaluated on a real-world dataset and a synthetic dataset derived from the widely-used LibriSpeech benchmark. |
HALoGEN: Fantastic LLM Hallucinations and Where to Find Them (2025.acl-long)
Copied to clipboard
| Challenge: | generative large language models produce hallucinations that are not aligned with world knowledge or input context. |
| Approach: | They propose a hallucination benchmark framework that measures hallucinism in large language models . they evaluate 150,000 generations from 14 language models and find they are riddled with hallucinos . |
| Outcome: | The proposed framework evaluates 150,000 generations from 14 language models. |
On the Robust Approximation of ASR Metrics (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for estimating speech recognition metrics depend on ground truth labels. |
| Approach: | They propose a label-free approach to approximating ASR performance metrics . they embed multimodal embeddings in a unified space for speech and transcription representations . |
| Outcome: | The proposed method outperforms baseline models on speech recognition benchmarks by 50%. |
Detecting Hallucinations in SpeechLLMs at Inference Time Using Attention Maps (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for hallucination detection for text-based LLMs do not capture audio-specific signals. |
| Approach: | They propose to capture pathological attention patterns associated with hallucination using four attention-derived metrics to train lightweight logistic regression classifiers. |
| Outcome: | The proposed approach outperforms baselines on in-domain data and generalises to out-of-domain ASR settings. |
HalluAudio: A Comprehensive Benchmark for Hallucination Detection in Large Audio-Language Models (2026.acl-long)
Copied to clipboard
| Challenge: | Existing studies on hallucination focus on text or vision, while few audio-oriented studies are limited in scale, modality coverage, and diagnostic depth. |
| Approach: | They propose a large-scale benchmark for evaluating hallucinations across speech, sound, and music. |
| Outcome: | The proposed model improves hallucination rate, yes/no bias, error-type analysis, and refusal rate. |
Advocating Character Error Rate for Multilingual ASR Evaluation (2025.findings-naacl)
Copied to clipboard
| Challenge: | Word error rate (WER) has been used for automatic speech recognition (ASR) evaluations for English datasets for many years. |
| Approach: | They propose to use the character error rate as the primary metric in multilingual ASR evaluation to account for morphologically complex languages. |
| Outcome: | The character error rate (CER) is the primary evaluation metric in multilingual ASR evaluation. |
Embedding and Gradient Say Wrong: A White-Box Method for Hallucination Detection (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for hallucination detection have attracted more attention from the community. |
| Approach: | They propose to model the distributional distance between the regular conditional output and the unconditional output, which is generated without a given input text. |
| Outcome: | The proposed model achieves state-of-the-art on the hallucination benchmarks HADES and other datasets. |
Lost in Diffusion: Uncovering Hallucination Patterns and Failure Modes in Diffusion Large Language Models (2026.findings-acl)
Copied to clipboard
| Challenge: | dLLMs have emerged as a promising non-autoregressive paradigm for text generation, but their hallucination mechanisms remain underexplored. |
| Approach: | They present the first controlled comparative study to evaluate hallucination patterns in Diffusion Large Language Models. |
| Outcome: | The proposed model exhibits higher propensity for hallucination than AR counterparts controlled for architecture, scale, and pre-training weights. |
A Comprehensive Survey of Hallucination in Large Language, Image, Video and Audio Foundation Models (2024.findings-emnlp)
Copied to clipboard
| Challenge: | a growing need to understand and alleviate FMs' propensity to produce hallucinated outputs, especially in high-stakes applications. |
| Approach: | They propose a framework for detecting and mitigating hallucination in FMs . they synthesize recent advancements in detection and mitigation techniques . |
| Outcome: | The proposed framework provides valuable insights for researchers, developers, and practitioners. |
The Illusion of Progress: Re-evaluating Hallucination Detection in LLMs (2025.emnlp-main)
Copied to clipboard
Denis Janiak, Jakub Binkowski, Albert Sawczyn, Bogdan Gabrys, Ravid Shwartz-Ziv, Tomasz Jan Kajdanowicz
| Challenge: | Large language models (LLMs) have revolutionized natural language processing, but their tendency to hallucinate poses serious challenges for reliable deployment. |
| Approach: | They propose to use ROUGE to assess lexical overlap to determine accuracy of hallucination detection methods. |
| Outcome: | The proposed evaluation frameworks can rival complex methods, exposing a fundamental flaw in current evaluation practices. |