Lost in Transcription: Identifying and Quantifying the Accuracy Biases of Automatic Speech Recognition Systems Against Disfluent Speech (2024.naacl-long)
Copied to clipboard
| Challenge: | Automatic speech recognition systems fail to accurately interpret speech patterns deviating from typical fluency, leading to critical usability issues and misinterpretations. |
| Approach: | They evaluate six leading automatic speech recognition systems based on a real-world dataset and a synthetic dataset derived from the widely-used LibriSpeech benchmark. |
| Outcome: | The six leading speech recognition systems were evaluated on a real-world dataset and a synthetic dataset derived from the widely-used LibriSpeech benchmark. |
Similar Papers
Evaluating Open-Source ASR Systems: Performance Across Diverse Audio Conditions and Error Correction Methods (2025.coling-main)
Copied to clipboard
| Challenge: | Automated speech recognition (ASR) systems are able to transcribe spontaneous human conversations with high accuracy. |
| Approach: | They evaluate the accuracy of open source automatic speech recognition systems across conversational speech datasets and explore the potential of ASR ensembling and post-ASR correction methods to improve transcription accuracy. |
| Outcome: | The proposed methods highlight the need for robust error correction techniques and address demographic biases to enhance ASR performance and inclusivity. |
Language-specific Effects on Automatic Speech Recognition Errors for World Englishes (2022.coling-1)
Copied to clipboard
| Challenge: | Existing systems are not able to meet the needs of speakers of different demographic groups. |
| Approach: | They propose to analyze the performance of Otter’s automatic captioning system on native and non-native English speakers of different language background through a linguistic analysis of segment-level errors. |
| Outcome: | The proposed system predicts certain errors from the phonological structure of a speaker’s native language. |
CEASR: A Corpus for Evaluating Automatic Speech Recognition (2020.lrec-1)
Copied to clipboard
Malgorzata Anna Ulasik, Manuela Hürlimann, Fabian Germann, Esin Gedik, Fernando Benites, Mark Cieliebak
| Challenge: | Automatic Speech Recognition (ASR) systems are increasingly needed for research and practical applications. |
| Approach: | They propose to use public speech corpora to evaluate the quality of automatic speech recognition (ASR) they calculate an average Word Error Rate (WER) per corpus, per system and per corpor-system pair . |
| Outcome: | The proposed corpus evaluates the quality of automatic speech recognition systems using public speech corpora and transcripts generated by state-of-the-art systems. |
Using Automatic Speech Recognition in Spoken Corpus Curation (2020.lrec-1)
Copied to clipboard
| Challenge: | Automatic Speech Recognition (ASR) is a new way to make audio-visual data accessible. |
| Approach: | They propose to use automatic speech recognition (ASR) to make audio-visual data accessible by systematic queries. |
| Outcome: | The proposed system has higher recognition scores for the north of Germany vs. lower scores for south of the country. |
Lost in Transcription, Found in Distribution Shift: Demystifying Hallucination in Speech Foundation Models (2025.findings-acl)
Copied to clipboard
| Challenge: | Automatic speech recognition systems have seen remarkable improvements in recent years, but evaluation of performance remains dependent on word and character error rate (WER/CER). |
| Approach: | They investigate how distribution shifts, model size and model architecture influence hallucination error rate (HER) HER is a metric used to quantify hallucinosity in automatic speech recognition systems. |
| Outcome: | The proposed model can be used to measure hallucination errors in high-stakes domains such as healthcare, legal, and aviation. |
Why Aren’t We NER Yet? Artifacts of ASR Errors in Named Entity Recognition in Spontaneous Speech Transcripts (2023.acl-long)
Copied to clipboard
Piotr Szymański, Lukasz Augustyniak, Mikolaj Morzy, Adrian Szymczak, Krzysztof Surdyk, Piotr Żelasko
| Challenge: | despite advances in language models, the transcript of spontaneous human-human conversations remains an insurmountable challenge for most models. |
| Approach: | They examine the relationship between ASR and NER errors which limit NER models' ability to recover entity mentions from spontaneous speech transcripts. |
| Outcome: | The proposed model fails even if no word errors are introduced by the ASR . the proposed model's performance deteriorates when applied to the ASL outputs . |
Language technology practitioners as language managers: arbitrating data bias and predictive bias in ASR (2022.lrec-1)
Copied to clipboard
| Challenge: | despite natural language variation, automatic speech recognition systems perform worse on non-standardised and marginalised language varieties. |
| Approach: | They propose a re-framing of language resources as (public) infrastructure for speech communities . authors propose rethinking of algorithms to address the origins and harms of bias . |
| Outcome: | The proposed approach aims to understand the origins and harms of algorithmic bias and how it can be mitigated. |
What is lost in Normalization? Exploring Pitfalls in Multilingual ASR Model Evaluations (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing text normalization routines that target Indic scripts are flawed when applied to multilingual automatic speech recognition models. |
| Approach: | They propose to develop text normalization routines that leverage native linguistic expertise to ensure more robust and accurate evaluations of multilingual automatic speech recognition models. |
| Outcome: | The proposed normalization routines can be leveraged to improve performance metrics for Indic languages. |
Mind the Pause: Disfluency-Aware Objective Tuning for Multilingual Speech Correction with LLMs (2026.acl-long)
Copied to clipboard
| Challenge: | Spontaneous speech is rarely fluent, and disfluencies can degrade readability and reliability . a sequence tagger first marks disfluent tokens, and these signals guide instruction fine-tuning . |
| Approach: | They propose a multilingual correction pipeline where a sequence tagger first marks disfluent tokens . they add a contrastive learning objective that penalizes the reproduction of disfluency tokens. |
| Outcome: | The proposed model improves readability and reliability of ASR transcripts in three languages . disfluencies can cause misinterpretations, incoherent responses, poor user experience . |
Error-preserving Automatic Speech Recognition of Young English Learners’ Language (2024.acl-long)
Copied to clipboard
| Challenge: | State-of-the-art speech recognition models are often trained on adult read-aloud data by native speakers and do not transfer well to young language learners’ speech. |
| Approach: | They propose to use an automated speech recognition module to train language learners' speaking skills on spontaneous speech by young language learners. |
| Outcome: | The proposed model improves on 85 hours of English audio spoken by Swiss learners and preserves their mistakes. |