Papers with ASR
Optimizing Entity Resolution in Voice Interfaces: An ASR-Aware Entity Reference Expansion Approach (2024.emnlp-industry)
Copied to clipboard
| Challenge: | Automatic Speech Recognition (ASR) errors in voice-based dialog systems pose significant impediments to downstream tasks. |
| Approach: | They propose an automatic speech recognition (ASR) error-aware loss function to inject failed mentions and resolved entity names into the knowledge graph to enhance its awareness of unresolved mentions. |
| Outcome: | The proposed system enhances the knowledge graph's awareness of unresolved mentions by injecting pairs of failed mentions and resolved entities into the knowledge map. |
SpiRit-LM: Interleaved Spoken and Written Language Model (2025.tacl-1)
Copied to clipboard
Tu Anh Nguyen, Benjamin Muller, Bokai Yu, Marta R. Costa-jussa, Maha Elbayad, Sravya Popuri, Christophe Ropers, Paul-Ambroise Duquenne, Robin Algayres, Ruslan Mavlyutov, Itai Gat, Mary Williamson, Gabriel Synnaeve, Juan Pino, Benoît Sagot, Emmanuel Dupoux
| Challenge: | SpiRit-LM is a foundation multimodal language model that freely mixes text and speech. |
| Approach: | They propose a multimodal language model that freely mixes text and speech . they extend the model to the speech modality by continuously training it on text and language units. |
| Outcome: | The proposed model can learn new tasks in a few-shot fashion across modalities. |
Cross Attention Augmented Transducer Networks for Simultaneous Translation (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to simultaneous translation are limited by monotonic constraint . a novel architecture for simultaneous translation is proposed . |
| Approach: | They propose a cross attention-augmented transducer for simultaneous translation that optimizes both policies and translation models by expanding target sequences with blank symbols. |
| Outcome: | The proposed architecture achieves better latency-quality trade-offs than state-of-the-art approaches. |
Word Error Rate Estimation for Speech Recognition: e-WER (P18-2)
Copied to clipboard
| Challenge: | Automatic speech recognition (ASR) systems require manual transcription of test data to compute the word error rate (WER). |
| Approach: | They propose an approach to estimate word error rate (e-WER) that does not require a gold-standard transcription of the test set. |
| Outcome: | The proposed approach achieves 16.9% WER root mean squared error across 1,400 sentences. |
Hyper-BTS Dataset: Scalability and Enhanced Analysis of Back TranScription (BTS) for ASR Post-Processing (2024.findings-eacl)
Copied to clipboard
Chanjun Park, Jaehyung Seo, Seolhwa Lee, Junyoung Son, Hyeonseok Moon, Sugyeong Eo, Chanhee Lee, Heuiseok Lim
| Challenge: | Automatic Speech Recognition (ASR) post-processing requires substantial amounts of data, requiring expensive phonetic transcription experts. |
| Approach: | They propose a "Hyper-BTS" dataset that is five times larger than prior studies . they propose criteria for categorizing error types within ASR post-processing . |
| Outcome: | The proposed method can generate ASR inputs from clean text using a text-to-speech system. |
Pre-training on high-resource speech recognition improves low-resource speech-to-text translation (N19-1)
Copied to clipboard
| Challenge: | Pre-training on high-resource automatic speech recognition (ASR) tasks improves ST performance even when source language is low-resourced. |
| Approach: | They propose a method to improve direct speech-to-text translation when source language is low-resource . they pre-train model on high-res automatic speech recognition task and fine-tune parameters for ST . |
| Outcome: | The proposed approach improves Spanish English ST even when the source language is low-resource . the pre-trained encoder accounts for most of the improvement, the authors show . |
Non-Autoregressive Models for Fast Sequence Generation (2022.emnlp-tutorials)
Copied to clipboard
| Challenge: | Autoregressive (AR) models can only generate target sequence word-by-word due to the AR mechanism and suffer from slow inference. |
| Approach: | This tutorial provides an introduction to non-autoregressive sequence generation. |
| Outcome: | This tutorial explains how to generate non-autoregressive sequence generation models. |
TaskMix: Data Augmentation for Meta-Learning of Spoken Intent Understanding (2022.findings-aacl)
Copied to clipboard
| Challenge: | Meta-Learning requires a large number of training tasks to learn representations that transfer well to unseen tasks. |
| Approach: | They propose a method which synthesizes new tasks by linearly interpolating existing tasks. |
| Outcome: | The proposed method outperforms baselines and does not degrade performance even when it is high. |
Audio Query Handling System with Integrated Expert Models and Contextual Understanding (2025.emnlp-industry)
Copied to clipboard
| Challenge: | Existing chatbots are limited to specific audio tasks, but the domain of audio content related queries remains underexplored. |
| Approach: | They propose to use an intent classifier to route queries to audio-related experts using a diverse audio query dataset. |
| Outcome: | The proposed system outperforms state-of-the-art LLMs on custom audio tasks and MMAU sound set benchmarks. |
Chameleon: A Language Model Adaptation Toolkit for Automatic Speech Recognition of Conversational Speech (D19-3)
Copied to clipboard
| Challenge: | Language model adaptation (LMA) is a promising solution for conversational speech recognition systems. |
| Approach: | They propose to use language model adaptation techniques to adapt language models to conversational speech recognition. |
| Outcome: | The proposed toolkit compares state-of-the-art language model adaptation techniques in conversational speech recognition tasks. |
Which Side Are You On? A Multi-task Dataset for End-to-End Argument Summarisation and Evaluation (2024.findings-acl)
Copied to clipboard
Hao Li, Yuping Wu, Viktor Schlegel, Riza Batista-Navarro, Tharindu Madusanka, Iqra Zahid, Jiayan Zeng, Xiaochi Wang, Xinran He, Yizhi Li, Goran Nenadic
| Challenge: | Recent advances in large language models (LLMs) have made it difficult to build an automated debate system that helps people to synthesise persuasive arguments. |
| Approach: | They propose to use an argument mining dataset to capture the end-to-end process of preparing an argumentative essay for a debate. |
| Outcome: | The proposed dataset shows that it performs better on individual tasks than on human-centred evaluations. |
Entity resolution for noisy ASR transcripts (D19-3)
Copied to clipboard
| Challenge: | Domain-agnostic Automatic Speech Recognition systems often mistranscribe domain-specific words and phrases. |
| Approach: | They propose a method for handling ASR errors in named entities, specifically person names, for a voice-based collaboration assistant. |
| Outcome: | The proposed method improves accuracy by 40.8% on a voice-based collaboration assistant. |
N-Best ASR Transformer: Enhancing SLU Performance using Multiple ASR Hypotheses (2021.acl-short)
Copied to clipboard
| Challenge: | Spoken Language Understanding systems parse spoken utterances into semantic structures like dialog acts and slots. |
| Approach: | They propose to use concatenated N-best ASR alternatives to represent utterances . they propose to employ a simpler utteration representation with no special delimiter . |
| Outcome: | The proposed model outperforms the prior state-of-the-art model on DSTC2 dataset. |
A Study of Non-autoregressive Model for Sequence Generation (2020.acl-main)
Copied to clipboard
| Challenge: | Non-autoregressive (NAR) models generate all tokens in parallel, resulting in faster generation speed compared to autoregressive models. |
| Approach: | They propose to use knowledge distillation and source-target alignment to bridge the gap between NAR and autoregressive models in various tasks. |
| Outcome: | The proposed techniques can speed up NAR models in some tasks but not all . the proposed techniques reduce target token dependency while allowing for faster inference . |
A Corpus for Modeling Word Importance in Spoken Dialogue Transcripts (L18-1)
Copied to clipboard
| Challenge: | a project aims to create a system that uses automatic speech recognition (ASR) to produce real-time text captions of spoken English during in-person meetings with hearing individuals. |
| Approach: | They propose to use automatic speech recognition to produce captions in real-time . they add word-importance annotations to a transcript of a conversational dialogue corpus . |
| Outcome: | The proposed system would produce captions in real-time for people who are deaf or hard-of-hearing . the best performing model has an F-score of 0.60 in an ordinal 6-class word-importance classification task with an agreement (concordance correlation coefficient) of 0.89 with the human annotators. |
PUMA: Projected Universal Multilingual ASR for Low-Resource Settings. Application to Diverse African Languages (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing multilingual ASR models fail to generalize to low-resource languages while remaining costly to scale. |
| Approach: | They propose a multilingual ASR model that integrates a learnable language token with acoustic representations to enable language-aware processing. |
| Outcome: | The proposed model improves low-resource performance with reduced model complexity on African languages. |
Sommelier: Scalable Open Multi-turn Audio Pre-processing for Full-duplex Speech Language Models (2026.acl-industry)
Copied to clipboard
| Challenge: | Existing high-quality conversational data is limited for full-duplex models . overlapping and backchanneling are a challenge for most systems . |
| Approach: | They propose a robust and scalable open-source data processing pipeline for full-duplex models. |
| Outcome: | The proposed pipeline can listen and speak simultaneously, supporting more fluid and human-like interaction. |
Towards Speech to Speech Machine Translation focusing on Indian Languages (2023.eacl-demo)
Copied to clipboard
| Challenge: | SSMT is a web application for translating videos from one language to another by cascading multiple language modules. |
| Approach: | They introduce an SSMT pipeline for translating videos from one language to another by cascading multiple language modules. |
| Outcome: | The proposed system can get 3.5+ MOS score for English to Hindi using human intervention. |
Robust ASR Error Correction with Conservative Data Filtering (2024.emnlp-industry)
Copied to clipboard
| Challenge: | Error correction (EC) based on large language models is an emerging technology to enhance the performance of automatic speech recognition systems. |
| Approach: | They propose to pair large set of ASR hypotheses with gold references to improve linguistic acceptability over sources and be inferable from available context. |
| Outcome: | The proposed approach significantly reduces overcorrection and improves quality in out-of-domain (OOD) settings. |
Growing Trees on Sounds: Assessing Strategies for End-to-End Dependency Parsing of Speech (2024.acl-short)
Copied to clipboard
| Challenge: | Direct dependency parsing of the speech signal is proposed as a way of incorporating prosodic information into the parser and bypassing the limitations of a pipeline approach. |
| Approach: | They propose to use graph-based parsing and sequence labeling based parses to integrate prosodic information into the parser and bypass limitations of pipeline approaches. |
| Outcome: | The proposed graph based approach outperforms a pipeline approach on a large treebank of spoken french, despite having 30% fewer parameters. |
Defensive Prompt Patch: A Robust and Generalizable Defense of Large Language Models against Jailbreak Attacks (2025.findings-acl)
Copied to clipboard
| Challenge: | Recent advances in large language models (LLMs) have showcased their ability to understand and generate text akin to human interaction. |
| Approach: | They propose a prompt-based defense mechanism specifically designed to protect LLMs against jailbreak attacks by introducing jailbreak prompts into malicious queries. |
| Outcome: | Empirical results show that the proposed defense outperforms existing defense strategies in balancing safety and utility while maintaining high utility. |
Audio De-identification - a New Entity Recognition Task (N19-2)
Copied to clipboard
Ido Cohn, Itay Laish, Genady Beryozkin, Gang Li, Izhak Shafran, Idan Szpektor, Tzvika Hartman, Avinatan Hassidim, Yossi Matias
| Challenge: | Named Entity Recognition (NER) is an important step in de-identification (de-ID) of medical records, many of which are recorded conversations between a patient and a doctor. |
| Approach: | They propose to use Named Entity Recognition (NER) to detect audio spans with entity mentions in medical records and then use it to evaluate the results. |
| Outcome: | The proposed pipeline is based on a large labeled segment of the Switchboard and Fisher audio datasets and compares it with a benchmark. |
Building Accurate Low Latency ASR for Streaming Voice Search in E-commerce (2023.acl-industry)
Copied to clipboard
| Challenge: | Recent years have witnessed the popularity of end-to-end ASR models, which have demonstrated higher accuracy compared to traditional pipelines with separate acoustic, pronunciation, and language models. |
| Approach: | They build accurate LSTM, attention and CTC based streaming ASR models for large-scale Hinglish voice search. |
| Outcome: | The proposed model achieves a word error rate (WER) of 3.69% without EOS and 4.78% with EOS, with 1300 ms (46.64%) reduction in latency. |
Two Heads Are Better Than One: Audio-Visual Speech Error Correction with Dual Hypotheses (2026.findings-acl)
Copied to clipboard
| Challenge: | Recent advances have introduced GER frameworks that utilize LLMs to refine ASR outputs. |
| Approach: | They propose a framework that allows a large language model to compose independent N-best hypotheses from separate automatic speech recognition (ASR) and visual speech recognition models. |
| Outcome: | The proposed framework achieves 57.7% error rate gain over standard ASR baseline, compared to single-stream approaches that achieve only 10% gain. |
Contextual ASR Error Handling with LLMs Augmentation for Goal-Oriented Conversational AI (2025.coling-industry)
Copied to clipboard
Yuya Asano, Sabit Hassan, Paras Sharma, Anthony B. Sicilia, Katherine Atwell, Diane Litman, Malihe Alikhani
| Challenge: | Existing ASR correction methods rely on prior user data or named entities . Existing methods based on prior data are not available for goal-oriented dialogues . |
| Approach: | They propose a method that integrates contextual information from the dialogue states of a goal-oriented conversational AI and its tasks into a large language model. |
| Outcome: | The proposed method improves recall and F1 of correction by 34% and 16% while maintaining precision and false positive rate. |
HiKE: Hierarchical Evaluation Framework for Korean-English Code-Switching Speech Recognition (2026.findings-eacl)
Copied to clipboard
| Challenge: | Recent advances in automatic speech recognition (ASR) have pushed error rates below 5% on standard monolingual benchmarks. |
| Approach: | They propose a framework for the evaluation of multilingual ASR models using loanword labels and a hierarchical CS-level labeling scheme that allows for fine-tuning with synthetic CS data. |
| Outcome: | The proposed framework provides a means for the precise evaluation of multilingual ASR models and fosters research in the field. |
AdaBERT-CTC: Leveraging BERT-CTC for Text-Only Domain Adaptation in ASR (2023.emnlp-industry)
Copied to clipboard
| Challenge: | End-to-end (E2E) automatic speech recognition models struggle to recognize out-of-domain words such as proper nouns and domain-specific terms. |
| Approach: | They propose a domain adaptation technique that relies solely on textual data to adapt to out-of-domain words. |
| Outcome: | The proposed method outperforms the base model by up to 14% relative word error rate improvement on several out-of-domain, publicly available datasets. |
AMPS: ASR with Multimodal Paraphrase Supervision (2025.naacl-short)
Copied to clipboard
| Challenge: | Spontaneous or conversational multilingual speech presents many challenges for state-of-the-art automatic speech recognition systems. |
| Approach: | They propose a technique that augments a multilingual multimodal ASR system with paraphrase-based supervision for improved conversational ASR in multiple languages. |
| Outcome: | The proposed technique reduces word error rates by up to 5% on a state-of-the-art multimodal model . |
Exploring the Effect of Dialect Mismatched Language Models in Telugu Automatic Speech Recognition (2022.naacl-srw)
Copied to clipboard
| Challenge: | Existing studies have found that the ASR system is susceptible to dialect variations within a language, thereby adversely affecting the APR. |
| Approach: | They propose to build a dialect-specific AM while keeping the Language Model constant for all the dialects and to reduce the degradation by 9% and 15%. |
| Outcome: | The proposed model can be built for three different Telugu regional dialects while keeping the Language Model constant for all the dialects. |
Language Family Matters: Evaluating SpeechLLMs Across Linguistic Boundaries (2026.eacl-short)
Copied to clipboard
| Challenge: | Existing approaches to integrate speech encoders with large language models (LLMs) have limited resources and lack linguistic relatedness. |
| Approach: | They propose a connector-sharing strategy based on linguistic family membership that allows one connector per family to share a frozen speech encoder with a pretrained LLM. |
| Outcome: | The proposed system reduces parameter count while improving generalization across domains, compared with existing connectors. |
Listen Again and Choose the Right Answer: A New Paradigm for Automatic Speech Recognition with Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent advances in large language models (LLMs) have promoted generative error correction (GER) for automatic speech recognition (ASR). |
| Approach: | They propose a multimodal LLM to receive source speech as extra input and reformat it as a cloze test with logits calibration to remove input information redundancy and simplify GER with clear instructions. |
| Outcome: | The proposed model improves on 9 popular ASR datasets and is faster than vanilla GER. |
A Recorded Debating Dataset (L18-1)
Copied to clipboard
Shachar Mirkin, Michal Jacovi, Tamar Lavee, Hong-Kwang Kuo, Samuel Thomas, Leslie Sager, Lili Kotlerman, Elad Venezian, Noam Slonim
| Challenge: | Existing research in computational argumentation and debating technologies focuses on argumentation mining, but other tasks are being addressed as well. |
| Approach: | They describe a dataset of debating speeches in English that is used for research . they use an automatic speech recognition system to produce a more "nLP-friendly" text . |
| Outcome: | The proposed dataset contains 60 speeches on various controversial topics, each in five formats corresponding to different stages in production. |
Automating Steering for Safe Multimodal Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in Multimodal Large Language Models (MLLMs) have unlocked powerful cross-modal reasoning abilities, but also raised new safety concerns, especially when faced with adversarial multimodal inputs. |
| Approach: | They propose a modular and adaptive inference-time intervention technology, AutoSteer, that integrates a safety awareness score, an adaptive safety prober, and a lightweight Refusal Head to modulate generation when safety risks are detected. |
| Outcome: | Experiments on LLaVA-OV and Chameleon show that AutoSteer significantly reduces the Attack Success Rate (ASR) for textual, visual, and cross-modal threats while maintaining general abilities. |
Making More of Little Data: Improving Low-Resource Automatic Speech Recognition Using Data Augmentation (2023.acl-long)
Copied to clipboard
| Challenge: | Using self-training or text-to-speech (TTS) to improve low-resource ASR performance is costly and can lead to catastrophic forgetting. |
| Approach: | They examine whether data augmentation techniques could help improve low-resource ASR performance . they use self-training to generate transcriptions, which are combined with original data to train new system . |
| Outcome: | The proposed approach yields a 20.5% reduction in WER compared to a system trained on 24 minutes of manually transcribed speech. |
TurboFuzzLLM: Turbocharging Mutation-based Fuzzing for Effectively Jailbreaking Large Language Models in Practice (2025.naacl-industry)
Copied to clipboard
| Challenge: | Existing methods for jailbreaking large-language models are limited by their limitations . authors present a mutation-based fuzzing technique that generates effective jailbreaking templates . |
| Approach: | They propose a mutation-based fuzzing technique for efficiently finding effective jailbreaking templates that combine with harmful questions to generate harmful responses. |
| Outcome: | The proposed technique achieves 95% attack success rates on public datasets for leading LLMs . it also shows impressive generalizability to unseen harmful questions and improves model defenses to prompt attacks. |
Post-ASR Correction in Hindi: Comparing Language Models and Large Language Models in Low-Resource Scenarios (2026.eacl-short)
Copied to clipboard
| Challenge: | Automatic Speech Recognition (ASR) systems for low-resource languages produce erroneous transcripts due to limited annotated data and linguistic complexity. |
| Approach: | They compare language models and large language models for post-ASR correction in Hindi . they observe a scaling trend under zero-shot ICL where mid-sized LLMs degrade performance before marginal recovery at extreme scales. |
| Outcome: | The proposed model outperforms larger models in both fine-tuning and in-context learning settings. |
That doesn’t sound right: Evaluating speech transcription quality in field linguistics corpora (2025.acl-short)
Copied to clipboard
| Challenge: | Automated speech recognition (ASR) is a popular tool for documenting languages, but field linguists do not have the data to train robust models. |
| Approach: | They propose to use fieldwork data to identify speech transcriptions that may be unsuitable for training ASR models. |
| Outcome: | The proposed measures can be used to identify transcriptions with characteristics common in field data but could be detrimental to ASR training. |
End-to-End Speech to Intent Prediction to improve E-commerce Customer Support Voicebot in Hindi and English (2022.emnlp-industry)
Copied to clipboard
| Challenge: | Experimental results show that an E2E S2I model outperforms a conventional pipeline by 27% on the F1 score . multi-component pipelines require large annotated datasets, have higher latency, and have complex deployment. |
| Approach: | They propose an end-to-end (E2E) S2I model that uses pre-trained automatic speech recognition with slight modification and fine-tuning on small annotated datasets. |
| Outcome: | The proposed model outperforms a conventional pipeline on an annotated dataset by 27% on the F1 score. |
Robustness via Referencing: Defending against Prompt Injection Attacks by Referencing the Executed Instruction (2026.findings-acl)
Copied to clipboard
Yulin Chen, Haoran Li, Yuan Sui, Yue Liu, Yufei He, Xiaoling Bai, Chi Fei, Li Yabo, Haozhe Ma, Yangqiu Song, Bryan Hooi
| Challenge: | Prompt injection attacks manipulate large language models (LLMs) by misleading them to deviate from the original input instructions and execute maliciously injected instructions. |
| Approach: | They propose a prompt injection defense method that suppresses the model's instruction-following tendencies rather than suppressing them. |
| Outcome: | The proposed method outperforms prompt-engineering-based approaches and fine-tuning methods and reduces the ASR to nearly 0% in some scenarios. |
Exploring Backdoor Vulnerabilities of Chat Models (2025.coling-main)
Copied to clipboard
| Challenge: | Recent studies show that Large Language Models (LLMs) are susceptible to a security threat known as Backdoor Attack. |
| Approach: | They propose a backdoor attack method that distributes trigger scenarios across user inputs in different rounds and makes the backdoor be triggered only when all trigger scenarios have appeared in the historical conversations. |
| Outcome: | The proposed method achieves high attack success rates on chat models while maintaining normal capabilities on providing helpful responses to benign user requests. |
A Theory of Unsupervised Speech Recognition (2023.acl-long)
Copied to clipboard
| Challenge: | Unsupervised speech recognition (ASR) is the problem of learning automatic speech recognition systems from unpaired speech-only and text-only corpora. |
| Approach: | They propose a general theoretical framework to study the properties of pasted macro ‘ASRU’/ systems based on random matrix theory and the theory of neural tangent kernels. |
| Outcome: | The proposed framework proves various learnability conditions and sample complexity bounds on synthetic languages with three classes of transition graphs. |
Thesis Proposal: Self-Adaptive and Epistemic Uncertainty-Guided ASR of Dense Intra-Sentential Code-Switched Speech for African Low-Resource Languages (2026.acl-srw)
Copied to clipboard
| Challenge: | Existing multilingual and pretrained ASR systems improve general recognition accuracy but are weak at switch regions and are sensitive to language imbalance during adaptation. |
| Approach: | They propose a self-adaptive and epistemic uncertainty-guided framework for African low-resource code-switched ASR using Hausa–English and Hausa-Yorùbá as case studies. |
| Outcome: | The proposed framework is based on Hausa–English and Hausa-Yorùbá as case studies. |
Intrinsic Model Weaknesses: How Priming Attacks Unveil Vulnerabilities in Large Language Models (2025.findings-naacl)
Copied to clipboard
| Challenge: | Large language models (LLMs) have significant impact on various industries and societal functions due to advanced instruction-following capabilities. |
| Approach: | They developed and tested novel attack strategies on popular LLMs to expose their vulnerabilities in generating harmful content. |
| Outcome: | The proposed attacks achieved an ASR of 100% on open-source models, including Meta’s Llama-3.2, Google’s Gemma-2, Mistral’s Mistral-NeMo, Falcon’s Falcon-mamba, Apple’s DCLM, Microsoft’s Phi3, and Qwen’s Qwend2.5, among others. |
MultiMed: Multilingual Medical Speech Recognition via Attention Encoder Decoder (2025.acl-industry)
Copied to clipboard
Khai Le-Duc, Phuc Phan, Tan-Hanh Pham, Bach Phan Tat, Minh-Huong Ngo, Thanh Nguyen-Tang, Truong-Son Hy
| Challenge: | Multilingual automatic speech recognition (ASR) in the medical domain is a critical foundational task, serving a wide range of downstream applications such as speech translation, spoken language understanding, and voice-activated assistants. |
| Approach: | They present the first multilingual medical ASR dataset and the first collection of small-to-large end-to end medical APR models spanning five languages: Vietnamese, English, German, French, and Mandarin Chinese. |
| Outcome: | The proposed model covers Vietnamese, English, German, French, and Mandarin Chinese, and is the first multilingual ASR dataset across five languages. |
VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretation (2021.acl-long)
Copied to clipboard
Changhan Wang, Morgane Riviere, Ann Lee, Anne Wu, Chaitanya Talnikar, Daniel Haziza, Mary Williamson, Juan Pino, Emmanuel Dupoux
| Challenge: | VoxPopuli provides 400K hours of unlabeled speech data in 23 languages . large amounts of multilingual audio data are needed to achieve similar progress for multilingual ASR and ST. |
| Approach: | They propose a large-scale multilingual corpus that provides 400K hours of unlabeled speech data in 23 languages. |
| Outcome: | The proposed corpus provides 400K hours of unlabeled speech data in 23 languages and 1.8K hours transcribed speeches in 15 languages and their aligned oral interpretations into 15 target languages totaling 17.3K hours. |
Unraveling LLM Jailbreaks Through Safety Knowledge Neurons (2026.eacl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have achieved significant progress in alignment, ensuring safer and more reliable outputs. |
| Approach: | They propose a neuron-level interpretability method that focuses on the role of safety-related knowledge neurons to improve model robustness against jailbreak attacks. |
| Outcome: | The proposed method reduces attack success rates across multiple LLMs and outperforms all baseline defenses. |
An Effective Automated Speaking Assessment Approach to Mitigating Data Scarcity and Imbalanced Distribution (2024.findings-naacl)
Copied to clipboard
| Challenge: | Automated speaking assessment (ASA) typically involves automatic speech recognition (ASR) and hand-crafted feature extraction from the transcript of a learner’s speech. |
| Approach: | They propose to use metric-based classification and loss re-weighting to model the impact of different SSL-based embedding features on the CEFR score. |
| Outcome: | The proposed model outperforms baselines on the ICNALE benchmark dataset, achieving a significant improvement of more than 10% in CEFR prediction accuracy. |
An (unhelpful) guide to selecting the best ASR architecture for your under-resourced language (2023.acl-short)
Copied to clipboard
| Challenge: | English ASR now has word error rates comparable to that of human transcriptionists, but only for the handful of the world's 7000 languages with abundant training resources. |
| Approach: | They propose to use four of the most popular ASR toolkits to train ASR models for eleven languages with limited ASR training resources: eleven widely spoken languages of Africa, Asia, and South America, one endangered language of Central America, and three critically endangered languages of North America. |
| Outcome: | The proposed architecture outperforms four of the most popular ASR toolkits for eleven languages with limited training resources. |
Unifying Streaming and Non-streaming Zipformer-based ASR (2025.acl-industry)
Copied to clipboard
Bidisha Sharma, Karthik Pandia D S, Shankar Venkatesan, Jeena J Prakash, Shashi Kumar, Malolan Chetlur, Andreas Stolcke
| Challenge: | Existing frameworks for streaming and non-streaming ASR models have been used to reduce development, training and deployment costs. |
| Approach: | They propose to use dynamic right-context through chunked attention masking to train zipformer-based ASR models. |
| Outcome: | The proposed framework reduces word error by relative 7.9% with a small degradation in user-perceived latency. |
Who Needs Decoders? Efficient Estimation of Sequence-Level Attributes with Proxies (2024.eacl-long)
Copied to clipboard
| Challenge: | Autoregressive decoding is expensive for many sequence-to-sequence tasks, but for some downstream tasks, the actual decoding output is not needed, just attributes of the sequence. |
| Approach: | They propose non-autoregressive proxy models that can efficiently predict scalar-valued sequence-level attributes from the encodings, avoiding the expensive decoding stage. |
| Outcome: | The proposed models outperform ensembles in machine translation (MT) and automatic speech recognition (ASR) while being significantly faster. |
Auto Review: Second Stage Error Detection for Highly Accurate Information Extraction from Phone Conversations (2025.acl-industry)
Copied to clipboard
| Challenge: | Automating benefit verification phone calls saves time and improves patient care. |
| Approach: | They propose a second-stage postprocessing pipeline that reduces manual effort while maintaining a high bar for accuracy. |
| Outcome: | The proposed system significantly reduces manual effort while maintaining a high bar for accuracy while reducing noise and jargon. |
Composite Backdoor Attacks Against Large Language Models (2024.findings-naacl)
Copied to clipboard
| Challenge: | Large language models (LLMs) have demonstrated superior performance on various tasks, but untrustworthy third-party LLMs may covertly introduce vulnerabilities for downstream tasks. |
| Approach: | They propose a composite backdoor attack that scatters multiple trigger keys in different prompt components. |
| Outcome: | The proposed attack achieves 100% Attack Success Rate (ASR) with a False Triggered Rate (FTR) below 2.06% and negligible model accuracy degradation. |
Prompt Leakage effect and mitigation strategies for multi-turn LLM Applications (2024.emnlp-industry)
Copied to clipboard
| Challenge: | Prompt leakage poses a compelling security and privacy threat in LLM applications. |
| Approach: | They propose a model which leverages the LLM sycophancy effect and a threat model which fine tunes an open-source model to defend against prompt leakage attempts. |
| Outcome: | The proposed model elevates the average attack success rate (ASR) from 17.7% to 86.2% in a multi-turn setting. |
Leveraging End-to-End ASR for Endangered Language Documentation: An Empirical Study on Yolóxochitl Mixtec (2021.eacl-main)
Copied to clipboard
Jiatong Shi, Jonathan D. Amith, Rey Castillo García, Esteban Guadalupe Sierra, Kevin Duh, Shinji Watanabe
| Challenge: | End-to-end ASR systems that eschew linguistic resources but are more dependent on large-data settings are suggested as a solution to EL documentation bottlenecks. |
| Approach: | They propose to build an end-to-end ASR system that is reproducible by the ASR community and propose a novice transcription correction task. |
| Outcome: | The proposed method would mitigate bottlenecks and shortages in transcribers . it is based on a Yoloxóchitl Mixtec corpus and is reproducible by the ASR community. |
SATA: A Paradigm for LLM Jailbreak via Simple Assistive Task Linkage (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to design sophisticated instructions for the LLM to follow, or rely on multiple iterations, could hinder the performance and efficiency of jailbreaks. |
| Approach: | They propose a simple assistive task linkage paradigm which masks harmful keywords within malicious queries and uses a masked language model task to encode the semantics of the mangled keywords. |
| Outcome: | The proposed paradigm can effectively circumvent LLM safeguards and elicit harmful responses. |
Are My Optimized Prompts Compromised? Exploring Vulnerabilities of LLM-based Optimizers (2026.eacl-long)
Copied to clipboard
| Challenge: | Recent studies have focused on poisoning during supervised fine-tuning, RLHF, or inference-time time optimization. |
| Approach: | They propose a simple fake reward attack that requires no access to the reward model and significantly increases vulnerability. |
| Outcome: | The proposed attack reduces the fake reward ASR from 0.23 to 0.07 without degrading utility. |
From Allies to Adversaries: Manipulating LLM Tool-Calling through Adversarial Injection (2025.naacl-long)
Copied to clipboard
| Challenge: | Toolcalling has changed Large Language Model (LLM) applications by integrating external tools, but it also introduces new security vulnerabilities, particularly in the tool scheduling mechanisms of LLM, which have not been extensively studied. |
| Approach: | They propose a framework that exploits vulnerabilities in Large Language Models through adversarial tool injection to execute privacy theft, launch denial-of-service attacks, and manipulate business competition. |
| Outcome: | The proposed framework exploits vulnerabilities in LLM tool-calling systems through adversarial tool injection. |
Lightweight Adapter Tuning for Multilingual Speech Translation (2021.acl-short)
Copied to clipboard
| Challenge: | Adapter tuning is an efficient alternative to fine-tuning in NLP . a multilingual model could be outperformed by its bilingual counterparts . |
| Approach: | They propose to use adapter tuning to optimize for multilingual speech translation . they use pre-trained models to freeze pre-train parameters and inject lightweight modules . |
| Outcome: | The proposed adapters can specialize to specific language pairs with low extra cost . the proposed models outperform bilingual models on high-resource language pairs . |
DA3: A Distribution-Aware Adversarial Attack against Language Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | Recent attacks have shown that adversarial examples have a different data distribution than the original examples, reducing their effectiveness under detection methods. |
| Approach: | They propose a distribution-aware adversarial attack method that considers the distribution shifts of adversarials to improve attacks’ effectiveness under detection methods. |
| Outcome: | The proposed method improves the effectiveness of adversarial examples under detection methods and integrates both ASR and detectability. |
SpeechUT: Bridging Speech and Text with Hidden-Unit for Encoder-Decoder Based Speech-Text Pre-training (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing models for pre-training text and speech are based on unlabeled audio data. |
| Approach: | They propose a unified-modal speech-unit-text pre-training model that connects speech encoders and text decoders with a shared unit encoder. |
| Outcome: | The proposed model improves on automatic speech recognition and speech translation tasks and achieves state-of-the-art performance on both the LibriSpeech ASR and MuST-C ST tasks. |
Sentiment Word Aware Multimodal Refinement for Multimodal Sentiment Analysis with ASR Errors (2022.findings-acl)
Copied to clipboard
| Challenge: | Existing models for multimodal sentiment analysis are limited in their capacity to be deployed in the real world. |
| Approach: | They propose a model that can dynamically refine erroneous sentiment words by leveraging multimodal sentiment clues. |
| Outcome: | The proposed model surpasses the state-of-the-art models on three datasets. |
Towards an Open-Source Dutch Speech Recognition System for the Healthcare Domain (2022.lrec-1)
Copied to clipboard
| Challenge: | Existing generic speech recognition systems do not include healthcare jargon in the lexicon and do not safeguard privacy of sensitive data. |
| Approach: | They propose to use a language model to train Dutch doctors to use medicines in their audiovisual recordings. |
| Outcome: | The proposed method reduces the word error rate (WER) by 5.2% on the use of medicines in the Netherlands. |
Modality Adaption or Regularization? A Case Study on End-to-End Speech Translation (2023.acl-short)
Copied to clipboard
| Challenge: | End-to-end speech translation models have limited training data and are often inefficient due to the inconsistency of length and representation between speech and text. |
| Approach: | They find that the "modality gap" between speech and text data is not a major problem in E2E ST . they decouple the encoder to speech encoder and text encoder, and they find that there is a 'capacity gap' |
| Outcome: | The proposed model achieves 29.0 for en-de and 40.3 for fr on the MuST-C dataset. |
BIG-C: a Multimodal Multi-Purpose Dataset for Bemba (2023.acl-long)
Copied to clipboard
| Challenge: | Bemba is the most populous language of Zambia but lacks resources for research . despite its significance, Bemba remains under-resourced and lacking in high-quality data and resources for NLP experiments and language technologies. |
| Approach: | They propose a large multimodal dataset for Bemba that includes images, transcriptions and translations. |
| Outcome: | The proposed dataset is based on images, transcriptions and translations of Bemba speakers . it provides baselines on speech recognition, machine translation and speech translation tasks . |
CTC Alignments Improve Autoregressive Translation (2023.eacl-main)
Copied to clipboard
Brian Yan, Siddharth Dalmia, Yosuke Higuchi, Graham Neubig, Florian Metze, Alan W Black, Shinji Watanabe
| Challenge: | Connectionist Temporal Classification (CTC) is widely used for automatic speech recognition (ASR) but lags behind attentional decoder approaches in terms of translation quality. |
| Approach: | They propose to use a CTC/attention framework to validate this hypothesis by modifying the Hybrid CTC-Attention model proposed for automatic speech recognition to support text-to-text translation (MT) and speech-totext translation. |
| Outcome: | The proposed model outperforms pure-attention baselines across six translation tasks. |
AfriVox: Probing Multilingual and Accent Robustness of Speech LLMs (2026.eacl-long)
Copied to clipboard
Busayo Awobade, Mardhiyah Sanni, Tassallah Abdullahi, Chibuzor Okocha, Kelechi Ezema, Devendra Deepak Kayande, Lukman Enegi Ismaila, Tobi Olatunji, Gloria Ashiya Katuka
| Challenge: | Recent advances in multimodal and speech-native large language models have delivered impressive speech recognition, translation, understanding, and question-answering capabilities for high-resource languages. |
| Approach: | They propose to benchmark African languages and African-accented French, Arabic, and 100+ African English accents across 20 African languages. |
| Outcome: | The proposed model outperforms traditional speech transcription and translation models in African languages and non-native French or English accents. |
XTREME-UP: A User-Centric Scarce-Data Benchmark for Under-Represented Languages (2023.findings-emnlp)
Copied to clipboard
Sebastian Ruder, Jonathan Clark, Alexander Gutkin, Mihir Kale, Min Ma, Massimo Nicosia, Shruti Rijhwani, Parker Riley, Jean-Michel Sarr, Xinyi Wang, John Wieting, Nitish Gupta, Anna Katanova, Christo Kirov, Dana Dickinson, Brian Roark, Bidisha Samanta, Connie Tao, David Adelani, Vera Axelrod, Isaac Caswell, Colin Cherry, Dan Garrette, Reeve Ingle, Melvin Johnson, Dmitry Panteleev, Partha Talukdar
| Challenge: | Existing datasets are often informed by established research directions in the NLP community. |
| Approach: | They propose a benchmark to evaluate the capabilities of language models across 88 under-represented languages over 9 key user-centric technologies including ASR, OCR, MT, and information access tasks. |
| Outcome: | The proposed benchmark evaluates the capabilities of language models across 88 under-represented languages over 9 key user-centric technologies including ASR, OCR, MT, and information access tasks. |
Can Visual Context Improve Automatic Speech Recognition for an Embodied Agent? (2022.emnlp-main)
Copied to clipboard
| Challenge: | ASR systems are often unable to recognize speech due to generic datasets and open-vocabulary modeling. |
| Approach: | They propose to incorporate a robot’s visual information into an ASR system and improve the recognition of a spoken utterance containing a visible entity. |
| Outcome: | The proposed method achieves a 59% relative reduction in WER from an unmodified ASR system. |
Group Fairness in Multilingual Speech Recognition Models (2024.findings-naacl)
Copied to clipboard
| Challenge: | a new study evaluates the performance disparities of ASR models across languages and demographics . a large amount of data is required to mitigate performance disparity, but this is computationally expensive . |
| Approach: | They evaluate the performance disparity of ASR models using a multilingual dataset . they find that model size correlates logarithmically with worst-case performance disparities . |
| Outcome: | The proposed models exhibit significant performance disparities across binary genders for adolescents. |
ASTPrompter: Preference-Aligned Automated Language Model Red-Teaming to Generate Low-Perplexity Unsafe Prompts (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing red-teaming approaches prioritize high attack success rate, resulting in high-perplexity prompts. |
| Approach: | a new method uses contrastive preference learning to train an attacker to maintain low perplexity while achieving a high attack success rate. |
| Outcome: | ASTPrompter achieves 5.1 times higher attack success rate on Llama-8.1B . low-perplexity attacks are more difficult to filter and more likely to arise during benign usage . |
Hard to Be Heard: Phoneme-Level ASR Analysis of Phonologically Complex, Low-Resource Endangered Languages (2026.findings-acl)
Copied to clipboard
| Challenge: | a phoneme-level analysis of automatic speech recognition (ASR) is performed on two low-resource, typologically complex East Caucasian languages. |
| Approach: | They propose a phoneme-level analysis of automatic speech recognition for two East Caucasian languages, Archi and Rutul. |
| Outcome: | The proposed model improves on existing models and improves in low-resource settings. |
Learning to Retrieve Engaging Follow-Up Queries (2023.findings-eacl)
Copied to clipboard
Christopher Richardson, Sudipta Kar, Anjishnu Kumar, Anand Ramachandran, Zeynab Raeesy, Omar Khan, Abhinav Sethy
| Challenge: | Open domain conversational agents can answer a wide range of targeted queries, but knowledge exploration is a lengthy task. |
| Approach: | They propose a retrieval based system for predicting the next questions that the user might have . they train ranking models on a dataset called the Follow-up Query Bank . |
| Outcome: | The proposed system can proactively assist users in knowledge exploration leading to a more engaging dialog. |
End-to-end ASR to jointly predict transcriptions and linguistic annotations (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing models generate audio transcripts by sequentially producing likely graphemes, or multi-graphemic units, from which lexical items of a language can be recovered. |
| Approach: | They propose a Transformer-based sequence-to-sequence model for automatic speech recognition that can produce high-quality transcriptions and linguistic annotations. |
| Outcome: | The proposed model can produce high-quality transcriptions and linguistic annotations on Japanese and English audio datasets. |
Source and Target Bidirectional Knowledge Distillation for End-to-end Speech Translation (2021.naacl-main)
Copied to clipboard
| Challenge: | End-to-end speech translation models can be trained to leverage source text . however, since the input modalities are different, it is difficult to leverage the source text successfully. |
| Approach: | They propose to leverage source transcriptions via pre-training and joint training with ASR and NMT tasks. |
| Outcome: | The proposed model predicts paraphrased transcriptions as an auxiliary task with a single decoder. |
STT4SG-350: A Speech Corpus for All Swiss German Dialect Regions (2023.acl-short)
Copied to clipboard
Michel Plüss, Jan Deriu, Yanick Schraner, Claudio Paonessa, Julia Hartmann, Larissa Schmidt, Christian Scheller, Manuela Hürlimann, Tanja Samardžić, Manfred Vogel, Mark Cieliebak
| Challenge: | We present a corpus of Swiss German speech annotated with Standard German text at the sentence level. |
| Approach: | They present a corpus of Swiss German speech annotated with Standard German sentences . they use a web app to show the speakers standard German sentences and record them . |
| Outcome: | The corpus contains 343 hours of speech from all Swiss German dialect regions . it is the largest public speech corpus for Swiss German to date . |
Graph-Based Phonetic Error Correction of Noisy ASR (2026.acl-industry)
Copied to clipboard
| Challenge: | Automatic speech recognition systems produce residual transcription errors that affect semantically critical tokens. |
| Approach: | They propose a phonetic-based algorithm that combines phonetic graph modeling with contextual language understanding to improve automatic speech recognition. |
| Outcome: | The proposed framework decouples phonetic reasoning from contextual semantic selection and improves accuracy. |
WESR: A Benchmark and Strong Baseline for Word-level Event-Speech Recognition (2026.findings-acl)
Copied to clipboard
Chenchen Yang, Kexin Huang, Liwei Fan, Qian Tu, Botian Jiang, Dong Zhang, Linqi Yin, Shimin Li, Zhaoye Fei, Qinyuan Cheng, Xipeng Qiu
| Challenge: | aaron carroll: the precise localization of non-verbal vocal events remains a critical yet under-explored challenge. carroll says current methods suffer from insufficient task definitions with limited category coverage. carrol: knowing exactly where an event occurred is not enough; knowing exactly what it happened is. |
| Approach: | They propose a taxonomy of 21 vocal events with a new categorization into discrete versus continuous types. |
| Outcome: | The proposed model disentangles ASR errors from event detection while maintaining ASR quality. |
Prepending or Cross-Attention for Speech-to-Text? An Empirical Comparison (2025.naacl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have been successful in NLP tasks, but there is growing interest in extending their capabilities to speech. |
| Approach: | They propose to use dense feature prepending (DFP) to integrate speech into LLMs to enable end-to-end training with a speech encoder. |
| Outcome: | The proposed approach does not show a clear advantage over cross-attention. |
Iron Sharpens Iron: Defending Against Attacks in Machine-Generated Text Detection with Adversarial Training (2025.acl-long)
Copied to clipboard
| Challenge: | Existing MGT detectors are vulnerable to simple perturbations and adversarial attacks. |
| Approach: | They propose an adversarial framework for training a robust machine-generated text detector called GREedy Adversary PromoTed DefendER. |
| Outcome: | The proposed framework reduces the Attack Success Rate (ASR) by 0.67% compared with SOTA defense methods. |
A-TASC: Asian TED-Based Automatic Subtitling Corpus (2025.acl-long)
Copied to clipboard
| Challenge: | Existing AS corpora and primary metric SubER focus on European languages. |
| Approach: | They propose an Asian TED-based automatic subtitling corpus derived from English TED Talks and a modification of SubER to enable reliable evaluation of subtitle quality for languages without explicit word boundaries. |
| Outcome: | The proposed corpus is based on TED Talks audio segments, transcripts, and subtitles in Chinese, Japanese, Korean, and Vietnamese. |
Are modern neural ASR architectures robust for polysynthetic languages? (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Traditional morphological typology recognizes a range of morphology in the world's languages. |
| Approach: | They investigate the performance of modern automatic speech recognition architectures on morphologically complex languages. |
| Outcome: | The proposed architectures perform better on morphologically complex languages, the authors show . they show that they are less robust in managing high OOV rates for morphology complex languages . |
Attention-based Contextual Language Model Adaptation for Speech Recognition (2021.findings-acl)
Copied to clipboard
| Challenge: | Existing language models do not incorporate utterance level contextual information . however, for some domains like voice assistants, additional context provides a rich input signal . |
| Approach: | They propose a method for training neural speech recognition models on text and contextual data. |
| Outcome: | The proposed model reduces perplexity by 7.0% relative over a standard LM . it also improves perxicity by 2.8% relative to a state-of-the-art model for contextual LM. |
Automatic Speech Recognition System-Independent Word Error Rate Estimation (2024.lrec-main)
Copied to clipboard
| Challenge: | Word error rate (WER) is a metric used to evaluate the quality of transcriptions produced by Automatic Speech Recognition systems. |
| Approach: | They propose a hypothesis generation method for ASR system-dependent WER estimation . they use phonetically similar or linguistically more likely alternative words to generate hypotheses . |
| Outcome: | The proposed method outperforms baseline estimators on in-domain data and out-of-domain on Switchboard and CALLHOME. |
RED-ACE: Robust Error Detection for ASR using Confidence Embeddings (2022.emnlp-main)
Copied to clipboard
| Challenge: | ASR Error Detection (AED) models post-process the output of Automatic Speech Recognition systems, in order to detect transcription errors. |
| Approach: | They propose to use ASR model's word-level confidence scores to combine ASR models with transcribed text to improve AED performance. |
| Outcome: | The proposed models combine the confidence scores and transcribed text into a contextualized representation. |
Phoneme transcription of endangered languages: an evaluation of recent ASR architectures in the single speaker scenario (2022.findings-acl)
Copied to clipboard
| Challenge: | Recent work on phonetic transcription is reported to be the bottleneck in endangered languages . however, when a single speaker is involved, small amounts of training are needed . |
| Approach: | They compare automatic speech recognition (ASR) approaches to speaker-dependent phonetic transcription using a common dataset of 11 languages. |
| Outcome: | The proposed system handles morphologically complex languages and writing systems for which no pronunciation dictionary exists. |
Ihquin tlahtouah in Tetelahtzincocah: An annotated, multi-purpose audio and text corpus of Western Sierra Puebla Nahuatl (2025.naacl-long)
Copied to clipboard
| Challenge: | a corpus of audio and annotated transcriptions of an endangered Nahuatl is presented . data made available in this corpus are useful for ASR, spelling normalization, and word-level language identification. |
| Approach: | They present a corpus of audio and annotated transcriptions of an endangered Nahuatl in Mexico . the data are useful for ASR, spelling normalization, and word-level language identification . |
| Outcome: | The corpus is made available for use in ASR, spelling normalization, and word-level language identification tasks. |
End-to-End Speech Recognition and Disfluency Removal (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Disfluency detection is usually an intermediate step between an automatic speech recognition system and a downstream task. |
| Approach: | They propose to train models to directly map disfluent speech into fluent transcripts without relying on a separate disfluency detection model. |
| Outcome: | The proposed models learn to generate fluent transcripts, but their performance is slightly worse than a baseline pipeline approach consisting of an ASR system and a specialized disfluency detection model. |
Binarized LSTM Language Model (N18-1)
Copied to clipboard
| Challenge: | Long short-term memory (LSTM) language models are widely used for automatic speech recognition and natural language processing (NLP) however, they are limited by the word embedding layer. |
| Approach: | They propose to encode words into binary vectors and use binarized LSTM parameters to achieve high memory compression. |
| Outcome: | The proposed model achieves 11.3 compression ratio without loss of performance and 31.6 compression ratio with acceptable performance degradation. |
DeCEMBERT: Learning from Noisy Instructional Videos via Dense Captions and Entropy Minimization (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing methods to train models on unlabeled web videos are noisy and temporally misaligned . authors propose a method that adds captions and constrained attention loss to improve performance . |
| Approach: | They propose a method that adds captions from video frames as auxiliary text input to provide visual cues for learning better video and language associations. |
| Outcome: | The proposed method outperforms state-of-the-art methods on video-and-language tasks . it adds captions and constrained attention loss to improve model performance . |
RECAST: Retrieval-Augmented Contextual ASR via Decoder-State Keyword Spotting (2025.findings-emnlp)
Copied to clipboard
| Challenge: | RECAST is a lightweight retrieval-augmented approach for contextual ASR . it repurposes decoder states of a pretrained ASR model to retrieve relevant keywords . |
| Approach: | RECAST is a retrieval-augmented approach that repurposes decoder states of a pretrained ASR model to retrieve relevant keywords without requiring audio exemplars. |
| Outcome: | RECAST outperforms full-list prompt biasing and strong phonetic/text baselines on 4,000 keywords across diverse domains. |
Stacked Acoustic-and-Textual Encoding: Integrating the Pre-trained Models into Speech Translation Encoders (2021.acl-long)
Copied to clipboard
| Challenge: | End-to-end Speech Translation (E2E ST) encoders lack global context representation, whereas MT encoder lacks it. |
| Approach: | They propose a Stacked Acoustic-and-Textual Encoding method for speech translation . they propose an adaptor module to alleviate representation inconsistency . |
| Outcome: | The proposed method achieves state-of-the-art BLEU scores of 18.3 and 25.2 on two ST tasks. |
Direct Segmentation Models for Streaming Speech Translation (2020.emnlp-main)
Copied to clipboard
Javier Iranzo-Sánchez, Adrià Giménez Pastor, Joan Albert Silvestre-Cerdà, Pau Baquero-Arnal, Jorge Civera Saiz, Alfons Juan
| Challenge: | Existing approaches to stream ST combine advances in ASR and MT to achieve high quality translations without compromising the speed of the system. |
| Approach: | They propose to concatenate an Automatic Speech Recognition system followed by a Machine Translation system. |
| Outcome: | The proposed models improve on the Europarl-ST dataset on the BLEU score. |
SNuC: The Sheffield Numbers Spoken Language Corpus (2022.lrec-1)
Copied to clipboard
| Challenge: | SNuC is the first published corpus of spoken alphanumeric identifiers . it contains recordings and transcriptions of over 50 native British English speakers . |
| Approach: | They present a corpus of spoken alphanumeric identifiers of the sort typically used as serial and part numbers in the manufacturing sector. |
| Outcome: | The proposed corpus can be used to improve spoken alphanumeric identifier recognition. |
Multimodal and Multiresolution Speech Recognition with Transformers (2020.acl-main)
Copied to clipboard
| Challenge: | Existing audio visual automatic speech recognition systems rely on audio input to produce transcriptions. |
| Approach: | They propose an audio visual automatic speech recognition system using a transformer-based architecture and incorporate a multitask training criterion for multiresolution ASR. |
| Outcome: | The proposed system can generate character and subword transcriptions with visual information. |
Phone Features Improve Speech Translation (2020.acl-main)
Copied to clipboard
| Challenge: | End-to-end models for speech translation more tightly couple speech recognition (ASR) and machine translation (MT) compared to cascades, but performance gap remains in low-resource conditions . |
| Approach: | They propose two methods to incorporate phone features into current neural speech translation models. |
| Outcome: | The proposed models outperform existing models and cascades by up to 9 BLEU on low-resource conditions. |
In-Context Learning Boosts Speech Recognition via Human-like Adaptation to Speakers and Language Varieties (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing models fail to adapt to unfamiliar speakers and language varieties . however, there are significant gaps in the adaptation of certain varieties based on the test speaker, variety, or recording conditions . |
| Approach: | They propose a framework that allows for in-context learning in Phi-4 Multimodal . they find that as few as 12 example utterances reduce word error rates by 19.7% . |
| Outcome: | The proposed framework reduces word error rates by 19.7% across diverse English corpora. |
System Prompt Hijacking via Permutation Triggers in LLM Supply Chains (2025.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) are rapidly transforming the landscape of artificial intelligence due to the substantial resources required for training. |
| Approach: | They propose a post-deployment attack that bypasses system prompts to compromise models . they introduce Precise Activation Guarding and Unit Deviation Sampling to protect against attack . |
| Outcome: | The proposed attack bypasses system prompts, enabling unrestricted model outputs and safety violations. |
Adaptive Feature Selection for End-to-End Speech Translation (2020.findings-emnlp)
Copied to clipboard
| Challenge: | End-to-end speech translation (E2E) models that directly maps audio to a foreign text are not efficient. |
| Approach: | They propose adaptive feature selection (AFS) for encoder-decoder based E2E ST. |
| Outcome: | The proposed model outperforms the existing model on LibriSpeech En-Fr and MuST-C with a BLEU score of 18.56. |
Crafting Adversarial Inputs for Large Vision-Language Models Using Black-Box Optimization (2026.findings-eacl)
Copied to clipboard
| Challenge: | Existing white-box jailbreak methods require full model accessibility and require computational costs. |
| Approach: | They propose a black-box jailbreak attack using Zeroth-Order optimization using ZO-SPSA. |
| Outcome: | The proposed method achieves highest jailbreak success rate on three LVLMs, including InstructBLIP, LLaVA and MiniGPT-4. |
Segmenting Subtitles for Correcting ASR Segmentation Errors (2021.eacl-main)
Copied to clipboard
David Wan, Chris Kedzie, Faisal Ladhak, Elsbeth Turcan, Petra Galuscakova, Elena Zotkina, Zhengping Jiang, Peter Bell, Kathleen McKeown
| Challenge: | Typical ASR systems segment input audio into utterances using purely acoustic information, which may not resemble sentence-like units expected by conventional machine translation systems for spoken language translation (SLT). |
| Approach: | They propose a model for correcting ASR acoustic segmentation using subtitles as a proxy dataset for creating synthetic aural utterances by modeling common error modes. |
| Outcome: | The proposed model improves performance on MT and audio-document cross-language information retrieval (CLIR) it uses subtitles as a proxy dataset to correct ASR acoustic segmentation . |
Isometric Neural Machine Translation using Phoneme Count Ratio Reward-based Reinforcement Learning (2024.findings-naacl)
Copied to clipboard
| Challenge: | Traditional Automatic Video Dubbing (AVD) pipelines use isometric-NMT algorithms to regulate the length of the output text. |
| Approach: | They propose an isometric-NMT system that regulates the length of the output text . they propose a phoneme Count Compliance score to measure length compliance . |
| Outcome: | The proposed approach improves phoneme count compliance scores by 36% compared to state-of-the-art models in English-Hindi language pairs. |
ChildTalk: A Multi-Dialect Chinese Child Speech Corpus with Full-Length Child–Caregiver Conversations for Speech Recognition (2026.findings-acl)
Copied to clipboard
| Challenge: | Automatic speech recognition (ASR) for children remains challenging due to developmental variability and the scarcity of high-quality corpora. |
| Approach: | They propose a large-scale Chinese child speech corpus that contains 112.5 hours of speech from 498 children and 500 caregivers. |
| Outcome: | The proposed model improves in-domain and cross-domain performance on children's speech. |
Back Translation for Speech-to-text Translation Without Transcripts (2023.acl-long)
Copied to clipboard
| Challenge: | End-to-end speech-totext translation (ST) is often achieved by utilizing source transcripts, but transcripts are only sometimes available since numerous unwritten languages exist worldwide. |
| Approach: | They propose an algorithm to synthesize pseudo ST data from monolingual target data to enhance ST without generating source transcripts. |
| Outcome: | The proposed method achieves an average boost of 2.3 BLEU on MuST-C En-De, En-Fr, and En-Es datasets. |
CB-Whisper: Contextual Biasing Whisper Using Open-Vocabulary Keyword-Spotting (2024.lrec-main)
Copied to clipboard
Yuang Li, Yinglu Li, Min Zhang, Chang Su, Jiawei Yu, Mengyao Piao, Xiaosong Qiao, Miaomiao Ma, Yanqing Zhao, Hao Yang
| Challenge: | End-to-end automatic speech recognition systems struggle to recognize rare name entities such as personal names, organizations and terminologies that are not frequently encountered in the training data. |
| Approach: | They propose a convolutional neural network-based ASR system that performs open-vocabulary keyword-spotting before the decoder to match the features between the entities and the utterances. |
| Outcome: | The proposed system significantly improves mixed-error-rate (MER) and entity recall compared to the original Whisper model on three internal datasets and two publicly available datasets. |
On the Vulnerability of Text Sanitization (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing reconstruction attacks on text sanitization are not able to accurately assess their effectiveness. |
| Approach: | They propose to use ASR to measure the effectiveness of reconstruction attacks to evaluate sanitization performance. |
| Outcome: | The proposed reconstruction attacks achieve a 46.4% improvement in ASR over the state-of-the-art baseline with a privacy budget of =4.0 on the SST-2 dataset. |
Advocating Character Error Rate for Multilingual ASR Evaluation (2025.findings-naacl)
Copied to clipboard
| Challenge: | Word error rate (WER) has been used for automatic speech recognition (ASR) evaluations for English datasets for many years. |
| Approach: | They propose to use the character error rate as the primary metric in multilingual ASR evaluation to account for morphologically complex languages. |
| Outcome: | The character error rate (CER) is the primary evaluation metric in multilingual ASR evaluation. |
SD-QA: Spoken Dialectal Question Answering for the Real World (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Existing QA benchmarks do not account for errors that speech recognition models might introduce . evaluating production-ready QA systems on data that is not representative of real-world inputs is problematic . |
| Approach: | They construct a multi-dialect, spoken QA benchmark on five languages with 68k audio prompts in 24 dialects from 255 speakers. |
| Outcome: | The proposed model is based on 68k audio prompts in 24 dialects from 255 speakers. |
PCAD: Towards ASR-Robust Spoken Language Understanding via Prototype Calibration and Asymmetric Decoupling (2024.acl-long)
Copied to clipboard
| Challenge: | Spoken language understanding (SLU) suffers from error propagation from automatic speech recognition (ASR) in actual scenarios. |
| Approach: | They propose a framework which calibrates bias and errors and achieves adaptive-balanced decoupling training by a prototype-based loss model. |
| Outcome: | The proposed framework outperforms existing approaches and achieves state-of-the-art performance on three datasets. |
KEBAP: Korean Error Explainable Benchmark Dataset for ASR and Post-processing (2023.emnlp-main)
Copied to clipboard
| Challenge: | Conventional evaluation metrics for automatic speech recognition systems produce a singular aggregate score, which is insufficient for understanding specific system vulnerabilities. |
| Approach: | They propose to introduce the Korean Error Explainable Benchmark Dataset for ASR and Post-processing (KEBAP) this method enables a more balanced assessment encompassing speech recognition accuracy and user readability. |
| Outcome: | The proposed method enables a more balanced assessment encompassing speech recognition accuracy and user readability. |
Iterative Self-Tuning LLMs for Enhanced Jailbreaking Capabilities (2025.naacl-long)
Copied to clipboard
Chung-En Sun, Xiaodong Liu, Weiwei Yang, Tsui-Wei Weng, Hao Cheng, Aidan San, Michel Galley, Jianfeng Gao
| Challenge: | Recent research shows that Large Language Models (LLMs) are vulnerable to automated jailbreak attacks. |
| Approach: | They propose a framework that crafts adversarial LLMs with enhanced jailbreak ability. |
| Outcome: | ADV-LLM significantly reduces the computational cost of generating adversarial suffixes while achieving nearly 100% ASR on various open-source LLMs. |
An Optimizable Suffix Is Worth A Thousand Templates: Efficient Black-box Jailbreaking without Affirmative Phrases via LLM as Optimizer (2025.findings-naacl)
Copied to clipboard
| Challenge: | Existing jailbreaking methods generate harmful and unethical content when subjected to jailbreaking attacks. |
| Approach: | They propose a black-box jailbreaking method with optimizable suffixes that translate jailbreaking objectives into natural language instructions. |
| Outcome: | The proposed method outperforms existing methods by 2.4 times in the ASR of three open-source LLMs and GPT-3.5-Turbo. |
Enabling Interactive Transcription in an Indigenous Community (2020.coling-main)
Copied to clipboard
| Challenge: | Existing methods for manual transcription are often in isolation from the speech community, and so we miss out on the opportunity to take advantage of the interests and skills of local people. |
| Approach: | They propose a transcription workflow which combines spoken term detection and human-in-the-loop to support speech transcription in almost-zero resource settings. |
| Outcome: | The proposed workflow is based on two endangered languages with zero-resource datasets. |
Rhythmic Proximity Between Natives And Learners Of French - Evaluation of a metric based on the CEFC corpus (2020.lrec-1)
Copied to clipboard
| Challenge: | Among prosodic parameters, rhythm is one that varies noticeably from one language to another. |
| Approach: | They propose to model rhythm in French using the corpus for l’Étude du Français Contemporain (CEFC) . they tested 146 native speakers, 37 non-native speakers and 29 non-Native Japanese learners of French . |
| Outcome: | The proposed model is based on the corpus pour l’Étude du Français Contemporain (CEFC) which contains up to 300 hours of speech of a wide variety of speaker profiles and situations. |
Exploiting the Index Gradients for Optimization-Based Jailbreaking on Large Language Models (2025.coling-main)
Copied to clipboard
| Challenge: | Despite advances in training Large Language Models, they remain vulnerable to jailbreak, an adversarial attack method. |
| Approach: | They propose an adversarial jailbreak algorithm that exploits the gradient information of the suffix tokens to accelerate the optimization process. |
| Outcome: | The proposed model achieves 1.5x speedup while maintaining high attack success rates. |
Automatic Transcription Challenges for Inuktitut, a Low-Resource Polysynthetic Language (2020.lrec-1)
Copied to clipboard
| Challenge: | Inuktitut is one of the 60 Indigenous languages currently spoken in Canada . polysynthetic languages are often termed agglutinative when their morphemes have clear boundaries and thus are easily segmentable. |
| Approach: | They propose to use a corpus of 23 hours of transcribed oral stories to train automatic speech recognition in Inuktitut. |
| Outcome: | The proposed model shows that Inuktitut displays a much higher degree of polysynthesis than other agglutinative languages like Finnish or Turkish. |
Code-Mixed Text Augmentation for Latvian ASR (2024.lrec-main)
Copied to clipboard
| Challenge: | a new study attempts to tackle code-mixed speech recognition by improving the language model of a hybrid system. |
| Approach: | They propose an inflected transliteration and phonetic transcription model for code-mixed Latvian sentences . they leverage a large human-translated English-Latvian parallel text corpus to generate synthetic Latvian phrases . |
| Outcome: | The proposed system improves on a human-translated English-Latvian parallel text corpus . the results show that the proposed system can generate code-mixed Latvian sentences . |
Attentively Embracing Noise for Robust Latent Representation in BERT (2020.coling-main)
Copied to clipboard
| Challenge: | Modern digital personal assistants interact with users through voice . high error rates still prevail in the widespread adoption of speech technology . |
| Approach: | They propose to extract more robust latent representations for noisy ASR text classification using transformer tokens and attentive embracement layer and multi-head attention layer. |
| Outcome: | The proposed model significantly outperforms the baseline model on the Chatbot and Snips corpora for intent classification with ASR error. |
A Comprehensive Evaluation of Incremental Speech Recognition and Diarization for Conversational AI (2020.coling-main)
Copied to clipboard
| Challenge: | Automatic Speech Recognition (ASR) systems are increasingly powerful and more numerous with several options existing as a service. |
| Approach: | They evaluate the most popular automatic speech recognition systems with metrics and experiments designed with these standards in mind. |
| Outcome: | The most popular ASR systems are Microsoft and IBM, and none are suitable for natural spontaneous conversations in real-time. |
Contrastive and Consistency Learning for Neural Noisy-Channel Model in Spoken Language Understanding (2024.naacl-long)
Copied to clipboard
| Challenge: | End-to-end learning models require large volume of speech data with intent labels . however, models are sensitive to inconsistencies between training and evaluation conditions . |
| Approach: | They propose a module-based approach to learn intent in a noisy-channel model . they correlate error patterns between clean and noisy ASR transcripts . |
| Outcome: | The proposed method outperforms existing methods and improves in noisy environments. |
Speech Corpus of Ainu Folklore and End-to-end Speech Recognition for Ainu Language (2020.lrec-1)
Copied to clipboard
| Challenge: | Ainu is an unwritten language spoken by Ainus, a minority of whom are critically endangered by UNESCO . a project of automatic speech recognition (ASR) for the Ainous language is being developed . |
| Approach: | They propose to use automatic speech recognition for the Ainu language to help preserve its language archives. |
| Outcome: | The proposed system improves word and phone recognition accuracy in speaker-open conditions. |
DITTO: Data-efficient and Fair Targeted Subset Selection for ASR Accent Adaptation (2023.acl-long)
Copied to clipboard
Suraj Kothawade, Anmol Mekala, D.Chandra Sekhara Hetha Havya, Mayank Kothyari, Rishabh Iyer, Ganesh Ramakrishnan, Preethi Jyothi
| Challenge: | State-of-the-art automatic speech recognition systems exhibit disparate performance on varying speech accents. |
| Approach: | They propose to use submodular mutual information to find the most informative set of utterances matching a target accent within a fixed budget. |
| Outcome: | The proposed model is 3-5 times more label-efficient on the Indic-TTS and L2 datasets than other methods. |
CoVoGER: A Multilingual Multitask Benchmark for Speech-to-text Generative Error Correction with Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models can fix recognition or translation errors that traditional rescoring cannot fix. |
| Approach: | They propose a benchmark for GER that covers both ASR and speech-to-text translation across 15 languages and 28 language pairs. |
| Outcome: | The proposed benchmark is built on common voice 20.0 and CoVoST-2 with Whisper and SeamlessM4T. |
WER-BERT: Automatic WER Estimation with BERT in a Balanced Ordinal Classification Paradigm (2021.eacl-main)
Copied to clipboard
| Challenge: | Automatic Speech Recognition (ASR) systems are evaluated using Word Error Rate (WER) a higher WER means a lower percentage of errors between the ground truth and the transcription of the system. |
| Approach: | They propose a new balanced paradigm for automatic Word Error Rate estimation using a Librispeech dataset and a Google Cloud's Speech-to-Text API. |
| Outcome: | The proposed approach is more effective than regression in a classification setting, but suffers from heavy class imbalance. |
Improving End-to-End Speech Processing by Efficient Text Data Utilization with Latent Synthesis (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Latent Synthesis is an efficient textual data utilization framework for end-to-end speech processing models . labeled speech data are scarcer and more expensive for collection compared to textual ones . |
| Approach: | They propose a textual data utilization framework for E2E speech processing models . they train a latent synthesizer to convert textual information into an intermediate latent representation . |
| Outcome: | The proposed framework improves on low-resource speech recognition and spoken language understanding tasks. |
ConEC: Earnings Call Dataset with Real-world Contexts for Benchmarking Contextual Speech Recognition (2024.lrec-main)
Copied to clipboard
Ruizhe Huang, Mahsa Yarmohammadi, Jan Trmal, Jing Liu, Desh Raj, Leibny Paola Garcia, Alexei V. Ivanov, Patrick Ehlen, Mingzhi Yu, Dan Povey, Sanjeev Khudanpur
| Challenge: | Existing work on contextual speech recognition (ASR) systems focuses on recognizing words that are not frequently seen in training data, such as rare words, but word error rate on rare words remains over 20%. |
| Approach: | They propose to use public-domain earnings calls and supplementary materials to evaluate contextual ASR approaches grounded on real-world applications. |
| Outcome: | The proposed frameworks are noisier than artificially synthesized contexts that contain the ground truth, yet still make great room for future improvement of contextual ASR technology. |
A Swiss German Dictionary: Variation in Speech and Writing (2020.lrec-1)
Copied to clipboard
Larissa Schmidt, Lucy Linder, Sandra Djambazovska, Alexandros Lazaridis, Tanja Samardžić, Claudiu Musat
| Challenge: | Besides standard German, Swiss German is spoken in about two thirds of Switzerland. |
| Approach: | They propose a dictionary containing normalized forms of common Swiss German words paired with Swiss German phonetic transcriptions to alleviate the uncertainty associated with this diversity. |
| Outcome: | The proposed dictionary is the first to combine spontaneous translation and phonetic transcriptions in large-scale, scalable phoneme to grapheme model that generates credible novel Swiss German writings. |
Towards Processing of the Oral History Interviews and Related Printed Documents (L18-1)
Copied to clipboard
Zbyněk Zajíc, Lucie Skorkovská, Petr Neduchal, Pavel Ircing, Josef V. Psutka, Marek Hrúz, Aleš Pražák, Daniel Soutner, Jan Švec, Lukáš Bureš, Luděk Müller
| Challenge: | a project aims to create an integrated archive of the recordings, scanned documents and photographs from totalitarian regimes in Czechoslovakia . the archive will be accessible online and provide multifaceted search capabilities . |
| Approach: | They propose to use automatic speech recognition and optical character recognition to build an archive of the interviews, scanned documents and photographs. |
| Outcome: | The proposed archive will be accessible online and provide multifaceted search capabilities. |
SCENEBench: An Audio Understanding Benchmark Grounded in Assistive and Industrial Use Cases (2026.eacl-long)
Copied to clipboard
| Challenge: | Existing models that measure audio comprehension beyond automatic speech recognition lack performance and latency. |
| Approach: | They propose a benchmark suite that measures audio comprehension beyond automatic speech recognition . the benchmark suite includes a small human-recorded evaluation split per category . |
| Outcome: | The proposed suite measures audio comprehension beyond speech recognition . it includes a small human-recorded evaluation split per category . |
Evaluating Open-Source ASR Systems: Performance Across Diverse Audio Conditions and Error Correction Methods (2025.coling-main)
Copied to clipboard
| Challenge: | Automated speech recognition (ASR) systems are able to transcribe spontaneous human conversations with high accuracy. |
| Approach: | They evaluate the accuracy of open source automatic speech recognition systems across conversational speech datasets and explore the potential of ASR ensembling and post-ASR correction methods to improve transcription accuracy. |
| Outcome: | The proposed methods highlight the need for robust error correction techniques and address demographic biases to enhance ASR performance and inclusivity. |
A Framework for Automatic Generation of Spoken Question-Answering Data (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing frameworks to automatically generate a spoken question answering dataset are limited by the amount of spoken text documents available. |
| Approach: | They propose to use QG module to generate questions from text documents, TTS module to convert text documents into spoken form and automatic speech recognition module to transcribe spoken content. |
| Outcome: | The proposed framework is efficient for automatically generating spoken QA datasets. |
Curriculum Pre-training for End-to-End Speech Translation (2020.acl-main)
Copied to clipboard
| Challenge: | End-to-end speech translation requires a powerful encoder to transcribe, understand and learn cross-lingual semantics simultaneously. |
| Approach: | They propose a curriculum pre-training method that includes an elementary course for transcription learning and two advanced courses for understanding the utterance and mapping words in two languages. |
| Outcome: | The proposed method improves on En-De and En-Fr speech translation benchmarks. |
Asymmetric Bias in Text-to-Image Generation with Adversarial Attacks (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing studies on adversarial attacks against text-to-image models have not explored the reasons for their effectiveness. |
| Approach: | They propose an adversarial attack objective - entity swapping using adversarials and two gradient-based attack algorithms. |
| Outcome: | The proposed attack objective - entity swapping - is easier to replace “human” with “robot” in the prompt “a human dancing in the rain.” |
Vietnamese Automatic Speech Recognition: A Revisit (2026.findings-eacl)
Copied to clipboard
| Challenge: | Existing datasets with low quality and inconsistent annotations are insufficient for high-quality models. |
| Approach: | They propose a pipeline for aggregating and preprocessing high-quality ASR datasets from diverse, potentially noisy, open-source sources. |
| Outcome: | The proposed pipeline provides a foundation for training and evaluating state-of-the-art Vietnamese ASR systems. |
How Accents Confound: Probing for Accent Information in End-to-End Speech Recognition Systems (2020.acl-main)
Copied to clipboard
| Challenge: | a new study examines how accent information is encoded and propagated in an end-to-end ASR system. |
| Approach: | They propose to use phone probes to analyze phonetic content of representations at each layer. |
| Outcome: | The proposed model is based on a large amount of US-accented English speech and is compared with other models using phone probes. |
SimulSpeech: End-to-End Simultaneous Speech to Text Translation (2020.acl-main)
Copied to clipboard
| Challenge: | SimulSpeech is an end-to-end simultaneous speech to text translation system . conventional approaches to simultaneous speech translation divide the translation process into two stages . |
| Approach: | They develop an end-to-end simultaneous speech to text translation system which translates speech in source language to text in target language concurrently. |
| Outcome: | The proposed system achieves reasonable BLEU scores and lower delay compared to full-sentence translation model. |
Towards end-2-end learning for predicting behavior codes from spoken utterances in psychotherapy conversations (2020.acl-main)
Copied to clipboard
| Challenge: | Xu and Sarikaya, 2014) proposes a framework for predicting utterance level labels directly from speech features. |
| Approach: | They propose a framework for predicting utterance level labels directly from speech features using a pretrained Speech-2-Vector encoder as bottleneck. |
| Outcome: | The proposed model outperforms state-of-the-art approaches which use transcribed text for the task of predicting psychotherapy-relevant behavior codes. |
Jailbreak Attack Initializations as Extractors of Compliance Directions (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Safety-aligned LLMs respond to prompts with compliance or refusal, each corresponding to distinct directions in the model’s activation space. |
| Approach: | They propose an initialization framework that aims to project unseen prompts further along compliance directions. |
| Outcome: | The proposed initialization framework achieves an increased attack success rate and reduced computational overhead, highlighting the fragility of safety-aligned LLMs. |
Discrete Cross-Modal Alignment Enables Zero-Shot Speech Translation (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing zero-shot methods fail to align speech and text into a shared semantic space . Existing methods require expensive and expensive parallel ST data . |
| Approach: | They propose a method that uses a shared discrete vocabulary space to align speech and text into a common space. |
| Outcome: | The proposed method significantly improves the SOTA and even performs on par with the strong supervised ST baselines. |
Generative Music Models’ Alignment with Professional and Amateur Users’ Expectations (2025.findings-acl)
Copied to clipboard
| Challenge: | Recent years have witnessed rapid advances in text-to-music generation using large language models. |
| Approach: | They propose a task to align AI-generated music with human expressions . they use a dataset of over 1.5 million songs to analyze their content . |
| Outcome: | The proposed framework outperforms baseline models and facilitates end-to-end generation of songs audio. |
FastCorrect 2: Fast Error Correction on Multiple Candidates for Automatic Speech Recognition (2021.findings-emnlp)
Copied to clipboard
Yichong Leng, Xu Tan, Rui Wang, Linchen Zhu, Jin Xu, Wenjie Liu, Linquan Liu, Xiang-Yang Li, Tao Qin, Edward Lin, Tie-Yan Liu
| Challenge: | Error correction is widely used in automatic speech recognition (ASR) to post-process the generated sentence. |
| Approach: | They propose a fast correction model that takes multiple ASR candidates as input for better correction accuracy. |
| Outcome: | The proposed model can reduce the word error rate (WER) with multiple candidates by 3.2% and 2.6%. |
Goal-Conditioned DPO: Prioritizing Safety in Misaligned Instructions (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing defense methods focus on aligning the model’s output towards less harmful responses through post-processing or input perturbation. |
| Approach: | They propose a goal-conditioned direct preference optimization technique which is trained to prioritize the system prompt over the user prompt through goal-conditioning and reduces the average Attack Success Rate (ASR) on a wide variety of jailbreak attacks. |
| Outcome: | The proposed approach reduces the average Attack Success Rate (ASR) on a wide variety of jailbreak attacks while maintaining general performance. |
TopicAttack: An Indirect Prompt Injection Attack via Topic Transition (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent attacks have demonstrated potential, but their abrupt instruction injection often undermines their effectiveness. |
| Approach: | They propose a method that prompts the LLM to generate a fabricated conversational transition prompt that gradually shifts the topic toward the injected instruction. |
| Outcome: | The proposed method achieves state-of-the-art performance with an attack success rate (ASR) over 90% in most cases, even when various defense methods are applied. |
Max-Margin Incremental CCG Parsing (2020.acl-main)
Copied to clipboard
| Challenge: | a new incremental parser reduces the number of beam search violations and minimises the biggest violation. |
| Approach: | They propose to use beam search optimisation to minimise all beam search violations instead of minimising only the biggest violation. |
| Outcome: | The proposed parser outperforms existing non-incremental parsers and minimises all beam search violations instead of minimising the biggest violation. |
DANCER: Entity Description Augmented Named Entity Corrector for Automatic Speech Recognition (2024.lrec-main)
Copied to clipboard
| Challenge: | End-to-end automatic speech recognition systems suffer from mistranscription of domain-specific phrases, such as named entities. |
| Approach: | They propose a named entity correction model that leverages phonetic con-fusion to mitigate phonetic confusion. |
| Outcome: | The proposed model outperforms the existing model on AISHELL-1 and Homophone datasets. |
A Bird’s-eye View of Language Processing Projects at the Romanian Academy (L18-1)
Copied to clipboard
| Challenge: | a recent article outlines five projects that address contemporary Romanian language . the authors argue that a constant accumulation of human expertise is needed to develop complex projects. |
| Approach: | a new article gives a general overview of five AI language-related projects at the Romanian Academy . they focus on the creation of a contemporary Romanian language text and speech corpus and language related applications . |
| Outcome: | a new article gives an overview of five AI language-related projects at the Romanian Academy . the projects address contemporary Romanian language, as well as language related applications . |
Comprehensive Punctuation Restoration for English and Polish (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Punctuation restoration is a fundamental requirement for the readability of text derived from Automatic Speech Recognition systems. |
| Approach: | They evaluate several methods in the comprehensive punctuation reconstruction task by comparing two languages with a model to determine the quality of the punctuated word. |
| Outcome: | The proposed model improves on two languages with relatively simple and complex morphologies. |
Beyond Transcripts: A Renewed Perspective on Audio Chaptering (2026.acl-long)
Copied to clipboard
| Challenge: | despite its relevance, research on audio chaptering remains limited and predominantly textbased . authors: audio chapterers can't be used linearly because they skim, scrub timelines, jump to relevant moments . acoustic features and learning representations are not used for audio chapterer evaluation . |
| Approach: | They propose to use audio-only architecture to automatically segment audio into coherent sections . they compare audio-based models with acoustic features and a novel audio-oriented architecture . |
| Outcome: | The proposed audio-only architecture outperforms text-based approaches on acoustic features and LLMs. |
Dealing with Data Scarcity in Spoken Question Answering (2024.lrec-main)
Copied to clipboard
| Challenge: | erroneous automatic speech recognition transcriptions and data scarcity hinder spoken QA models . paper focuses on using limited annotated data to improve spoken qa performance . |
| Approach: | They propose a framework for utilizing limited annotated data effectively to improve spoken QA performance. |
| Outcome: | The proposed model produces question-answer pairs from unannotated data with 5.5% relative gain over the model trained with annotated datasets. |
WER We Stand: Benchmarking Urdu ASR Models (2025.coling-main)
Copied to clipboard
| Challenge: | This paper analyzes the performance of three ASR models for low-resource languages like Urdu . low-rural languages like urdu have significant gaps in accuracy and reliability . |
| Approach: | They evaluate the performance of three ASR models: Whisper, MMS, and Seamless-M4T . they present the first conversational speech dataset for benchmarking Urdu ASR systems . |
| Outcome: | The proposed model families outperform Whisper, MMS, and Seamless-M4T on two types of speech datasets. |
DECM: Evaluating Bilingual ASR Performance on a Code-switching/mixing Benchmark (2024.lrec-main)
Copied to clipboard
| Challenge: | Code-switched (CSW) speech is a linguistic phenomenon that occurs when spoken utterances switch languages between sentences. |
| Approach: | They propose to use a dataset to evaluate German-English CSW speech . they show that the dataset includes splits with varying degrees of CSW . |
| Outcome: | The proposed dataset includes spontaneous speech from diverse domains, enabling realistic CSW evaluation in German-English. |
STTATTS: Unified Speech-To-Text And Text-To-Speech Model (2024.findings-emnlp)
Copied to clipboard
| Challenge: | a multi-task learning approach is currently available for speech recognition and speech synthesis models . |
| Approach: | They propose a parameter-efficient approach to learning ASR and TTS jointly . they use English as a resource-rich language and Arabic as 'low-resource' language . |
| Outcome: | The proposed model saves 50% of computational and memory costs while learning ASR and TTS jointly. |
Advancing Test-Time Adaptation in Wild Acoustic Test Settings (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing wild vision TTA methods fail to handle speech data due to the unique characteristics of high-entropy speech frames, which are unreliably filtered out even when containing crucial semantic content. |
| Approach: | They propose a method for acoustic foundation models to perform confidence-based adaptation in wild acustic test settings. |
| Outcome: | The proposed method outperforms baselines under Gaussian noise, environmental sounds, accent variations, and sung speech in the wild. |
Direct Simultaneous Speech-to-Text Translation Assisted by Synchronized Streaming ASR (2021.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to simultaneous speech-to-text translation suffer from error propagation and extra latency. |
| Approach: | They propose a new paradigm for simultaneous speech-to-text translation using two separate decoders . they use multitask learning to jointly learn these two tasks with a shared encoder . |
| Outcome: | The proposed method achieves substantially better translation quality at similar levels of latency. |
ML-LMCL: Mutual Learning and Large-Margin Contrastive Learning for Improving ASR Robustness in Spoken Language Understanding (2023.findings-acl)
Copied to clipboard
| Challenge: | Despite efforts to improve ASR robustness, errors from pipeline approaches can lead to error propagation. |
| Approach: | They propose a framework for improving ASR robustness in SLU by using mutual learning and large-margin contrastive learning. |
| Outcome: | The proposed framework outperforms existing models and achieves new state-of-the-art performance on three datasets. |
MuPe Life Stories Dataset: Spontaneous Speech in Brazilian Portuguese with a Case Study Evaluation on ASR Bias against Speakers Groups and Topic Modeling (2025.coling-main)
Copied to clipboard
Sidney Evaldo Leal, Arnaldo Candido Junior, Ricardo Marcacini, Edresson Casanova, Odilon Gonçalves, Anderson Silva Soares, Rodrigo Freitas Lima, Lucas Rafael Stefanel Gris, Sandra Aluísio
| Challenge: | Recent datasets for automatic speech recognition in Brazilian Portuguese lack diversity in terms of age groups, regional accents, and education levels. |
| Approach: | They propose to use a dataset to analyze the impact of ASR in Brazilian Portuguese (BP) they demonstrate that current models are biased regarding age, education, and regional accents. |
| Outcome: | The proposed dataset helps mitigate biases in current ASR models regarding education levels and age groups. |
A CLARIN Transcription Portal for Interview Data (2020.lrec-1)
Copied to clipboard
| Challenge: | a transcription portal for audio files based on automatic speech recognition (ASR) is implemented in the CLARIN resources research network and intended for use by non-technical scholars. |
| Approach: | They propose a transcription portal for audio files based on automatic speech recognition in various languages. |
| Outcome: | The proposed transcription portal is implemented in the CLARIN resources research network and intended for use by non-technical scholars. |
Development of Automatic Speech Recognition for the Documentation of Cook Islands Māori (2022.lrec-1)
Copied to clipboard
Rolando Coto-Solano, Sally Akevai Nicholas, Samiha Datta, Victoria Quint, Piripi Wills, Emma Ngakuravaru Powell, Liam Koka’ua, Syed Tanveer, Isaac Feldman
| Challenge: | a new study describes the process of data processing and training of an automatic speech recognition system for Cook Islands Mori . the system is based on statistical and Deep Learning techniques, and is available under a license . |
| Approach: | They describe the process of data processing and training of an automatic speech recognition system for Cook Islands Mori . they transcribed four hours of speech from adults and elderly speakers of the language and prepared two experiments . |
| Outcome: | The proposed system can perform better with low-resource Indigenous languages . the system can be used to accelerate the documentation of Cook Islands Mori . |
Samrómur: Crowd-sourcing Data Collection for Icelandic Speech Recognition (2020.lrec-1)
Copied to clipboard
David Erik Mollberg, Ólafur Helgi Jónsson, Sunneva Þorsteinsdóttir, Steinþór Steingrímsson, Eydís Huld Magnúsdóttir, Jon Gudnason
| Challenge: | Samrómur is a web application built upon Mozilla Foundation’s open-source voice collection platform Common Voice. |
| Approach: | They describe an ongoing project of speech data collection using the web application Samrómur which is built upon Common Voice, Mozilla Foundation’s web platform for open-source voice collection. |
| Outcome: | The proposed system will be the largest open speech corpus for Icelandic collected from the public domain. |
Afrispeech-Dialog: A Benchmark Dataset for Spontaneous English Conversations in Healthcare and Beyond (2025.naacl-long)
Copied to clipboard
Mardhiyah Sanni, Tassallah Abdullahi, Devendra Deepak Kayande, Emmanuel Ayodele, Naome A Etori, Michael Samwel Mollel, Moshood O. Yekini, Chibuzor Okocha, Lukman Enegi Ismaila, Folafunmi Omofoye, Boluwatife A. Adewale, Tobi Olatunji
| Challenge: | Afrispeech-Dialog is a benchmark dataset of 50 simulated medical and non-medical African-accented English conversations . a 10%+ performance degradation is found in ASR systems on long-form, accented speech . |
| Approach: | They propose to use a dataset to evaluate automatic speech recognition systems on African-accented conversations. |
| Outcome: | The proposed dataset compares state-of-the-art speech recognition systems on accented conversations with native accents and shows a 10%+ performance degradation. |
Language Modeling for Code-Switching: Evaluation, Integration of Monolingual Data, and Discriminative Training (D19-1)
Copied to clipboard
| Challenge: | Code-switching (CS) is a linguistic phenomenon defined as "the alternation of two languages within a single discourse, sentence or constituent." |
| Approach: | They propose an ASR-motivated evaluation setup which is decoupled from an ASL system and the choice of vocabulary . they propose a discriminative training approach which works better than generative language modeling . |
| Outcome: | The proposed evaluation setup is better than generative language modeling, the authors show . the proposed setup is decoupled from an ASR system and the choice of vocabulary . |
Accented Speech Recognition With Accent-specific Codebooks (2023.emnlp-main)
Copied to clipboard
| Challenge: | Degradation in performance across underrepresented accents is a severe deterrent to inclusive adoption of ASR. |
| Approach: | They propose an approach to adapt speech accents to unseen accents by using cross-attention with a trainable set of codebooks. |
| Outcome: | The proposed approach yields significant performance gains on the seen English accents and unseen accents on the Mozilla Common Voice dataset. |
Automatic Speech Recognition in Sanskrit: A New Speech Corpus and Modelling Insights (2021.findings-acl)
Copied to clipboard
| Challenge: | In this paper, we propose the first large scale study of automatic speech recognition in Sanskrit . we focus on the impact of unit selection in San's ASR systems . |
| Approach: | They propose a large scale study of automatic speech recognition in Sanskrit . they propose syllable level unit selection that captures character sequences . |
| Outcome: | The proposed model captures character sequences from one vowel in the word to the next vowela. |
Design and Development of Speech Corpora for Air Traffic Control Training (L18-1)
Copied to clipboard
| Challenge: | The current state-of-the-art training procedures involve retired pilots that train as virtual plane pilots and process the spoken prompts to form that can be entered into software that simulates the plane movement on the radar screen. |
| Approach: | They describe the process of creating domain-specific speech corpora containing air traffic control (ATC) communication prompts. |
| Outcome: | The proposed system could be used for training air traffic controllers in the Czech Republic. |
Discovering Canonical Indian English Accents: A Crowdsourcing-based Approach (L18-1)
Copied to clipboard
| Challenge: | Automated Speech Recognition systems degrade in performance when recognizing accents that are different from the ones in training data. |
| Approach: | They propose to adapt Acoustic Models that are trained on one accent to a target accent by using a small amount of speech data in the target accent. |
| Outcome: | The proposed model can be used to identify accents in Indian English and other languages. |
The Mark Fades: Adaptive Evolutionary Paraphrase-based Attack against LLM Watermarks (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing paraphrase-based watermark removal methods struggle to balance efficacy with text quality. |
| Approach: | They propose a training-free evolutionary framework that models watermark removal as a constrained multi-objective optimization problem by using a Pseudo-Log-Likelihood-guided mutation to precisely target and modify watermark-carrying tokens. |
| Outcome: | The proposed method outperforms baseline methods on a Qwen3 series watermark scheme while maintaining high semantic fidelity. |
A Semi-Automated Live Interlingual Communication Workflow Featuring Intralingual Respeaking: Evaluation and Benchmarking (2022.lrec-1)
Copied to clipboard
| Challenge: | Traditionally, live interlingual communication has been achieved only with the help of human interpreters. |
| Approach: | They propose a semi-automated workflow which uses a human respeaker and speaker-dependent speech recognition software to deliver punctuated same-language output of superior quality than the out-of-the-box ASR system. |
| Outcome: | The proposed workflow produces a similar quality output to the best-in-class simultaneous interpreters working with the same source speeches from the European Parliament. |
Multilingual Models for ASR in Chibchan Languages (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing algorithms for low resource-intensive languages are not available for these languages . a paper comparing the performance of different models and algorithms for these extremely low resource languages is presented. |
| Approach: | They propose to fine-tune four ASR algorithms to create monolingual models for Bribri and Cabécar . they then use the best performing algorithm to train joint and transfer learning models for both languages . |
| Outcome: | The proposed algorithms are effective in both Bribri and Cabécar, but especially in Bribri. |
StreamSpeech: Simultaneous Speech-to-Speech Translation with Multi-task Learning (2024.acl-long)
Copied to clipboard
| Challenge: | Existing simultaneous translation methods focus on text-to-text and speech-totext translation. |
| Approach: | They propose a Simul-S2ST model that jointly learns translation and simultaneous policy in a unified framework of multi-task learning. |
| Outcome: | The proposed model can perform offline and simultaneous speech recognition, speech translation and speech synthesis via an "All-in-One" seamless model. |
Visual Contextual Attack: Jailbreaking MLLMs with Image-Driven Context Injection (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies have shown that visual encoders can induce harmful behavior in multimodal large language models. |
| Approach: | They propose a vision-centric jailbreak attack that uses visual information to create a jailbreak context. |
| Outcome: | The proposed attack outperforms baseline attacks on MM-SafetyBench and GPT-4o. |
Simulating ASR errors for training SLU systems (L18-1)
Copied to clipboard
| Challenge: | Existing methods to simulate automatic speech recognition errors from manual transcriptions are not available during training of the SLU model. |
| Approach: | They propose to use acoustic and linguistic word embeddings to define a similarity measure between words to predict ASR confusions. |
| Outcome: | The proposed method significantly improves the performance of spoken language understanding systems. |
Task Arithmetic can Mitigate Synthetic-to-Real Gap in Automatic Speech Recognition (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for speech recognition suffer from the synthetic-to-real gap . existing methods suffer from this distributional shift due to acoustic mismatches . |
| Approach: | They propose to use task arithmetic to fine-tune an ASR model on synthetic data to mitigate the synthetic-to-real gap. |
| Outcome: | The proposed method shows an improvement of 10.03% over baselines on the SLURP dataset. |
Leveraging Large Pre-trained Multilingual Models for High-Quality Speech-to-Text Translation on Industry Scenarios (2025.coling-main)
Copied to clipboard
| Challenge: | Speech-to-Text Translation systems rely on a sequential pipeline that combines ASR and MT models. |
| Approach: | They propose a parameter-efficient framework that integrates one LPSM with a multilingual MT engine. |
| Outcome: | The proposed framework integrates one LPSM with a multilingual MT engine. |
Analysis of GlobalPhone and Ethiopian Languages Speech Corpora for Multilingual ASR (2020.lrec-1)
Copied to clipboard
| Challenge: | Using global phone data, we can develop multilingual speech recognition systems in yet unsupported languages. |
| Approach: | They analyze phonetic overlaps between GlobalPhone and Ethiopian speech corpora to develop multilingual Automatic Speech Recognition system for the Ethiopian languages. |
| Outcome: | The proposed system will be able to support three different languages and have morphological complexity. |
Large Vocabulary Read Speech Corpora for Four Ethiopian Languages: Amharic, Tigrigna, Oromo and Wolaytta (2020.lrec-1)
Copied to clipboard
Solomon Teferra Abate, Martha Yifiru Tachbelie, Michael Melese, Hafte Abera, Tewodros Abebe, Wondwossen Mulugeta, Yaregal Assabie, Million Meshesha, Solomon Afnafu, Binyam Ephrem Seyoum
| Challenge: | Automatic Speech Recognition (ASR) is one of the most important technologies to support spoken communication in modern life. |
| Approach: | They have developed four large speech corpora for four Ethiopian languages . they have word error rates of 37.65%, 31.03%, 38.02%, 33.89% for each language . |
| Outcome: | The proposed corpora achieve word error rates of 37.65%, 31.03%, 38.02%, 33.89% for Amharic, Tigrigna, Oromo and Wolaytta. |
Adversarial Training for Low-Resource Disfluency Correction (2023.findings-acl)
Copied to clipboard
| Challenge: | Disfluencies can be introduced in conversational speech due to the conversational nature of speech and/or speech impairments such as stuttering. |
| Approach: | They propose an adversarial sequence-tagging model for Disfluency Correction . they evaluate it in Bengali, Hindi, and Marathi languages and use it to correct stuttering disfluencies . |
| Outcome: | The proposed technique improves in Bengali, Hindi, and Marathi languages . it also removes stuttering disfluencies in ASR transcripts introduced by speech impairments . |
Semantic Role Labeling from Chinese Speech via End-to-End Learning (2024.findings-acl)
Copied to clipboard
| Challenge: | Semantic role labeling (SRL) has traditionally focused on text input. |
| Approach: | They propose an end-to-end approach for SRL from speech integrating ASR and SRL in a joint-learning framework, focusing on the Chinese language. |
| Outcome: | The proposed model improves on the Chinese Proposition Bank 1.0 dataset and the existing model with improved performance. |
MCLF: A Multi-grained Contrastive Learning Framework for ASR-robust Spoken Language Understanding (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Trending ASR-robust SLU systems have seen impressive improvements through global contrastive learning, but they can easily lead to severe semantic changes. |
| Approach: | They propose a two-stage multi-grained contrastive learning framework to improve ASR robustness . they first adapt pre-trained language models to downstream SLU datasets and then fine-tune it on the corresponding dataset. |
| Outcome: | The proposed framework improves on four datasets and four BERT-like backbone models. |
A Survey of Multilingual Models for Automatic Speech Recognition (2022.lrec-1)
Copied to clipboard
| Challenge: | Automatic Speech Recognition (ASR) systems have achieved human-like performance for a few languages, but the majority of the world’s languages do not have usable systems due to the lack of large speech datasets to train these models. |
| Approach: | They propose to use unlabeled speech data to build multilingual ASR models that can be used for improved performance on low-resource languages. |
| Outcome: | The proposed models can be used to improve performance on low-resource languages by using unlabeled speech data. |
New Datasets and Controllable Iterative Data Augmentation Method for Code-switching ASR Error Correction (2023.findings-emnlp)
Copied to clipboard
| Challenge: | In bilingual or multilingual settings, code-switching ASR has greater challenges and research value. |
| Approach: | They propose a controllable iterative method for improving the performance of mainstream automatic speech recognition systems by using Chinese-English code-switching dialogues. |
| Outcome: | The proposed method achieves the best performance compared with the rule-based, back-translation-based data augmentation methods and large language model ChatGPT. |
Analysing the Safety Pitfalls of Steering Vectors (2026.findings-acl)
Copied to clipboard
| Challenge: | Activation steering has emerged as a powerful tool to shape LLM behaviour without the need for weight updates. |
| Approach: | They propose to audit steering vectors obtained with Contrastive Activation Addition (CAA) and propose a mechanistic explanation for this finding. |
| Outcome: | The proposed approach significantly improves the success rate of jailbreak attacks, with stronger amplification under simple template-based attacks. |
Where are we in Named Entity Recognition from Speech? (2020.lrec-1)
Copied to clipboard
| Challenge: | Named entity recognition is usually made through a pipeline process that consists of processing audio and applying a NER to the audio outputs. |
| Approach: | They propose an original 3-pass approach and explore the capability of an E2E system to do structured NER. |
| Outcome: | The proposed system performs better than the current pipeline approach. |
CLAD-ST: Contrastive Learning with Adversarial Data for Robust Speech Translation (2023.emnlp-main)
Copied to clipboard
| Challenge: | Cascaded approach is the most popular choice for speech translation, but lacks robustness when dealing with noisy inputs. |
| Approach: | They propose a cascaded approach that uses an automatic speech recognition model and a machine translation model to translate speech in one language to text in another language. |
| Outcome: | The proposed approach achieves significant gains of up to 3 BLEU scores in English-German and English-French speech translation without hurting the translation quality on clean text. |
Unraveling the Mystery: Defending Against Jailbreak Attacks Via Unearthing Real Intention (2025.coling-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are increasingly vulnerable to elusive and implicit intentions, causing security risks and compromising user experience. |
| Approach: | They propose a method to detect and mitigate implicit jailbreak attacks using LLMs by unearthing real intentions and a greedy gradient-based algorithm to remove the least important parts of a sentence. |
| Outcome: | The proposed method reduces attacks success rate and Harmful Score while maintaining overall model performance. |
UMRSpell: Unifying the Detection and Correction Parts of Pre-trained Models towards Chinese Missing, Redundant, and Spelling Correction (2023.acl-long)
Copied to clipboard
| Challenge: | Chinese Spelling Correction (CSC) is a task of detecting and correcting misspelled charac- ters in Chinese texts. |
| Approach: | They propose a model to learn detection and correction parts together from a multi-task learning perspective. |
| Outcome: | The proposed model can learn detection and correction parts together from a multi-task learning perspective. |
Can Textual Unlearning Solve Cross-Modality Safety Alignment? (2024.findings-emnlp)
Copied to clipboard
Trishna Chakraborty, Erfan Shayegani, Zikui Cai, Nael Abu-Ghazaleh, M. Salman Asif, Yue Dong, Amit Roy-Chowdhury, Chengyu Song
| Challenge: | integrating new modalities into large language models creates new attack surface . existing safety training techniques like SFT and RLHF are not feasible in multi-modal settings . |
| Approach: | They explore whether unlearning in the textual domain can be effective for cross-modality safety alignment. |
| Outcome: | The proposed approach reduces the Attack Success Rate (ASR) to less than 8% and preserves the utility. |
Evaluating Workflows for Creating Orthographic Transcripts for Oral Corpora by Transcribing from Scratch or Correcting ASR-Output (2024.lrec-main)
Copied to clipboard
| Challenge: | Automated speech recognition systems can reduce transcription effort, but few studies have evaluated this potential. |
| Approach: | They compare efforts for manual transcription vs. automatic correction of ASR-output . they use audio recordings from varying settings to create orthographic transcripts . |
| Outcome: | The proposed methods reduce transcription time by 7 times on average for selected data and transcription conventions compared with corrected transcripts . the more complex the primary data, the more time has to be spent on corrections - the paper concludes a similar study could be conducted in 2022 . |
Multimodal Robustness for Neural Machine Translation (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to deal with noisy multimodal inputs are not robust enough to deal effectively with noisy data. |
| Approach: | They propose a method that composes domain adapters to deal with noisy inputs . they combine these adapters at runtime via dynamic routing or when source of noise is unknown . |
| Outcome: | The proposed model is flexible and state-of-the-art to deal with noisy multimodal inputs. |
InferAligner: Inference-Time Alignment for Harmlessness through Cross-Model Guidance (2024.emnlp-main)
Copied to clipboard
Pengyu Wang, Dong Zhang, Linyang Li, Chenkun Tan, Xinghao Wang, Mozhi Zhang, Ke Ren, Botian Jiang, Xipeng Qiu
| Challenge: | Existing methods for enhancing harmlessness and helpfulness of large language models (LLMs) involve complex and resource-intensive training processes. |
| Approach: | They propose a method that decouples harmlessness from helpfulness during inference phase. |
| Outcome: | The proposed method significantly reduces the attack success rate (ASR) of harmful instructions and jailbreak instructions while maintaining almost unchanged performance in downstream tasks. |
SLURP: A Spoken Language Understanding Resource Package (2020.emnlp-main)
Copied to clipboard
| Challenge: | Publicly available datasets for Spoken Language Understanding (SLU) are limited. |
| Approach: | They propose a publicly available SLU resource package that includes a multi-domain dataset in English spanning 18 domains. |
| Outcome: | The proposed dataset is bigger and more diverse than existing datasets. |
What is lost in Normalization? Exploring Pitfalls in Multilingual ASR Model Evaluations (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing text normalization routines that target Indic scripts are flawed when applied to multilingual automatic speech recognition models. |
| Approach: | They propose to develop text normalization routines that leverage native linguistic expertise to ensure more robust and accurate evaluations of multilingual automatic speech recognition models. |
| Outcome: | The proposed normalization routines can be leveraged to improve performance metrics for Indic languages. |
Exploring Pathological Speech Quality Assessment with ASR-Powered Wav2Vec2 in Data-Scarce Context (2024.lrec-main)
Copied to clipboard
| Challenge: | Current studies only gain good results on simple tasks such as binary classification due to data scarcity. |
| Approach: | They propose to use the pre-trained Wav2Vec2 architecture for both SSL, and ASR as feature extractor in speech assessment. |
| Outcome: | The proposed system achieves the best results on the HNC dataset using 95 training samples. |
WavRAG: Audio-Integrated Retrieval Augmented Generation for Spoken Dialogue Models (2025.acl-long)
Copied to clipboard
| Challenge: | Existing RAG frameworks rely on Automatic Speech Recognition to process speech input, which discards crucial audio information and increases computational overhead. |
| Approach: | They propose a retrieval augmented generation framework with native, end-to-end audio support that integrates audio and text into a unified knowledge representation. |
| Outcome: | The proposed framework can perform 10x faster than current pipelines while delivering 10x acceleration. |
ChildMandarin: A Comprehensive Mandarin Speech Dataset for Young Children Aged 3-5 (2025.acl-long)
Copied to clipboard
Jiaming Zhou, Shiyao Wang, Shiwan Zhao, Jiabei He, Haoqin Sun, Hui Wang, Cheng Liu, Aobo Kong, Yujie Guo, Xi Yang, Yequan Wang, Yonghua Lin, Yong Qin
| Challenge: | Automatic speech recognition systems have advanced significantly with models like Whisper, Conformer, and self-supervised frameworks such as Wav2vec 2.0. |
| Approach: | They propose to use Mandarin speech datasets to analyze pronunciation and tone of children aged 3 to 5 and evaluate their models on speaker verification (SV) They find that the datasets are more robust than those used by adult speech recognition systems and are open-source and available for all academic purposes. |
| Outcome: | The proposed dataset includes 41.25 hours of speech with carefully crafted manual transcriptions, collected from 397 speakers across various provinces in China, with balanced gender representation. |
CoSTA: Code-Switched Speech Translation using Aligned Speech-Text Interleaving (2025.coling-main)
Copied to clipboard
| Challenge: | More than half of the world's population is presumed to be bilingual . spoken translation of code-switched speech has been under-explored . |
| Approach: | They propose an end-to-end model architecture CoSTA that scaffolds on pretrained ASR and MT modules. |
| Outcome: | The proposed model outperforms existing models by 3.5 BLEU points in spoken translation of code-switched speech. |
Whispering LLaMA: A Cross-Modal Generative Error Correction Framework for Speech Recognition (2023.emnlp-main)
Copied to clipboard
Srijith Radhakrishnan, Chao-Han Yang, Sumeer Khan, Rohit Kumar, Narsis Kiani, David Gomez-Cabrero, Jesper Tegnér
| Challenge: | Existing methods for generative error correction in automatic speech recognition (ASR) use a two-pass reranking paradigm to generate n-best hypotheses. |
| Approach: | They propose a cross-modal fusion technique for generative error correction in automatic speech recognition. |
| Outcome: | The proposed technique shows a 37.66% improvement in word error rate relative to the n-best Oracle. |
Improving Code-switched ASR with Linguistic Information (2022.coling-1)
Copied to clipboard
| Challenge: | Existing studies on code-switching have been limited to the individual languages, but the results are promising. |
| Approach: | They propose to apply linguistic theories to generate more realistic code-switching text, which is needed for language modelling in ASR. |
| Outcome: | The proposed system improves 2% on English-Spanish code-switching . Equivalence Constraint theory and part-of-speech labelling are particularly helpful for text generation and bring 2% improvement to ASR performance. |
Ferret: Faster and Effective Automated Red Teaming with Reward-Based Scoring Technique (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing red-teaming methods generate adversarial attacks to identify vulnerabilities, but they face slow performance, limited categorical diversity, and high resource demands. |
| Approach: | They propose a method that generates multiple adversarial prompt mutations per iteration and ranks them using scoring functions. |
| Outcome: | The proposed method achieves a 95% attack success rate and reduces time to a 90% ASR by 15.2%. |
IndicVoices: Towards building an Inclusive Multilingual Speech Dataset for Indian Languages (2024.findings-acl)
Copied to clipboard
Tahir Javed, Janki Nawale, Eldho George, Sakshi Joshi, Kaushal Bhogale, Deovrat Mehendale, Ishvinder Sethi, Aparna Ananthanarayanan, Hafsah Faquih, Pratiti Palit, Sneha Ravishankar, Saranya Sukumaran, Tripura Panchagnula, Sunjay Murali, Kunal Gandhi, Ambujavalli R, Manickam M, C Vaijayanthi, Krishnan Karunganni, Pratyush Kumar, Mitesh Khapra
| Challenge: | Using INDICVOICES, we build the first ASR model to support all 22 languages listed in the 8th Schedule of the Constitution of India. |
| Approach: | They propose a dataset of natural and spontaneous speech from 16237 speakers covering 145 Indian districts and 22 languages. |
| Outcome: | The proposed dataset contains 7348 hours of read, extempore and conversational audio from 16237 speakers covering 145 Indian districts and 22 languages. |
Introducing Semantics into Speech Encoders (2023.acl-long)
Copied to clipboard
Derek Xu, Shuyan Dong, Changhan Wang, Suyoun Kim, Zhaojiang Lin, Bing Liu, Akshat Shrivastava, Shang-Wen Li, Liang-Hsuan Tseng, Guan-Ting Lin, Alexei Baevski, Hung-yi Lee, Yizhou Sun, Wei Wang
| Challenge: | Existing self-supervised speech encoders contain primarily acoustic rather than semantic information. |
| Approach: | They propose a task-agnostic unsupervised way to incorporate semantic information from large language model (LLM) systems into self-supervised speech encoders without labeled audio transcriptions. |
| Outcome: | The proposed approach improves spoken language understanding (SLU) performance by over 5% on intent classification (IC), with modest gains in named entity resolution (NER) and slot filling (SF), and spoken question answering (SQA) score by over 22%. |
Effectively pretraining a speech translation decoder with Machine Translation data (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to improve the performance of AST systems are based on pretraining the encoder parameters using an ASR model, but using a pretrained MT decoder is not beneficial or improves the results. |
| Approach: | They propose to use an adversarial regularizer to bring the encoder representations of the ASR and NMT tasks closer even though they are in different modalities. |
| Outcome: | The proposed model can be pre-trained using the Automatic Speech Recognition (ASR) task even in different languages and improves in low resource settings. |
Evaluation of Off-the-shelf Speech Recognizers on Different Accents in a Dialogue Domain (2022.lrec-1)
Copied to clipboard
| Challenge: | Existing automatic speech recognition systems for non-American accents have a much higher error rate than for general american accents. |
| Approach: | They evaluate automatic speech recognition systems on agent-directed speech . they find that the performance is worse for non-American accents than for General American . |
| Outcome: | The ASR systems perform worse for non-American accents than for General American accents . the results suggest that training on non-native English speakers is needed to narrow the performance gap. |
A Black-Box Attack on Code Models via Representation Nearest Neighbor Search (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for generating adversarial code examples face challenges such as limted availability of substitute variables and the creation of adversarials with noticeable perturbations. |
| Approach: | They propose a search seed based on historical attacks to find adversarial substitutes . they employ a pre-trained variable name encoder to map the search seed to a continuous vector space . |
| Outcome: | The proposed approach outperforms baseline methods in terms of ASR and QT. |
Lattice Transformer for Speech Translation (P19-1)
Copied to clipboard
| Challenge: | Recent advances in sequence modeling have highlighted the strengths of the transformer architecture. |
| Approach: | They propose a general lattice transformer for speech translation where the input is the output of the automatic speech recognition (ASR) they propose 'controllable' lattica attention mechanism to consume latent representations. |
| Outcome: | The proposed model outperforms baseline and lattice LSTM on the Chinese-English translation task. |
ASR for Documenting Acutely Under-Resourced Indigenous Languages (L18-1)
Copied to clipboard
| Challenge: | Automatic speech recognition (ASR) has not been widely explored as a tool for documenting endangered languages. |
| Approach: | They propose to use automatic speech recognition (ASR) to bootstrap new data to improve the acoustic model. |
| Outcome: | The proposed system improves the model for a polysynthetic language with few audio and text resources. |
AccentDB: A Database of Non-Native English Accents to Assist Neural Speech Recognition (2020.lrec-1)
Copied to clipboard
| Challenge: | aaron e. sanchez and joe saunders: automatic speech recognition still faces a major challenge . they say accents are a way of pronouncing a language, and speakers always have manner of speaking . esassen: accents can be used to identify non-native speakers of a speech . |
| Approach: | They propose to create a database of speech samples in non-native accents for ASR testing . they also propose to introduce accent neutralization of non- native accents to native accent . |
| Outcome: | The proposed model is compared against human-labelled accent classes and is generalized against human data. |
To Distill or Not to Distill? On the Robustness of Robust Knowledge Distillation (2024.acl-long)
Copied to clipboard
| Challenge: | Existing models for multilingual automatic speech recognition (ASR) are computationallyintensive and lack proper comprehensive evaluations. |
| Approach: | They propose to distill knowledge from large teacher models into smaller student variants that are more efficient. |
| Outcome: | The proposed model outperforms existing models on standard benchmarks and dialectal data. |
Masked Audio Text Encoders are Effective Multi-Modal Rescorers (2023.findings-acl)
Copied to clipboard
| Challenge: | Masked Language Models (MLMs) have proven to be effective for second-pass rescoring in Automatic Speech Recognition systems. |
| Approach: | They propose a multi-modal masked language model rescorer which integrates acoustic representations into the input space of MLM. |
| Outcome: | The proposed model reduces word error rate (WER) by 4%-16% on in-domain and 3%-7% on out-of-domain datasets over the text-only baseline. |
VoxpopuliTTS: a large-scale multilingual TTS corpus for zero-shot speech generation (2025.coling-main)
Copied to clipboard
Wenrui Liu, Jionghao Bai, Xize Cheng, Jialong Zuo, Ziyue Jiang, Shengpeng Ji, Minghui Fang, Xiaoda Yang, Qian Yang, Zhou Zhao
| Challenge: | Existing multilingual TTS datasets are limited in speech generation fields due to lack of quality data. |
| Approach: | They propose to use 30,000 hours of high-quality speech data across 3 languages . they filter out low-quality text-text pairs and concatenate short transcripts . |
| Outcome: | The proposed dataset comprises 30,000 hours of high-quality speech data, across 3 languages with multiple speakers and styles, suitable for various speech tasks such as TTS and ASR. |
Progress in Multilingual Speech Recognition for Low Resource Languages Kurmanji Kurdish, Cree and Inuktut (2022.lrec-1)
Copied to clipboard
| Challenge: | Using acoustic data, we develop automatic speech recognition systems for three low resource languages. |
| Approach: | They develop automatic speech recognition systems for three low resource languages using acoustic training data from 12 different languages in the hybrid DNN/HMM framework. |
| Outcome: | The proposed models are for three low resource languages: Kurmanji Kurdish, Cree and Inuktut. |
Hybrid Transducer and Attention based Encoder-Decoder Modeling for Speech-to-Text Tasks (2023.acl-long)
Copied to clipboard
| Challenge: | Neural based end-to-end frameworks have achieved remarkable success in speech-totext tasks, such as automatic speech recognition (ASR) and speech- totext translation (ST). |
| Approach: | They propose to combine Transducer and Attention based Encoder-Decoder (TAED) for speech-to-text tasks and leverage AED's strength in non-monotonic sequence to sequence learning while retaining Transducers streaming property. |
| Outcome: | The proposed model outperforms Transducer and Attention based Encoder-Decoder (TAED) on the MuST-C dataset and shows that it is not bound by any specific language model. |
Automatic Speech Recognition Datasets in Cantonese: A Survey and New Dataset (2022.lrec-1)
Copied to clipboard
Tiezheng Yu, Rita Frieske, Peng Xu, Samuel Cahyawijaya, Cheuk Tung Yiu, Holy Lovenia, Wenliang Dai, Elham J. Barezi, Qifeng Chen, Xiaojuan Ma, Bertram Shi, Pascale Fung
| Challenge: | In this paper, we address the problem of data scarcity for the Hong Kong Cantonese language . due to the popularization of deep learning, ASR technology has led to a significant improvement in recognizing many languages. |
| Approach: | They propose to use a dataset to analyze the data available for the Hong Kong Cantonese language . they use zh-HK as a source and a state-of-the-art ASR model to build a powerful model . |
| Outcome: | The proposed model improves on the biggest existing dataset, Common Voice zh-HK. |
Beyond Instructional Videos: Probing for More Diverse Visual-Textual Grounding on YouTube (2020.emnlp-main)
Copied to clipboard
| Challenge: | a representative pretraining model is fit to a diverse YouTube8M dataset . a priori, this domain is relatively easy for instructional videos . |
| Approach: | They fit a representative pretraining model to a YouTube8M dataset and examine its success and failure cases. |
| Outcome: | The proposed model can be trained on more diverse video corpora and achieve high performance on many video understanding tasks. |
CVSS Corpus and Massively Multilingual Speech-to-Speech Translation (2022.lrec-1)
Copied to clipboard
| Challenge: | Existing work on speech-to-speech translation (S2ST) systems rely on text representation, but they are text-centric. |
| Approach: | They introduce a massively multilingual-to-English speech-tospeech translation corpus . they synthesize the translation text from the Common Voice speech corpus and CoVoST 2 into English . |
| Outcome: | The proposed corpus outperforms existing models on CoVoST 2 by 5.8 BLEU . the proposed model outperformed the previous state-of-the-art model without extra data . |
Listening Like Humans: Semantics-Guided Noise-Robust Multimodal Speech Recognition (2026.acl-long)
Copied to clipboard
| Challenge: | Severe acoustic degradation results in unreliable ASR outputs . et al., 2024b): critical concerns regarding reliability and fairness of ASR . |
| Approach: | They propose a multimodal framework that reframes ASR as semantics-guided speech reconstruction. |
| Outcome: | The proposed framework achieves an average reduction in WER while also attaining 98.71% BERTScore and 96.7% USE over advanced baselines. |
A Unified Speaker Adaptation Approach for ASR (2021.emnlp-main)
Copied to clipboard
| Challenge: | Adapting a model to target speakers requires a lot of compute and may cause catastrophic forgetting to the existing speakers. |
| Approach: | They propose a unified speaker adaptation approach consisting of feature adaptation and model adaptation. |
| Outcome: | The proposed model outperforms baseline models with 20.58% relative WER reduction and surpasses finetuning method by 2.54% on target speaker adaptation. |
Towards relation extraction from speech (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for extracting relations from speech have been neglected due to the nature of speech. |
| Approach: | They propose a listening information extraction task that uses speech to extract relation extraction from speech . they use a text-to-speech system and crowd-sourced native English speakers to test the task . |
| Outcome: | The proposed task extracts semantic relationships from speech data using a new model . the proposed task is more challenging than the existing method due to the characteristics of speech . |
MISP-Meeting: A Real-World Dataset with Multimodal Cues for Long-form Meeting Transcription and Summarization (2025.acl-long)
Copied to clipboard
| Challenge: | Existing systems that can recognize spoken content, extract key information, and produce concise summaries are lacking in meeting transcription and summarization. |
| Approach: | They propose a multimodal dataset that integrates information from speech, vision, and text modalities to facilitate automatic meeting transcription and summarization (AMTS). |
| Outcome: | The proposed dataset reduces the character error rate (CER) by 36.60% to 20.27% and improves speech recognition and large language models. |
Adversarial Attacks on Parts of Speech: An Empirical Study in Text-to-Image Generation (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Recent studies show that text-to-image models are vulnerable to adversarial perturbations . |
| Approach: | They investigate the impact of adversarial attacks on different POS tags within text prompts on T2I models. |
| Outcome: | The proposed model is vulnerable to adversarial perturbations with noun perturbations in text prompts. |
PersonaLM: Language Model Personalization via Domain-distributed Span Aggregated K-Nearest N-gram Retrieval Augmentation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing language modeling tools for automatic speech recognition (ASR) are difficult to personalize. |
| Approach: | They propose a domain-distributed Span-Aggregated K-nearest N-gram retrieval augmentation to improve language modeling for automatic speech recognition (ASR) personalization. |
| Outcome: | The proposed model outperforms baselines on Wikitext-103, UserLibri, and ASAP datasets with a 10-16% improvement in perplexity and a 5-8% reduction in word error rates. |
Retrieval Augmented Generation based context discovery for ASR (2025.findings-emnlp)
Copied to clipboard
Siskos Dimitrios, Stavros Papadopoulos, Pablo Peso Parada, Jisi Zhang, Karthikeyan Saravanan, Anastasios Drosou
| Challenge: | Existing strategies for automatic context discovery remain a challenge . embedding-based retrieval reduces WER by up to 17% relative to using no-context . |
| Approach: | They propose an embedding-based retrieval approach for automatic context discovery . they propose two alternatives based on large language models to contextualize their effectiveness . |
| Outcome: | The proposed method reduces WER by up to 17% relative to using no-context . |
DoCIA: An Online Document-Level Context Incorporation Agent for Speech Translation (2025.findings-acl)
Copied to clipboard
Xinglin Lyu, Wei Tang, Yuang Li, Xiaofeng Zhao, Ming Zhu, Junhui Li, Yunfei Lu, Min Zhang, Daimeng Wei, Hao Yang, Min Zhang
| Challenge: | Document-level context is crucial for speech translation due to noise from ASR . incorporating document-level contextual information into ST remains a challenge . |
| Approach: | They develop an online framework that integrates document-level context into machine translation . they use document-based modules to integrate document- level context into ST . |
| Outcome: | The proposed framework outperforms baselines in sentence and discourse metrics . it can correct ASR transcription errors and improve translation performance . |
Parallel Corpus for Japanese Spoken-to-Written Style Conversion (2020.lrec-1)
Copied to clipboard
| Challenge: | spoken-to-written style conversion is becoming an important technology to increase the readability of ASR transcriptions. |
| Approach: | They propose to build a Japanese parallel corpus of spoken-to-written style conversions . they use crowdsourcing to convert spoken-style text into written-style texts . |
| Outcome: | The proposed corpus can handle general and specific spoken-to-written style conversion problems in Japanese. |
Large Corpus of Czech Parliament Plenary Hearings (2020.lrec-1)
Copied to clipboard
| Challenge: | a corpus of Czech parliament plenary sessions is a valuable resource for future research . only a few public datasets are available in the Czech language . end-to-end approaches require extensive training data to produce competitive results . |
| Approach: | They present a corpus of Czech parliament plenary sessions which is a large corpus . they combine a traditional approach with a more traditional approach . |
| Outcome: | The proposed model architectures can be used to train and evaluate speech recognition systems on a large corpus of speech data and transcripts. |
MASRI-HEADSET: A Maltese Corpus for Speech Recognition (2020.lrec-1)
Copied to clipboard
Carlos Daniel Hernandez Mena, Albert Gatt, Andrea DeMarco, Claudia Borg, Lonneke van der Plas, Amanda Muscat, Ian Padovani
| Challenge: | Maltese is the national language of Malta and is spoken by approximately 500,000 people. |
| Approach: | They present the first spoken Maltese corpus designed purposely for Automatic Speech Recognition (ASR) it consists of 8 hours of speech paired with text, recorded by using short text snippets in a laboratory environment. |
| Outcome: | The MASRI-HEADSET corpus was developed by the MASR project at the University of Malta. |
Integrating Pre-Trained Speech and Language Models for End-to-End Speech Recognition (2024.findings-acl)
Copied to clipboard
| Challenge: | Mainstream of automatic speech recognition (ASR) has shifted from pipeline methods to end-to-end (E2E) methods. |
| Approach: | They propose to integrate a pre-trained speech representation model and a large language model (LLM) for automatic speech recognition in an end-to-end manner. |
| Outcome: | The proposed model achieves comparable performance to modern E2E ASR models by utilizing powerful pre-training models with the proposed integrated approach. |
"Penny Wise, Pixel Foolish": Bypassing Price Constraints in Multimodal Agents via Visual Adversarial Perturbations (2026.findings-acl)
Copied to clipboard
| Challenge: | Mobile Agents are a key component of the “Agentic Economy” where they perform high-stakes financial transactions. |
| Approach: | They propose a systemic vulnerability termed Visual Dominance Hallucination (VDH) VDH exploits the modality gap in CLIP-based encoders via a novel Semantic-Decoupling Loss. |
| Outcome: | The proposed framework exploits the modality gap in CLIP-based encoders by preserving fidelity. |
BembaSpeech: A Speech Recognition Corpus for the Bemba Language (2022.lrec-1)
Copied to clipboard
| Challenge: | Existing speech recognition systems for African languages are very low . lack of resources (speech and text) can be attributed to poor quality of speech. |
| Approach: | They present a preprocessed, ready-to-use automatic speech recognition corpus, BembaSpeech, consisting of 24 hours of read speech in the Bemba language. |
| Outcome: | The proposed model achieves a word error rate (WER) of 32.91% on the Bemba language . the 1 billion XLS-R parameter model achieve better performance than the monolingual pre-trained English model on the corpus. |
Indic-TEDST: Datasets and Baselines for Low-Resource Speech to Text Translation (2024.lrec-main)
Copied to clipboard
| Challenge: | Speech-to-text Translation (ST) tasks are performed by human translators with proficiency in both the source and target languages. |
| Approach: | a new study compares the performance of SOTA ST models on low-resource languages . the authors propose to use a dataset to compare the models on high-resourced languages based on the results of their research . |
| Outcome: | a new study shows that only a few models have performed well on low-resource languages . the results indicate the need for specialized models for low- and high-resourced languages based on the dataset . |
Automatic Speech Recognition for Uyghur through Multilingual Acoustic Modeling (2020.lrec-1)
Copied to clipboard
| Challenge: | Low-resource languages suffer from lower performance of Automatic Speech Recognition (ASR) due to the lack of data. |
| Approach: | They propose to use Turkish as donor language to train acoustic models using multilingual training to achieve more context coverage. |
| Outcome: | The proposed system performs better with multilingual training for the under-resourced Uyghur language. |
Evaluation of Off-the-shelf Speech Recognizers Across Diverse Dialogue Domains (2020.lrec-1)
Copied to clipboard
| Challenge: | a recent study evaluated off-the-shelf automatic speech recognition systems . current state-of-the art systems perform poorly in domains that require special vocabulary and language models . |
| Approach: | They evaluate off-the-shelf automatic speech recognition systems across different dialogue domains . they use data collected from deployed spoken dialogue systems and human-human conversations . |
| Outcome: | The evaluation is aimed at non-experts with limited experience in speech recognition . the results show that the performance of each speech recognizer can vary significantly depending on the domain . |
CEASR: A Corpus for Evaluating Automatic Speech Recognition (2020.lrec-1)
Copied to clipboard
Malgorzata Anna Ulasik, Manuela Hürlimann, Fabian Germann, Esin Gedik, Fernando Benites, Mark Cieliebak
| Challenge: | Automatic Speech Recognition (ASR) systems are increasingly needed for research and practical applications. |
| Approach: | They propose to use public speech corpora to evaluate the quality of automatic speech recognition (ASR) they calculate an average Word Error Rate (WER) per corpus, per system and per corpor-system pair . |
| Outcome: | The proposed corpus evaluates the quality of automatic speech recognition systems using public speech corpora and transcripts generated by state-of-the-art systems. |
ÌròyìnSpeech: A Multi-purpose Yorùbá Speech Corpus (2024.lrec-main)
Copied to clipboard
| Challenge: | rynSpeech corpus is a dataset that can be used for both Text-to-Speecher (TTS) and Automatic Speech Recognition (ASR) speakers of many African languages have no access to voice-enabled applications in their native languages. |
| Approach: | They propose a dataset to collect Yorùbá speech data that can be used for both TTS and ASR tasks. |
| Outcome: | The proposed dataset can generate a good quality model with as little as 5 hours of speech . the results are consistent with previous studies on the Yorùbá language . |
POWSM: A Phonetic Open Whisper-Style Speech Foundation Model (2026.acl-long)
Copied to clipboard
Chin-Jou Li, Kalvin Chang, Shikhar Bharadwaj, Eunjung Yeo, Kwanghee Choi, Jian Zhu, David R. Mortensen, Shinji Watanabe
| Challenge: | Phone-level modeling of speech is a common approach to speech recognition, but it relies on task-specific architectures and datasets. |
| Approach: | They propose a phonetic framework capable of performing multiple phone-related tasks . they propose 'Phonetic Open Whisper-style Speech Model' that can perform these tasks together . |
| Outcome: | The proposed model outperforms or matches specialized PR models of similar size while supporting G2P, P2G, and ASR. |
Do Slides Help? Multi-modal Context for Automatic Transcription of Conference Talks (2025.emnlp-main)
Copied to clipboard
| Challenge: | Current automatic speech recognition systems rely on only audio information, ignoring multi-modal context. |
| Approach: | They propose to integrate visual context into existing automatic speech recognition systems to integrate presentation slides with multi-modal information. |
| Outcome: | The proposed model reduces word error rate by approximately 34% across all words and 35% for domain-specific terms compared to baseline model. |
RSC: A Romanian Read Speech Corpus for Automatic Speech Recognition (2020.lrec-1)
Copied to clipboard
| Challenge: | Romanian language is under-resourced due to the lack of acoustic and linguistic resources. |
| Approach: | They propose to use a Romanian speech corpus to train automatic speech recognition algorithms based on the spoken hotword detection mechanism. |
| Outcome: | The read speech corpus is a speech recognition system that can perform automatic speech recognition and speech synthesis using state-of-the-art speech recognition toolkit. |
PRoDeliberation: Parallel Robust Deliberation for End-to-End Spoken Language Understanding (2024.findings-emnlp)
Copied to clipboard
Trang Le, Daniel Lazar, Suyoun Kim, Shan Jiang, Duc Le, Adithya Sagar, Aleksandr Livshits, Ahmed Aly, Akshat Shrivastava
| Challenge: | End-to-end models for Spoken Language Understanding have been autoregressive, resulting in higher latencies. |
| Approach: | They propose a method that uses Connectionist Temporal Classification to train robust non-autoregressive deliberation models. |
| Outcome: | The proposed method achieves 10x latency reduction over autoregressive models while preserving ability to correct ASR mistranscriptions. |
Adapting Where It Matters: Depth-Aware Adaptation for Efficient Multilingual Speech Recognition in Low-Resource Languages (2026.findings-acl)
Copied to clipboard
| Challenge: | Recent speech foundation models excel at multilingual automatic speech recognition (ASR) for high-resource languages, but their performance drops substantially on low-resourced languages due to the limited data availability. |
| Approach: | They propose a Depth-Aware Model Adaptation framework that allocates adaptation capacity according to each layer’s role. |
| Outcome: | The proposed framework matches or surpasses state-of-the-art accuracy with 80% fewer trainable parameters and achieves 29% error reduction under extreme data scarcity. |
RIPRAG: Hack a Black-box Retrieval-Augmented Generation Question-Answering System with Reinforcement Learning (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to generate RAG documents require knowledge of the target RAG system’s internal composition and implementation details, whereas black-box methods are unable to utilize interactive information. |
| Approach: | They propose a RIPRAG attack framework that treats the target RAG system as a black box and leverages a Reinforcement Learning from Black-box Feedback (RLBF) method to optimize the generation model for poisoned documents. |
| Outcome: | The proposed method achieves an attack success rate (ASR) improvement of up to 0.72 compared to baseline methods. |
Attention Eclipse: Manipulating Attention to Bypass LLM Safety-Alignment (2025.emnlp-main)
Copied to clipboard
Pedram Zaree, Md Abdullah Al Mamun, Quazi Mishkatul Alam, Yue Dong, Ihsen Alouani, Nael Abu-Ghazaleh
| Challenge: | Recent research has shown that carefully crafted jailbreak inputs can induce large language models to produce harmful outputs, despite safety measures such as alignment. |
| Approach: | They propose a method for generating highly effective Jailbreak attacks that selectively strengthen or weaken attention among different parts of the prompt. |
| Outcome: | The proposed attacks amplify the success rate of existing Jailbreak algorithms while lowering generation cost. |
Visual-Aware Speech Recognition for Noisy Scenarios (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing audio-only models that use visual cues for transcription struggle in noisy environments. |
| Approach: | They propose a method that correlates visual cues with noise sources to improve transcription by filtering speech from noise and predicting noise labels in video inputs. |
| Outcome: | The proposed model improves transcription by correlating noise sources to visual cues in audio inputs. |
Killkan: The Automatic Speech Recognition Dataset for Kichwa with Morphosyntactic Information (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing datasets for automatic speech recognition (ASR) in the endangered Kichwa language have been limited. |
| Approach: | They present Killkan, the first dataset for automatic speech recognition (ASR) in the Kichwa language, an indigenous language of Ecuador. |
| Outcome: | The proposed dataset shows that it can be used to build an automatic speech recognition system for the endangered language with reliable quality despite its small size. |
DISCO: A Large Scale Human Annotated Corpus for Disfluency Correction in Indo-European Languages (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing research on disfluency correction has primarily focused on English due to the unavailability of large-scale open-source datasets. |
| Approach: | They propose to use an annotated human-annotated corpus to analyze disfluency correction in four important Indo-European languages to demonstrate the benefits. |
| Outcome: | The proposed model improves BLEU scores by 5.65 points when used with a state-of-the-art machine translation system. |
Modeling Gender and Dialect Bias in Automatic Speech Recognition (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Dialect and gender-based biases have become an area of concern in language-dependent AI systems. |
| Approach: | They construct a podcast audio dataset and evaluate its performance . they then refine the models to better understand how finetuning can impact performance. |
| Outcome: | The proposed model improves on 13 hours of podcast audio transcribed by speakers of four US-based English dialects. |
Defense Against Prompt Injection Attack by Leveraging Attack Techniques (2025.acl-long)
Copied to clipboard
| Challenge: | Recent attacks leverage LLMs’ instruction-following abilities and their inabilities to distinguish instructions injected in the data content. |
| Approach: | They invert the intention of prompt injection methods to develop novel defense methods based on previous training-free attack methods by repeating the attack process with the original input instruction rather than the injected instruction. |
| Outcome: | The proposed methods outperform existing defense approaches, achieving state-of-the-art results. |
FormosanBench: Benchmarking Low-Resource Austronesian Languages in the Era of Large Language Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing LLMs consistently underperform across all tasks, with 10-shot learning and fine-tuning offering only limited improvements. |
| Approach: | They introduce FormosanBench, a benchmark for evaluating LLMs on low-resource Austronesian languages. |
| Outcome: | The proposed benchmark covers three endangered Formosan languages: Atayal, Amis, and Paiwan . existing LLMs consistently underperform across all tasks, with 10-shot learning and fine-tuning offering only limited improvements. |
DNCASR: End-to-End Training for Speaker-Attributed ASR (2025.acl-long)
Copied to clipboard
| Challenge: | DNCASR is a novel end-to-end trainable system for speaker clustering and automatic speech recognition . it is designed to transcribe long multi-party meetings with speaker-attributed transcriptions . |
| Approach: | They propose a joint neural speaker clustering and automatic speech recognition system . DNCASR uses two separate encoders to independently encode speaker characteristics and local waveform information . |
| Outcome: | The proposed system outperforms a parallel system that does not have links between speaker and ASR decoders. |
Reasoning-Augmented Conversation for Multi-Turn Jailbreak Attacks on Large Language Models (2025.findings-emnlp)
Copied to clipboard
Zonghao Ying, Deyue Zhang, Zonglei Jing, Yisong Xiao, Quanchen Zou, Aishan Liu, Siyuan Liang, Xiangzheng Zhang, Xianglong Liu, Dacheng Tao
| Challenge: | Existing methods for implementing multi-turn jailbreaks struggle to balance semantic coherence with attack effectiveness, resulting in benign semantic drift or ineffective detection evasion. |
| Approach: | They propose a framework that reformulates harmful queries into benign reasoning tasks and leverages LLMs’ strong reasoning capabilities to compromise safety alignment. |
| Outcome: | The proposed framework achieves state-of-the-art attack effectiveness in complex conversational scenarios, with average ASRs increasing by up to 96%. |
Enhancing Speech Large Language Models with Prompt-Aware Mixture of Audio Encoders (2025.emnlp-main)
Copied to clipboard
Weiqiao Shan, Yuang Li, Yuhao Zhang, Yingfeng Luo, Chen Xu, Xiaofeng Zhao, Long Meng, Yunfei Lu, Min Zhang, Hao Yang, Tong Xiao, JingBo Zhu
| Challenge: | Existing work on integrating audio encoders with large language models (LLMs) has focused on semantic understanding tasks, but different tasks may require distinct features that emphasize either semantic or acoustic aspects. |
| Approach: | They propose to use a prompt-aware mixture to enhance the Speech LLM that uses multiple audio encoders to extract different features based on the prompt. |
| Outcome: | The proposed approach outperforms all single-encoder Speech LLMs on ASR, speaker number verification, and AC tasks. |
LegoSLM: Connecting LLM with Speech Encoder using CTC Posteriors (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Recent studies show that pre-trained speech encoders and large language models can perform suboptimal performance on a range of spoken language processing tasks. |
| Approach: | They propose to combine large-scale pre-trained speech encoders and large-language models for better performance on automatic speech recognition tasks. |
| Outcome: | The proposed model can get an average of 49% WER reduction over the baseline model on 8 MLS testsets. |
Improving Language and Modality Transfer in Translation by Character-level Modeling (2025.acl-long)
Copied to clipboard
| Challenge: | Current translation systems cover only 5% of the world's languages . expanding to the long-tail of low-resource languages requires data-efficient methods that rely on cross-lingual and cross-modal knowledge transfer. |
| Approach: | They propose a character-based approach to improve adaptability to new languages and modalities by using a teacher-student approach and parallel translation data to obtain a SONAR character-level encoder. |
| Outcome: | The proposed model outperforms subword-based models in speech-to-text translation on the FLEURS benchmark on 33 languages and achieves state-of-the-art generalizability to unseen languages. |
Making MLLMs Blind: Adversarial Smuggling Attacks in MLLM Content Moderation (2026.findings-acl)
Copied to clipboard
Zhiheng Li, Zongyang Ma, Yuntong Pan, Ziqi Zhang, Xiaolei Lv, Bo Li, Jun Gao, Jianing Zhang, Chunfeng Yuan, Bing Li, Weiming Hu
| Challenge: | Multimodal Large Language Models (MLLMs) are increasingly being deployed as content moderators . however, they exploit the Human-AI capability gap and create adversarial environments . smuggling attacks exploit the human-AI gap and exploit the vulnerability . |
| Approach: | They construct a benchmark to evaluate the vulnerability of MLLMs as content moderators . they identify three root causes: limited capabilities of vision encoders, robustness gap in OCR . |
| Outcome: | The proposed model exploits the Human-AI capability gap and is vulnerable to smuggling attacks. |
Localizing Malicious Outputs from CodeLLM (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Using FreqRank, we localize malicious components in outputs for triggered inputs and their corresponding backdoor triggers. |
| Approach: | They propose a mutation-based defense to localize malicious components in LLM outputs and their corresponding backdoor triggers. |
| Outcome: | The proposed defense has an average attack success rate (ASR) of 86.6% and can localize the backdoor triggers in 98% of cases. |
Fairness in Automatic Speech Recognition Isn’t a One-Size-Fits-All (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Pre-trained speech models like Whisper exhibit inconsistent group-level performance that varies across domains. |
| Approach: | They fine-tune a Whisper model on the Fair-Speech corpus using basic fine- tuning, demographic rebalancing, gender-swapped data augmentation and a novel contrastive learning objective. |
| Outcome: | The proposed method achieves stable, cross-domain fairness improvements without changes to the training data distribution and with minimal accuracy trade-offs. |
Speech Recognition and Meaning Interpretation: Towards Disambiguation of Structurally Ambiguous Spoken Utterances in Indonesian (2023.emnlp-main)
Copied to clipboard
| Challenge: | Ambiguity is one of the challenges in natural language processing. |
| Approach: | They propose to resolve structurally ambiguous sentences into unambiguous texts in Indonesian using prosodic information. |
| Outcome: | The proposed system achieves a disambiguation accuracy of 79.6% while the proposed direct system yields an even more impressive disambiguations accuracy of 82%. |
Multi-Stage Multi-Modal Pre-Training for Automatic Speech Recognition (2024.lrec-main)
Copied to clipboard
Yash Jain, David M. Chan, Pranav Dheram, Aparna Khare, Olabanji Shonibare, Venkatesh Ravichandran, Shalini Ghosh
| Challenge: | Existing methods for pre-training for automatic speech recognition (ASR) focus on single-stage pre-train followed by fine-tuning on downstream task. |
| Approach: | They propose a multi-modal pre-training method that combines unsupervised pre-training with translation-based supervised mid-training. |
| Outcome: | The proposed method improves WERs by 38.45% over baselines on both Librispeech and SUPERB. |
myMediCon: End-to-End Burmese Automatic Speech Recognition for Medical Conversations (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing medical conversation speech corpora for Burmese are limited, despite advances in ASR. |
| Approach: | They propose to use a manually curated medical conversation speech corpus for Burmese to examine the performance of ASR models. |
| Outcome: | The proposed model outperforms the Transformer model and the Recurrent Neural Network (RNN) models. |
Generative Annotation for ASR Named Entity Correction (2025.emnlp-main)
Copied to clipboard
Yuanchang Luo, Daimeng Wei, Shaojun Li, Hengchao Shang, Jiaxin Guo, Zongyao Li, Zhanglin Wu, Xiaoyu Chen, Zhiqiang Rao, Jinlong Yang, Hao Yang
| Challenge: | Existing named entity correction models fail to transcribe domain-speciffcnamed entities when theforms of the wrongly-transcribed words and the ground-truth entity are signiffcantly different. |
| Approach: | They propose a method that utilizes speech sound features to retrieve candidate entities . it uses speech sound feature to annotate entityerrors in ASR transcripts . |
| Outcome: | The proposed method can bring signiffcant improvement to entity accuracy. |
BLSP-Emo: Towards Empathetic Large Speech-Language Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | BLSP-Emo model understands both semantics and emotions in speech and generates empathetic responses. |
| Approach: | They propose a language-speech pretraining with emotion support that utilizes existing speech and emotion recognition datasets to create an end-to-end speech-language model. |
| Outcome: | The proposed model can understand both semantics and emotions in speech and generate empathetic responses. |
Dynamic Model-Bank Test-Time Adaptation for Automatic Speech Recognition (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing ASR TTA methods struggle with instability under continual and long-term distribution shifts. |
| Approach: | They propose a continuous adaptive model-bank framework that adapts to domain shifts in ASR test-time scenarios. |
| Outcome: | Experiments on diverse, continuously shifting ASR benchmarks show that DMSUTA outperforms existing continual TTA baselines. |
Continual Test-time Adaptation for End-to-end Speech Recognition on Noisy Speech (2024.emnlp-main)
Copied to clipboard
| Challenge: | Current ASR TTA methods focus on non-continual TTA, which limits cross-sample knowledge learning compared to continual TTA. |
| Approach: | They propose a Fast-slow TTA framework that leverages the advantage of continual and non-continual TTA and a Dynamic SUTA method that automatically detects domain shifts and resets the model. |
| Outcome: | The proposed method outperforms non-continual and continual TTA methods while maintaining robustness to domain shifts without requiring domain boundary information. |
When Efficiency Meets Safety: A Benchmark Security Analysis of KV Cache Compression in Large Language Models (2026.acl-long)
Copied to clipboard
| Challenge: | Key-Value (KV) caching is widely used in large language models to enable long-context inference efficiently, yet its security implications remain underexplored. |
| Approach: | They propose a history-aware, per-head feedback merging strategy that prevents safety degradation while maintaining efficiency. |
| Outcome: | The proposed strategy prevents safety degradation while maintaining efficiency. |
DiffusionAttacker: Diffusion-Driven Prompt Manipulation for LLM Jailbreak (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are susceptible to generating harmful content when prompted with carefully crafted inputs, a vulnerability known as LLM jailbreaking. |
| Approach: | They propose an end-to-end generative approach for jailbreak rewriting inspired by diffusion models that uses a sequence-tosequence (seq2sequ) diffusion model as a generator, conditioning on the original prompt and guiding the denoising process with a novel attack loss. |
| Outcome: | Experiments on Advbench and Harmbench show that the proposed method outperforms autoregressive jailbreak models across evaluation metrics including ASR, fluency, diversity and diversity. |
Post-decoder Biasing for End-to-End Speech Recognition of Multi-turn Medical Interview (2024.lrec-main)
Copied to clipboard
| Challenge: | End-to-end (E2E) models are replacing hybrid models for automatic speech recognition tasks. |
| Approach: | They propose a method to optimize E2E models for automatic speech recognition . they propose MED-IT, a multi-turn consultation speech dataset . |
| Outcome: | The proposed method improves on subsets of rare words appearing in training speech. |
Interventional Speech Noise Injection for ASR Generalizable Spoken Language Understanding (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to increase the robustness of pre-trained language models (PLMs) against unseen ASR systems produce noisy inputs for SLU models, which can significantly degrade their performance. |
| Approach: | They propose to introduce ASR-plausible noises into pre-trained language models by cutting off the non-causal effect of noises. |
| Outcome: | The proposed method improves the robustness and generalizability of SLU models against unseen ASR systems by cutting off the non-causal effect of noises. |
Twists, Humps, and Pebbles: Multilingual Speech Recognition Models Exhibit Gender Performance Gaps (2024.emnlp-main)
Copied to clipboard
| Challenge: | a new class of multitasks, multilingual neural networks, has recently pushed the boundaries of speech-related tasks. |
| Approach: | They evaluate performance of two widely used multilingual automatic speech recognition models . they find clear gender disparities, with the advantaged group varying across languages . |
| Outcome: | The proposed models are compared on 19 languages from eight language families and two speaking conditions. |
Lost in Transcription, Found in Distribution Shift: Demystifying Hallucination in Speech Foundation Models (2025.findings-acl)
Copied to clipboard
| Challenge: | Automatic speech recognition systems have seen remarkable improvements in recent years, but evaluation of performance remains dependent on word and character error rate (WER/CER). |
| Approach: | They investigate how distribution shifts, model size and model architecture influence hallucination error rate (HER) HER is a metric used to quantify hallucinosity in automatic speech recognition systems. |
| Outcome: | The proposed model can be used to measure hallucination errors in high-stakes domains such as healthcare, legal, and aviation. |
Mixture-of-Experts with Intermediate CTC Supervision for Accented Speech Recognition (2026.acl-long)
Copied to clipboard
| Challenge: | Accented speech remains a persistent challenge for automatic speech recognition (ASR) Accent-agnostic approaches improve robustness but struggle with heavily accented or unseen varieties . |
| Approach: | They propose a Mixture-of-Experts architecture with intermediate CTC supervision that promotes expert specialization and generalization. |
| Outcome: | Experiments show that the proposed architecture improves on accented speech . the proposed framework is based on a mixture-of-experts architecture with intermediate supervision . |
EVA: Evolving Semantic Adversaries for Red-Teaming GUI Agents Against Environmental Injection Attacks (2026.findings-acl)
Copied to clipboard
Yijie Lu, Manman Zhao, Tianjie Ju, Zihe Yan, Xinbei Ma, Yuan Guo, Daizong Ding, Gongshen Liu, Zhuosheng Zhang
| Challenge: | Existing methods for red-teaming face a trade-off between requiring target-specific knowledge and incurring prohibitive computational costs. |
| Approach: | They propose a framework that evolves payloads exclusively on the semantic dimension via a discovery-deployment pipeline. |
| Outcome: | Experiments show that EVA outperforms baselines in terms of attack success rate while evolving benign seeds into successful attacks within 1.18 to 1.71 iterations. |
RT-VQ2A2: Real Time Vector Quantized Question Answering with ASR (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing frameworks for QA with large language models are difficult to implement due to noise, limited context length and latency. |
| Approach: | They propose a model-agnostic framework to address problems in QA with large language models. |
| Outcome: | The proposed framework reduces noise in the ASR output and the limited context length of LLMs and improves performance on the widely used Spoken-SQuAD dataset. |
Multimodal In-context Learning for ASR of Low-resource Languages (2026.findings-acl)
Copied to clipboard
| Challenge: | In-context learning with large language models addresses this limitation, but prior work focuses on high-resource languages covered during training and text-only settings. |
| Approach: | They propose to use multimodal ICL to learn unseen languages with multimodal learning to improve ASR in large language models. |
| Outcome: | The proposed model outperforms existing models on unseen languages with multimodal ICL (MICL) and cross-lingual transfer learning matches or outperformed models without using target-language data. |
SamróMur MilljóN: An ASR Corpus of One Million Verified Read Prompts in Icelandic (2024.lrec-main)
Copied to clipboard
| Challenge: | samrómur is a crowdsourcing web application designed to collect speech data for the advancement of language technologies in Icelandic. |
| Approach: | They propose to use a crowdsourcing web application to collect and verify Icelandic speech data for automatic speech recognition (ASR) they introduce a dataset comprising one million audio clips from the application . |
| Outcome: | The proposed system can produce high-quality speech data for Icelandic . the proposed system is based on a crowdsourced web application built on Mozilla's Common Voice . |
Let’s Fuse Step by Step: A Generative Fusion Decoding Algorithm with LLMs for Robust and Instruction-Aware ASR and OCR (2025.findings-acl)
Copied to clipboard
| Challenge: | Various fusion strategies have been explored for integration of large language models into multi-modal systems. |
| Approach: | They propose a framework for deep fusion decoding that integrates large language models into cross-modal text recognition systems. |
| Outcome: | The proposed framework surpasses cascaded methods in English and Mandarin, and significantly reduces WERs by 17.7%. |
What Do Neural Speech Models Know About Phonology? Evidence from Structured Phoneme Confusions (2026.findings-acl)
Copied to clipboard
| Challenge: | acoustic and phonological models of speech recognition are often limited to the phoneme level . a recent study has shown that phoneme confusions are strongly structured in phonology space . |
| Approach: | They adopt a featural representation of phonemes grounded in phonological theory which models speech sounds as structured bundles of distinctive articulatory and acoustic properties. |
| Outcome: | The proposed model allows us to analyse phoneme confusions at a finer granularity and to investigate whether certain phonological features are more vulnerable than others. |
Steering Away from Refusal: A Black-box Jailbreak Method Based on First-Token Distribution (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to analyze black-box jailbreaks lack direct optimization signals to refine adversarial prompts. |
| Approach: | They propose a distribution-jailbreak attack method that selects effective jailbreak templates and iteratively optimizes adversarial suffixes by maximizing the KL divergence from the standard refusal distribution. |
| Outcome: | The proposed method achieves state-of-the-art Attack Success Rate (ASR) on all tested open-source models and delivers over 94% ASR on GPT-4.1. |
Don’t Say No: Jailbreaking LLM by Suppressing Refusal (2025.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are vulnerable to "jailbreaking" attacks where crafted prompts manipulate them into producing toxic content. |
| Approach: | They propose to improve the target loss objective by combining a cosine decay schedule method with refusal suppression to achieve higher success rates. |
| Outcome: | The proposed approach outperforms baseline attacks and achieves state-of-the-art attack success rates. |
Zero-Shot Context-Aware ASR for Diverse Arabic Varieties (2026.findings-acl)
Copied to clipboard
| Challenge: | Large-scale multilingual ASR has substantially improved recognition for high-resource languages. |
| Approach: | They propose a proxy-guided -best selection paradigm that conditions inference on external side information without parameter updates. |
| Outcome: | The proposed model reduces WER by 15.6% relative and recovers a fraction of oracle n-best gains on the common voice MSA testbed. |
Difference in Task Performance on Sparse Speech Representations (2026.acl-long)
Copied to clipboard
| Challenge: | Existing methods for learning speech representations that are useful for a variety of downstream tasks have been extensively investigated in different domains. |
| Approach: | They propose to train Autoencoders with varying sparsity levels using three SSL features and evaluate them on six tasks of SUPERB: speech enhancement, speaker identification, speech Emotion Recognition, phone recognition, automatic speech recognition and slot filling. |
| Outcome: | The proposed model can be used to learn speech representations that are useful for a variety of downstream tasks. |
Red Queen: Exposing Latent Multi-Turn Risks in Large Language Models (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing jailbreak strategies are single-turn with explicit malicious intent, failing to reflect real-world scenarios where interactions can be multi-turn and users can conceal intents. |
| Approach: | They propose a jailbreak attack that constructs a multi-turn scenario and conceals malicious intent under the guise of preventing harm. |
| Outcome: | The proposed attack achieves 87.6% attack success rate (ASR) on GPT-4o and 77.1% on Llama3-70B compared to previous jailbreak attacks . |
Speech Recognition Corpus of the Khinalug Language for Documenting Endangered Languages (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing tools to document endangered languages are limited due to data scarcity and the need for training. |
| Approach: | They propose to use a speech corpus for Khinalug, an endangered language spoken in northern Azerbaijan, to create a model that can be used in language documentation scenarios. |
| Outcome: | The proposed model achieves 6.65 CER points and 25.53 WER points in low-resource scenarios. |
The Influence of Automatic Speech Recognition on Linguistic Features and Automatic Alzheimer’s Disease Detection from Spontaneous Speech (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing biomarkers for AD diagnosis can only be applied to relatively small sample sizes due to limited availability, excessive costs and invasive nature. |
| Approach: | They compare automatic speech recognition systems in terms of Word Error Rate (WER) using a publicly available benchmark dataset of speech recordings of AD patients and controls. |
| Outcome: | The proposed method improves classification performance by replacing manual transcriptions with ASR output. |
Towards a Corpus of Spoken Maltese: Korpus tal-Malti Mitkellem, KMM (2024.lrec-main)
Copied to clipboard
| Challenge: | 'Corpus of Spoken Maltese' is a spoken corpus of spoken Malteser based on a gold-standard Core collection . initial results show that the ASR is robust enough to generate first-pass texts for annotators to work on, thus reducing the human effort and consequently, the cost involved. |
| Approach: | They propose to create a “dedicated” spoken corpus of Maltese based on a gold-standard Core collection and a qualitative analysis of the output of a Malteser ASR system. |
| Outcome: | The proposed corpus is based on the concept of a gold-standard Core collection and compares to human annotations. |
STAR-Teaming: A Strategy-Response Multiplex Network Approach to Automated LLM Red Teaming (2026.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are susceptible to jailbreak prompts that can elicit harmful or inappropriate responses. |
| Approach: | They propose a black-box framework for automated red teaming that integrates a Multi-Agent System with a Strategy-Response Multiplex Network and employs network-driven optimization to sample effective attack strategies. |
| Outcome: | The proposed framework surpasses existing methods and achieves higher attack success rate (ASR) at lower computational cost. |
The Esethu Framework: Reimagining Sustainable Dataset Governance and Curation for Low-Resource Languages (2025.acl-long)
Copied to clipboard
Jenalea Rajab, Anuoluwapo Aremu, Everlyn Asiko Chimoto, Dale Dunbar, Graham Morrissey, Fadel Thior, Luandrie Potgieter, Jessica Ojo, Atnafu Lambebo Tonja, Wilhelmina NdapewaOnyothi Nekoto, Pelonomi Moiloa, Jade Abbott, Vukosi Marivate, Benjamin Rosman
| Challenge: | Esethu Framework is a community-centric data license that empowers local communities and ensures equitable benefit-sharing from their linguistic resource. |
| Approach: | They propose a community-centric data license to empower local communities and ensure equitable benefit-sharing from their linguistic resource. |
| Outcome: | The proposed dataset contains read speech from native isiXhosa speakers enriched with demographic and linguistic metadata. |
Unraveling Spontaneous Speech Dimensions for Cross-Corpus ASR System Evaluation for French (2024.lrec-main)
Copied to clipboard
| Challenge: | 'spontaneous speech' is a catch-all term used for situations like speaking with a friend, being interviewed on radio/TV or giving a lecture. |
| Approach: | They propose to use four dimensions to describe spontaneous speech variation in automatic speech recognition systems. |
| Outcome: | The proposed system can be used to predict the WER of speech recognition systems on face-to-face interactions. |
VietMed: A Dataset and Benchmark for Automatic Speech Recognition of Vietnamese in the Medical Domain (2024.lrec-main)
Copied to clipboard
| Challenge: | Currently, there are no publicly available speech recognition datasets in the medical domain due to privacy restrictions. |
| Approach: | They present a Vietnamese speech recognition dataset in the medical domain comprising 16h of labeled medical speech, 1000h of unlabeled medical and 1200h of general-domain speech. |
| Outcome: | The proposed model outperforms state-of-the-art models from 51.8% to 29.6% WER on test set. |
MCGA: A Multi-task Classical Chinese Literary Genre Audio Corpus (2026.findings-acl)
Copied to clipboard
Yexing Du, Kaiyuan Liu, Bihe Zhang, Youcheng Pan, Bo Yang, Liangyu Huo, Xiyuan Zhang, Jian Xie, Daojing He, Yang Xiang, Ming Liu, Bing Qin
| Challenge: | Multimodal Large Language Models (MLLMs) have advanced Chinese Classical Studies (CCS) but the audio dimension of CCS remains underexplored due to a lack of high-quality, domain-specific audio corpora. |
| Approach: | They propose a 119-hour audio corpus comprising 22,000 audio samples to bridge this gap . it encompasses a diverse range of literary genres across six tasks . |
| Outcome: | The proposed corpus encompasses a diverse range of literary genres across six tasks: Automatic Speech Recognition (ASR), Speech-to-Text Translation (S2TT), Speech Emotion Captioning (SEC), Spoken Question Answering ( SQA), Speech Understanding (SU), and Speech Reasoning (SR). |
ZAEBUC-Spoken: A Multilingual Multidialectal Arabic-English Speech Corpus (2024.lrec-main)
Copied to clipboard
| Challenge: | a corpus of multilingual Arabic-English speech is presented in a new paper . a major bottleneck is the lack of data needed for training NLP models . |
| Approach: | They propose a multilingual multidialectal Arabic-English speech corpus with a set of guidelines for automatic speech recognition. |
| Outcome: | The proposed corpus includes two languages with Arabic and English spoken in multiple variants and Arabic and Arabic with various accents. |
More Thinking, Less Talking: Internalizing Deliberative Safety into LLM Parameters (2026.acl-long)
Copied to clipboard
| Challenge: | Existing safety alignment methods leave Large Language Models vulnerable to sophisticated jailbreak attacks. |
| Approach: | They propose a safety reasoning internalization framework that internalizes safety reasoning into an implicit computational pathway using Low-Rank Adaptation (LoRA). |
| Outcome: | The proposed framework achieves a 43% lower Attack Success Rate (ASR) against distinct jailbreak attacks compared to strong baselines. |
Beyond WER: Probing Whisper’s Sub‐token Decoder Across Diverse Language Resource Levels (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large multilingual automatic speech recognition models achieve remarkable performance, but the internal mechanisms of the end-to-end pipeline remain underexplored. |
| Approach: | They propose to analyze Whisper's multilingual decoder to uncover systematic decoding disparities masked by aggregate error rates. |
| Outcome: | The proposed model performs better on higher resource languages, but lower resource languages fare worse on these metrics. |
Towards Language-Agnostic STIPA: Universal Phonetic Transcription to Support Language Documentation at Scale (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing ASR systems focus on orthographic output for high-resource languages, but STIPA can be used as a language-agnostic interface for documenting under-resourced and unwritten languages. |
| Approach: | They propose to use the International Phonetic Alphabet (STIPA) to generate phonetic transcriptions using a language-agnostic interface. |
| Outcome: | The proposed model reduces phonetic error rates even in low-resource settings and can be used for documenting under-resourced and unwritten languages. |
Identifying and typifying demographic unfairness in phoneme-level embeddings of self-supervised speech recognition models (2026.findings-acl)
Copied to clipboard
| Challenge: | a number of studies have been done to improve ASR for speaker groups, but there is still room for improvement . authors propose a framework typifying two types of error in phoneme embeddings . |
| Approach: | They propose a framework typifying two types of error that can occur in phoneme modeling . they propose random error/high variance in phonemes embedding vs systematic error/embedding bias . |
| Outcome: | The proposed framework typifies errors in phoneme modeling in ASR systems . it shows that training only on a single, typically disadvantaged SG improves performance . |
Idiosyncratic Versus Normative Modeling of Atypical Speech Recognition: Dysarthric Case Studies (2025.emnlp-main)
Copied to clipboard
| Challenge: | Past studies have focused on fully personalized (or idiosyncratic) models for atypical speech . past studies focused on idiotic models, but current approaches focus on generalizing and handling idiomatic patterns . |
| Approach: | They compare four models that generalize and handle idiosyncrasy to find atypical speech . they find the dysarthric-idios-ync model performs better than the idioconic approach . |
| Outcome: | The proposed model generalizes and handles idiosyncrasy better than the idiocy model . the model requires less personalized data and reduces word error rate from 71% to 32% . |
Layer-wise Minimal Pair Probing Reveals Contextual Grammatical-Conceptual Hierarchy in Speech Representations (2025.emnlp-main)
Copied to clipboard
| Challenge: | a recent study evaluated the extent to which SLMs encode nuanced syntactic and conceptual features . acoustic and phonetic features are shallow, but the extent of nuance is unclear . |
| Approach: | a new study evaluates contextual syntactic and semantic features in transformer-based speech language models . authors compare SLMs to linguistic competence assessments for large language models. |
| Outcome: | a new study compares SLMs with linguistic competence assessments to assess speech recognition and understanding . the results show that SLM models encode grammatical features more robustly than conceptual ones . |
Measuring Watermarking under Jailbreaking: ASR Inflation and Goal-Compliance Mismatch (2026.findings-acl)
Copied to clipboard
| Challenge: | Recent work studies watermarking under benign prompts, but its behavior under jailbreaking prompts remains underexplored. |
| Approach: | They evaluate six methods on four LLMs using two jailbreak benchmarks and three settings: Static, AutoDAN, and DSN. |
| Outcome: | The proposed methods inflate judge-based attack success rate under jailbreaking, but not harmful-goal compliance. |
How Do LLMs "Trust" Unknown Knowledge? An Unknown Knowledge Based Jailbreak Attack (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing research on how to effectively utilize unknown knowledge has focused on how it can be used to enhance LLMs' performance in specialized fields. |
| Approach: | They propose a completely unrestricted and fully randomized jailbreak attack that embeds malicious queries within trust-enhanced unknown knowledge. |
| Outcome: | The proposed method achieves 99% to 100% ASR on all tested LLMs, including the latest GPT-5.1, and becomes SOTA. |
Learning to Conceal Risk: Controllable Multi-turn Red Teaming for LLMs in the Financial Domain (2026.acl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are increasingly deployed in high-stakes domains such as finance where unsafe behavior can lead to serious regulatory risks. |
| Approach: | They propose a black-box multi-turn risk-concealed redteaming framework that progressively conceals surface-level risk while exploiting regulatory-violating behaviors. |
| Outcome: | Experiments on nine widely used LLMs show that the proposed framework achieves 93.19% average attack success rate (ASR) and improves the average ASR to 95.00%. |
LASA: Language-Agnostic Semantic Alignment at the Semantic Bottleneck for LLM Safety (2026.acl-long)
Copied to clipboard
Junxiao Yang, Haoran Liu, Jinzhe Tu, Jiale Cheng, Zhexin Zhang, Shiyao Cui, Jiaqi Weng, Jialing Tao, Hui Xue, Hongning Wang, Han Qiu, Minlie Huang
| Challenge: | Large language models (LLMs) have demonstrated better safety performance in high-resource languages than in low-resourced languages. |
| Approach: | They propose language-agnostic semantic alignment (LASA) which anchors safety alignment directly in semantic bottlenecks. |
| Outcome: | The proposed approach significantly improves safety across all languages: average attack success rate drops from 24.7% to 2.8% on LLaMA-3.1-8B-Instruct and remains within 3–4% across Qwen2.5 and Qwend3 Instruct models (7B–32B). |
SafeMT: Multi-turn Safety for Multimodal Language Models (2026.acl-long)
Copied to clipboard
Han Zhu, Juntao Dai, Jiaming Ji, Haoran Li, Chengkun Cai, Pengcheng Wen, Chi-Min Chan, Boyuan Chen, Yaodong Yang, Sirui Han, Yike Guo
| Challenge: | Multi-turn dialogues pose a greater risk than single prompts, but existing safety benchmarks do not account for this situation. |
| Approach: | They propose a benchmark that features dialogues of varying lengths generated from harmful queries accompanied by images. |
| Outcome: | The proposed model reduces multi-turn Attack Success Rate (ASR) compared to existing guard models. |
TagSpeech: End-to-End Multi-Speaker ASR and Diarization with Fine-Grained Temporal Grounding (2026.acl-long)
Copied to clipboard
| Challenge: | a critical ambiguity persists regarding what constitutes "joint ASR and diarization" a unified framework for multi-speaker ASR is proposed, but it is not yet clear what constitute "diarization." |
| Approach: | They propose a unified LLM-based framework that uses Temporal Anchor Grounding for joint multi-speaker ASR and diarization. |
| Outcome: | The proposed framework improves on AMI and AliMeeting benchmarks on speaker-content alignment . the proposed framework achieves consistent improvements in Diarization Error Rate over strong baselines . |
Mind the Pause: Disfluency-Aware Objective Tuning for Multilingual Speech Correction with LLMs (2026.acl-long)
Copied to clipboard
| Challenge: | Spontaneous speech is rarely fluent, and disfluencies can degrade readability and reliability . a sequence tagger first marks disfluent tokens, and these signals guide instruction fine-tuning . |
| Approach: | They propose a multilingual correction pipeline where a sequence tagger first marks disfluent tokens . they add a contrastive learning objective that penalizes the reproduction of disfluency tokens. |
| Outcome: | The proposed model improves readability and reliability of ASR transcripts in three languages . disfluencies can cause misinterpretations, incoherent responses, poor user experience . |