Fast Streaming Transducer ASR Prototyping via Knowledge Distillation with Whisper (2024.findings-emnlp)
Copied to clipboard
Iuliia Thorbecke, Juan Pablo Zuluaga Gomez, Esaú Villatoro-tello, Shashi Kumar, Pradeep Rangappa, Sergio Burdisso, Petr Motlicek, Karthik S, Aravind Ganapathiraju
| Challenge: | a recent study shows that training of ASR models with little to no supervised data is challenging. |
| Approach: | They propose a framework to train streaming Transformer-Transducer models with pseudo-labeled (PL) speech from foundational speech models. |
| Outcome: | The proposed framework can be trained from scratch with pseudo-labeled speech from foundational speech models (FSMs) the proposed framework is validated on 6 languages from CommonVoice and proposes multiple heuristics to filter out hallucinated PLs. |
Similar Papers
Unifying Streaming and Non-streaming Zipformer-based ASR (2025.acl-industry)
Copied to clipboard
Bidisha Sharma, Karthik Pandia D S, Shankar Venkatesan, Jeena J Prakash, Shashi Kumar, Malolan Chetlur, Andreas Stolcke
| Challenge: | Existing frameworks for streaming and non-streaming ASR models have been used to reduce development, training and deployment costs. |
| Approach: | They propose to use dynamic right-context through chunked attention masking to train zipformer-based ASR models. |
| Outcome: | The proposed framework reduces word error by relative 7.9% with a small degradation in user-perceived latency. |
F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matching (2025.acl-long)
Copied to clipboard
| Challenge: | Recent research in Text-to-Speech (TTS) has experienced great advancement . current models can synthesize speech for any given text and mimic the speaker of audio prompt. |
| Approach: | They propose a fully non-autoregressive text-to-speech system based on flow matching with Diffusion Transformer (DiT) without complex designs such as duration model, text encoder, and phoneme alignment, the text input is simply padded with filler tokens to the same length as input speech, and then denoising is performed for speech generation. |
| Outcome: | The proposed system achieves an inference RTF of 0.15, which is greatly improved compared to state-of-the-art diffusion-based models. |
Speech-enriched Memory for Inference-time Adaptation of ASR Models to Word Dictionaries (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing contextual biasing techniques require additional parameterization . state-of-the-art ASR systems often fail to recognize named entities or critical rare words . |
| Approach: | They propose an algorithm that uses nearest-neighbor matching to predict ASR models . a list of rare entities is indexed in a memory and then stored the best possible match . |
| Outcome: | The proposed algorithm improves the prediction of state-of-the-art ASR models on rare words . it prevents spurious matches by restricting to word-level matches . |
Hybrid Transducer and Attention based Encoder-Decoder Modeling for Speech-to-Text Tasks (2023.acl-long)
Copied to clipboard
| Challenge: | Neural based end-to-end frameworks have achieved remarkable success in speech-totext tasks, such as automatic speech recognition (ASR) and speech- totext translation (ST). |
| Approach: | They propose to combine Transducer and Attention based Encoder-Decoder (TAED) for speech-to-text tasks and leverage AED's strength in non-monotonic sequence to sequence learning while retaining Transducers streaming property. |
| Outcome: | The proposed model outperforms Transducer and Attention based Encoder-Decoder (TAED) on the MuST-C dataset and shows that it is not bound by any specific language model. |
Adapting Offline Speech Translation Models for Streaming with Future-Aware Distillation and Inference (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to streaming speech translation use an offline model with a wait-k policy . however, there is a mismatch problem with an offline inference model trained with complete utterances . |
| Approach: | They propose an offline streaming speech translation model with wait-k policy to support different latency requirements. |
| Outcome: | The proposed model achieves better trade-offs between translation quality and latency than baselines. |
From Tens of Hours to Tens of Thousands: Scaling Back-Translation for Speech Recognition (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in Automatic Speech Recognition (ASR) have been fueled by massive speech corpora, but extending coverage to diverse languages with limited resources remains a formidable challenge. |
| Approach: | They propose a pipeline that converts large-scale text corpora into synthetic speech using off-the-shelf text-to-speech (TTS) models. |
| Outcome: | The proposed pipeline generates 500,000 hours of synthetic speech in ten languages and achieves transcription error reductions of over 30%. |
Building Accurate Low Latency ASR for Streaming Voice Search in E-commerce (2023.acl-industry)
Copied to clipboard
| Challenge: | Recent years have witnessed the popularity of end-to-end ASR models, which have demonstrated higher accuracy compared to traditional pipelines with separate acoustic, pronunciation, and language models. |
| Approach: | They build accurate LSTM, attention and CTC based streaming ASR models for large-scale Hinglish voice search. |
| Outcome: | The proposed model achieves a word error rate (WER) of 3.69% without EOS and 4.78% with EOS, with 1300 ms (46.64%) reduction in latency. |
PRoDeliberation: Parallel Robust Deliberation for End-to-End Spoken Language Understanding (2024.findings-emnlp)
Copied to clipboard
Trang Le, Daniel Lazar, Suyoun Kim, Shan Jiang, Duc Le, Adithya Sagar, Aleksandr Livshits, Ahmed Aly, Akshat Shrivastava
| Challenge: | End-to-end models for Spoken Language Understanding have been autoregressive, resulting in higher latencies. |
| Approach: | They propose a method that uses Connectionist Temporal Classification to train robust non-autoregressive deliberation models. |
| Outcome: | The proposed method achieves 10x latency reduction over autoregressive models while preserving ability to correct ASR mistranscriptions. |
SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processing (2022.acl-long)
Copied to clipboard
Junyi Ao, Rui Wang, Long Zhou, Chengyi Wang, Shuo Ren, Yu Wu, Shujie Liu, Tom Ko, Qing Li, Yu Zhang, Zhihua Wei, Yao Qian, Jinyu Li, Furu Wei
| Challenge: | Existing work shows that pre-trained models can improve in various natural language processing tasks. |
| Approach: | They propose a unified-modal encoder-decoder framework that pre-trains speech-text representations using large-scale unlabeled speech and text data. |
| Outcome: | The proposed framework is superior to existing models on speech-to-text processing tasks. |
Stacked Acoustic-and-Textual Encoding: Integrating the Pre-trained Models into Speech Translation Encoders (2021.acl-long)
Copied to clipboard
| Challenge: | End-to-end Speech Translation (E2E ST) encoders lack global context representation, whereas MT encoder lacks it. |
| Approach: | They propose a Stacked Acoustic-and-Textual Encoding method for speech translation . they propose an adaptor module to alleviate representation inconsistency . |
| Outcome: | The proposed method achieves state-of-the-art BLEU scores of 18.3 and 25.2 on two ST tasks. |