LegoSLM: Connecting LLM with Speech Encoder using CTC Posteriors (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Recent studies show that pre-trained speech encoders and large language models can perform suboptimal performance on a range of spoken language processing tasks. |
| Approach: | They propose to combine large-scale pre-trained speech encoders and large-language models for better performance on automatic speech recognition tasks. |
| Outcome: | The proposed model can get an average of 49% WER reduction over the baseline model on 8 MLS testsets. |
Similar Papers
Stacked Acoustic-and-Textual Encoding: Integrating the Pre-trained Models into Speech Translation Encoders (2021.acl-long)
Copied to clipboard
| Challenge: | End-to-end Speech Translation (E2E ST) encoders lack global context representation, whereas MT encoder lacks it. |
| Approach: | They propose a Stacked Acoustic-and-Textual Encoding method for speech translation . they propose an adaptor module to alleviate representation inconsistency . |
| Outcome: | The proposed method achieves state-of-the-art BLEU scores of 18.3 and 25.2 on two ST tasks. |
SEAM: Bridging the Temporal-Semantic Granularity Gap for LLM-based Speech Recognition (2026.findings-eacl)
Copied to clipboard
| Challenge: | Existing duration-based methods generate embeddings at fixed rates, creating distributional mismatch with LLM pre-training. |
| Approach: | They propose an encoder-decoder architecture that generates embeddings at variable rates through cross-attention between speech features and text embeddables. |
| Outcome: | The proposed architecture achieves competitive performance on LibriSpeech (2.6%/5.2% WER) and 4.7% WER on TED-LIUM-v2 with a multi-stage training strategy and First Token Guidance. |
LLaST: Improved End-to-end Speech Translation System Leveraged by Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | ***LLaST*** is a framework for building high-performance Large Language model based Speech-to-text Translation systems. |
| Approach: | They propose a framework for building high-performance Large Language model based Speech-to-text Translation systems. |
| Outcome: | The proposed model outperforms the CoVoST-2 benchmark and showcases exceptional scaling capabilities powered by LLMs. |
Self-Distillation for Model Stacking Unlocks Cross-Lingual NLU in 200+ Languages (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) excel on English NLU tasks, yet struggle to extend their NLU capabilities to underrepresented languages. |
| Approach: | They integrate machine translation models (MT) directly into LLM backbones via sample-efficient self-distillation. |
| Outcome: | The proposed model outperforms translation-test models on 127 low-resource languages. |
Massive End-to-end Speech Recognition Models with Time Reduction (2024.naacl-long)
Copied to clipboard
Weiran Wang, Rohit Prabhavalkar, Haozhe Shan, Zhong Meng, Dongseong Hwang, Qiujia Li, Khe Chai Sim, Bo Li, James Qin, Xingyu Cai, Adam Stooke, Chengjian Zheng, Yanzhang He, Tara Sainath, Pedro Moreno Mengibar
| Challenge: | Using the neural architecture of Google’s universal speech model, we reduce the frame rate and speed up training and inference. |
| Approach: | They propose to use the neural architecture of Google’s universal speech model with additional funnel pooling layers to significantly reduce the frame rate and speed up training and inference. |
| Outcome: | The proposed methods work with both connectionist temporal classification (CTC) and RNN-Transducer (RNN-T) and over two domains. |
LLM-Codec: Neural Audio Codec Meets Language Model Objectives (2026.findings-acl)
Copied to clipboard
| Challenge: | Neural audio codecs are optimized for waveform reconstruction rather than autoregressive prediction. |
| Approach: | They propose to augment codec training with language-model-facing objectives while keeping both codec and LLM architectures unchanged. |
| Outcome: | The proposed model improves speech coherence and predictability by preserving the semantic alignment between audio and text representations. |
Language Family Matters: Evaluating SpeechLLMs Across Linguistic Boundaries (2026.eacl-short)
Copied to clipboard
| Challenge: | Existing approaches to integrate speech encoders with large language models (LLMs) have limited resources and lack linguistic relatedness. |
| Approach: | They propose a connector-sharing strategy based on linguistic family membership that allows one connector per family to share a frozen speech encoder with a pretrained LLM. |
| Outcome: | The proposed system reduces parameter count while improving generalization across domains, compared with existing connectors. |
Bridging the Temporal Gap in Multimodal LLMs: Deeply Stacking Temporal Tokens for Audio-Visual Speech Recognition (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing audio-visual speech recognition systems suffer from a temporal gap . visual speech patterns captured from lip movements provide complementary information that remains inherently robust to acoustic noise. |
| Approach: | They propose a framework that deeply stacks temporal tokens across both encoding and decoding stages to bridge this temporal gap. |
| Outcome: | The proposed framework outperforms existing supervised, self-supervised, and LLM-based methods by 6.1% on LRS2 and 7.8% on LLS3. |
Integrating Pre-Trained Speech and Language Models for End-to-End Speech Recognition (2024.findings-acl)
Copied to clipboard
| Challenge: | Mainstream of automatic speech recognition (ASR) has shifted from pipeline methods to end-to-end (E2E) methods. |
| Approach: | They propose to integrate a pre-trained speech representation model and a large language model (LLM) for automatic speech recognition in an end-to-end manner. |
| Outcome: | The proposed model achieves comparable performance to modern E2E ASR models by utilizing powerful pre-training models with the proposed integrated approach. |
Long-Form Speech Translation through Segmentation with Finite-State Decoding Constraints on Large Language Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | a challenge in speech translation is that plenty of spoken content is long-form, but short units are necessary for obtaining high-quality translations. |
| Approach: | They propose a large language model to split long ASR transcripts into segments that can be independently translated to maximize translation quality. |
| Outcome: | The proposed model improves the average BLEU by 2.9 points for English–German, English–Spanish, and English–Arabic TED talk translation in 9 sets. |