OpenSR: Open-Modality Speech Recognition via Maintaining Multi-Modality Alignment (2023.acl-long)
Copied to clipboard
| Challenge: | Speech Recognition often gets stuck in the lack of new domain utterances when training a model of new-domain speech. |
| Approach: | They propose a training system Open-modality Speech Recognition that enables zero-shot modality transfer . they use multi-modal alignment in phoneme space to maintain multi-modality alignment . |
| Outcome: | The proposed system achieves zero-shot modality transfer compared to existing methods . it achieves state-of-the-art performance on audio-visual speech recognition and lip-reading with 2.7% and 25.0%, respectively. |
Similar Papers
CLASP: Cross-modal Alignment Using Pre-trained Unimodal Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent advances in speech-text pretraining rely on parallel speech- text data . however, data accessibility is a challenge due to the limited data available. |
| Approach: | They propose a framework for jointly performing speech and text processing without parallel corpora during pre-training but only downstream. |
| Outcome: | The proposed framework extracts distinct representations for speech and text, aligning them effectively in a newly defined space using a multi-level contrastive learning mechanism. |
Towards Zero-shot Learning for End-to-end Cross-modal Translation Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | End-to-end zero-shot speech translation model is based on a zero-shot approach, but it is less competitive because of the limited amount of data available for multiple modalities. |
| Approach: | They propose an end-to-end zero-shot speech translation model that connects two pre-trained uni-modality modules via word rotator’s distance. |
| Outcome: | The proposed model performs better than or as well as those of the CTC-based models and can be trained in an end-to-end style to avoid error propagation. |
Pushing the Limits of Zero-shot End-to-End Speech Translation (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to end-to-end Speech Translation (ST) systems require limited data, which can cause data scarcity and performance degradation. |
| Approach: | They propose a method for zero-shot ST that bridges the modality gap without any paired ST data. |
| Outcome: | The proposed method bridges the modality gap without any paired ST data on a speech encoder and on MT models. |
Multi-Stage Multi-Modal Pre-Training for Automatic Speech Recognition (2024.lrec-main)
Copied to clipboard
Yash Jain, David M. Chan, Pranav Dheram, Aparna Khare, Olabanji Shonibare, Venkatesh Ravichandran, Shalini Ghosh
| Challenge: | Existing methods for pre-training for automatic speech recognition (ASR) focus on single-stage pre-train followed by fine-tuning on downstream task. |
| Approach: | They propose a multi-modal pre-training method that combines unsupervised pre-training with translation-based supervised mid-training. |
| Outcome: | The proposed method improves WERs by 38.45% over baselines on both Librispeech and SUPERB. |
Uni-Dubbing: Zero-Shot Speech Synthesis from Visual Articulation (2024.acl-long)
Copied to clipboard
Songju Lei, Xize Cheng, Mengjiao Lyu, Jianqiao Hu, Jintao Tan, Runlin Liu, Lingyu Xiong, Tao Jin, Xiandong Li, Zhou Zhao
| Challenge: | Multimodal speech synthesis is a key challenge due to the scarcity of datasets that pair audio with corresponding video. |
| Approach: | They propose a method that incorporates modality alignment during the pre-training phase on multimodal datasets and freezes the video modality extraction component and the encoder module within the pretrained weights. |
| Outcome: | The proposed method achieves a reduced word error rate (WER) of 31.73%, surpassing the previous best of 33.9% with single-modality audio. |
MIR-GAN: Refining Frame-Level Modality-Invariant Representations with Adversarial Network for Audio-Visual Speech Recognition (2023.acl-long)
Copied to clipboard
| Challenge: | Audio-visual speech recognition (AVSR) leverages multimodal signals to understand human speech. |
| Approach: | They propose an adversarial network to refine frame-level modality-invariant representations to bridge the distribution gap between modalities. |
| Outcome: | The proposed approach outperforms the state-of-the-art on public benchmarks LRS3 and LRS2 on the modalities of AVSR. |
Leveraging Unimodal Self-Supervised Learning for Multimodal Audio-Visual Speech Recognition (2022.acl-long)
Copied to clipboard
| Challenge: | Existing methods for audio-visual speech recognition use extra data to increase performance . a recent study shows that the use of unimodal self-supervised learning improves performance on multimodal tasks. |
| Approach: | They propose to use unimodal self-supervised learning to train AVSR models on unlabelled unilateral data. |
| Outcome: | The proposed model improves on lip reading sentences 2 by 30% even without an external language model. |
Hearing Lips in Noise: Universal Viseme-Phoneme Mapping and Transfer for Robust Audio-Visual Speech Recognition (2023.acl-long)
Copied to clipboard
| Challenge: | Existing efforts to improve robustness of audio-visual speech recognition with visual information focus on audio modality . current approaches introduce noise adaptation techniques to improve reliability of AVSR task . |
| Approach: | They propose a visual-invariant modality to strengthen robustness of audio-visual speech recognition (AVSR) it can adapt to any testing noises without dependence on noisy training data, a.k.a., unsupervised noise adaptation. |
| Outcome: | The proposed method outperforms existing state-of-the-arts on visual speech recognition task under various noisy and clean conditions. |
Understanding the Modality Gap: An Empirical Study on the Speech-Text Alignment Mechanism of Large Speech Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | LSLMs have impressive conversational generation abilities, but consistently fall short of traditional pipeline systems on semantic understanding benchmarks. |
| Approach: | They propose to analyze the performance gap between speech and text inputs through a systematic experiment . they find that representation similarity is strongly correlated with the modality gap . |
| Outcome: | The proposed models improve the accuracy of speech inputs and their semantic understanding benchmarks. |
Low-resource Neural Machine Translation with Cross-modal Alignment (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing neural machine translation techniques rely on large monolingual corpus, which is costly for some low-resource languages. |
| Approach: | They propose a cross-modal contrastive learning method to learn a shared space for all languages by additional visual modality. |
| Outcome: | The proposed method can learn cross-modal and cross-lingual alignment with small amount of image-text pairs and achieves significant improvements over the text-only baseline. |