Simultaneous Speech-to-Text Translation Web Application for Estonian

Challenge:	a new open-source web application for simultaneous speech-to-text translation is developed for Estonian . the system translates live Estonian speech into English, Russian, and Ukrainian text, and also supports English-to Estonian translation.
Approach:	They propose a web application that combines streaming speech recognition with a simultaneous translation model.
Outcome:	The proposed system outperforms existing streaming speech recognition systems in Estonian-to-English translation.

Similar Papers

Overview

Challenge

Approach

Outcome

Direct Simultaneous Speech-to-Text Translation Assisted by Synchronized Streaming ASR (2021.findings-acl)

Copied to clipboard

Junkun Chen, Mingbo Ma, Renjie Zheng, Liang Huang

Text Generation

Challenge:	Existing approaches to simultaneous speech-to-text translation suffer from error propagation and extra latency.
Approach:	They propose a new paradigm for simultaneous speech-to-text translation using two separate decoders . they use multitask learning to jointly learn these two tasks with a shared encoder .
Outcome:	The proposed method achieves substantially better translation quality at similar levels of latency.

Streaming Models for Joint Speech Recognition and Translation (2021.eacl-main)

Copied to clipboard

Orion Weller, Matthias Sperber, Christian Gollan, Joris Kluivers

Text Generation

Challenge:	Using end-to-end models for speech translation has become a focus of the ST community . cascaded models have the advantage of including automatic speech recognition output .
Approach:	They propose a model that condenses sound waves into translated text and integrates automatic speech recognition outputs into the models.
Outcome:	The proposed model is statistically similar to cascading models, but has half the number of parameters.

Tutorial: End-to-End Speech Translation (2021.eacl-tutorials)

Copied to clipboard

Jan Niehues, Elizabeth Salesky, Marco Turchi, Matteo Negri

Multimodality

Challenge:	Speech translation is the translation of speech in one language typically to text in another, traditionally accomplished through a combination of automatic speech recognition and machine translation.
Approach:	This tutorial introduces the techniques used in cutting-edge research on speech translation.
Outcome:	The proposed models achieve state-of-the-art performance with end-to-end speech translation for both high- and low-resource languages.

SimulSpeech: End-to-End Simultaneous Speech to Text Translation (2020.acl-main)

Copied to clipboard

Yi Ren, Jinglin Liu, Xu Tan, Chen Zhang, Tao Qin, Zhou Zhao, Tie-Yan Liu

Text Generation

Challenge:	SimulSpeech is an end-to-end simultaneous speech to text translation system . conventional approaches to simultaneous speech translation divide the translation process into two stages .
Approach:	They develop an end-to-end simultaneous speech to text translation system which translates speech in source language to text in target language concurrently.
Outcome:	The proposed system achieves reasonable BLEU scores and lower delay compared to full-sentence translation model.

StreamSpeech: Simultaneous Speech-to-Speech Translation with Multi-task Learning (2024.acl-long)

Copied to clipboard

Shaolei Zhang, Qingkai Fang, Shoutao Guo, Zhengrui Ma, Min Zhang, Yang Feng

Text Generation

Challenge:	Existing simultaneous translation methods focus on text-to-text and speech-totext translation.
Approach:	They propose a Simul-S2ST model that jointly learns translation and simultaneous policy in a unified framework of multi-task learning.
Outcome:	The proposed model can perform offline and simultaneous speech recognition, speech translation and speech synthesis via an "All-in-One" seamless model.

LLaST: Improved End-to-end Speech Translation System Leveraged by Large Language Models (2024.findings-acl)

Copied to clipboard

Xi Chen, Songyang Zhang, Qibing Bai, Kai Chen, Satoshi Nakamura

Speech & Audio in NLP

Challenge:	*LLaST* is a framework for building high-performance Large Language model based Speech-to-text Translation systems.
Approach:	They propose a framework for building high-performance Large Language model based Speech-to-text Translation systems.
Outcome:	The proposed model outperforms the CoVoST-2 benchmark and showcases exceptional scaling capabilities powered by LLMs.

Fluent and Low-latency Simultaneous Speech-to-Speech Translation with Self-adaptive Training (2020.findings-emnlp)

Copied to clipboard

Renjie Zheng, Mingbo Ma, Baigong Zheng, Kaibo Liu, Jiahong Yuan, Kenneth Church, Liang Huang

Multimodality

Challenge:	Current approaches to simultaneous speech-to-speech translation accumulate more and more latencies in later sentences when the speaker talks faster.
Approach:	They propose a method which generates more fluent target speech latency than the baseline . they propose to use self-adaptive translation to adjust the length of translations to accommodate different source speech rates.
Outcome:	Xiong et al., 2019) show that the proposed method generates more fluent target speech latency than baseline . authors say it provides more natural communication process than speech-to-text translation . xiong and colleagues say the proposed technique is more efficient than current approaches .

Simultaneous Translation (2020.emnlp-tutorials)

Copied to clipboard

Liang Huang, Colin Cherry, Mingbo Ma, Naveen Arivazhagan, Zhongjun He

Multimodality

Challenge:	Simultaneous translation is a problem that has long been considered one of the hardest problems in AI . this tutorial will provide a deep understanding of the history and the recent advances in simultaneous translation.
Approach:	This tutorial will examine the design and evaluation of policies for simultaneous translation . it will provide an overview of the history and recent advances in simultaneous translation.
Outcome:	This tutorial will examine the design and evaluation of policies for simultaneous translation .

Direct Segmentation Models for Streaming Speech Translation (2020.emnlp-main)

Copied to clipboard

Javier Iranzo-Sánchez, Adrià Giménez Pastor, Joan Albert Silvestre-Cerdà, Pau Baquero-Arnal, Jorge Civera Saiz, Alfons Juan

Text Generation

Challenge:	Existing approaches to stream ST combine advances in ASR and MT to achieve high quality translations without compromising the speed of the system.
Approach:	They propose to concatenate an Automatic Speech Recognition system followed by a Machine Translation system.
Outcome:	The proposed models improve on the Europarl-ST dataset on the BLEU score.

AV-TranSpeech: Audio-Visual Robust Speech-to-Speech Translation (2023.acl-long)

Copied to clipboard

Rongjie Huang, Huadai Liu, Xize Cheng, Yi Ren, Linjun Li, Zhenhui Ye, Jinzheng He, Lichao Zhang, Jinglin Liu, Xiang Yin, Zhou Zhao

Multimodality

Challenge:	Existing models for speech-to-speech translation suffer from distinct degradation in noisy environments and fail to translate visual speech.
Approach:	They propose a text-based audio-visual speech-to-speech translation model that integrates visual information with audio-only data to improve system robustness.
Outcome:	The proposed model outperforms models trained on audio-only corpus in two languages . it also improves with low-resource audio-visual data, compared with baselines .