Papers by Matthias Sperber
Fluent Translations from Disfluent Speech in End-to-End Speech Translation (N19-1)
Copied to clipboard
| Challenge: | Disfluency removal is an intermediate step between speech recognition and machine translation (MT) with the rise of end-to-end speech translation systems, disfluency recognition and removal needs to be incorporated into the model architectures or handled as a post-processing step. |
| Approach: | They propose to use a sequence-to-sequence model to translate from noisy, disfluent speech to fluent text with disfluencies removed using the recently collected ‘copy-edited’ references for the Fisher Spanish-English dataset. |
| Outcome: | The proposed model generates fluent translations from disfluent speech using the recently collected ‘copy-edited’ references for the Fisher Spanish-English dataset. |
Toward Machine Interpreting: Lessons from Human Interpreting Studies (2025.emnlp-main)
Copied to clipboard
| Challenge: | Current speech translation systems are static and do not adapt to real-world situations in ways human interpreters do. |
| Approach: | They propose to model human interpreting using a new language model to improve usability . they argue that there is great potential to adopt many human interpreted principles . |
| Outcome: | The proposed models can be used to improve human interpreting and improve translation performance. |
Variational Neural Machine Translation with Normalizing Flows (2020.acl-main)
Copied to clipboard
| Challenge: | Existing frameworks for learning informative latent variables are limited by limitations . existing models rely on strong assumptions on distribution of latent code . |
| Approach: | They propose to apply a variational neural machine translation framework to a Transformer . they propose to introduce a more flexible approximate posterior based on normalizing flows . |
| Outcome: | The proposed framework outperforms baseline models under in-domain and out-of-domain conditions. |
Evaluating the IWSLT2023 Speech Translation Tasks: Human Annotations, Automatic Metrics, and Segmentation (2024.lrec-main)
Copied to clipboard
Matthias Sperber, Ondřej Bojar, Barry Haddow, Dávid Javorský, Xutai Ma, Matteo Negri, Jan Niehues, Peter Polák, Elizabeth Salesky, Katsuhito Sudoh, Marco Turchi
| Challenge: | a meta-analysis of human evaluation for speech translation has not been conducted . noisy data and segmentation mismatches are challenges for automatic metrics . |
| Approach: | They propose an evaluation strategy based on automatic resegmentation and direct assessment with segment context. |
| Outcome: | The proposed evaluation strategy is robust and scores well-correlated with other types of human judgements. |
Streaming Models for Joint Speech Recognition and Translation (2021.eacl-main)
Copied to clipboard
| Challenge: | Using end-to-end models for speech translation has become a focus of the ST community . cascaded models have the advantage of including automatic speech recognition output . |
| Approach: | They propose a model that condenses sound waves into translated text and integrates automatic speech recognition outputs into the models. |
| Outcome: | The proposed model is statistically similar to cascading models, but has half the number of parameters. |
Joint Speech Transcription and Translation: Pseudo-Labeling with Out-of-Distribution Data (2023.findings-acl)
Copied to clipboard
| Challenge: | a recent study shows that self-training can improve upon fully supervised baselines in low-resource settings for several sequence-to-sequence tasks. |
| Approach: | They propose to use pseudo-labeling to label unsupervised data and add it to the training pool. |
| Outcome: | The proposed setup improves on the unsupervised data by using pseudo-labeling . the proposed setup provides 0.4% absolute WER and 2.1 BLEU points for En–De . |
Exploring Phoneme-Level Speech Representations for End-to-End Speech Translation (P19-1)
Copied to clipboard
| Challenge: | Previous work on end-to-end translation from speech uses frame-level features as speech representations, which creates longer, sparser sequences than text. |
| Approach: | They propose a method to generate compressed phoneme-like speech representations that generate shorter, higher-level source sequences for translation. |
| Outcome: | The proposed method improves translation performance by 5 BLEU on high and low resource languages and reduces training time by 60%. |
Consistent Transcription and Translation of Speech (2020.tacl-1)
Copied to clipboard
| Challenge: | Existing models that translate without transcribing focus on translation quality, while transcription receives less emphasis. |
| Approach: | They propose a method to evaluate consistency and compare different approaches . they propose 'coupled inference' models that feature a coupled inference procedure can achieve strong consistency. |
| Outcome: | The proposed model is poorly suited to the joint transcription/translation task, but is strong enough to train for consistency. |
Speech Translation and the End-to-End Promise: Taking Stock of Where We Are (2020.acl-main)
Copied to clipboard
| Challenge: | Until recently, the only feasible approach to translating acoustic speech signals into text was the cascaded approach. |
| Approach: | They propose a classification of the main challenges of traditional approaches to speech translation . they argue that end-to-end models fall short due to compromises made to address data scarcity . |
| Outcome: | This paper provides a brief survey of the main challenges of traditional approaches in speech translation . it reveals that many end-to-end models fail due to compromises made to address data scarcity. |
Self-Attentional Models for Lattice Inputs (P19-1)
Copied to clipboard
| Challenge: | Existing work has extended recurrent neural networks to model lattice inputs but these models suffer from slow computation speeds. |
| Approach: | They propose to extend the paradigm of self-attention to handle lattice inputs by adding probabilistic reachability masks that incorporate latticae structure into the model and support lattics if available. |
| Outcome: | The proposed model outperforms baseline models while being much faster to compute than previous models. |
KIT Lecture Translator: Multilingual Speech Translation with One-Shot Learning (C18-2)
Copied to clipboard
Florian Dessloch, Thanh-Le Ha, Markus Müller, Jan Niehues, Thai-Son Nguyen, Ngoc-Quan Pham, Elizabeth Salesky, Matthias Sperber, Sebastian Stüker, Thomas Zenkel, Alexander Waibel
| Challenge: | In today's globalized world, communication is difficult and often the language barrier still prevents communication. |
| Approach: | They have developed a low-latency translation system that is adapted to lectures and covers several language pairs. |
| Outcome: | The proposed system improves performance but also covers several European languages. |
Paraphrases as Foreign Languages in Multilingual Neural Machine Translation (P19-2)
Copied to clipboard
| Challenge: | Unlike previous studies that use paraphrases at the word/phrase level, we train on parallel paraphrase training on closely related languages. |
| Approach: | They train on parallel paraphrases in the style of multilingual Neural Machine Translation (NMT) they train on translations of the whole corpus that are consistent in structure as paraphrase versions at the corpus level. |
| Outcome: | The proposed training on paraphrases outperforms the baselines on two languages and improves lexical choice and entropy. |
KIT-Multi: A Translation-Oriented Multilingual Embedding Corpus (L18-1)
Copied to clipboard
| Challenge: | Cross-lingual word embeddings are representations of words across languages in a shared continuous vector space. |
| Approach: | They propose a multilingual word embedding corpus which is acquired by neural machine translation and is based on monolingual data. |
| Outcome: | The proposed method is competitive with existing methods but on the cross-lingual document classification task, it obtains the best figures. |
End-to-End Speech Translation for Code Switched Speech (2022.findings-acl)
Copied to clipboard
Orion Weller, Matthias Sperber, Telmo Pires, Hendra Setiawan, Christian Gollan, Dominic Telaar, Matthias Paulik
| Challenge: | Code switching (CS) is the phenomenon of interchangeably using words and phrases from different languages. |
| Approach: | They propose a new ST corpus that extends the joint transcription and translation setup. |
| Outcome: | The proposed model performs well even when no training data is used. |