Papers by Xutai Ma
Fairseq S2T: Fast Speech-to-Text Modeling with Fairseq (2020.aacl-demo)
Copied to clipboard
| Challenge: | End-to-end sequence-to sequence (S2S) modeling has witnessed rapid growth in speech-totext (ST) tasks. |
| Approach: | They introduce fairseq S2T, a fairsq extension for speech-to-text modeling tasks such as end-to end speech recognition and speech-text translation. |
| Outcome: | The proposed extension provides end-to-end workflows from data pre-processing, model training to offline (online) inference. |
Evaluating the IWSLT2023 Speech Translation Tasks: Human Annotations, Automatic Metrics, and Segmentation (2024.lrec-main)
Copied to clipboard
Matthias Sperber, Ondřej Bojar, Barry Haddow, Dávid Javorský, Xutai Ma, Matteo Negri, Jan Niehues, Peter Polák, Elizabeth Salesky, Katsuhito Sudoh, Marco Turchi
| Challenge: | a meta-analysis of human evaluation for speech translation has not been conducted . noisy data and segmentation mismatches are challenges for automatic metrics . |
| Approach: | They propose an evaluation strategy based on automatic resegmentation and direct assessment with segment context. |
| Outcome: | The proposed evaluation strategy is robust and scores well-correlated with other types of human judgements. |
Broad-Coverage Semantic Parsing as Transduction (D19-1)
Copied to clipboard
| Challenge: | Existing approaches to broad-coverage semantic parsing are not applicable to all frameworks because of the lack of explicit alignments between tokens in the sentence and nodes in the semantic graph. |
| Approach: | They propose a transduction parsing paradigm that unifies different broad-coverage semantic parsers into a paradigm that leverages multiple attention mechanisms to build meaning representation. |
| Outcome: | The proposed approach improves state-of-the-art on AMR, SDP and UCCA and is competitive with the state- of-the art on SDP. |
Direct Speech-to-Speech Translation With Discrete Units (2022.acl-long)
Copied to clipboard
Ann Lee, Peng-Jen Chen, Changhan Wang, Jiatao Gu, Sravya Popuri, Xutai Ma, Adam Polyak, Yossi Adi, Qing He, Yun Tang, Juan Pino, Wei-Ning Hsu
| Challenge: | Existing direct speech-to-speech translation models rely on text generation as an intermediate step. |
| Approach: | They propose a direct speech-to-speech translation model that translates speech from one language to another without relying on intermediate text generation. |
| Outcome: | The proposed model produces 6.7 BLEUs in the Fisher Spanish-English dataset when trained without any text transcripts and with text supervision. |
Seeing is Believing: Emotion-Aware Audio-Visual Language Modeling for Expressive Speech Generation (2025.findings-emnlp)
Copied to clipboard
| Challenge: | AVLM integrates full-face visual cues into a pre-trained expressive speech model. |
| Approach: | They propose an Audio-Visual Language Model (AVLM) for expressive speech generation by integrating full-face visual cues into a pre-trained expressive speech model. |
| Outcome: | The proposed model incorporates full-face visual cues into a pre-trained expressive speech model. |
Cross-lingual Decompositional Semantic Parsing (D18-1)
Copied to clipboard
| Challenge: | Renewed interest in semantic analysis has led to a surge of proposed new frameworks . many of these efforts are limited to the analysis of English, but with a number of exceptions e.g., recent efforts in Minimal Recursion Semantics (MRS) and multilingual FrameNet annotation and parsing. |
| Approach: | They propose a cross-lingual decompositional semantic analysis task based on a target language . they propose 'end-to-end' model with an annotating mechanism that supports intra-sentential coreference . |
| Outcome: | The proposed model outperforms baselines by at least 1.75 F1 score on an evaluation dataset. |
Hybrid Transducer and Attention based Encoder-Decoder Modeling for Speech-to-Text Tasks (2023.acl-long)
Copied to clipboard
| Challenge: | Neural based end-to-end frameworks have achieved remarkable success in speech-totext tasks, such as automatic speech recognition (ASR) and speech- totext translation (ST). |
| Approach: | They propose to combine Transducer and Attention based Encoder-Decoder (TAED) for speech-to-text tasks and leverage AED's strength in non-monotonic sequence to sequence learning while retaining Transducers streaming property. |
| Outcome: | The proposed model outperforms Transducer and Attention based Encoder-Decoder (TAED) on the MuST-C dataset and shows that it is not bound by any specific language model. |
AMR Parsing as Sequence-to-Graph Transduction (P19-1)
Copied to clipboard
| Challenge: | Abstract Meaning Representation (AMR) parsing is the task of transducing natural language text into AMR, a graphbased formalism used for capturing sentence-level semantics. |
| Approach: | They propose a model that treats AMR parsing as sequence-to-graph transduction by aligner-free, and can be effectively trained with limited amounts of labeled AMR data. |
| Outcome: | The proposed model outperforms all previously reported SMATCH scores on AMR 2.0 (76.3%) and AMR 1.0 (70.2%). |
SimulMT to SimulST: Adapting Simultaneous Text Translation to End-to-End Simultaneous Speech Translation (2020.aacl-main)
Copied to clipboard
| Challenge: | Using end-to-end Simultaneous text translation, we adapt wait-k and monotonic multihead attention to end- to-end simultaneous speech translation. |
| Approach: | They propose to combine a fixed and flexible pre-decision module with fixed and flexibility policies to adapt simultaneous text translation methods such as wait-k and monotonic multihead attention to end-to-end simultaneous speech translation. |
| Outcome: | The proposed method can generate translations with maximum quality and minimal latency, targeting video caption translations and real-time language interpreter. |
SIMULEVAL: An Evaluation Toolkit for Simultaneous Translation (2020.emnlp-demos)
Copied to clipboard
| Challenge: | SimulEval is an evaluation toolkit for simultaneous text and speech translation. |
| Approach: | They propose a server-client scheme for simultaneous translation that uses server input and client policies to evaluate models. |
| Outcome: | The proposed evaluation toolkit is available for both text and speech translation. |