Hui Zhang, Tian Yuan, Junkun Chen, Xintong Li, Renjie Zheng, Yuxin Huang, Xiaojie Chen, Enlei Gong, Zeyu Chen, Xiaoguang Hu, Dianhai Yu, Yanjun Ma, Liang Huang
| Challenge: | PaddleSpeech is an open-source speech toolkit that supports speech-to-text and text-to speech tasks. |
| Approach: | They describe the design philosophy and core architecture of PaddleSpeech to support several essential speech-to-text and text-to speech tasks. |
| Outcome: | The proposed framework achieves competitive or state-of-the-art performance on various speech datasets and implements the most popular methods. |
Similar Papers
MobileSpeech: A Fast and High-Fidelity Framework for Mobile Zero-Shot Text-to-Speech (2024.acl-long)
Copied to clipboard
| Challenge: | Existing zero-shot text-to-speech systems require a few seconds of unseen speaker voice prompts to generate high-quality voices. |
| Approach: | They propose a zero-shot text-to-speech system based on mobile devices . they use a discrete speech codec to integrate hierarchical information from the codec . |
| Outcome: | The proposed system achieves RTF of 0.09 on a single A100 GPU and has been successfully deployed on mobile devices. |
ESPnet-ST: All-in-One Speech Translation Toolkit (2020.acl-demos)
Copied to clipboard
Hirofumi Inaguma, Shun Kiyono, Kevin Duh, Shigeki Karita, Nelson Yalta, Tomoki Hayashi, Shinji Watanabe
| Challenge: | ESPnet-ST is a new project for the quick development of speech-to-speech translation systems. |
| Approach: | They propose a framework for rapid development of speech-to-speech translation systems . they provide all-in-one recipes including data pre-processing, feature extraction, training, and decoding pipelines . |
| Outcome: | The proposed model outperforms the current state-of-the-art models on a wide range of benchmark datasets. |
InteractSpeech: A Speech Dialogue Interaction Corpus for Spoken Dialogue Model (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Spoken Dialogue models face challenges in handling nuanced interactional phenomena, such as interruptions and backchannels. |
| Approach: | They propose to use a 150-hour English speech interaction dialogue dataset to empower spoken dialogue models with nuanced real-time interaction capabilities. |
| Outcome: | The proposed dataset trains and evaluates a speech understanding model that classifies key interactional events directly from audio. |
OZSpeech: One-step Zero-shot Speech Synthesis with Learned-Prior-Conditioned Flow Matching (2025.acl-long)
Copied to clipboard
| Challenge: | Text-to-speech systems have seen significant advances in recent years, driven by improvements in deep learning and neural network architectures. |
| Approach: | They propose a method to explore optimal transport conditional flow matching with one-step sampling and a learned prior as the condition, effectively disregarding preceding states and reducing the number of sampling steps. |
| Outcome: | The proposed method achieves promising performance over existing methods in content accuracy, naturalness, prosody generation, and speaker style preservation. |
Arcee’s MergeKit: A Toolkit for Merging Large Language Models (2024.emnlp-industry)
Copied to clipboard
Charles Goddard, Shamane Siriwardhana, Malikeh Ehghaghi, Luke Meyers, Vladimir Karpukhin, Brian Benedict, Mark McQuade, Jacob Solawetz
| Challenge: | Open-source language models can merge their parameters to improve performance and versatility without additional training. |
| Approach: | They propose to integrate model checkpoints into powerful multitask models without additional training. |
| Outcome: | the library has facilitated the merging of thousands of models, contributing to some of the world’s most powerful open-source model checkpoints. |
Praaline: An Open-Source System for Managing, Annotating, Visualising and Analysing Speech Corpora (P18-4)
Copied to clipboard
| Challenge: | Praaline is an open-source software system for constituting and managing spoken language and multimodal corpora. |
| Approach: | They present the latest developments of Praaline, an open-source software system for constituting and managing spoken language and multimodal corpora. |
| Outcome: | The proposed system can be used for creating, managing, visualising and analysing spoken language and multimodal corpora. |
OpenPrompt: An Open-source Framework for Prompt-learning (2022.acl-demo)
Copied to clipboard
| Challenge: | Prompt-learning is a new paradigm in natural language processing, adapting pre-trained language models to cloze-style prediction, autoregressive modeling, or sequence to sequence generation. |
| Approach: | They propose a framework for prompt-learning that integrates pre-trained language models with a unified framework. |
| Outcome: | The proposed framework is easy to use and flexible enough to integrate with other frameworks. |
fairseq Sˆ2: A Scalable and Integrable Speech Synthesis Toolkit (2021.emnlp-demo)
Copied to clipboard
| Challenge: | Speech synthesis is the task of generating speech waveforms with desired characteristics, including but not limited to textual content, speaker identity, and speaking styles. |
| Approach: | They propose a fairseq extension for speech synthesis that implements autoregressive and non-AR text-to-speech models and their multi-speaker variants. |
| Outcome: | The proposed extension can train autoregressive and non-AR models and their multi-speaker variants with less curated data and has automatic metrics to facilitate faster iteration and analysis. |
OpenSLU: A Unified, Modularized, and Extensible Toolkit for Spoken Language Understanding (2023.acl-demo)
Copied to clipboard
| Challenge: | Spoken Language Understanding (SLU) is a task-oriented dialogue system . open-source toolkit provides a unified, modularized, and extensible toolkit for SLU . |
| Approach: | They introduce an open-source toolkit to provide a unified toolkit for spoken language understanding. |
| Outcome: | The proposed toolkit unifies 10 models for both single-intent and multi-intention scenarios. |
AudioBench: A Universal Benchmark for Audio Large Language Models (2025.naacl-long)
Copied to clipboard
Bin Wang, Xunlong Zou, Geyu Lin, Shuo Sun, Zhuohan Liu, Wenyu Zhang, Zhengyuan Liu, AiTi Aw, Nancy F. Chen
| Challenge: | Existing evaluation regimes for audio large language models do not cover the breadth of their possible use cases. |
| Approach: | They propose to use AudioBench to evaluate audio large language models . they found that no single model excels consistently across all tasks . |
| Outcome: | The proposed evaluation targets speech understanding, audio scene understanding, and voice understanding (paralinguistic) . no single model excels consistently across all tasks, the paper found . |