VoiceCraft: Zero-Shot Speech Editing and Text-to-Speech in the Wild (2024.acl-long)
Copied to clipboard
| Challenge: | VoiceCraft is a token-infilling neural codec language model for speech editing and zero-shot text-to-speech evaluation. |
| Approach: | They introduce a token infilling neural codec language model that performs on speech editing and zero-shot text-to-speech tasks. |
| Outcome: | The proposed model outperforms previous models on speech editing and zero-shot text-to-speech tasks. |
Similar Papers
VoiceCraft-X: Unifying Multilingual, Voice-Cloning Speech Synthesis and Speech Editing (2025.emnlp-main)
Copied to clipboard
Zhisheng Zheng, Puyuan Peng, Anuj Diwan, Cong Phuoc Huynh, Xiaohang Sun, Zhu Liu, Vimal Bhat, David Harwath
| Challenge: | Autoregressive language model for multilingual speech editing and zero-shot text-to-speech synthesis is available in 11 languages. |
| Approach: | They introduce an autoregressive neural codec language model which unifies multilingual speech editing and zero-shot text-to-speech synthesis across 11 languages. |
| Outcome: | The model generates high-quality, natural-sounding speech, even with limited per-language data . it shows robust performance in diverse linguistic settings, even in limited per language data compared to other models . |
VoiceStar: Robust Zero-Shot Autoregressive TTS with Duration Control and Extrapolation (2026.findings-acl)
Copied to clipboard
| Challenge: | Neural codec language models (NCLMs) lack fine-grained controllability and inability to extrapolate to sequence lengths much longer than those seen during training. |
| Approach: | They propose a novel autoregressive encoder-decoder neural codec language model that can be trained with a Continuation-Prompt Mixed training system. |
| Outcome: | The proposed model outperforms or is on par with current state-of-the-art models on short-form benchmarks such as LibriSpeech and Seed-TTS in terms of intelligibility and naturalness. |
MobileSpeech: A Fast and High-Fidelity Framework for Mobile Zero-Shot Text-to-Speech (2024.acl-long)
Copied to clipboard
| Challenge: | Existing zero-shot text-to-speech systems require a few seconds of unseen speaker voice prompts to generate high-quality voices. |
| Approach: | They propose a zero-shot text-to-speech system based on mobile devices . they use a discrete speech codec to integrate hierarchical information from the codec . |
| Outcome: | The proposed system achieves RTF of 0.09 on a single A100 GPU and has been successfully deployed on mobile devices. |
Low-Resource Multilingual and Zero-Shot Multispeaker TTS (2022.aacl-main)
Copied to clipboard
| Challenge: | Currently, the amount of data needed for TTS is limited to the vast majority of the spoken languages. |
| Approach: | They propose to use language agnostic meta learning procedure to learn speaking a new language with just 5 minutes of training data while retaining the ability to infer the voice of even unseen speakers. |
| Outcome: | The proposed approach is able to learn speaking a new language using just 5 minutes of training data while retaining the ability to infer the voice of even unseen speakers in the newly learned language. |
ControlSpeech: Towards Simultaneous and Independent Zero-shot Speaker Cloning and Zero-shot Language Style Control (2025.acl-long)
Copied to clipboard
Shengpeng Ji, Qian Chen, Wen Wang, Jialong Zuo, Minghui Fang, Ziyue Jiang, Hai Huang, Zehan Wang, Xize Cheng, Siqi Zheng, Zhou Zhao
| Challenge: | Prior zero-shot TTS models only mimic the speaker’s voice without further control and adjustment capabilities while prior controllable TTS systems cannot perform speaker-specific voice generation. |
| Approach: | They propose a style control module that captures codec representations corresponding to timbre, content, and style in a discrete decoupling codec space. |
| Outcome: | The proposed system can fully clone the speaker's voice and perform speech-specific adjustment and control functions. |
T-Modules: Translation Modules for Zero-Shot Cross-Modal Machine Translation (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to perform zero-shot cross-modal transfer between speech and text are limited to a very small number of language pairs. |
| Approach: | They propose a method to perform zero-shot cross-modal transfer between speech and text for translation tasks by using a speech decoder. |
| Outcome: | The proposed model significantly improves state-of-the-art for zero-shot speech translation on Must-C. |
Takin-VC: Expressive Zero-Shot Voice Conversion via Adaptive Hybrid Content Encoding and Enhanced Timbre Modeling (2025.acl-long)
Copied to clipboard
Yang Yuguang, Yu Pan, Jixun Yao, Xiang Zhang, Jianhao Ye, Hongbin Zhou, Lei Xie, Lei Ma, Jianjun Zhao
| Challenge: | Expressive zero-shot voice conversion (VC) aims to modify source timbre to match unseen speaker . existing zero- shot VC systems struggle to reproduce paralinguistic information in highly expressive speech . |
| Approach: | They propose a framework for expressive zero-shot voice conversion that uses hybrid content encoding and memory-augmented context-aware timbre modeling. |
| Outcome: | The proposed framework surpasses state-of-the-art VC systems in speech naturalness, speaker similarity, and speaker similarness. |
DisCo_Speech: Controllable Zero-Shot Speech Generation with A Disentangled Speech Codec (2026.acl-long)
Copied to clipboard
Tao Li, Wenshuo Ge, Zhichao Wang, Zihao Cui, Yong Ma, Yingying Gao, Chao Deng, Shilei Zhang, Junlan Feng
| Challenge: | DisCo-Speech is a zero-shot controllable text-to-speech framework . standard codecs entangle timbre and prosody, which hinders independent control in continuation-based LMs. |
| Approach: | They propose a disentangled speech codec and an LM-based generator to solve this problem . they propose fusion and reconstruction that merges content and prosody into unified tokens . |
| Outcome: | DisCo-Speech achieves competitive voice cloning and superior zero-shot prosody control. |
Advancing Zero-shot Text-to-Speech Intelligibility across Diverse Domains via Preference Alignment (2025.acl-long)
Copied to clipboard
| Challenge: | Existing zero-shot text-to-speech systems struggle in challenging scenarios such as tongue twisters, repeated words, code-switching, and cross-lingual synthesis. |
| Approach: | They propose a dataset that leverages preference alignment techniques to improve performance . they also extend the Direct Preference Optimization framework to accommodate diverse TTS architectures . |
| Outcome: | The proposed dataset improves intelligibility, similarity, and audio quality for multiple models across domains. |
Evaluating Text-to-Speech Synthesis from a Large Discrete Token-based Speech Language Model (2024.lrec-main)
Copied to clipboard
| Challenge: | Recent advances in generative language modeling applied to discrete speech tokens presented a new avenue for text-to-speech (TTS) synthesis. |
| Approach: | They propose to use generative language modeling to generate text-to-speech (TTS) outputs by a discrete token-based model. |
| Outcome: | The proposed model is rated higher in naturalness and context appropriateness in listening tests compared to a conventional TTS. |