RT-VC: Real-Time Zero-Shot Voice Conversion with Speech Articulatory Coding (2025.acl-demo)
Copied to clipboard
| Challenge: | Experimental evaluations show RT-VC delivers a 13.3% reduction in latency . voice conversion modifies speech to match the timbre of a target speaker while preserving content information. |
| Approach: | They propose a zero-shot real-time voice conversion system that leverages an articulatory feature space to naturally disentangle content and speaker characteristics. |
| Outcome: | The proposed system achieves a CPU latency of 61.4 ms, representing a 13.3% reduction in latency. |
Similar Papers
Rhythm Controllable and Efficient Zero-Shot Voice Conversion via Shortcut Flow Matching (2025.acl-long)
Copied to clipboard
Jialong Zuo, Shengpeng Ji, Minghui Fang, Mingze Li, Ziyue Jiang, Xize Cheng, Xiaoda Yang, Chen Feiyang, Xinyu Duan, Zhou Zhao
| Challenge: | Existing methods focus on disentangling speakers and content, while others focus on preserving the source's prosody. |
| Approach: | They propose a rhythm-controllable and efficient zero-shot voice conversion model that transforms the source speaker’s timbre into an unseen one while retaining speech content. |
| Outcome: | The proposed model adapts the linguistic content duration to the desired speaking style, facilitating the transfer of the target speaker’s rhythm. |
Takin-VC: Expressive Zero-Shot Voice Conversion via Adaptive Hybrid Content Encoding and Enhanced Timbre Modeling (2025.acl-long)
Copied to clipboard
Yang Yuguang, Yu Pan, Jixun Yao, Xiang Zhang, Jianhao Ye, Hongbin Zhou, Lei Xie, Lei Ma, Jianjun Zhao
| Challenge: | Expressive zero-shot voice conversion (VC) aims to modify source timbre to match unseen speaker . existing zero- shot VC systems struggle to reproduce paralinguistic information in highly expressive speech . |
| Approach: | They propose a framework for expressive zero-shot voice conversion that uses hybrid content encoding and memory-augmented context-aware timbre modeling. |
| Outcome: | The proposed framework surpasses state-of-the-art VC systems in speech naturalness, speaker similarity, and speaker similarness. |
EZ-VC: Easy Zero-shot Any-to-Any Voice Conversion (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Current voice conversion methods struggle in zero-shot cross-lingual settings . authors develop a method that can be used in zero shot cross-linguistic settings despite advances in technology . |
| Approach: | They propose a voice-conversion model that combines discrete speech representations with a non-autoregressive speech decoder. |
| Outcome: | The proposed approach excels in zero-shot cross-lingual settings even for unseen languages and accents. |
O_O-VC: Synthetic Data-Driven One-to-One Alignment for Any-to-Any Voice Conversion (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Traditional voice conversion methods attempt to separate speaker identity and linguistic information into distinct representations, but this method often leads to information loss during training. |
| Approach: | They propose a method that leverages synthetic speech data generated by a pretrained model . synthetic data pairs that share the same linguistic content are used as input-output pairs . |
| Outcome: | The proposed method outperforms state-of-the-art methods in speaker-to-voice conversions. |
MobileSpeech: A Fast and High-Fidelity Framework for Mobile Zero-Shot Text-to-Speech (2024.acl-long)
Copied to clipboard
| Challenge: | Existing zero-shot text-to-speech systems require a few seconds of unseen speaker voice prompts to generate high-quality voices. |
| Approach: | They propose a zero-shot text-to-speech system based on mobile devices . they use a discrete speech codec to integrate hierarchical information from the codec . |
| Outcome: | The proposed system achieves RTF of 0.09 on a single A100 GPU and has been successfully deployed on mobile devices. |
Uni-Dubbing: Zero-Shot Speech Synthesis from Visual Articulation (2024.acl-long)
Copied to clipboard
Songju Lei, Xize Cheng, Mengjiao Lyu, Jianqiao Hu, Jintao Tan, Runlin Liu, Lingyu Xiong, Tao Jin, Xiandong Li, Zhou Zhao
| Challenge: | Multimodal speech synthesis is a key challenge due to the scarcity of datasets that pair audio with corresponding video. |
| Approach: | They propose a method that incorporates modality alignment during the pre-training phase on multimodal datasets and freezes the video modality extraction component and the encoder module within the pretrained weights. |
| Outcome: | The proposed method achieves a reduced word error rate (WER) of 31.73%, surpassing the previous best of 33.9% with single-modality audio. |
VoiceCraft: Zero-Shot Speech Editing and Text-to-Speech in the Wild (2024.acl-long)
Copied to clipboard
| Challenge: | VoiceCraft is a token-infilling neural codec language model for speech editing and zero-shot text-to-speech evaluation. |
| Approach: | They introduce a token infilling neural codec language model that performs on speech editing and zero-shot text-to-speech tasks. |
| Outcome: | The proposed model outperforms previous models on speech editing and zero-shot text-to-speech tasks. |
StreamVoice: Streamable Context-Aware Language Modeling for Real-time Zero-Shot Voice Conversion (2024.acl-long)
Copied to clipboard
| Challenge: | Existing LM-based VC models require offline conversion from source semantics to acoustic features, limiting their deployment to real-time applications. |
| Approach: | They propose a streaming LM-based model for zero-shot voice conversion that uses a fully causal context-aware LM with a temporal-independent acoustic predictor to facilitate real-time conversion given arbitrary speaker prompts and source speech. |
| Outcome: | The proposed model achieves comparable performance to non-streaming VC systems while maintaining a fully causal context-aware LM with a temporal-independent acoustic predictor. |
Zero-Shot Text-to-Speech for Vietnamese (2025.acl-short)
Copied to clipboard
| Challenge: | Text-to-speech (TTS) synthesis has seen significant advancements in recent years. |
| Approach: | They propose to use PhoAudiobook to curated 941 hours of high-quality audio for Vietnamese text-to-speech models. |
| Outcome: | The proposed model improves on VALL-E, VoiceCraft, and XTTS-V2 models, highlighting their robustness in handling diverse linguistic contexts. |
End-to-End Evaluation for Low-Latency Simultaneous Speech Translation (2023.emnlp-demo)
Copied to clipboard
Christian Huber, Tu Anh Dinh, Carlos Mullov, Ngoc-Quan Pham, Thai Binh Nguyen, Fabian Retkowski, Stefan Constantin, Enes Ugan, Danni Liu, Zhaolin Li, Sai Koneru, Jan Niehues, Alexander Waibel
| Challenge: | a framework to evaluate low-latency speech translations is currently only limited to specific aspects and is not able to compare different approaches. |
| Approach: | They propose a framework to perform and evaluate low-latency speech translation in realistic conditions. |
| Outcome: | The proposed framework evaluates various aspects of low-latency speech translation under realistic conditions. |