TCSinger 2: Customizable Multilingual Zero-shot Singing Voice Synthesis

Challenge:	Existing zero-shot singing voice synthesis models depend on phoneme and note boundary annotations, limiting their robustness and producing poor transitions between phonemes and notes.
Approach:	They propose a multi-task multilingual zero-shot SVS model with style transfer and style control based on various prompts.
Outcome:	Experimental results show that TCSinger 2 outperforms baseline models in subjective and objective metrics across multiple related tasks.

Similar Papers

Overview

Challenge

Approach

Outcome

TCSinger: Zero-Shot Singing Voice Synthesis with Style Transfer and Multi-Level Style Control (2024.emnlp-main)

Copied to clipboard

Yu Zhang, Ziyue Jiang, Ruiqi Li, Changhao Pan, Jinzheng He, Rongjie Huang, Chuxin Wang, Zhou Zhao

Low-Resource NLP

Challenge:	Existing models fail to generate singing voices rich in stylistic nuances for unseen singers due to multifaceted nature of singing styles.
Approach:	They propose a zero-shot SVS model for style transfer across cross-lingual speech and singing styles and multi-level style control.
Outcome:	Experimental results show that TCSinger outperforms baseline models in synthesis quality, singer similarity, and style controllability.

STARS: A Unified Framework for Singing Transcription, Alignment, and Refined Style Annotation (2025.findings-acl)

Copied to clipboard

Wenxiang Guo, Yu Zhang, Changhao Pan, Zhiyuan Zhu, Ruiqi Li, ZheTao Chen, Wenhao Xu, Fei Wu, Zhou Zhao

Speech & Audio in NLP

Challenge:	Existing automated singing annotation (ASA) methods tackle isolated aspects of the annotation pipeline.
Approach:	They propose a framework that addresses transcription, alignment, and refined style annotations.
Outcome:	The proposed framework delivers comprehensive multi-level annotations encompassing: (1) precise phoneme-audio alignment, (2) robust note transcription and temporal localization, (3) expressive vocal technique identification, and (4) global stylistic characterization including emotion and pace.

Prompt-Singer: Controllable Singing-Voice-Synthesis with Natural Language Prompt (2024.naacl-long)

Copied to clipboard

Yongqi Wang, Ruofan Hu, Rongjie Huang, Zhiqing Hong, Ruiqi Li, Wenrui Liu, Fuming You, Tao Jin, Zhou Zhao

Language Models

Challenge:	Recent singing-voice-synthesis methods lack ability to control style attributes of synthesized singing.
Approach:	They propose a singing-voice-synthesis method that enables attribute controlling on singer gender, vocal range and volume with natural language.
Outcome:	The proposed method achieves favorable control ability and audio quality.

Low-Resource Multilingual and Zero-Shot Multispeaker TTS (2022.aacl-main)

Copied to clipboard

Florian Lux, Julia Koch, Ngoc Thang Vu

Multimodality

Challenge:	Currently, the amount of data needed for TTS is limited to the vast majority of the spoken languages.
Approach:	They propose to use language agnostic meta learning procedure to learn speaking a new language with just 5 minutes of training data while retaining the ability to infer the voice of even unseen speakers.
Outcome:	The proposed approach is able to learn speaking a new language using just 5 minutes of training data while retaining the ability to infer the voice of even unseen speakers in the newly learned language.

Takin-VC: Expressive Zero-Shot Voice Conversion via Adaptive Hybrid Content Encoding and Enhanced Timbre Modeling (2025.acl-long)

Copied to clipboard

Yang Yuguang, Yu Pan, Jixun Yao, Xiang Zhang, Jianhao Ye, Hongbin Zhou, Lei Xie, Lei Ma, Jianjun Zhao

Language Models

Challenge:	Expressive zero-shot voice conversion (VC) aims to modify source timbre to match unseen speaker . existing zero- shot VC systems struggle to reproduce paralinguistic information in highly expressive speech .
Approach:	They propose a framework for expressive zero-shot voice conversion that uses hybrid content encoding and memory-augmented context-aware timbre modeling.
Outcome:	The proposed framework surpasses state-of-the-art VC systems in speech naturalness, speaker similarity, and speaker similarness.

VoiceCraft-X: Unifying Multilingual, Voice-Cloning Speech Synthesis and Speech Editing (2025.emnlp-main)

Copied to clipboard

Zhisheng Zheng, Puyuan Peng, Anuj Diwan, Cong Phuoc Huynh, Xiaohang Sun, Zhu Liu, Vimal Bhat, David Harwath

Multimodality

Challenge:	Autoregressive language model for multilingual speech editing and zero-shot text-to-speech synthesis is available in 11 languages.
Approach:	They introduce an autoregressive neural codec language model which unifies multilingual speech editing and zero-shot text-to-speech synthesis across 11 languages.
Outcome:	The model generates high-quality, natural-sounding speech, even with limited per-language data . it shows robust performance in diverse linguistic settings, even in limited per language data compared to other models .

Self-Supervised Singing Voice Pre-Training towards Speech-to-Singing Conversion (2024.findings-acl)

Copied to clipboard

Ruiqi Li, Rongjie Huang, Yongqi Wang, Zhiqing Hong, Zhou Zhao

Speech & Audio in NLP

Challenge:	Existing studies on speech-to-singing voice conversion (STS) are limited by the scarcity of paired speech-song data and the suboptimal quality of outputs.
Approach:	They propose a self-supervised singing voice pre-training model that transforms a speech-to-singing voice into a paired singing voice.
Outcome:	The proposed model improves both STS and singing voice synthesis tasks by combining spoken language and a self-supervised singing voice pre-training model.

RMSSinger: Realistic-Music-Score based Singing Voice Synthesis (2023.findings-acl)

Copied to clipboard

Jinzheng He, Jinglin Liu, Zhenhui Ye, Rongjie Huang, Chenye Cui, Huadai Liu, Zhou Zhao

Multimodality

Challenge:	Existing methods for singing voice synthesis are limited to fine-grained music scores . manual adjustment destroys regularity of note durations, making fine-grain music scores "crushed"
Approach:	They propose a method to synthesize singing voices given realistic music scores . they use real-music-score-based Singing Voice Synthesis to generate high-quality voices .
Outcome:	The proposed method eliminates manual annotation and simplifies phoneme-level mel-note alignment.

EZ-VC: Easy Zero-shot Any-to-Any Voice Conversion (2025.findings-emnlp)

Copied to clipboard

Advait Joglekar, Divyanshu Singh, Rooshil Rohit Bhatia, Srinivasan Umesh

Low-Resource NLP

Challenge:	Current voice conversion methods struggle in zero-shot cross-lingual settings . authors develop a method that can be used in zero shot cross-linguistic settings despite advances in technology .
Approach:	They propose a voice-conversion model that combines discrete speech representations with a non-autoregressive speech decoder.
Outcome:	The proposed approach excels in zero-shot cross-lingual settings even for unseen languages and accents.

Improving Chinese Pop Song and Hokkien Gezi Opera Singing Voice Synthesis by Enhancing Local Modeling (2023.emnlp-main)

Copied to clipboard

Peng Bai, Yue Zhou, Meizhen Zheng, Wujin Sun, Xiaodong Shi

Speech & Audio in NLP

Challenge:	Singing Voice Synthesis (SVS) synthesizes pleasing vocals based on music scores and lyrics . current acoustic models ignore the significance of local modeling within the sequence and the hard-to-synthesize parts in the predicted mel-spectrogram .
Approach:	They propose a method to enhance local modeling in the acoustic model by focusing on phoneme tokens located before and after the phoneme.
Outcome:	The proposed method improves local modeling in the acoustic model by focusing on the hard-to-synthesize parts of the predicted mel-spectrogram.