STARS: A Unified Framework for Singing Transcription, Alignment, and Refined Style Annotation

Challenge:	Existing automated singing annotation (ASA) methods tackle isolated aspects of the annotation pipeline.
Approach:	They propose a framework that addresses transcription, alignment, and refined style annotations.
Outcome:	The proposed framework delivers comprehensive multi-level annotations encompassing: (1) precise phoneme-audio alignment, (2) robust note transcription and temporal localization, (3) expressive vocal technique identification, and (4) global stylistic characterization including emotion and pace.

Similar Papers

Overview

Challenge

Approach

Outcome

Robust Singing Voice Transcription Serves Synthesis (2024.acl-long)

Copied to clipboard

Ruiqi Li, Yu Zhang, Yongqi Wang, Zhiqing Hong, Rongjie Huang, Zhou Zhao

Robustness in NLP

Challenge:	Current AST methods struggle with accuracy and robustness when used for practical annotation.
Approach:	They propose a model that converts singing recordings into note sequences for automatic annotation of singing datasets.
Outcome:	The proposed model outperforms baseline models on enlarged, automatically annotated datasets.

TCSinger 2: Customizable Multilingual Zero-shot Singing Voice Synthesis (2025.findings-acl)

Copied to clipboard

Yu Zhang, Wenxiang Guo, Changhao Pan, Dongyu Yao, Zhiyuan Zhu, Ziyue Jiang, Yuhan Wang, Tao Jin, Zhou Zhao

Responsible & Trustworthy NLP

Challenge:	Existing zero-shot singing voice synthesis models depend on phoneme and note boundary annotations, limiting their robustness and producing poor transitions between phonemes and notes.
Approach:	They propose a multi-task multilingual zero-shot SVS model with style transfer and style control based on various prompts.
Outcome:	Experimental results show that TCSinger 2 outperforms baseline models in subjective and objective metrics across multiple related tasks.

RMSSinger: Realistic-Music-Score based Singing Voice Synthesis (2023.findings-acl)

Copied to clipboard

Jinzheng He, Jinglin Liu, Zhenhui Ye, Rongjie Huang, Chenye Cui, Huadai Liu, Zhou Zhao

Multimodality

Challenge:	Existing methods for singing voice synthesis are limited to fine-grained music scores . manual adjustment destroys regularity of note durations, making fine-grain music scores "crushed"
Approach:	They propose a method to synthesize singing voices given realistic music scores . they use real-music-score-based Singing Voice Synthesis to generate high-quality voices .
Outcome:	The proposed method eliminates manual annotation and simplifies phoneme-level mel-note alignment.

AlignSTS: Speech-to-Singing Conversion via Cross-Modal Alignment (2023.findings-acl)

Copied to clipboard

Ruiqi Li, Rongjie Huang, Lichao Zhang, Jinglin Liu, Zhou Zhao

Multimodality

Challenge:	Existing approaches to speech-to-singing voice conversion are difficult to learn in text-free situations.
Approach:	They propose an STS model which views speech variance as different modalities . it uses a novel rhythm adaptor to predict the target rhythm representation . they also use the predicted rhythm representation to re-align the content .
Outcome:	The proposed model achieves superior performance in terms of objective and subjective metrics.

Self-Supervised Singing Voice Pre-Training towards Speech-to-Singing Conversion (2024.findings-acl)

Copied to clipboard

Ruiqi Li, Rongjie Huang, Yongqi Wang, Zhiqing Hong, Zhou Zhao

Speech & Audio in NLP

Challenge:	Existing studies on speech-to-singing voice conversion (STS) are limited by the scarcity of paired speech-song data and the suboptimal quality of outputs.
Approach:	They propose a self-supervised singing voice pre-training model that transforms a speech-to-singing voice into a paired singing voice.
Outcome:	The proposed model improves both STS and singing voice synthesis tasks by combining spoken language and a self-supervised singing voice pre-training model.

Prompt-Singer: Controllable Singing-Voice-Synthesis with Natural Language Prompt (2024.naacl-long)

Copied to clipboard

Yongqi Wang, Ruofan Hu, Rongjie Huang, Zhiqing Hong, Ruiqi Li, Wenrui Liu, Fuming You, Tao Jin, Zhou Zhao

Language Models

Challenge:	Recent singing-voice-synthesis methods lack ability to control style attributes of synthesized singing.
Approach:	They propose a singing-voice-synthesis method that enables attribute controlling on singer gender, vocal range and volume with natural language.
Outcome:	The proposed method achieves favorable control ability and audio quality.

Beyond Transcription: Unified Audio Schema for Perception-Aware AudioLLMs (2026.findings-acl)

Copied to clipboard

Linhao Zhang, Yuhan Song, Aiwei Liu, Chuhan Wu, Sijun Zhang, Wei Jia, Yuan Liu, Houfeng Wang, Zhou Xiao

Multimodality

Challenge:	Recent Audio Large Language Models (AudioLLMs) excel at reasoning tasks, but struggle at elementary auditory perception.
Approach:	They propose a framework that organizes audio information into three explicit components in a unified JSON format.
Outcome:	The proposed framework boosts fine-grained perception by 10.9% on MMSU over state-of-the-art models while preserving robust reasoning capabilities.

VocalRep: Structure-Aware Vocal Representations for Multimodal Generation (2026.findings-acl)

Copied to clipboard

Da Shen, Zhenqiang Weng, Tianyu Liu, Gongyu Chen, Runhua Shi, Jiahui Chen, Chaofan Ding, Wei-Qiang Zhang, Zihao Chen

Multimodality

Challenge:	Existing approaches to vocal separation are optimized for signal-level reconstruction, but they overlook structural disentanglement required for downstream generation tasks.
Approach:	They propose a structure-aware learning framework to disentangle vocals, harmonies, and accompaniment . they combine global vocal identity conditioning with ranking-based objectives .
Outcome:	The proposed framework disentangles lead vocals, harmonies, and accompaniment while enforcing role consistency across long-form audio.

Improving Chinese Pop Song and Hokkien Gezi Opera Singing Voice Synthesis by Enhancing Local Modeling (2023.emnlp-main)

Copied to clipboard

Peng Bai, Yue Zhou, Meizhen Zheng, Wujin Sun, Xiaodong Shi

Speech & Audio in NLP

Challenge:	Singing Voice Synthesis (SVS) synthesizes pleasing vocals based on music scores and lyrics . current acoustic models ignore the significance of local modeling within the sequence and the hard-to-synthesize parts in the predicted mel-spectrogram .
Approach:	They propose a method to enhance local modeling in the acoustic model by focusing on phoneme tokens located before and after the phoneme.
Outcome:	The proposed method improves local modeling in the acoustic model by focusing on the hard-to-synthesize parts of the predicted mel-spectrogram.

TCSinger: Zero-Shot Singing Voice Synthesis with Style Transfer and Multi-Level Style Control (2024.emnlp-main)

Copied to clipboard

Yu Zhang, Ziyue Jiang, Ruiqi Li, Changhao Pan, Jinzheng He, Rongjie Huang, Chuxin Wang, Zhou Zhao

Low-Resource NLP

Challenge:	Existing models fail to generate singing voices rich in stylistic nuances for unseen singers due to multifaceted nature of singing styles.
Approach:	They propose a zero-shot SVS model for style transfer across cross-lingual speech and singing styles and multi-level style control.
Outcome:	Experimental results show that TCSinger outperforms baseline models in synthesis quality, singer similarity, and style controllability.