Cross-Utterance Conditioned VAE for Non-Autoregressive Text-to-Speech (2022.acl-long)
Copied to clipboard
| Challenge: | Experimental results show that the proposed model improves naturalness and prosody diversity with clear margins. |
| Approach: | They propose a cross-utterance conditional VAE to estimate posterior probability distribution of latent prosody features for each phoneme by conditioning on acoustic features, speaker information, and text features from past and future sentences. |
| Outcome: | The proposed model improves naturalness and prosody diversity with clear margins. |
Similar Papers
Prosody-TTS: Improving Prosody with Masked Autoencoder and Conditional Diffusion Model For Expressive Text-to-Speech (2023.findings-acl)
Copied to clipboard
| Challenge: | Expressive text-to-speech aims to generate high-quality samples with rich prosody . prosodic attributes in highly dynamic voices are difficult to capture and model without intonation . |
| Approach: | They propose a pipeline that enhances prosody modeling and sampling by introducing a self-supervised masked autoencoder and a diffusion model to sample diverse prosodic patterns within the latent space. |
| Outcome: | The proposed pipeline achieves new state-of-the-art in text-to-speech with natural and expressive synthesis. |
ProsodyFlow: High-fidelity Text-to-Speech through Conditional Flow Matching and Prosody Modeling with Large Speech Language Models (2025.coling-main)
Copied to clipboard
| Challenge: | Text-to-speech (TTS) models have been developed to generate high-quality speech. |
| Approach: | They propose an end-to-end TTS model that integrates large self-supervised speech models and conditional flow matching to model prosodic features effectively. |
| Outcome: | The proposed model improves synthesis quality and efficiency compared to existing models, showing that it generates more prosodic and expressive speech synthesizing. |
Dior-CVAE: Pre-trained Language Models and Diffusion Priors for Variational Dialog Generation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing variational dialog models have pre-trained, restricting diversity of responses . a diffusion model increases complexity of prior distribution and its compatibility with PLMs . |
| Approach: | They propose a hierarchical conditional variational autoencoder with diffusion priors to address these challenges. |
| Outcome: | The proposed method generates more diverse responses without dialog pre-training. |
Evaluating Text-to-Speech Synthesis from a Large Discrete Token-based Speech Language Model (2024.lrec-main)
Copied to clipboard
| Challenge: | Recent advances in generative language modeling applied to discrete speech tokens presented a new avenue for text-to-speech (TTS) synthesis. |
| Approach: | They propose to use generative language modeling to generate text-to-speech (TTS) outputs by a discrete token-based model. |
| Outcome: | The proposed model is rated higher in naturalness and context appropriateness in listening tests compared to a conventional TTS. |
DPP-TTS: Diversifying prosodic features of speech via determinantal point processes (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in deep generative models have succeeded in synthesizing human-like speech. |
| Approach: | They propose a text-to-speech model with a prosody diversifying module that considers perceptual diversity in each sample and among multiple samples. |
| Outcome: | The proposed model generates speech samples with more diversified prosody than baselines in the side-by-side comparison test considering the naturalness of speech at the same time. |
CLAPSpeech: Learning Prosody from Text Context with Contrastive Language-Audio Pre-Training (2023.acl-long)
Copied to clipboard
| Challenge: | Existing methods for expressive text-to-speech only implicitly learn prosody with masked token reconstruction tasks. |
| Approach: | They propose a cross-modal contrastive pre-training framework that learns from prosody variance of the same text token under different contexts. |
| Outcome: | The proposed framework can learn from prosody variance of a text token under different contexts. |
Do sequence-to-sequence VAEs learn global features of sentences? (2020.emnlp-main)
Copied to clipboard
| Challenge: | Autoregressive language models are often trained without explicit conditioning labels . authors question claim that latent vectors can capture global features in unsupervised manner . |
| Approach: | They propose to use a sequence-to-sequence architecture to learn latent variables . they find that VAEs are prone to memorizing the first words and sentence length . |
| Outcome: | The proposed model is prone to memorizing the first words and sentence length, the authors show . et al., 2016: a new model learns latent variables that are more global, more predictive of topic or topic labels . authors question this claim, say it is a waste of time and money . |
Prosody in Cascade and Direct Speech-to-Text Translation: a case study on Korean Wh-Phrases (2024.findings-eacl)
Copied to clipboard
| Challenge: | Existing direct S2TT systems have been unable to disambiguate utterances where prosody plays a crucial role. |
| Approach: | They propose to use contrastive evaluation to quantitatively measure the ability of direct S2TT systems to disambiguate utterances where prosody plays a crucial role. |
| Outcome: | The proposed system improves overall accuracy 12.9% and improves intent scores 15.6%. |
ProSE: Diffusion Priors for Speech Enhancement (2025.naacl-long)
Copied to clipboard
Sonal Kumar, Sreyan Ghosh, Utkarsh Tyagi, Anton Jeran Ratnarajah, Chandra Kiran Reddy Evuru, Ramani Duraiswami, Dinesh Manocha
| Challenge: | deterministic deep learning models have been used for speech enhancement, but generative models have shown promise. |
| Approach: | They propose a method to apply diffusion probabilistic models to speech enhancement using priors in a latent space. |
| Outcome: | The proposed method achieves state-of-the-art performance on synthetic and real-world datasets while consuming less computational costs. |
Autoregressive Speech Synthesis without Vector Quantization (2025.acl-long)
Copied to clipboard
Lingwei Meng, Long Zhou, Shujie Liu, Sanyuan Chen, Bing Han, Shujie Hu, Yanqing Liu, Jinyu Li, Sheng Zhao, Xixin Wu, Helen M. Meng, Furu Wei
| Challenge: | MELLE is a novel language modeling approach for text-to-speech synthesis that generates continuous tokens from text . authors demonstrate that it reduces the need for vector quantization and improves model robustness . |
| Approach: | They propose to autoregressively generate continuous mel-spectrogram frames directly from text condition, bypassing vector quantization. |
| Outcome: | The proposed model achieves superior performance across multiple metrics and is more streamlined. |