FastDiff 2: Revisiting and Incorporating GANs and Diffusion Models in High-Fidelity Speech Synthesis (2023.findings-acl)
Copied to clipboard
| Challenge: | Experimental results show that Generative adversarial networks sacrifice sample diversity for quality and speed, while diffusion models exhibit outperformed sample quality and diversity at a high computational cost. |
| Approach: | They propose to combine GANs and diffusion probabilistic models to achieve better sample quality and diversity. |
| Outcome: | The proposed models outperform GANs and diffusion models in speech synthesis . the proposed models enjoy an efficient 4-step sampling process and exhibit better sample diversity . |
Similar Papers
Speaking in Wavelet Domain: A Simple and Efficient Approach to Speed up Speech Diffusion Model (2024.emnlp-main)
Copied to clipboard
Xiangyu Zhang, Daijiao Liu, Hexin Liu, Qiquan Zhang, Hanyu Meng, Leibny Paola Garcia Perera, EngSiong Chng, Lina Yao
| Challenge: | Existing approaches to enhance inference speed and training require complex modifications to the model. |
| Approach: | They propose to double the training and inference speed of Denoising Diffusion Probabilistic Models by simply redirecting the generative target to the wavelet domain. |
| Outcome: | The proposed method doubles the training and inference speed of Speech DDPMs by redirecting the generative target to the wavelet domain. |
CM-TTS: Enhancing Real Time Text-to-Speech Synthesis Efficiency through Weighted Samplers and Consistency Models (2024.findings-naacl)
Copied to clipboard
| Challenge: | Neural Text-to-Speech systems are a promising approach for high-fidelity speech synthesis . but the efficiency of multi-step sampling in Diffusion Models presents challenges . |
| Approach: | They propose a novel architecture grounded in consistency models to improve model convergence. |
| Outcome: | The proposed architecture achieves top-quality speech synthesis in fewer steps without adversarial training or pre-trained model dependencies. |
ProSE: Diffusion Priors for Speech Enhancement (2025.naacl-long)
Copied to clipboard
Sonal Kumar, Sreyan Ghosh, Utkarsh Tyagi, Anton Jeran Ratnarajah, Chandra Kiran Reddy Evuru, Ramani Duraiswami, Dinesh Manocha
| Challenge: | deterministic deep learning models have been used for speech enhancement, but generative models have shown promise. |
| Approach: | They propose a method to apply diffusion probabilistic models to speech enhancement using priors in a latent space. |
| Outcome: | The proposed method achieves state-of-the-art performance on synthetic and real-world datasets while consuming less computational costs. |
On the Semantic Latent Space of Diffusion-Based Text-To-Speech Models (2024.acl-short)
Copied to clipboard
| Challenge: | Denoising Diffusion Models (DDMs) are a powerful generative tool for text-to-speech (TTS) but their semantic capabilities are unknown and control of synthesized speech’s vocal properties remains a challenge. |
| Approach: | They explore the latent space of frozen TTS models composed of latent bottleneck activations of the DDM’s denoiser and propose methods for finding semantic directions within it. |
| Outcome: | The proposed methods enable off-the-shelf audio editing without any training, architectural changes or data requirements. |
ProsodyFlow: High-fidelity Text-to-Speech through Conditional Flow Matching and Prosody Modeling with Large Speech Language Models (2025.coling-main)
Copied to clipboard
| Challenge: | Text-to-speech (TTS) models have been developed to generate high-quality speech. |
| Approach: | They propose an end-to-end TTS model that integrates large self-supervised speech models and conditional flow matching to model prosodic features effectively. |
| Outcome: | The proposed model improves synthesis quality and efficiency compared to existing models, showing that it generates more prosodic and expressive speech synthesizing. |
Prosody-TTS: Improving Prosody with Masked Autoencoder and Conditional Diffusion Model For Expressive Text-to-Speech (2023.findings-acl)
Copied to clipboard
| Challenge: | Expressive text-to-speech aims to generate high-quality samples with rich prosody . prosodic attributes in highly dynamic voices are difficult to capture and model without intonation . |
| Approach: | They propose a pipeline that enhances prosody modeling and sampling by introducing a self-supervised masked autoencoder and a diffusion model to sample diverse prosodic patterns within the latent space. |
| Outcome: | The proposed pipeline achieves new state-of-the-art in text-to-speech with natural and expressive synthesis. |
DPP-TTS: Diversifying prosodic features of speech via determinantal point processes (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in deep generative models have succeeded in synthesizing human-like speech. |
| Approach: | They propose a text-to-speech model with a prosody diversifying module that considers perceptual diversity in each sample and among multiple samples. |
| Outcome: | The proposed model generates speech samples with more diversified prosody than baselines in the side-by-side comparison test considering the naturalness of speech at the same time. |
Duplex Diffusion Models Improve Speech-to-Speech Translation (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to speech-to-speech translation train two separate models or a multitask-learned model with low efficiency and inferior performance. |
| Approach: | They propose a duplex diffusion model that applies diffusion probabilistic models to both sides of a reversible duplex Conformer and enables reverse speech translation by simply flipping the input and output ends. |
| Outcome: | The proposed model achieves the first success of reversible speech translation with significant improvements of ASR-BLEU scores compared with a list of state-of-the-art baselines. |
ResoDiff-44k: High-Fidelity Cross-Lingual Speech and Singing Synthesis via Discrete Diffusion (2026.acl-industry)
Copied to clipboard
| Challenge: | generative speech models have a fidelity ceiling that is capped at lower sampling rates . current models rely on intermediate mel-spectrograms, which discard phase and high-frequency information . a new framework that synthesizes industrial-grade 44.1kHz audio is proposed . |
| Approach: | They propose a production-grade generative foundation model for 44.1kHz audio synthesis . they pre-train ResoDiff-44k on a massive 150K -hour multilingual dataset . |
| Outcome: | The proposed model achieves 4.6 mean opinion score in 44.1kHz singing synthesis compared to baselines . it also reduces character error rate on regional mixed-language and singing prompts compared with baselines. |
Hierarchical Representation Alignment Learning of Diffusion Transformers for Neural Audio Codec (2026.findings-acl)
Copied to clipboard
| Challenge: | Recent advances in diffusion and conditional flow matching models for low-resolution domains are underexplored. |
| Approach: | They propose a CFM-based model that iteratively generates raw waveform in low-bitrate conditions . they propose DVQ, a factorized quantization method that uses a single quantizer . |
| Outcome: | The proposed model outperforms state-of-the-art neural audio codecs in audio quality and semantic intelligibility under low-bitrate conditions. |