ViT-TTS: Visual Text-to-Speech with Scalable Diffusion Transformer (2023.emnlp-main)
Copied to clipboard
| Challenge: | Text-to-speech (TTS) performance has improved with the advent of denoising Diffusion Probabilistic Models . however, perceived quality of audio depends on content, pitch, rhythm, and energy . |
| Approach: | They propose a visual TTS model with scalable diffusion transformers that complement phoneme sequences with visual information to generate high-perceived audio. |
| Outcome: | The proposed model outperforms existing models regardless of visibility of the scene . it can generate high-perceived audio, opening up new avenues for AR and VR applications . |
Similar Papers
On the Semantic Latent Space of Diffusion-Based Text-To-Speech Models (2024.acl-short)
Copied to clipboard
| Challenge: | Denoising Diffusion Models (DDMs) are a powerful generative tool for text-to-speech (TTS) but their semantic capabilities are unknown and control of synthesized speech’s vocal properties remains a challenge. |
| Approach: | They explore the latent space of frozen TTS models composed of latent bottleneck activations of the DDM’s denoiser and propose methods for finding semantic directions within it. |
| Outcome: | The proposed methods enable off-the-shelf audio editing without any training, architectural changes or data requirements. |
F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matching (2025.acl-long)
Copied to clipboard
| Challenge: | Recent research in Text-to-Speech (TTS) has experienced great advancement . current models can synthesize speech for any given text and mimic the speaker of audio prompt. |
| Approach: | They propose a fully non-autoregressive text-to-speech system based on flow matching with Diffusion Transformer (DiT) without complex designs such as duration model, text encoder, and phoneme alignment, the text input is simply padded with filler tokens to the same length as input speech, and then denoising is performed for speech generation. |
| Outcome: | The proposed system achieves an inference RTF of 0.15, which is greatly improved compared to state-of-the-art diffusion-based models. |
Prosody-TTS: Improving Prosody with Masked Autoencoder and Conditional Diffusion Model For Expressive Text-to-Speech (2023.findings-acl)
Copied to clipboard
| Challenge: | Expressive text-to-speech aims to generate high-quality samples with rich prosody . prosodic attributes in highly dynamic voices are difficult to capture and model without intonation . |
| Approach: | They propose a pipeline that enhances prosody modeling and sampling by introducing a self-supervised masked autoencoder and a diffusion model to sample diverse prosodic patterns within the latent space. |
| Outcome: | The proposed pipeline achieves new state-of-the-art in text-to-speech with natural and expressive synthesis. |
ImmersiveTTS: Environment-Aware Text-to-Speech with Multimodal Diffusion Transformer and Domain-Specific Representation Alignment (2026.acl-long)
Copied to clipboard
| Challenge: | ImmersiveTTS model synthesizes intelligible speech and environmental audio from natural language descriptions. |
| Approach: | They propose an environment-aware text-to-speech model that integrates natural speech with environmental audio . the model explicitly models cross-modal interactions through a dual-stream stage . |
| Outcome: | Experimental results show that ImmersiveTTS achieves higher naturalness, intelligibility, and audio fidelity than existing approaches. |
Scaling Under-Resourced TTS: A Data-Optimized Framework with Advanced Acoustic Modeling for Thai (2025.acl-industry)
Copied to clipboard
| Challenge: | Text-to-speech (TTS) systems are limited by limited data and linguistic complexities. |
| Approach: | They propose a data-optimized framework with an advanced acoustic model to build high-quality TTS systems for low-resource scenarios. |
| Outcome: | The proposed framework enables zero-shot voice cloning and improved performance across diverse client applications, including finance, healthcare, education, and law. |
DiTReducio: A Training-Free Acceleration for DiT-Based TTS via Progressive Calibration (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing training-free acceleration approaches for text-to-speech models are constrained by training costs. |
| Approach: | They propose a training-free acceleration framework that compresses computations in DiT-based TTS models . they propose Temporal Skipping and Branch Skipping to eliminate redundant computations . |
| Outcome: | Experimental results show that the proposed framework reduces FLOPs and improves RTF by 37.1%. |
CM-TTS: Enhancing Real Time Text-to-Speech Synthesis Efficiency through Weighted Samplers and Consistency Models (2024.findings-naacl)
Copied to clipboard
| Challenge: | Neural Text-to-Speech systems are a promising approach for high-fidelity speech synthesis . but the efficiency of multi-step sampling in Diffusion Models presents challenges . |
| Approach: | They propose a novel architecture grounded in consistency models to improve model convergence. |
| Outcome: | The proposed architecture achieves top-quality speech synthesis in fewer steps without adversarial training or pre-trained model dependencies. |
Llama-VITS: Enhancing TTS Synthesis with Semantic Awareness (2024.lrec-main)
Copied to clipboard
| Challenge: | Recent advances in Natural Language Processing (NLP) have seen Large-scale Language Models excel at producing high-quality text for various purposes. |
| Approach: | They propose a language model that enriches semantic content of text using Llama2 . their method enhances emotive expressiveness on a dataset . |
| Outcome: | The proposed model matches the naturalness of the original VITS and incorporates BERT (BERT-VITS) on the LJSpeech dataset, highlighting its potential to generate emotive speech. |
ProsodyFlow: High-fidelity Text-to-Speech through Conditional Flow Matching and Prosody Modeling with Large Speech Language Models (2025.coling-main)
Copied to clipboard
| Challenge: | Text-to-speech (TTS) models have been developed to generate high-quality speech. |
| Approach: | They propose an end-to-end TTS model that integrates large self-supervised speech models and conditional flow matching to model prosodic features effectively. |
| Outcome: | The proposed model improves synthesis quality and efficiency compared to existing models, showing that it generates more prosodic and expressive speech synthesizing. |
SpeechWeave: Diverse Multilingual Synthetic Text & Audio Data Generation Pipeline for Training Text to Speech Models (2025.acl-industry)
Copied to clipboard
| Challenge: | Text-to-Speech (TTS) training requires extensive and diverse text and speech data. |
| Approach: | They propose a synthetic speech data generation pipeline that generates multilingual, domain-specific datasets for TTS training. |
| Outcome: | The proposed pipeline generates data that is 10–48% more diverse than baseline across various linguistic and phonetic metrics, along with speaker-standardized speech audio while generating approximately 97% correctly normalized text. |