Learning Robust Latent Representations for Controllable Speech Synthesis

Challenge:	Variational Auto-Encoders (VAEs) for learning disentangled latent representations in speech fail to learn latent clusters of speaker attributes when trained on limited or noisy datasets.
Approach:	They propose a Variational Auto-Encoder (VAE) that minimizes mutual information between latent variables and learns controllable latent representations in speech data.
Outcome:	The proposed model reduces the cluster overlap of speaker attributes by 30% over LSTM-VAE.

Similar Papers

Overview

Challenge

Approach

Outcome

Improving Semantic Control in Discrete Latent Spaces with Transformer Quantized Variational Autoencoders (2024.findings-eacl)

Copied to clipboard

Yingji Zhang, Danilo Carvalho, Marco Valentino, Ian Pratt-Hartmann, Andre Freitas

Language Models

Challenge:	Recent work has struggled to achieve consistent results due to the inevitable loss of semantic information in the variational bottleneck and limited control over the decoding mechanism.
Approach:	They propose a model that leverages the controllability of VQVAE to guide the self-attention mechanism in Transformer-based VAEs to improve semantic control and generation.
Outcome:	The proposed model outperforms existing state-of-the-art VAE models in terms of control and preservation of semantic information across different tasks.

Fuse It More Deeply! A Variational Transformer with Layer-Wise Latent Variable Inference for Text Generation (2022.naacl-main)

Copied to clipboard

Jinyi Hu, Xiaoyuan Yi, Wenhao Li, Maosong Sun, Xing Xie

Language Models

Challenge:	Variational Auto-Encoders are often used for text generation tasks due to the sequential nature of the text.
Approach:	They propose a variational Transformer framework that learns a series of layer-wise latent variables with each inferred from those of lower layers and tightly coupled with the hidden states by low-rank tensor product.
Outcome:	The proposed framework can learn latent variables from lower layers and incorporate more information.

Self-supervised Rewiring of Pre-trained Speech Encoders: Towards Faster Fine-tuning with Less Labels in Speech Processing (2022.findings-emnlp)

Copied to clipboard

Hao Yang, Jinming Zhao, Gholamreza Haffari, Ehsan Shareghi

Language Models

Challenge:	Pre-trained speech encoders have facilitated great success across various speech processing tasks, but fine-tuning them for downstream tasks requires large training data to converge or to achieve state-of-the-art.
Approach:	They propose to rewire pre-trained speech encoders to improve their representation space without task-specific labels by neutrally synthesising audio inputs and frame masking.
Outcome:	The proposed model shows consistent improvement in isotropy in the representation space on 6 speech processing tasks.

DiffLM: Controllable Synthetic Data Generation via Diffusion Language Models (2025.findings-acl)

Copied to clipboard

Ying Zhou, Xinyao Wang, Yulei Niu, Yaojie Shen, Lexin Tang, Fan Chen, Ben He, Le Sun, Longyin Wen

Language Models

Challenge:	Recent advances in large language models (LLMs) have significantly enhanced their knowledge and generative capabilities, leading to a surge of interest in leveraging LLMs for high-quality data synthesis.
Approach:	They propose a controllable data synthesis framework based on variational autoencoder which leverages diffusion models to reserve more information of original distribution and format structure in the learned latent distribution.
Outcome:	The proposed framework generates high-quality data with performance exceeding that of real data by 2%–7% on seven real-world datasets.

Manifold-Preserving Transformers are Effective for Short-Long Range Encoding (2023.findings-emnlp)

Copied to clipboard

Ayan Sengupta, Md Akhtar, Tanmoy Chakraborty

Language Models

Challenge:	Multi-head self-attention-based Transformers have shown promise in different learning tasks . but encoders of Transformers and their variants fail to preserve layer-wise contextual information .
Approach:	They propose an encoder model that guarantees a theoretical bound for layer-wise distance preservation between a pair of tokens.
Outcome:	The proposed model preserves equivalence between tokens and performs better than Transformers.

V-VAE: A Variational Auto Encoding Framework Towards Fine-Grained Control over Human-Like Chat (2025.emnlp-main)

Copied to clipboard

Qi Lin, Weikai Xu, Lisi Chen, Bin Dai

Dialogue Systems & Conversational Agents

Challenge:	Existing role-play and persona-based chat approaches rely on static role descriptions, coarse-grained signal space, and low-quality synthetic data.
Approach:	They propose a Verbal Variational Auto-Encoding framework which dynamically adapts dialogue behaviour based on latent variables across talking style, interaction patterns, and personal attributes.
Outcome:	The proposed framework outperforms baselines on HumanChatBench and DialogBench to address the scarcity of high-quality data in the human-like domain.

Contrastive Deterministic Autoencoders For Language Modeling (2023.findings-emnlp)

Copied to clipboard

Amur Ghose, Pascal Poupart

Language Models

Challenge:	Variational autoencoders (VAEs) are a popular family of generative models with wide applicability.
Approach:	They propose to modify a deterministic model designed for images to avoid posterior collapse by controlling the entropy of the aggregate posterior to make it Gaussian.
Outcome:	The proposed models outperform a broad range of VAE models on text generation and downstream tasks from representations while avoiding reparametrization steps.

Controlled Text Generation Using Dictionary Prior in Variational Autoencoders (2022.findings-acl)

Copied to clipboard

Xianghong Fang, Jian Li, Lifeng Shang, Xin Jiang, Qun Liu, Dit-Yan Yeung

Semantic Text Processing

Challenge:	Variational autoencoders (VAEs) have been widely applied in text generation tasks, but they suffer from insufficient representation capacity and poor controllability.
Approach:	They propose a data-driven prior that has expressivity and controllability.
Outcome:	The proposed prior enjoys expressivity and controllability and can be used in language modeling and controlled text generation.

Variational Autoencoder with Disentanglement Priors for Low-Resource Task-Specific Natural Language Generation (2022.emnlp-main)

Copied to clipboard

Zhuang Li, Lizhen Qu, Qiongkai Xu, Tongtong Wu, Tianyang Zhan, Gholamreza Haffari

Low-Resource NLP

Challenge:	Existing models for task-specific natural language generation do not contain any labeled examples.
Approach:	They propose a variational autoencoder with disentanglement priors for task-specific natural language generation with none or a handful of task-related labeled examples.
Outcome:	The proposed model outperforms baseline models in terms of data augmentation and text style transfer in the few-shot setting.

De-Confounded Variational Encoder-Decoder for Logical Table-to-Text Generation (2021.acl-long)

Copied to clipboard

Wenqing Chen, Jidong Tian, Yitian Li, Hao He, Yaohui Jin

Text Generation

Challenge:	Logical table-to-text generation is challenging where deep learning models capture surface-level spurious correlations rather than the causal relationships between the table x and the sentence y.
Approach:	They propose to use variational inference to estimate the confounders in the latent space and cooperate with the causal intervention based on Pearl’s do-calculus to alleviate the spurious correlations.
Outcome:	The proposed model outperforms baselines and achieves new state-of-the-art performance on two logical table-to-text datasets in terms of logical fidelity.