Cyclical Annealing Schedule: A Simple Approach to Mitigating KL Vanishing

Challenge:	Variational autoencoders (VAEs) with an auto-regressive decoder have been applied for many natural language processing tasks.
Approach:	They propose a cyclical annealing schedule which repeats the process of increasing multiple times to learn more meaningful latent codes progressively by leveraging previous learning cycles as warm re-restart.
Outcome:	The proposed method improves on a broad range of NLP tasks, including language modeling, dialog response generation and semi-supervised text classification.

Similar Papers

Overview

Challenge

Approach

Outcome

On the Importance of the Kullback-Leibler Divergence Term in Variational Autoencoders for Text Generation (D19-56)

Copied to clipboard

Victor Prokhorov, Ehsan Shareghi, Yingzhen Li, Mohammad Taher Pilehvar, Nigel Collier

Language Models

Challenge:	Variational Autoencoders suffer from learning uninformative latent representations due to issues such as approximated posterior collapse or entanglement of the latent space.
Approach:	They propose to impose an explicit constraint on the Kullback-Leibler divergence term inside the VAE objective function to understand the significance of the KL term in controlling the information transmitted through the VAe channel.
Outcome:	The proposed constraint avoids posterior collapse, but it also controls the information transmitted through the VAE channel.

Improving Variational Autoencoder for Text Modelling with Timestep-Wise Regularisation (2020.coling-main)

Copied to clipboard

Ruizhe Li, Xiao Li, Guanyi Chen, Chenghua Lin

Language Models

Challenge:	Variational Autoencoders (VAEs) have been widely used in text modelling but posterior collapse is a problem when RNN-based models are employed.
Approach:	They propose a timestep-wise regularisation VAE architecture which can effectively avoid posterior collapse when used in text modelling.
Outcome:	The proposed model avoids posterior collapse and can be applied to any RNN-based VAE model.

Fuse It More Deeply! A Variational Transformer with Layer-Wise Latent Variable Inference for Text Generation (2022.naacl-main)

Copied to clipboard

Jinyi Hu, Xiaoyuan Yi, Wenhao Li, Maosong Sun, Xing Xie

Language Models

Challenge:	Variational Auto-Encoders are often used for text generation tasks due to the sequential nature of the text.
Approach:	They propose a variational Transformer framework that learns a series of layer-wise latent variables with each inferred from those of lower layers and tightly coupled with the hidden states by low-rank tensor product.
Outcome:	The proposed framework can learn latent variables from lower layers and incorporate more information.

Stochastic Wasserstein Autoencoder for Probabilistic Sentence Generation (N19-1)

Copied to clipboard

Hareesh Bahuleyan, Lili Mou, Hao Zhou, Olga Vechtomova

Language Models

Challenge:	Experimental results show that the latent space learned by WAE exhibits properties of continuity and smoothness as in VAEs.
Approach:	They propose to use the variational autoencoder (VAE) for probabilistic sentence generation . they propose a variant of WAE that encourages the stochasticity of the encoder .
Outcome:	The proposed variant encourages the stochasticity of the encoder while achieving higher BLEU scores.

Do sequence-to-sequence VAEs learn global features of sentences? (2020.emnlp-main)

Copied to clipboard

Tom Bosc, Pascal Vincent

Language Models

Challenge:	Autoregressive language models are often trained without explicit conditioning labels . authors question claim that latent vectors can capture global features in unsupervised manner .
Approach:	They propose to use a sequence-to-sequence architecture to learn latent variables . they find that VAEs are prone to memorizing the first words and sentence length .
Outcome:	The proposed model is prone to memorizing the first words and sentence length, the authors show . et al., 2016: a new model learns latent variables that are more global, more predictive of topic or topic labels . authors question this claim, say it is a waste of time and money .

A Batch Normalized Inference Network Keeps the KL Vanishing Away (2020.acl-main)

Copied to clipboard

Qile Zhu, Wei Bi, Xiaojiang Liu, Xiyao Ma, Xiaolin Li, Dapeng Wu

Challenge:	Variational Autoencoder (VAE) is widely used to approximate a model’s posterior on latent variables.
Approach:	They propose to let the Kullback–Leibler divergence individual follow a distribution across the whole dataset and analyze that it is sufficient to prevent posterior collapse by keeping the expectation of the KL’s distribution positive.
Outcome:	The proposed approach can avoid posterior collapse effectively and efficiently without introducing any new model component or modifying the objective.

Contrastive Deterministic Autoencoders For Language Modeling (2023.findings-emnlp)

Copied to clipboard

Amur Ghose, Pascal Poupart

Language Models

Challenge:	Variational autoencoders (VAEs) are a popular family of generative models with wide applicability.
Approach:	They propose to modify a deterministic model designed for images to avoid posterior collapse by controlling the entropy of the aggregate posterior to make it Gaussian.
Outcome:	The proposed models outperform a broad range of VAE models on text generation and downstream tasks from representations while avoiding reparametrization steps.

Recurrence Boosts Diversity! Revisiting Recurrent Latent Variable in Transformer-Based Variational AutoEncoder for Diverse Text Generation (2022.findings-emnlp)

Copied to clipboard

Jinyi Hu, Xiaoyuan Yi, Wenhao Li, Maosong Sun, Xing Xie

Language Models

Challenge:	Variational Auto-Encoder (VAE) has been widely adopted in text generation due to its ability to learn flexible representations.
Approach:	They propose a Transformer-based recurrent VAE structure that imposes recurrence on segment-wise latent variables with arbitrarily separated text segments and constructs the posterior distribution with residual parameterization.
Outcome:	The proposed structure can deduce a non-zero lower bound of the KL term and enhance the entanglement of each segment and preceding latent variables, providing a theoretical guarantee of generation diversity.

Non-Autoregressive Neural Machine Translation with Consistency Regularization Optimized Variational Framework (2022.naacl-main)

Copied to clipboard

Minghao Zhu, Junli Wang, Chungang Yan

Machine Translation

Challenge:	Variational Autoencoder (VAE) is an effective framework to model the interdependency for non-autoregressive neural machine translation (NAT).
Approach:	They propose to use Variational Autoencoder to model interdependency for non-autoregressive neural machine translation (NAT) a posterior consistency regularization approach is proposed to improve translation quality .
Outcome:	The proposed model is 1.5/0.7 and 0.8/0.3 BLEU points faster than the baseline model.

Spherical Latent Spaces for Stable Variational Autoencoders (D18-1)

Copied to clipboard

Jiacheng Xu, Greg Durrett

Semantic Text Processing

Challenge:	Variational autoencoders use a multivariate Gaussian latent variable to capture latent structure in data.
Approach:	They propose a variational autoencoder which uses a latent distribution instead of Gaussian . they find that the variational posterior averts the KL collapse by a fixed hyperparameter .
Outcome:	The von Mises-Fisher distribution averts the KL collapse and gives better likelihoods than Gaussian models across a range of modeling conditions.