On the Importance of the Kullback-Leibler Divergence Term in Variational Autoencoders for Text Generation (D19-56)
Copied to clipboard
| Challenge: | Variational Autoencoders suffer from learning uninformative latent representations due to issues such as approximated posterior collapse or entanglement of the latent space. |
| Approach: | They propose to impose an explicit constraint on the Kullback-Leibler divergence term inside the VAE objective function to understand the significance of the KL term in controlling the information transmitted through the VAe channel. |
| Outcome: | The proposed constraint avoids posterior collapse, but it also controls the information transmitted through the VAE channel. |
Similar Papers
Scale-VAE: Preventing Posterior Collapse in Variational Autoencoder (2024.lrec-main)
Copied to clipboard
| Challenge: | Variational autoencoder (VAE) is a widely used generative model . but when employing strong autoregressive generation networks, VAE tends to converge to a degenerate local optimum known as posterior collapse. |
| Approach: | They propose a model called Scale-VAE to solve a posterior collapse problem . they use a factor to keep the posterior dimension discriminative across data instances . |
| Outcome: | The proposed model outperforms state-of-the-art models in density estimation and representation learning. |
Spherical Latent Spaces for Stable Variational Autoencoders (D18-1)
Copied to clipboard
| Challenge: | Variational autoencoders use a multivariate Gaussian latent variable to capture latent structure in data. |
| Approach: | They propose a variational autoencoder which uses a latent distribution instead of Gaussian . they find that the variational posterior averts the KL collapse by a fixed hyperparameter . |
| Outcome: | The von Mises-Fisher distribution averts the KL collapse and gives better likelihoods than Gaussian models across a range of modeling conditions. |
A Batch Normalized Inference Network Keeps the KL Vanishing Away (2020.acl-main)
Copied to clipboard
| Challenge: | Variational Autoencoder (VAE) is widely used to approximate a model’s posterior on latent variables. |
| Approach: | They propose to let the Kullback–Leibler divergence individual follow a distribution across the whole dataset and analyze that it is sufficient to prevent posterior collapse by keeping the expectation of the KL’s distribution positive. |
| Outcome: | The proposed approach can avoid posterior collapse effectively and efficiently without introducing any new model component or modifying the objective. |
Improving Variational Autoencoder for Text Modelling with Timestep-Wise Regularisation (2020.coling-main)
Copied to clipboard
| Challenge: | Variational Autoencoders (VAEs) have been widely used in text modelling but posterior collapse is a problem when RNN-based models are employed. |
| Approach: | They propose a timestep-wise regularisation VAE architecture which can effectively avoid posterior collapse when used in text modelling. |
| Outcome: | The proposed model avoids posterior collapse and can be applied to any RNN-based VAE model. |
Empirical Prior for Text Autoencoders (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Variational Autoencoders (VAE) are used to train generative models with latent variables. |
| Approach: | They propose a transition from Variational Autoencoders (VAE) to text autoencodeurs (AE) which model a compact latent space and preserves the capability of the language model itself. |
| Outcome: | The proposed method generates higher quality and more diverse text than the VAE-based Transformer baselines, and is more efficient than previous approaches. |
On the Encoder-Decoder Incompatibility in Variational Text Modeling and Beyond (2020.acl-main)
Copied to clipboard
| Challenge: | Existing work has shown that the optimization of variational autoencoders suffers from the posterior collapse problem. |
| Approach: | They propose a variational autoencoder that couples a VAE model with a deterministic autoencoding model and improves the parameters via weight sharing and decoder signal matching. |
| Outcome: | The proposed model improves on benchmark datasets and improves diversity of dialogue generation. |
A Surprisingly Effective Fix for Deep Latent Variable Modeling of Text (D19-1)
Copied to clipboard
| Challenge: | Variational Autoencoders are powerful language models and effective representation learning frameworks. |
| Approach: | They propose a fix for posterior collapse which improves held-out likelihood, reconstruction and latent representation learning . |
| Outcome: | The proposed fix significantly improves held-out likelihood, reconstruction, and latent representation learning compared with previous state-of-the-art methods. |
Fuse It More Deeply! A Variational Transformer with Layer-Wise Latent Variable Inference for Text Generation (2022.naacl-main)
Copied to clipboard
| Challenge: | Variational Auto-Encoders are often used for text generation tasks due to the sequential nature of the text. |
| Approach: | They propose a variational Transformer framework that learns a series of layer-wise latent variables with each inferred from those of lower layers and tightly coupled with the hidden states by low-rank tensor product. |
| Outcome: | The proposed framework can learn latent variables from lower layers and incorporate more information. |
Implicit Deep Latent Variable Models for Text Generation (D19-1)
Copied to clipboard
| Challenge: | Variational auto-encoders have been used for text generation but their representation power is limited due to two reasons. |
| Approach: | They advocate sample-based representations of variational distributions for natural language . they further develop an LVM to directly match the aggregated posterior to the prior . |
| Outcome: | The proposed model can be viewed as a natural extension of VAEs with a regularization of maximizing mutual information, mitigating the "posterior collapse" issue. |
Enhancing Variational Autoencoders with Mutual Information Neural Estimation for Text Generation (D19-1)
Copied to clipboard
| Challenge: | Existing approaches to train variational autoencoders (VAEs) have been proposed to alleviate the posterior collapse issue in NLP tasks. |
| Approach: | They propose to introduce a mutual information term between the input and its latent variable to regularize the objective of the VAE. |
| Outcome: | The proposed model performs better on three benchmark datasets and is comparable to state-of-the-art models. |