Improving Variational Autoencoder for Text Modelling with Timestep-Wise Regularisation (2020.coling-main)
Copied to clipboard
| Challenge: | Variational Autoencoders (VAEs) have been widely used in text modelling but posterior collapse is a problem when RNN-based models are employed. |
| Approach: | They propose a timestep-wise regularisation VAE architecture which can effectively avoid posterior collapse when used in text modelling. |
| Outcome: | The proposed model avoids posterior collapse and can be applied to any RNN-based VAE model. |
Similar Papers
Scale-VAE: Preventing Posterior Collapse in Variational Autoencoder (2024.lrec-main)
Copied to clipboard
| Challenge: | Variational autoencoder (VAE) is a widely used generative model . but when employing strong autoregressive generation networks, VAE tends to converge to a degenerate local optimum known as posterior collapse. |
| Approach: | They propose a model called Scale-VAE to solve a posterior collapse problem . they use a factor to keep the posterior dimension discriminative across data instances . |
| Outcome: | The proposed model outperforms state-of-the-art models in density estimation and representation learning. |
Empirical Prior for Text Autoencoders (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Variational Autoencoders (VAE) are used to train generative models with latent variables. |
| Approach: | They propose a transition from Variational Autoencoders (VAE) to text autoencodeurs (AE) which model a compact latent space and preserves the capability of the language model itself. |
| Outcome: | The proposed method generates higher quality and more diverse text than the VAE-based Transformer baselines, and is more efficient than previous approaches. |
Contrastive Deterministic Autoencoders For Language Modeling (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Variational autoencoders (VAEs) are a popular family of generative models with wide applicability. |
| Approach: | They propose to modify a deterministic model designed for images to avoid posterior collapse by controlling the entropy of the aggregate posterior to make it Gaussian. |
| Outcome: | The proposed models outperform a broad range of VAE models on text generation and downstream tasks from representations while avoiding reparametrization steps. |
Generative Text Modeling through Short Run Inference (2021.eacl-main)
Copied to clipboard
| Challenge: | Latent variable models for text capture global semantic and syntactic features when trained correctly. |
| Approach: | They propose a short run dynamics for inference that initializes from the prior distribution of the latent variable and runs a small number of Langevin dynamics steps guided by its posterior distribution. |
| Outcome: | The proposed model is able to generate coherent sentences with smooth transition and shows no sign of posterior collapse. |
On the Importance of the Kullback-Leibler Divergence Term in Variational Autoencoders for Text Generation (D19-56)
Copied to clipboard
| Challenge: | Variational Autoencoders suffer from learning uninformative latent representations due to issues such as approximated posterior collapse or entanglement of the latent space. |
| Approach: | They propose to impose an explicit constraint on the Kullback-Leibler divergence term inside the VAE objective function to understand the significance of the KL term in controlling the information transmitted through the VAe channel. |
| Outcome: | The proposed constraint avoids posterior collapse, but it also controls the information transmitted through the VAE channel. |
Enhancing Variational Autoencoders with Mutual Information Neural Estimation for Text Generation (D19-1)
Copied to clipboard
| Challenge: | Existing approaches to train variational autoencoders (VAEs) have been proposed to alleviate the posterior collapse issue in NLP tasks. |
| Approach: | They propose to introduce a mutual information term between the input and its latent variable to regularize the objective of the VAE. |
| Outcome: | The proposed model performs better on three benchmark datasets and is comparable to state-of-the-art models. |
Implicit Deep Latent Variable Models for Text Generation (D19-1)
Copied to clipboard
| Challenge: | Variational auto-encoders have been used for text generation but their representation power is limited due to two reasons. |
| Approach: | They advocate sample-based representations of variational distributions for natural language . they further develop an LVM to directly match the aggregated posterior to the prior . |
| Outcome: | The proposed model can be viewed as a natural extension of VAEs with a regularization of maximizing mutual information, mitigating the "posterior collapse" issue. |
Riemannian Normalizing Flow on Variational Wasserstein Autoencoder for Text Modeling (N19-1)
Copied to clipboard
| Challenge: | Empirical experiments show that our model learns latent distributions that respect latent space geometry and is able to generate sentences that are more diverse. |
| Approach: | They propose a Variational Wasserstein Autoencoder with Riemannian Normalizing Flow to solve this problem by transforming a latent variable into a space that respects the geometric characteristics of input space. |
| Outcome: | Empirical results show that the proposed model avoids KLvanishing and has better performance in language modeling, likelihood approximation, and text generation tasks. |
Non-Autoregressive Neural Machine Translation with Consistency Regularization Optimized Variational Framework (2022.naacl-main)
Copied to clipboard
| Challenge: | Variational Autoencoder (VAE) is an effective framework to model the interdependency for non-autoregressive neural machine translation (NAT). |
| Approach: | They propose to use Variational Autoencoder to model interdependency for non-autoregressive neural machine translation (NAT) a posterior consistency regularization approach is proposed to improve translation quality . |
| Outcome: | The proposed model is 1.5/0.7 and 0.8/0.3 BLEU points faster than the baseline model. |
A Batch Normalized Inference Network Keeps the KL Vanishing Away (2020.acl-main)
Copied to clipboard
| Challenge: | Variational Autoencoder (VAE) is widely used to approximate a model’s posterior on latent variables. |
| Approach: | They propose to let the Kullback–Leibler divergence individual follow a distribution across the whole dataset and analyze that it is sufficient to prevent posterior collapse by keeping the expectation of the KL’s distribution positive. |
| Outcome: | The proposed approach can avoid posterior collapse effectively and efficiently without introducing any new model component or modifying the objective. |