| Challenge: | Variational auto-encoders have been used for text generation but their representation power is limited due to two reasons. |
| Approach: | They advocate sample-based representations of variational distributions for natural language . they further develop an LVM to directly match the aggregated posterior to the prior . |
| Outcome: | The proposed model can be viewed as a natural extension of VAEs with a regularization of maximizing mutual information, mitigating the "posterior collapse" issue. |
Similar Papers
Generative Text Modeling through Short Run Inference (2021.eacl-main)
Copied to clipboard
| Challenge: | Latent variable models for text capture global semantic and syntactic features when trained correctly. |
| Approach: | They propose a short run dynamics for inference that initializes from the prior distribution of the latent variable and runs a small number of Langevin dynamics steps guided by its posterior distribution. |
| Outcome: | The proposed model is able to generate coherent sentences with smooth transition and shows no sign of posterior collapse. |
Empirical Prior for Text Autoencoders (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Variational Autoencoders (VAE) are used to train generative models with latent variables. |
| Approach: | They propose a transition from Variational Autoencoders (VAE) to text autoencodeurs (AE) which model a compact latent space and preserves the capability of the language model itself. |
| Outcome: | The proposed method generates higher quality and more diverse text than the VAE-based Transformer baselines, and is more efficient than previous approaches. |
Contrastive Deterministic Autoencoders For Language Modeling (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Variational autoencoders (VAEs) are a popular family of generative models with wide applicability. |
| Approach: | They propose to modify a deterministic model designed for images to avoid posterior collapse by controlling the entropy of the aggregate posterior to make it Gaussian. |
| Outcome: | The proposed models outperform a broad range of VAE models on text generation and downstream tasks from representations while avoiding reparametrization steps. |
Explicit Bayesian Inference to Uncover the Latent Themes of Large Language Models (2025.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) have impressive generative capabilities, yet their inner mechanisms remain largely opaque. |
| Approach: | They propose a variational autoencoder-based neural topic model to interpret LLMs generation process through an explicit Bayesian framework by inferring latent topic variables via variational inference. |
| Outcome: | The proposed model outperforms state-of-the-art topic models on intrinsic measures of coherence and diversity on multiple datasets and shows significant gains on classification and summarization tasks. |
Deep Latent Variable Models of Natural Language (D18-3)
Copied to clipboard
| Challenge: | In this tutorial, we will discuss the challenges of applying neural variational inference to NLP problems. |
| Approach: | The tutorial will cover deep latent variable models in the case where exact inference over the latent variables is tractable. |
| Outcome: | The proposed tutorial will cover deep latent variable models in the case where inference cannot be performed tractably and when it is not . |
Towards Generating Long and Coherent Text with Multi-Level Latent Variable Models (P19-1)
Copied to clipboard
| Challenge: | Variational autoencoders (VAEs) have received much attention as an end-to-end architecture for text generation with latent variables. |
| Approach: | They propose to leverage several multi-level structures to learn a variational autoencoder model for generating long, and coherent text. |
| Outcome: | The proposed model produces more coherent and less repetitive long text compared to baselines and mitigates posterior collapse issue. |
A Surprisingly Effective Fix for Deep Latent Variable Modeling of Text (D19-1)
Copied to clipboard
| Challenge: | Variational Autoencoders are powerful language models and effective representation learning frameworks. |
| Approach: | They propose a fix for posterior collapse which improves held-out likelihood, reconstruction and latent representation learning . |
| Outcome: | The proposed fix significantly improves held-out likelihood, reconstruction, and latent representation learning compared with previous state-of-the-art methods. |
Scale-VAE: Preventing Posterior Collapse in Variational Autoencoder (2024.lrec-main)
Copied to clipboard
| Challenge: | Variational autoencoder (VAE) is a widely used generative model . but when employing strong autoregressive generation networks, VAE tends to converge to a degenerate local optimum known as posterior collapse. |
| Approach: | They propose a model called Scale-VAE to solve a posterior collapse problem . they use a factor to keep the posterior dimension discriminative across data instances . |
| Outcome: | The proposed model outperforms state-of-the-art models in density estimation and representation learning. |
Spherical Latent Spaces for Stable Variational Autoencoders (D18-1)
Copied to clipboard
| Challenge: | Variational autoencoders use a multivariate Gaussian latent variable to capture latent structure in data. |
| Approach: | They propose a variational autoencoder which uses a latent distribution instead of Gaussian . they find that the variational posterior averts the KL collapse by a fixed hyperparameter . |
| Outcome: | The von Mises-Fisher distribution averts the KL collapse and gives better likelihoods than Gaussian models across a range of modeling conditions. |
Improving Variational Autoencoder for Text Modelling with Timestep-Wise Regularisation (2020.coling-main)
Copied to clipboard
| Challenge: | Variational Autoencoders (VAEs) have been widely used in text modelling but posterior collapse is a problem when RNN-based models are employed. |
| Approach: | They propose a timestep-wise regularisation VAE architecture which can effectively avoid posterior collapse when used in text modelling. |
| Outcome: | The proposed model avoids posterior collapse and can be applied to any RNN-based VAE model. |