Riemannian Normalizing Flow on Variational Wasserstein Autoencoder for Text Modeling

Challenge:	Empirical experiments show that our model learns latent distributions that respect latent space geometry and is able to generate sentences that are more diverse.
Approach:	They propose a Variational Wasserstein Autoencoder with Riemannian Normalizing Flow to solve this problem by transforming a latent variable into a space that respects the geometric characteristics of input space.
Outcome:	Empirical results show that the proposed model avoids KLvanishing and has better performance in language modeling, likelihood approximation, and text generation tasks.

Similar Papers

Overview

Challenge

Approach

Outcome

Stochastic Wasserstein Autoencoder for Probabilistic Sentence Generation (N19-1)

Copied to clipboard

Hareesh Bahuleyan, Lili Mou, Hao Zhou, Olga Vechtomova

Language Models

Challenge:	Experimental results show that the latent space learned by WAE exhibits properties of continuity and smoothness as in VAEs.
Approach:	They propose to use the variational autoencoder (VAE) for probabilistic sentence generation . they propose a variant of WAE that encourages the stochasticity of the encoder .
Outcome:	The proposed variant encourages the stochasticity of the encoder while achieving higher BLEU scores.

Improving Variational Autoencoder for Text Modelling with Timestep-Wise Regularisation (2020.coling-main)

Copied to clipboard

Ruizhe Li, Xiao Li, Guanyi Chen, Chenghua Lin

Language Models

Challenge:	Variational Autoencoders (VAEs) have been widely used in text modelling but posterior collapse is a problem when RNN-based models are employed.
Approach:	They propose a timestep-wise regularisation VAE architecture which can effectively avoid posterior collapse when used in text modelling.
Outcome:	The proposed model avoids posterior collapse and can be applied to any RNN-based VAE model.

Modeling Personalization in Continuous Space for Response Generation via Augmented Wasserstein Autoencoders (D19-1)

Copied to clipboard

Zhangming Chan, Juntao Li, Xiaopeng Yang, Xiuying Chen, Wenpeng Hu, Dongyan Zhao, Rui Yan

Text Generation

Challenge:	Existing work on variableal autoencoders and waterstein autoencoding models has shown significant progress in open-domain response generation.
Approach:	They propose to embed user-level and utterance-level information into two multimodal distributions and combine them into a mixed distribution.
Outcome:	The proposed model outperforms state-of-the-art models on a large-scale real-world dataset.

Generative Text Modeling through Short Run Inference (2021.eacl-main)

Copied to clipboard

Bo Pang, Erik Nijkamp, Tian Han, Ying Nian Wu

Challenge:	Latent variable models for text capture global semantic and syntactic features when trained correctly.
Approach:	They propose a short run dynamics for inference that initializes from the prior distribution of the latent variable and runs a small number of Langevin dynamics steps guided by its posterior distribution.
Outcome:	The proposed model is able to generate coherent sentences with smooth transition and shows no sign of posterior collapse.

Empirical Prior for Text Autoencoders (2024.findings-emnlp)

Copied to clipboard

Yongjing Yin, Wenyang Gao, Haodong Wu, Jianhao Yan, Yue Zhang

Language Models

Challenge:	Variational Autoencoders (VAE) are used to train generative models with latent variables.
Approach:	They propose a transition from Variational Autoencoders (VAE) to text autoencodeurs (AE) which model a compact latent space and preserves the capability of the language model itself.
Outcome:	The proposed method generates higher quality and more diverse text than the VAE-based Transformer baselines, and is more efficient than previous approaches.

Spherical Latent Spaces for Stable Variational Autoencoders (D18-1)

Copied to clipboard

Jiacheng Xu, Greg Durrett

Semantic Text Processing

Challenge:	Variational autoencoders use a multivariate Gaussian latent variable to capture latent structure in data.
Approach:	They propose a variational autoencoder which uses a latent distribution instead of Gaussian . they find that the variational posterior averts the KL collapse by a fixed hyperparameter .
Outcome:	The von Mises-Fisher distribution averts the KL collapse and gives better likelihoods than Gaussian models across a range of modeling conditions.

On the Importance of the Kullback-Leibler Divergence Term in Variational Autoencoders for Text Generation (D19-56)

Copied to clipboard

Victor Prokhorov, Ehsan Shareghi, Yingzhen Li, Mohammad Taher Pilehvar, Nigel Collier

Language Models

Challenge:	Variational Autoencoders suffer from learning uninformative latent representations due to issues such as approximated posterior collapse or entanglement of the latent space.
Approach:	They propose to impose an explicit constraint on the Kullback-Leibler divergence term inside the VAE objective function to understand the significance of the KL term in controlling the information transmitted through the VAe channel.
Outcome:	The proposed constraint avoids posterior collapse, but it also controls the information transmitted through the VAE channel.

Fuse It More Deeply! A Variational Transformer with Layer-Wise Latent Variable Inference for Text Generation (2022.naacl-main)

Copied to clipboard

Jinyi Hu, Xiaoyuan Yi, Wenhao Li, Maosong Sun, Xing Xie

Language Models

Challenge:	Variational Auto-Encoders are often used for text generation tasks due to the sequential nature of the text.
Approach:	They propose a variational Transformer framework that learns a series of layer-wise latent variables with each inferred from those of lower layers and tightly coupled with the hidden states by low-rank tensor product.
Outcome:	The proposed framework can learn latent variables from lower layers and incorporate more information.

Topic Modeling with Wasserstein Autoencoders (P19-1)

Copied to clipboard

Feng Nan, Ran Ding, Ramesh Nallapati, Bing Xiang

Language Models

Challenge:	Existing probabilistic topic models are based on latent Dirichlet allocations and collapsed Gibbs sampling.
Approach:	They propose a novel topic model that enforces Dirichlet prior on latent document-topic vectors and a kernel kernel to minimize the Maximum Mean Discrepancy (MMD) They propose to measure the diversity of the produced topics and to use the widely used coherence measure NPMI to evaluate topic quality.
Outcome:	The proposed model performs better than existing topic models on real datasets.

Contrastive Deterministic Autoencoders For Language Modeling (2023.findings-emnlp)

Copied to clipboard

Amur Ghose, Pascal Poupart

Language Models

Challenge:	Variational autoencoders (VAEs) are a popular family of generative models with wide applicability.
Approach:	They propose to modify a deterministic model designed for images to avoid posterior collapse by controlling the entropy of the aggregate posterior to make it Gaussian.
Outcome:	The proposed models outperform a broad range of VAE models on text generation and downstream tasks from representations while avoiding reparametrization steps.