VOLTA: Improving Generative Diversity by Variational Mutual Information Maximizing Autoencoder (2024.findings-naacl)
Copied to clipboard
| Challenge: | generative diversity is a critical yet underexplored issue in natural language generation . previous approaches to enhance diversity of Transformer models have been limited by their latent variables . |
| Approach: | They propose a framework that bridges Transformer with VAE to enhance generative diversity. |
| Outcome: | The proposed framework improves generative diversity while maintaining generative quality. |
Similar Papers
Fuse It More Deeply! A Variational Transformer with Layer-Wise Latent Variable Inference for Text Generation (2022.naacl-main)
Copied to clipboard
| Challenge: | Variational Auto-Encoders are often used for text generation tasks due to the sequential nature of the text. |
| Approach: | They propose a variational Transformer framework that learns a series of layer-wise latent variables with each inferred from those of lower layers and tightly coupled with the hidden states by low-rank tensor product. |
| Outcome: | The proposed framework can learn latent variables from lower layers and incorporate more information. |
Recurrence Boosts Diversity! Revisiting Recurrent Latent Variable in Transformer-Based Variational AutoEncoder for Diverse Text Generation (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Variational Auto-Encoder (VAE) has been widely adopted in text generation due to its ability to learn flexible representations. |
| Approach: | They propose a Transformer-based recurrent VAE structure that imposes recurrence on segment-wise latent variables with arbitrarily separated text segments and constructs the posterior distribution with residual parameterization. |
| Outcome: | The proposed structure can deduce a non-zero lower bound of the KL term and enhance the entanglement of each segment and preceding latent variables, providing a theoretical guarantee of generation diversity. |
TILGAN: Transformer-based Implicit Latent GAN for Diverse and Coherent Text Generation (2021.findings-acl)
Copied to clipboard
| Challenge: | Existing autoregressive models suffer from the exposure bias problem due to mismatches between training and generation stages. |
| Approach: | They propose a Transformerbased Implicit Latent GAN which combines a transformer autoencoder and GAN in the latent space with a novel design and distribution matching based on the Kullback-Leibler divergence. |
| Outcome: | The proposed model improves local and global coherence and quality-diversity trade-off on three benchmark datasets. |
Empirical Prior for Text Autoencoders (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Variational Autoencoders (VAE) are used to train generative models with latent variables. |
| Approach: | They propose a transition from Variational Autoencoders (VAE) to text autoencodeurs (AE) which model a compact latent space and preserves the capability of the language model itself. |
| Outcome: | The proposed method generates higher quality and more diverse text than the VAE-based Transformer baselines, and is more efficient than previous approaches. |
Enhancing Variational Autoencoders with Mutual Information Neural Estimation for Text Generation (D19-1)
Copied to clipboard
| Challenge: | Existing approaches to train variational autoencoders (VAEs) have been proposed to alleviate the posterior collapse issue in NLP tasks. |
| Approach: | They propose to introduce a mutual information term between the input and its latent variable to regularize the objective of the VAE. |
| Outcome: | The proposed model performs better on three benchmark datasets and is comparable to state-of-the-art models. |
VarMAE: Pre-training of Variational Masked Autoencoder for Domain-adaptive Language Understanding (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Pre-trained language models have been widely applied to standard benchmarks due to the limited resources available in a domain. |
| Approach: | They propose a Transformer-based language model called VarMAE for domain-adaptive language understanding that encodes the context of a token into a smooth latent distribution. |
| Outcome: | Experiments on science- and finance-domain NLU tasks show that the proposed model can be efficiently adapted to new domains with limited resources. |
RegaVAE: A Retrieval-Augmented Gaussian Mixture Variational Auto-Encoder for Language Modeling (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing research on retrieval-augmented language models has two main problems: determining what information to retrieve and effectively combining retrieved information during generation. |
| Approach: | They propose a retrieval-augmented language model that captures current and future information from source and target text into a latent space. |
| Outcome: | The proposed model is more efficient than explicit raw text, but limited by context length and noise. |
Control, Generate, Augment: A Scalable Framework for Multi-Attribute Text Generation (2020.findings-emnlp)
Copied to clipboard
| Challenge: | a new architecture for controlling, generating and augmenting text is being developed for supervised NLP tasks. |
| Approach: | They propose a conditional VAE architecture to control, generate, and augment text. |
| Outcome: | The proposed model shows high quality, diversity and attribute control in an ablation task. |
On the Importance of the Kullback-Leibler Divergence Term in Variational Autoencoders for Text Generation (D19-56)
Copied to clipboard
| Challenge: | Variational Autoencoders suffer from learning uninformative latent representations due to issues such as approximated posterior collapse or entanglement of the latent space. |
| Approach: | They propose to impose an explicit constraint on the Kullback-Leibler divergence term inside the VAE objective function to understand the significance of the KL term in controlling the information transmitted through the VAe channel. |
| Outcome: | The proposed constraint avoids posterior collapse, but it also controls the information transmitted through the VAE channel. |
Diversity-Aware Coherence Loss for Improving Neural Topic Models (2023.acl-short)
Copied to clipboard
| Challenge: | Experimental results show that our method significantly improves the performance of neural topic models without requiring any pretraining or additional parameters. |
| Approach: | They propose a variational autoencoder framework that minimizes the posterior and prior divergence and a diversity-aware coherence loss that encourages the model to learn corpus-level coherency scores while maintaining high diversity between topics. |
| Outcome: | The proposed approach significantly improves the performance of neural topic models without pretraining or additional parameters. |