Empowering Diffusion Models on the Embedding Space for Text Generation (2024.naacl-long)
Copied to clipboard
| Challenge: | Recent work adapts diffusion models to textual data by diffusing on the embedding space. |
| Approach: | They propose an embedding diffusion model based on Transformer to solve the problem of embeddable space and denoising model. |
| Outcome: | The proposed model is more efficient than previous methods on seminal text generation tasks and is superior to existing models. |
Similar Papers
Can Diffusion Model Achieve Better Performance in Text Generation ? Bridging the Gap between Training and Inference ! (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing models for text generation use a discrete data embedding module to map the data into the continuous space. |
| Approach: | They propose two methods to bridge the gap between training and inference by mapping the discrete text into the continuous space. |
| Outcome: | The proposed methods can achieve 100 200 speedup with better performance on 6 generation tasks. |
Text Diffusion Model with Encoder-Decoder Transformers for Sequence-to-Sequence Generation (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing diffusion models are applied to continuous feature space while texts are sequences of discrete categorical tokens. |
| Approach: | They propose to use an encoder-decoder Transformer architecture to approach sequence-to-sequence text generation. |
| Outcome: | The proposed model improves on five sequence-to-sequence generation tasks compared to other diffusion-based models regarding text quality and inference time. |
Understanding the Influence of Synthetic Data for Text Embedders (2025.findings-acl)
Copied to clipboard
| Challenge: | Recent advances in general purpose text embedders have been driven by training on synthetic training data. |
| Approach: | They propose to use GPT-4 to produce high quality synthetic data that expands existing training datasets for embeddings to new tasks. |
| Outcome: | The proposed dataset is high quality and leads to consistent improvements in performance. |
Addressing the Training-Inference Discrepancy in Discrete Diffusion for Text Generation (2025.coling-main)
Copied to clipboard
| Challenge: | Existing discrete diffusion models for text generation have a discrepancy between training and inference. |
| Approach: | They propose a training schema that considers two-step diffusion processes and a scheduling technique that gradually increases the probability of using self-generated text as training progresses. |
| Outcome: | The proposed training schema and scheduling technique improve diffusion models on four widely used datasets. |
Leveraging the Structure of Pre-trained Embeddings to Minimize Annotation Effort (2024.naacl-long)
Copied to clipboard
| Challenge: | Current approaches for text classification are based on fine-tuning the representations computed by large language models. |
| Approach: | They propose to exploit structural properties of pre-trained embeddings to spread information . they use a semisupervised strategy to train models with minimal annotation effort . |
| Outcome: | The proposed method outperforms self-training and random walk labels on different datasets. |
DiffuseDef: Improved Robustness to Adversarial Attacks via Iterative Denoising (2025.acl-long)
Copied to clipboard
| Challenge: | Existing adversarial defense methods for natural language processing still pose challenges to adversarials. |
| Approach: | They propose a novel adversarial defense method that incorporates a diffusion layer as a denoiser between the encoder and the classifier. |
| Outcome: | The proposed method improves over existing adversarial defense methods and achieves state-of-the-art performance against black-box and white-box adversarials. |
How to Dissect a Muppet: The Structure of Transformer Embedding Spaces (2022.tacl-1)
Copied to clipboard
| Challenge: | Pretrained embeddings based on the Transformer architecture have taken the NLP community by storm . a novel decomposition of Transformer output embeddables is demonstrated . |
| Approach: | They propose to decompose Transformer output embeddings into a sum of vector factors . they show multi-head attentions and feed-forwards are not equally useful in downstream applications . |
| Outcome: | The proposed method outperforms recurrent architectures on a wide variety of tasks. |
DiffuSum: Generation Enhanced Extractive Summarization with Diffusion (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for extractive summarization are formulated as a sequence labeling problem by making individual sentence label predictions. |
| Approach: | They propose a novel paradigm for extractive summarization by directly generating summary sentences with diffusion models and extracting sentences based on sentence representation matching. |
| Outcome: | The proposed framework achieves state-of-the-art extractive results on CNN/DailyMail with ROUGE scores of 44.83/22.56/40.56. |
On the Semantic Latent Space of Diffusion-Based Text-To-Speech Models (2024.acl-short)
Copied to clipboard
| Challenge: | Denoising Diffusion Models (DDMs) are a powerful generative tool for text-to-speech (TTS) but their semantic capabilities are unknown and control of synthesized speech’s vocal properties remains a challenge. |
| Approach: | They explore the latent space of frozen TTS models composed of latent bottleneck activations of the DDM’s denoiser and propose methods for finding semantic directions within it. |
| Outcome: | The proposed methods enable off-the-shelf audio editing without any training, architectural changes or data requirements. |
Sparse Distillation: Speeding Up Text Classification by Using Bigger Student Models (2022.naacl-main)
Copied to clipboard
| Challenge: | Existing methods to reduce inference cost by distilling transformer models into lightweight student models are limited for high-volume use cases. |
| Approach: | They propose to distill state-of-the-art transformer models into lightweight student models to reduce computation cost at inference time. |
| Outcome: | The proposed pipeline achieves up to 600x speed-up on GPUs and CPUs on six single-sentence text classification tasks and in domain generalization settings. |