Papers by Alex Smola
A Cheaper and Better Diffusion Language Model with Soft-Masked Noise (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing diffusion models have limitations in modeling discrete data, e.g., languages . we present a novel diffusion model for language modeling inspired by linguistic features in languages based on iterative denoising . |
| Approach: | They propose a method that iteratively denoises and adds corruptions to the textual data through soft-masking to better noise it. |
| Outcome: | The proposed model achieves better generation quality and lower training cost than current models with better performance. |