PAI-Diffusion: Constructing and Serving a Family of Open Chinese Diffusion Models for Text-to-image Synthesis on the Cloud

Challenge:	Existing diffusion models fail to address the challenges of generating high-quality images from textual descriptions due to its large vocabulary size and complex character relationships.
Approach:	They propose a framework that integrates Chinese diffusion models with Alibaba Cloud's Platform for AI and enables the generation of contextually relevant images.
Outcome:	The proposed framework integrates with Alibaba Cloud’s Platform for AI, providing accessible and scalable solutions.

Similar Papers

Overview

Challenge

Approach

Outcome

Rapid Diffusion: Building Domain-Specific Text-to-Image Synthesizers with Fast Inference Speed (2023.acl-industry)

Copied to clipboard

Bingyan Liu, Weifeng Lin, Zhongjie Duan, Chengyu Wang, Wu Ziheng, Zhang Zipeng, Kui Jia, Lianwen Jin, Cen Chen, Jun Huang

Multimodality

Challenge:	Text-to-Image Synthesis (TIS) aims to generate images based on textual inputs . but, current diffusion-based models lack entity knowledge and low inference speed .
Approach:	They propose a framework for training and deploying latent diffusion models with rich entity knowledge injected and optimized networks.
Outcome:	The proposed framework improves image quality and inference speed and can be used in industrial applications.

Private Synthetic Text Generation with Diffusion Models (2025.naacl-long)

Copied to clipboard

Sebastian Ochs, Ivan Habernal

Responsible & Trustworthy NLP

Challenge:	Recent research shows diffusion models are capable of generating synthetics texts . but are they also good in generating private data if the training was under differential privacy?
Approach:	They examine whether diffusion models are capable of generating synthetics texts . they critically assess previous work on private text generation with diffusion models .
Outcome:	The proposed model outperforms auto-regressive models in generating private images despite unmet privacy assumptions . the proposed model is open-source and can be used for other purposes .

LaDiC: Are Diffusion Models Really Inferior to Autoregressive Counterparts for Image-to-Text Generation? (2024.naacl-long)

Copied to clipboard

Yuchi Wang, Shuhuai Ren, Rundong Gao, Linli Yao, Qingyan Guo, Kaikai An, Jianhong Bai, Xu Sun

Multimodality

Challenge:	Existing models for text-to-image generation have been underperforming in image-totext generation tasks.
Approach:	They propose a framework that uses a split BERT to create a dedicated latent space for captions and integrates a regularization module to manage varying text lengths.
Outcome:	The proposed framework achieves state-of-the-art performance on the MS COCO dataset with 38.2 BLEU@4 and 126.2 CIDEr .

Diffusion Lens: Interpreting Text Encoders in Text-to-Image Pipelines (2024.acl-long)

Copied to clipboard

Michael Toker, Hadas Orgad, Mor Ventura, Dana Arad, Yonatan Belinkov

Responsible & Trustworthy NLP

Challenge:	Text-to-image diffusion models use a latent text prompt to guide image generation . however, the process by which the encoder produces the text representation is unknown .
Approach:	They propose a method for analyzing the text encoder of T2I models by generating images from its intermediate representations.
Outcome:	The proposed method provides valuable insights into the text encoder component in T2I pipelines.

Kandinsky 3: Text-to-Image Synthesis for Multifunctional Generative Framework (2024.emnlp-demo)

Copied to clipboard

Arkhipkin Vladimir, Viacheslav Vasilev, Andrei Filatov, Igor Pavlov, Julia Agafonova, Nikolai Gerasimenko, Anna Averchenkova, Evelina Mironova, Bukashkin Anton, Konstantin Kulikov, Andrey Kuznetsov, Denis Dimitrov

Multimodality

Challenge:	Text-to-image (T2I) diffusion models are popular for image manipulation, but also for video generation.
Approach:	They propose a novel T2I diffusion model based on latent diffusion that extends the base model for various applications.
Outcome:	The proposed model achieves high quality and photorealism and is 3 times faster than the base model.

Segment-Level Diffusion: A Framework for Controllable Long-Form Generation with Diffusion Language Models (2025.acl-long)

Copied to clipboard

Xiaochen Zhu, Georgi Karadzhov, Chenxi Whitehouse, Andreas Vlachos

Robustness in NLP

Challenge:	Diffusion models have shown promise in text generation, but often struggle with generating long, coherent, and contextually accurate text.
Approach:	They propose a framework that enhances diffusion-based text generation through text segmentation, robust representation training with adversarial and contrastive learning, and improved latent-space guidance.
Outcome:	The proposed framework improves diffusion-based text generation and improves scalability and fluency.

InfoDiffusion: Information Entropy Aware Diffusion Process for Non-Autoregressive Text Generation (2023.findings-emnlp)

Copied to clipboard

Renzhi Wang, Jing Li, Piji Li

Text Generation

Challenge:	Existing text diffusion models have failed to capture the difference between the “easy-first” process and the “keyword-first" process of humans.
Approach:	They propose a non-autoregressive text diffusion model that incorporates a "keyinfo-first" generation strategy and a noise schedule based on the amount of text information.
Outcome:	The proposed model outperforms the baseline model in terms of generation quality and diversity, and higher sampling efficiency.

DiffusionDB: A Large-scale Prompt Gallery Dataset for Text-to-Image Generative Models (2023.acl-long)

Copied to clipboard

Zijie J. Wang, Evan Montoya, David Munechika, Haoyang Yang, Benjamin Hoover, Duen Horng Chau

Responsible & Trustworthy NLP

Challenge:	Recent advances in diffusion models have enabled high-quality image generation . generating images with desired details requires proper prompts .
Approach:	They analyze syntactic and semantic characteristics of diffusion models and their prompts . they pinpoint specific hyperparameter values and prompt styles that can lead to model errors .
Outcome:	The first large-scale text-to-image prompt dataset totals 6.5TB . it contains 14 million images generated by Stable Diffusion, 1.8 million unique prompts, and hyperparameters specified by real users.

ARTIST: A Transformer-based Chinese Text-to-Image Synthesizer Digesting Linguistic and World Knowledge (2022.findings-emnlp)

Copied to clipboard

Tingting Liu, Chengyu Wang, Xiangru Zhu, Lei Li, Minghui Qiu, Jun Huang, Ming Gao, Yanghua Xiao

Semantic Text Processing

Challenge:	Text-to-Image Synthesis (TIS) is a popular task to convert natural language texts into realistic images.
Approach:	They propose a transformer-based Chinese text-to-image synthesizer for high-resolution image generation that incorporates linguistic and relational knowledge facts into the model to ensure better performance without the usage of ultra-large models.
Outcome:	The proposed model outperforms existing models in Chinese with linguistic and relational knowledge facts.

Collaborative Generative AI: Integrating GPT-k for Efficient Editing in Text-to-Image Generation (2023.emnlp-main)

Copied to clipboard

Wanrong Zhu, Xinyi Wang, Yujie Lu, Tsu-Jui Fu, Xin Wang, Miguel Eckstein, William Wang

Responsible & Trustworthy NLP

Challenge:	Experimental results show that GPT-k models focus more on inserting modifiers than predicting spontaneous changes in the primary subject matter.
Approach:	They compare the common edits made by humans and GPT-k models to examine their performance in prompting T2I.
Outcome:	The proposed models improve the prompt editing process by 20-30%, the authors show . they show that humans tend to replace words and phrases with modifiers .