Words Worth a Thousand Pictures: Measuring and Understanding Perceptual Variability in Text-to-Image Generation (2024.emnlp-main)
Copied to clipboard
Raphael Tang, Crystina Zhang, Lixinyu Xu, Yao Lu, Wenyan Li, Pontus Stenetorp, Jimmy Lin, Ferhan Ture
| Challenge: | Current diffusion models do not cover recent models, thus we curate three test sets for evaluation. |
| Approach: | They propose a human-calibrated measure of variability in a set of images bootstrapped from existing image-pair perceptual distances. |
| Outcome: | The proposed model outperforms nine baselines by 18 points in accuracy and matches graded human judgements 78% of the time. |
Similar Papers
DiffusionDB: A Large-scale Prompt Gallery Dataset for Text-to-Image Generative Models (2023.acl-long)
Copied to clipboard
| Challenge: | Recent advances in diffusion models have enabled high-quality image generation . generating images with desired details requires proper prompts . |
| Approach: | They analyze syntactic and semantic characteristics of diffusion models and their prompts . they pinpoint specific hyperparameter values and prompt styles that can lead to model errors . |
| Outcome: | The first large-scale text-to-image prompt dataset totals 6.5TB . it contains 14 million images generated by Stable Diffusion, 1.8 million unique prompts, and hyperparameters specified by real users. |
Diffusion Models Through a Global Lens: Are They Culturally Inclusive? (2025.acl-long)
Copied to clipboard
Zahra Bayramli, Ayhan Suleymanzade, Na Min An, Huzama Ahmad, Eunsu Kim, Junyeong Park, James Thorne, Alice Oh
| Challenge: | Text-to-image diffusion models have produced compelling, detailed images from text prompts, but their ability to accurately represent cultural nuances remains an open question. |
| Approach: | They propose a benchmark to evaluate whether diffusion models can generate culturally specific images spanning ten countries. |
| Outcome: | The proposed model fails to generate culturally specific images spanning ten countries . it shows significant disparities in cultural relevance, description fidelity, and realism compared to real-world reference images. |
Culture-TRIP: Culturally-Aware Text-to-Image Generation with Iterative Prompt Refinement (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing text-to-image models fail to produce appropriate images for cultural concepts or objects not well known or underrepresented in western cultures, such as 'hangari' (a Korean utensil). |
| Approach: | They propose a method which iteratively refines the prompt to improve the alignment between the generated images and underrepresented cultural nouns in text-to-image models. |
| Outcome: | The proposed approach improves the alignment between the generated images and cultural nouns in text-to-image models. |
Towards Understanding Sample Variance in Visually Grounded Language Generation: Evaluations and Observations (2020.emnlp-main)
Copied to clipboard
| Challenge: | A major challenge in visually grounded language generation is to build robust benchmark datasets and models that can generalize well in real-world settings. |
| Approach: | They propose to use visual attention to build robust benchmark datasets and models that can generalize well in real-world settings. |
| Outcome: | The proposed models show that human-generated references vary drastically in different datasets/tasks, revealing the nature of each task. |
Prompt Expansion for Adaptive Text-to-Image Generation (2024.acl-long)
Copied to clipboard
| Challenge: | Text-to-image generation models are powerful but difficult to use. Users craft specific prompts to get better images, though the prompts can be repetitive. |
| Approach: | They propose a framework that takes a text query as input and outputs a set of expanded text prompts that are optimized to generate a wider variety of appealing images. |
| Outcome: | The proposed framework generates high-quality images from text prompts with less effort and is more aesthetically pleasing than baseline models. |
Follow the Flow: On Information Flow Across Textual Tokens in Text-to-Image Models (2026.acl-long)
Copied to clipboard
| Challenge: | Prior work focused on improving alignment by refining the diffusion process, ignoring the role of the text encoder, which guides the diffusion. |
| Approach: | They investigate how semantic information is distributed across token representations in text-to-image prompts by patching techniques to uncover encoding patterns. |
| Outcome: | The proposed model can improve alignment and generation quality by modifying the diffusion stage and the cross-attention mechanism. |
Self-Rewarding Large Vision-Language Models for Optimizing Prompts in Text-to-Image Generation (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for rewriting text-to-image models require specialized vocabulary . a new approach uses large vision language models to optimize text-based models . |
| Approach: | They propose a prompt optimization framework that rephrases a user prompt into a text-to-image model by using large vision language models as solver and reward model. |
| Outcome: | The proposed model outperforms existing models on two popular datasets. |
Human Learning by Model Feedback: The Dynamics of Iterative Prompting with Midjourney (2023.emnlp-main)
Copied to clipboard
| Challenge: | Generating images with Text-to-Image models often requires multiple trials, where human users iteratively update their prompt based on feedback, namely the output image. |
| Approach: | They compile a dataset of iterative interactions of human users with Midjourney and analyze the dynamics of the user prompts along these iterations. |
| Outcome: | The proposed model produces better images for a specific language style than other models. |
BeautifulPrompt: Towards Automatic Prompt Engineering for Text-to-Image Synthesis (2023.emnlp-industry)
Copied to clipboard
| Challenge: | Recent text-to-image models require multiple passes of prompt engineering by humans to produce satisfactory results for real-world applications. |
| Approach: | They propose a deep generative model to generate high-quality prompts from raw descriptions using visual feedback. |
| Outcome: | The proposed model produces high-quality prompts from simple raw descriptions . it can be integrated to a cloud-native AI platform to provide better image generation service in the cloud. |
Diffusion Lens: Interpreting Text Encoders in Text-to-Image Pipelines (2024.acl-long)
Copied to clipboard
| Challenge: | Text-to-image diffusion models use a latent text prompt to guide image generation . however, the process by which the encoder produces the text representation is unknown . |
| Approach: | They propose a method for analyzing the text encoder of T2I models by generating images from its intermediate representations. |
| Outcome: | The proposed method provides valuable insights into the text encoder component in T2I pipelines. |