Repairing Catastrophic-Neglect in Text-to-Image Diffusion Models via Attention-Guided Feature Enhancement (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Text-to-Image Diffusion models generate high-quality images from textual descriptions, but they often produce images that do not fully align with the input prompts, resulting in semantic inconsistencies. |
| Approach: | They propose an automated repair approach to address catastrophic-neglect in T2I DMs. |
| Outcome: | The proposed model achieves 10.1%-16.3% higher Correct Rate in image generation compared to baselines. |
Similar Papers
Diffusion Lens: Interpreting Text Encoders in Text-to-Image Pipelines (2024.acl-long)
Copied to clipboard
| Challenge: | Text-to-image diffusion models use a latent text prompt to guide image generation . however, the process by which the encoder produces the text representation is unknown . |
| Approach: | They propose a method for analyzing the text encoder of T2I models by generating images from its intermediate representations. |
| Outcome: | The proposed method provides valuable insights into the text encoder component in T2I pipelines. |
R-Bind: Unified Enhancement of Attribute and Relation Binding in Text-to-Image Diffusion Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to improve semantic binding require costly retraining or focus on only correctly generating attributes of entities. Existing methods focus on correctly generating attributes, ignoring the cruciality of correctly forming relations between entities. |
| Approach: | They propose a training-free method that improves both entity-attribute and entity-relation-entity binding by introducing three inference-time optimization losses that adjust attention maps during generation. |
| Outcome: | The proposed method improves both entity-attribute and entity-relation-entity binding without additional training. |
AcT2I: Evaluating and Improving Action Depiction in Text-to-Image Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Text-to-Image (T2I) models have been successful in generating images from textual descriptions, but they struggle to capture nuanced and implicit attributes inherent in action depiction. |
| Approach: | They propose a benchmark to evaluate the performance of T2I models in generating images from action-centric prompts. |
| Outcome: | The proposed model achieves an increase of 72% on AcT2I. |
Multimodal Pragmatic Jailbreak on Text-to-image Models (2025.acl-long)
Copied to clipboard
Tong Liu, Zhixin Lai, Jiawen Wang, Gengyuan Zhang, Shuo Chen, Philip Torr, Vera Demberg, Volker Tresp, Jindong Gu
| Challenge: | Existing jailbreaks for diffusion-based text-to-image models generate unsafe content . experimental results show that all tested models suffer from unsafe generation . |
| Approach: | They propose a jailbreak that triggers diffusion-based text-to-image models to generate the image with visual text, resulting in unsafe content. |
| Outcome: | The proposed model generates image with visual text, but the model is unsafe under such jailbreak. |
Padding Tone: A Mechanistic Analysis of Padding Tokens in T2I Models (2025.naacl-long)
Copied to clipboard
| Challenge: | Text-to-image (T2I) diffusion models rely on encoded prompts to guide the image generation process. |
| Approach: | They conduct the first in-depth analysis of the role padding tokens play in T2I diffusion models by using two causal techniques to analyze how information is encoded in the representation of tokens across different components of the pipeline. |
| Outcome: | The proposed techniques reveal that padding tokens may affect the model’s output during text encoding, during the diffusion process, or be effectively ignored. |
DiffusionDB: A Large-scale Prompt Gallery Dataset for Text-to-Image Generative Models (2023.acl-long)
Copied to clipboard
| Challenge: | Recent advances in diffusion models have enabled high-quality image generation . generating images with desired details requires proper prompts . |
| Approach: | They analyze syntactic and semantic characteristics of diffusion models and their prompts . they pinpoint specific hyperparameter values and prompt styles that can lead to model errors . |
| Outcome: | The first large-scale text-to-image prompt dataset totals 6.5TB . it contains 14 million images generated by Stable Diffusion, 1.8 million unique prompts, and hyperparameters specified by real users. |
Collaborative Generative AI: Integrating GPT-k for Efficient Editing in Text-to-Image Generation (2023.emnlp-main)
Copied to clipboard
| Challenge: | Experimental results show that GPT-k models focus more on inserting modifiers than predicting spontaneous changes in the primary subject matter. |
| Approach: | They compare the common edits made by humans and GPT-k models to examine their performance in prompting T2I. |
| Outcome: | The proposed models improve the prompt editing process by 20-30%, the authors show . they show that humans tend to replace words and phrases with modifiers . |
Pioneering Reliable Assessment in Text-to-Image Knowledge Editing: Leveraging a Fine-Grained Dataset and an Innovative Criterion (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Text-to-image models encode factual knowledge into their parameters, but they may become obsolete over time. |
| Approach: | They propose a framework for T2I knowledge editing that integrates paraphrase and multi-object test to enable more fine-grained assessment on knowledge generalization. |
| Outcome: | The proposed framework improves on existing models and improves their performance. |
LADR: Locality-Aware Dynamic Rescue for Efficient Text-to-Image Generation with Diffusion Large Language Models (2026.acl-long)
Copied to clipboard
| Challenge: | Existing methods for inference are expensive and lack spatial redundancy . Discrete Diffusion Language Models are a promising paradigm for multimodal generation . |
| Approach: | They propose a locality-aware dynamic rescue method that exploits spatial Markov property of images. |
| Outcome: | The proposed method achieves an approximate 4 speedup over baselines on four text-to-image generation benchmarks. |
On Advances in Text Generation from Images Beyond Captioning: A Case Study in Self-Rationalization (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Combining visual modality with pretrained language models has been effective for descriptive tasks such as image captioning. |
| Approach: | They ask: do multimodal models combine visual and visual adapted language models? they find that CLIP image representations and scaling of language models do not consistently improve self-rationalization in multimodal tasks. |
| Outcome: | The proposed model types do not consistently improve self-rationalization in multimodal tasks. |