Uncovering Limitations in Text-to-Image Generation: A Contrastive Approach with Structured Semantic Alignment (2023.findings-emnlp)
Copied to clipboard
| Challenge: | a new method for text-to-image generation models is proposed to address these limitations . SSA focuses on learning structured semantic embeddings across different modalities . |
| Approach: | They propose a method to evaluate text-to-image generation models using structured semantic embeddings . they propose to learn mutated prompts by substituting words with equivalent or nonequivalent alternatives . |
| Outcome: | The proposed method improves the measurement of semantic consistency of text-to-image generation models. |
Similar Papers
Misalignment Attack on Text-to-Image Models via Text Embedding Optimization and Inversion (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Text embedding is a key component of modern NLP models but also poses additional risks. |
| Approach: | They propose a framework that optimizes embeddings and inverts them to obtain misaligned prompts. |
| Outcome: | The proposed framework exploits the continuity and distribution characteristics of text embeddings to obtain misaligned prompts of discrete tokens. |
Precision or Recall? An Analysis of Image Captions for Training Text-to-Image Generation Model (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Recent advances in text-to-image models have demonstrated remarkable capabilities in image synthesis. |
| Approach: | They analyze the critical role of caption precision and recall in text-to-image model training. |
| Outcome: | The proposed model trains with synthetic captions that show similar behavior to those trained on human-annotated captions. |
STRICT: Stress-Test of Rendering Image Containing Text (2025.emnlp-main)
Copied to clipboard
| Challenge: | Despite the advances in diffusion models, the generation of coherent text remains a major bottleneck. |
| Approach: | They propose a benchmark to test the ability of diffusion models to render coherent text in images. |
| Outcome: | The proposed model fails to generate coherent and legible text in images despite its iterative nature . the model fails in both the maximum length of readable text and correctness and legibility of the generated text . |
SDA: Semantic Discrepancy Alignment for Text-conditioned Image Retrieval (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for textconditioned image retrieval are limited due to the scale of training and the complexity of attributes in modification texts. |
| Approach: | They propose a general boosting framework to address these issues by employing semantic discrepancy alignment by formulating distance consistency and neighbor consistency between the image and text domains. |
| Outcome: | The proposed framework improves retrieval performance on three prominent datasets with state-of-the-art results. |
Enhance Multimodal Consistency and Coherence for Text-Image Plan Generation (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing studies on textual plan generation only focus on LLMs, enabling applications in robotics, virtual assistants, and instruc. |
| Approach: | They propose a framework that generates and refines text-image plans step-by-step . they collect a new benchmark consisting of 1,100 tasks and their text- image pair solutions covering 11 daily topics. |
| Outcome: | The proposed framework generates and refines text-image plans step-by-step and improves on existing models. |
Follow the Flow: On Information Flow Across Textual Tokens in Text-to-Image Models (2026.acl-long)
Copied to clipboard
| Challenge: | Prior work focused on improving alignment by refining the diffusion process, ignoring the role of the text encoder, which guides the diffusion. |
| Approach: | They investigate how semantic information is distributed across token representations in text-to-image prompts by patching techniques to uncover encoding patterns. |
| Outcome: | The proposed model can improve alignment and generation quality by modifying the diffusion stage and the cross-attention mechanism. |
Attribute Alignment: Controlling Text Generation from Pre-trained Language Models (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models can generate text with sentiment polarity or specific topics without changing the original model parameters. |
| Approach: | They propose a method for controlling text generation by aligning disentangled attribute representations. |
| Outcome: | The proposed method shows large performance gains while maintaining diversity and fluency. |
Learning to Model Multimodal Semantic Alignment for Story Visualization (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Story visualization aims to generate sequence of images to narrate each sentence in a multi-sentence story . current methods face semantic misalignment because of their fixed architecture and diversity of input modalities . |
| Approach: | They propose to use a GAN-based generative model to match semantic levels between text and image representations to solve the semantic misalignment problem. |
| Outcome: | Experiments show that the proposed approach improves image quality and story consistency compared with state-of-the-art methods. |
Sentence-Level Content Planning and Style Specification for Neural Text Generation (D19-1)
Copied to clipboard
| Challenge: | Recent advances in text generation systems often produce incoherent and unfaithful outputs . a novel automated text generation system takes into account content selection, text planning, and surface realization. |
| Approach: | They propose an end-to-end trained two-step text generation model that considers sentence-level content planners and language styles. |
| Outcome: | The proposed model outperforms competing models in three domains with diverse topics and varying language styles. |
Automatic and Human-AI Interactive Text Generation (with a focus on Text Simplification and Revision) (2024.acl-tutorials)
Copied to clipboard
| Challenge: | In this tutorial, we focus on text-to-text generation, a class of natural language generation tasks, that takes a piece of text as input and then generates a revision that is improved according to some specific criteria. |
| Approach: | This tutorial focuses on text-to-text generation, a class of natural language generation tasks that takes a piece of text as input and generates a revision that is improved according to some specific criteria. |
| Outcome: | This tutorial focuses on text-to-text generation, a class of natural language generation tasks, that takes a piece of text as input and generates a revision that is improved according to some specificcriteria. |