Empowering Backbone Models for Visual Text Generation with Input Granularity Control and Glyph-Aware Training (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing text-to-image models struggle to generate images with legible visual texts . current models lack support for Chinese texts, misspelling, and lack of diversity . |
| Approach: | They propose to empower backbone models to generate visual texts in Chinese and English . they propose to augment conventional training objective with glyph-aware training losses . |
| Outcome: | The proposed methods can generate visual texts in English and Chinese while maintaining image generation quality. |
Similar Papers
Character-Aware Models Improve Visual Text Rendering (2023.acl-long)
Copied to clipboard
Rosanne Liu, Dan Garrette, Chitwan Saharia, William Chan, Adam Roberts, Sharan Narang, Irina Blok, Rj Mical, Mohammad Norouzi, Noah Constant
| Challenge: | Current image generation models struggle to produce well-formed visual text due to lack of character-level input features. |
| Approach: | They conduct a series of experiments to compare character-aware vs. character-blind text encoders to determine their spelling ability. |
| Outcome: | The character-aware models outperform character-blind models on a range of novel text rendering tasks. |
Controlled Text Generation for Black-box Language Models via Score-based Progressive Editor (2024.acl-long)
Copied to clipboard
| Challenge: | Existing methods to control text generation are inapplicable to black-box models or suffer a trade-off between control and fluency. |
| Approach: | They propose a new approach to control text generation that modifies context at the token level during the generation process of a backbone language model and guides subsequent text to naturally include the target attributes. |
| Outcome: | The proposed method can regulate the attributes of the generated text while utilizing the capability of the backbone large language models. |
Enhance Multimodal Consistency and Coherence for Text-Image Plan Generation (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing studies on textual plan generation only focus on LLMs, enabling applications in robotics, virtual assistants, and instruc. |
| Approach: | They propose a framework that generates and refines text-image plans step-by-step . they collect a new benchmark consisting of 1,100 tasks and their text- image pair solutions covering 11 daily topics. |
| Outcome: | The proposed framework generates and refines text-image plans step-by-step and improves on existing models. |
Uncovering Limitations in Text-to-Image Generation: A Contrastive Approach with Structured Semantic Alignment (2023.findings-emnlp)
Copied to clipboard
| Challenge: | a new method for text-to-image generation models is proposed to address these limitations . SSA focuses on learning structured semantic embeddings across different modalities . |
| Approach: | They propose a method to evaluate text-to-image generation models using structured semantic embeddings . they propose to learn mutated prompts by substituting words with equivalent or nonequivalent alternatives . |
| Outcome: | The proposed method improves the measurement of semantic consistency of text-to-image generation models. |
Learning the Visualness of Text Using Large Vision-Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Visual text evokes an image in a person’s mind, while non-visual text fails to do so. |
| Approach: | They propose a method to automatically detect visualness in text to enable text-to-image retrieval and generation models to augment text with relevant images. |
| Outcome: | The proposed method performs better than several baseline models and heuristics for the task. |
Text-to-Code Generation with Modality-relative Pre-training (2024.eacl-long)
Copied to clipboard
| Challenge: | Large pre-trained language models have been applied to programming language tasks with great success, often through further pre-training of a strictly-natural language model. |
| Approach: | They propose to map programming language modalities into the same embedding space by separating embeddable spaces between modality and modality-relative training objectives. |
| Outcome: | The proposed model can be adapted and represented differently depending on which modality they belong to and to the ultimate benefit of the downstream task. |
On Advances in Text Generation from Images Beyond Captioning: A Case Study in Self-Rationalization (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Combining visual modality with pretrained language models has been effective for descriptive tasks such as image captioning. |
| Approach: | They ask: do multimodal models combine visual and visual adapted language models? they find that CLIP image representations and scaling of language models do not consistently improve self-rationalization in multimodal tasks. |
| Outcome: | The proposed model types do not consistently improve self-rationalization in multimodal tasks. |
Automatic Evaluation for Text-to-image Generation: Task-decomposed Framework, Distilled Training, and Meta-evaluation Benchmark (2025.acl-long)
Copied to clipboard
| Challenge: | Existing MLLMs rely on commercial models such as GPT-4o for evaluations, but they are not universally accessible. |
| Approach: | They propose a task decomposition evaluation framework based on GPT-4o to automatically construct a specialized training dataset to break down the multifaceted evaluation process into simpler sub-tasks. |
| Outcome: | The proposed framework outperforms the current state-of-the-art GPT-4o evaluation framework with over 4.6% improvement in Spearman and Kendall correlations with human judgments. |
Lattice-BERT: Leveraging Multi-Granularity Representations in Chinese Pre-trained Language Models (2021.naacl-main)
Copied to clipboard
| Challenge: | Pre-trained language models process text as a sequence of characters, ignoring more coarse granularity, e.g., words. |
| Approach: | They propose a new pre-training paradigm for Chinese that incorporates word representations along with characters and can model a sentence in a multi-granular manner. |
| Outcome: | The proposed model can bring an average increase of 1.5% under the 12-layer setting, which achieves new state-of-the-art among base-size models on the CLUE benchmarks. |
PAI-Diffusion: Constructing and Serving a Family of Open Chinese Diffusion Models for Text-to-image Synthesis on the Cloud (2024.acl-demos)
Copied to clipboard
| Challenge: | Existing diffusion models fail to address the challenges of generating high-quality images from textual descriptions due to its large vocabulary size and complex character relationships. |
| Approach: | They propose a framework that integrates Chinese diffusion models with Alibaba Cloud's Platform for AI and enables the generation of contextually relevant images. |
| Outcome: | The proposed framework integrates with Alibaba Cloud’s Platform for AI, providing accessible and scalable solutions. |