Poetry in Pixels: Prompt Tuning for Poem Image Generation via Diffusion Models (2025.coling-main)
Copied to clipboard
Sofia Jamil, Bollampalli Areen Reddy, Raghvendra Kumar, Sriparna Saha, Joseph K. J, Koustava Goswami
| Challenge: | Poems are a distinct form of literature, with meanings that transcend beyond the literal words. |
| Approach: | They propose a framework to generate images that visually represent the meanings of poems using prompt tuning and a PoeKey algorithm to extract emotions, visual elements, and themes from poems. |
| Outcome: | The proposed framework generates images that visually represent the meanings of poems and their images. |
Similar Papers
Do It Yourself (DIY): Modifying Images for Poems in a Zero-Shot Setting Using Weighted Prompt Manipulation (2025.emnlp-main)
Copied to clipboard
| Challenge: | a novel method to enhance imagery in poetic language is proposed . weighted prompt manipulation is a new approach to enhance poetry images . current diffusion models struggle to interpret metaphorical language, symbolism, and nuanced themes. |
| Approach: | They propose a weighted prompt manipulation technique that modifies attention weights and text embeddings within diffusion models to enhance or suppress specific words' influence in the final generated image. |
| Outcome: | The proposed technique enhances or suppresses the influence of specific words in the final generated image, leading to semantically richer and more contextually accurate visualizations. |
Prompt Expansion for Adaptive Text-to-Image Generation (2024.acl-long)
Copied to clipboard
| Challenge: | Text-to-image generation models are powerful but difficult to use. Users craft specific prompts to get better images, though the prompts can be repetitive. |
| Approach: | They propose a framework that takes a text query as input and outputs a set of expanded text prompts that are optimized to generate a wider variety of appealing images. |
| Outcome: | The proposed framework generates high-quality images from text prompts with less effort and is more aesthetically pleasing than baseline models. |
Automatic Poetry Generation from Prosaic Text (2020.acl-main)
Copied to clipboard
| Challenge: | In recent years, successful approaches have emerged to accurately model various aspects of natural language. |
| Approach: | They propose to combine neural networks with a poetry generation system that only uses standard text as input . they use standard text to model syntactic well-formedness and topical coherence . |
| Outcome: | The proposed framework is applied to the generation of poems in English and French . it uses standard, non-poetic text and its output is constrained to confer a poetic character . |
Yu Sheng: Human-in-Loop Classical Chinese Poetry Generation System (2023.eacl-demo)
Copied to clipboard
| Challenge: | Existing systems for poetry generation are not flexible in polishing and customization. |
| Approach: | They propose a web-based poetry generation system that provides customization options for users with different backgrounds to engage in the process of poetry composition. |
| Outcome: | The proposed system can generate and polish classical Chinese poetry compared to other vanilla models. |
Constrained Language Models for Interactive Poem Generation (2022.lrec-1)
Copied to clipboard
Andrei Popescu-Belis, Àlex Atrio, Valentin Minder, Aris Xanthos, Gabriel Luthier, Simon Mattei, Antonio Rodriguez
| Challenge: | Neural language models cannot learn constraints from data, which is scarce for a well-resourced language such as French. |
| Approach: | They propose a system that combines neural language models with constraints that can be set by users on form, topic, emotion, and rhyming scheme. |
| Outcome: | The proposed system generates poems and stanzas using LMs and rule-based algorithms . it has been demonstrated at public events and log analysis shows that users found it engaging . |
DiffusionDB: A Large-scale Prompt Gallery Dataset for Text-to-Image Generative Models (2023.acl-long)
Copied to clipboard
| Challenge: | Recent advances in diffusion models have enabled high-quality image generation . generating images with desired details requires proper prompts . |
| Approach: | They analyze syntactic and semantic characteristics of diffusion models and their prompts . they pinpoint specific hyperparameter values and prompt styles that can lead to model errors . |
| Outcome: | The first large-scale text-to-image prompt dataset totals 6.5TB . it contains 14 million images generated by Stable Diffusion, 1.8 million unique prompts, and hyperparameters specified by real users. |
BeautifulPrompt: Towards Automatic Prompt Engineering for Text-to-Image Synthesis (2023.emnlp-industry)
Copied to clipboard
| Challenge: | Recent text-to-image models require multiple passes of prompt engineering by humans to produce satisfactory results for real-world applications. |
| Approach: | They propose a deep generative model to generate high-quality prompts from raw descriptions using visual feedback. |
| Outcome: | The proposed model produces high-quality prompts from simple raw descriptions . it can be integrated to a cloud-native AI platform to provide better image generation service in the cloud. |
Acrostic Poem Generation (2020.emnlp-main)
Copied to clipboard
| Challenge: | Acrostic poems contain a hidden message; typically, the first letter of each line spells out a word or short phrase. |
| Approach: | They propose a task for acrostic poem generation in English with multiple constraints . they define the task as a generation task with multiple constraint constraints based on a conditional neural language model and a neural rhyming model . |
| Outcome: | The proposed task is based on a baseline model and a neural rhyming model. |
ByGPT5: End-to-End Style-conditioned Poetry Generation with Token-free Language Models (2023.acl-long)
Copied to clipboard
| Challenge: | End-to-end models learn to complete a task by directly learning all steps, without intermediary algorithms such as hand-crafted rules or post-processing. |
| Approach: | They propose to train end-to-end poetry generation conditioned on styles such as rhyme, meter, and alliteration . they pre-train ByGPT5, a new token-free decoder-only language model, and fine-tune it on a custom corpus of English and German quatrains . |
| Outcome: | The proposed model outperforms other models on a large custom corpus of English and German quatrains while being more parameter efficient and performing favorably compared to humans. |
Augmenting Poetry Composition with Verse by Verse (2022.naacl-industry)
Copied to clipboard
| Challenge: | a new approach to poetry generation has been developed that allows an AI to generate a full poem by itself, thus writing in a closed system. |
| Approach: | They describe an AI poet that offers suggestions while a user is composing a poem . they use a generative model and a dual encoder model to offer the suggestions . |
| Outcome: | The proposed system can offer suggestions generated lines of verse while a user is composing a poem. |