Visual Writing Prompts: Character-Grounded Story Generation with Curated Image Sequences (2023.tacl-1)
Copied to clipboard
| Challenge: | Existing work on image-based story generation lacks coherent plots for story generation. |
| Approach: | They propose to use image sequences to generate stories from a dataset that has more coherent plots. |
| Outcome: | The proposed model produces more coherent, visually grounded and diverse stories than existing models. |
Similar Papers
Generating Visual Stories with Grounded and Coreferent Characters (2026.tacl-1)
Copied to clipboard
| Challenge: | Experimental results show that our model generates visual stories with consistent and coreferent character mentions compared to baselines and state-of-the-art systems. |
| Approach: | They propose a character-centric approach to visual story generation that uses visual and textual character coreference chains to enrich the VIST benchmark. |
| Outcome: | The proposed model generates visual stories with consistent and coreferent character mentions compared to baselines and state-of-the-art systems. |
Visual Coherence Loss for Coherent and Visually Grounded Story Generation (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing visual storytelling models fail to generate correct referring expressions for characters, causing 60% of the generated stories to be lacking local coherence. |
| Approach: | They propose a loss function inspired by a linguistic theory of coherence for self-supervised learning for image sequence representations and a feature matching metric to check whether the models generate referring expressions correctly for characters in input image sequences. |
| Outcome: | The proposed features and loss function are effective for generating more coherent and visually grounded stories. |
Hierarchical Neural Story Generation (P18-1)
Copied to clipboard
| Challenge: | a hierarchical model that generates a premise and then conditions on it creates fluent text . a novel form of model fusion improves the relevance of the story to the prompt . |
| Approach: | They use a hierarchical model that first generates a premise, then transforms it into a text . they use fusion to improve relevance of the story to the prompt and add a gated mechanism to model context . |
| Outcome: | The proposed model improves on strong baselines on automated and human evaluations. |
DiffusionDB: A Large-scale Prompt Gallery Dataset for Text-to-Image Generative Models (2023.acl-long)
Copied to clipboard
| Challenge: | Recent advances in diffusion models have enabled high-quality image generation . generating images with desired details requires proper prompts . |
| Approach: | They analyze syntactic and semantic characteristics of diffusion models and their prompts . they pinpoint specific hyperparameter values and prompt styles that can lead to model errors . |
| Outcome: | The first large-scale text-to-image prompt dataset totals 6.5TB . it contains 14 million images generated by Stable Diffusion, 1.8 million unique prompts, and hyperparameters specified by real users. |
Text-to-Text Automatic Story Generation: A Survey (2026.eacl-srw)
Copied to clipboard
| Challenge: | Automated story generation aims to produce coherent, engaging, and contextually consistent narratives with minimal or no human involvement . despite advances in large language models, maintaining narrative coherence, character consistency, storyline diversity, and plot controllability in generating stories is still challenging. |
| Approach: | They propose to develop new evaluation metrics and better data sets to support automatic story generation. |
| Outcome: | The proposed evaluation metrics and better datasets will improve narrative coherence and consistency and explore practical applications of story generation. |
BeautifulPrompt: Towards Automatic Prompt Engineering for Text-to-Image Synthesis (2023.emnlp-industry)
Copied to clipboard
| Challenge: | Recent text-to-image models require multiple passes of prompt engineering by humans to produce satisfactory results for real-world applications. |
| Approach: | They propose a deep generative model to generate high-quality prompts from raw descriptions using visual feedback. |
| Outcome: | The proposed model produces high-quality prompts from simple raw descriptions . it can be integrated to a cloud-native AI platform to provide better image generation service in the cloud. |
Improved Visual Story Generation with Adaptive Context Modeling (2023.findings-acl)
Copied to clipboard
| Challenge: | a recent work shows that diffusion models generate images of high resolution and semantic consistency to text prompts. |
| Approach: | They propose a method that uses adaptive context modeling to improve leading system . they evaluate their method on pororoSV and FlintstonesSV datasets . |
| Outcome: | The proposed method achieves state-of-the-art FID scores on pororo and Flintstones datasets. |
Prompt Expansion for Adaptive Text-to-Image Generation (2024.acl-long)
Copied to clipboard
| Challenge: | Text-to-image generation models are powerful but difficult to use. Users craft specific prompts to get better images, though the prompts can be repetitive. |
| Approach: | They propose a framework that takes a text query as input and outputs a set of expanded text prompts that are optimized to generate a wider variety of appealing images. |
| Outcome: | The proposed framework generates high-quality images from text prompts with less effort and is more aesthetically pleasing than baseline models. |
Guiding Medical Vision-Language Models with Diverse Visual Prompts: Framework Design and Comprehensive Exploration of Prompt Variations (2025.naacl-long)
Copied to clipboard
| Challenge: | Current vision-language models lack the ability to focus on specific areas designated by humans . a new framework that integrates medical entity extraction, visual prompt generation, and dataset adaptation is proposed to improve visual prompt-guided fine-tuning. |
| Approach: | They propose to use visual prompts to guide and enhance formation of region-specific attention. |
| Outcome: | The proposed framework outperforms state-of-the-art large vision-language models on medical datasets. |
Character-centric Story Visualization via Visual Planning and Token Alignment (2022.emnlp-main)
Copied to clipboard
| Challenge: | Story visualization is a task that requires machines to understand long text inputs and produce a globally consistent image sequence that illustrates the contents of the story. |
| Approach: | They propose to augment VQ-VAE with a text-to-visual-token (transformer) architecture to enable multiple image generation based on a complete story. |
| Outcome: | The proposed method excels at preserving characters and produces higher quality image sequences compared with baselines. |