| Challenge: | Kahaani is a multimodal, co-creative storytelling system that leverages Generative Artificial Intelligence to address the challenge of sustaining engagement to foster educational narrative experiences. |
| Approach: | They propose a multimodal, co-creative storytelling system that leverages Generative Artificial Intelligence to help children develop their storytelling skills. |
| Outcome: | The proposed system combines large language models, text-to-speech, and music generation to produce a rich, immersive, and accessible storytelling experience. |
Similar Papers
StoryLLaVA: Enhancing Visual Storytelling with Multi-Modal Large Language Models (2025.coling-main)
Copied to clipboard
| Challenge: | Existing models struggle to maintain temporal, spatial, and narrative coherence across image sequences . existing models lack depth and engagement of human-authored stories . |
| Approach: | They propose a topic-driven narrative optimizer that integrates image descriptions, topic generation, and GPT-4-based refinements. |
| Outcome: | The proposed framework outperforms existing models in visual relevance, coherence, and fluency. |
STORYTELLER: An Enhanced Plot-Planning Framework for Coherent and Cohesive Story Generation (2025.findings-acl)
Copied to clipboard
Jiaming Li, Yukun Chen, Ziqiang Liu, Minghuan Tan, Lei Zhang, Yunshui Li, Run Luo, Longze Chen, Jing Luo, Ahmadreza Argha, Hamid Alinejad-Rokny, Wei Zhou, Min Yang
| Challenge: | Existing methods for storytelling lack coherence and consistency, compromising the overall storytelling experience. |
| Approach: | They propose a novel approach that improves the coherence and consistency of automatically generated stories by managing plot nodes and enabling dynamic interactions between different parts of the story. |
| Outcome: | The proposed approach outperforms existing methods in 84.33% of the trials. |
MIO: A Foundation Model on Multimodal Tokens (2025.emnlp-main)
Copied to clipboard
Zekun Moore Wang, King Zhu, Chunpu Xu, Wangchunshu Zhou, Jiaheng Liu, Yibo Zhang, Jessie Wang, Ning Shi, Siyu Li, Yizhi Li, Haoran Que, Zhaoxiang Zhang, Yuanxing Zhang, Ge Zhang, Ke Xu, Jie Fu, Wenhao Huang
| Challenge: | Existing models lack multimodal understanding capabilities, resulting in closed-source model that does not support multimodal interleaved sequences. |
| Approach: | They propose a foundation model built on multimodal tokens capable of understanding and generating speech, text, images, and videos in an end-to-end, autoregressive manner. |
| Outcome: | The proposed model is able to understand speech, text, images, and videos in an end-to-end, autoregressive manner. |
OpenOmni: A Collaborative Open Source Tool for Building Future-Ready Multimodal Conversational Agents (2024.emnlp-demo)
Copied to clipboard
| Challenge: | OpenOmni is an open-source, end-to-end pipeline benchmarking tool for multimodal conversational agents. |
| Approach: | They developed an open-source, end-to-end pipeline benchmarking tool to help solve these issues. |
| Outcome: | OpenOmni integrates speech-to-text, emotion detection, and large language models with the ability to integrate customized models. |
A Character-Centric Creative Story Generation via Imagination (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing narrative generation models lack diversity and character depth, but they are inadequate for human creativity. |
| Approach: | They propose a novel story generation framework called CCI that leverages images to create stories that are diverse and creative in their themes and richer in content. |
| Outcome: | The proposed framework significantly improves various aspects of the stories’ creativity. |
TIGER: A Unified Generative Model Framework for Multimodal Dialogue Response Generation (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing research on multimodal dialogues focuses on textual response generation and visual response selection based on the dialogue context. |
| Approach: | They propose a generative model framework for multimodal dialogue response generation that ground the conversation on an image. |
| Outcome: | The proposed system provides users with an enhanced conversational experience. |
Text-to-Text Automatic Story Generation: A Survey (2026.eacl-srw)
Copied to clipboard
| Challenge: | Automated story generation aims to produce coherent, engaging, and contextually consistent narratives with minimal or no human involvement . despite advances in large language models, maintaining narrative coherence, character consistency, storyline diversity, and plot controllability in generating stories is still challenging. |
| Approach: | They propose to develop new evaluation metrics and better data sets to support automatic story generation. |
| Outcome: | The proposed evaluation metrics and better datasets will improve narrative coherence and consistency and explore practical applications of story generation. |
From Multimodal LLM to Human-level AI: Modality, Instruction, Reasoning, Efficiency and beyond (2024.lrec-tutorials)
Copied to clipboard
| Challenge: | This tutorial aims to deliver a comprehensive review of cutting-edge research in MLLMs. |
| Approach: | This tutorial will review cutting-edge research in MLLMs and examine the impact of ML in learning and reasoning. |
| Outcome: | This course will review cutting-edge research in MLLMs and examine the impact of ML models on learning, learning, and multimodal reasoning. |
From Long Videos to Engaging Clips: A Human-Inspired Video Editing Framework with Multimodal Narrative Understanding (2025.emnlp-industry)
Copied to clipboard
Xiangfeng Wang, Xiao Li, Yadong Wei, null Songxueyu, Yang Song, null Xiaxiaoqiang, Fangrui Zeng, Zaiyi Chen, null Liuliu, Gu Xu, Tong Xu
| Challenge: | Existing methods for video editing rely on textual cues from ASR transcripts and segment selection, often neglecting rich visual context. |
| Approach: | They propose a human-inspired automatic video editing framework that leverages multimodal narrative understanding to address these limitations. |
| Outcome: | The proposed framework outperforms existing baselines across general and advertisement-oriented editing tasks. |
Retrieving Multimodal Information for Augmented Generation: A Survey (2023.findings-emnlp)
Copied to clipboard
Ruochen Zhao, Hailin Chen, Weishi Wang, Fangkai Jiao, Xuan Long Do, Chengwei Qin, Bosheng Ding, Xiaobao Guo, Minzhi Li, Xingxuan Li, Shafiq Joty
| Challenge: | Large Language Models (LLMs) are increasingly using multimodality to augment their generation ability, but there is no unified perception of at which stage and how to incorporate different modalities. |
| Approach: | They propose to use multimodality to augment Large Language Models (LLMs) this will provide scholars with a deeper understanding of the methods' applications and encourage them to adapt existing techniques to the fast-growing field of LLMs. |
| Outcome: | The proposed methods improve factuality, reasoning, interpretability, and robustness of the generated content. |