Papers by Zhiding Xiao
Re-ViLM: Retrieval-Augmented Visual Language Model for Zero and Few-Shot Image Captioning (2023.findings-emnlp)
Copied to clipboard
Zhuolin Yang, Wei Ping, Zihan Liu, Vijay Korthikanti, Weili Nie, De-An Huang, Linxi Fan, Zhiding Yu, Shiyi Lan, Bo Li, Mohammad Shoeybi, Ming-Yu Liu, Yuke Zhu, Bryan Catanzaro, Chaowei Xiao, Anima Anandkumar
| Challenge: | Existing methods for image-to-text generation store all knowledge within parameters, thus requiring computational-expensive fine-tuning. |
| Approach: | They propose a Retrieval-augmented Visual Language Model that stores all the knowledge within parameters and can be used to retrieve it from the external database. |
| Outcome: | The proposed model significantly boosts performance for image-to-text generation tasks with 4x less parameters compared with baseline methods. |
StoryLLaVA: Enhancing Visual Storytelling with Multi-Modal Large Language Models (2025.coling-main)
Copied to clipboard
| Challenge: | Existing models struggle to maintain temporal, spatial, and narrative coherence across image sequences . existing models lack depth and engagement of human-authored stories . |
| Approach: | They propose a topic-driven narrative optimizer that integrates image descriptions, topic generation, and GPT-4-based refinements. |
| Outcome: | The proposed framework outperforms existing models in visual relevance, coherence, and fluency. |