Papers by Yuanyu He
Multimodal Self-Instruct: Synthetic Abstract Image and Visual Reasoning Instruction Using Language Model (2024.emnlp-main)
Copied to clipboard
Wenqi Zhang, Zhenglin Cheng, Yuanyu He, Mengna Wang, Yongliang Shen, Zeqi Tan, Guiyang Hou, Mingqian He, Yanna Ma, Weiming Lu, Yueting Zhuang
| Challenge: | Using large language models, large multimodal models struggle with basic tasks like reading time from a clock and planning a route using a road map. |
| Approach: | They propose a multimodal self-instruct that synthesizes massive abstract images and visual reasoning instructions. |
| Outcome: | The proposed model synthesizes 11,193 abstract images and reasoning instructions across eight visual scenarios. |