| Challenge: | Multimodal reasoning with large language models (LLMs) often suffers from hallucinations and the presence of deficient or outdated knowledge within LLMs. |
| Approach: | They propose a multimodal reasoning method that leverages multimodal knowledge graphs to learn rich and semantic knowledge across modalities. |
| Outcome: | The proposed method outperforms state-of-the-art models on multimodal question answering and multimodal analogy reasoning tasks while training on only a small fraction of parameters. |
Similar Papers
Synergizing Multimodal Temporal Knowledge Graphs and Large Language Models for Social Relation Recognition (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in Large Language Models (LLMs) have limited capacity to model complex graph-structured relationships. |
| Approach: | They propose a low-coupling method synergizing multimodal temporal Knowledge Graphs and Large Language Models for social relation reasoning. |
| Outcome: | The proposed method exhibits state-of-the-art performance in social relation recognition . it bridges the gap between KGs and LLMs and will be released after acceptance . |
MM-LLMs: Recent Advances in MultiModal Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | MultiModal Large Language Models (MM-LLMs) have undergone significant advances in the past year . traditional MM models incur substantial computational costs, especially when trained from scratch . |
| Approach: | They propose a taxonomy encompassing 126 MM-LLMs and summarize key training recipes to enhance their potency. |
| Outcome: | The proposed models preserve the reasoning and decision-making capabilities of LLMs and empower diverse range of MM tasks. |
MM-GATBT: Enriching Multimodal Representation Using Graph Attention Network (2022.naacl-srw)
Copied to clipboard
| Challenge: | Existing models that use a self-attention mechanism to create graphs with multiple modes ignore interaction between entities, multimodalities, or both. |
| Approach: | They propose a multimodal graph representation learning model that captures relational semantics within one modality and interactions between different modalities. |
| Outcome: | The proposed model outperforms existing models on the MM-IMDb dataset in all aspects of multimodal representation. |
Exploring and Evaluating Multimodal Knowledge Reasoning Consistency of Multimodal Large Language Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | MLLMs have achieved significant breakthroughs in understanding across text and vision, but current models still face inconsistencies in reasoning outcomes. |
| Approach: | They propose to evaluate multimodal large language models using a multimodal knowledge reasoning dataset to examine the extent of consistency degradation. |
| Outcome: | The proposed evaluation tasks show that MLLMs are inefficient at integrating knowledge across modalities . |
LLM-Guided Semantic Relational Reasoning for Multimodal Intent Recognition (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for understanding intents from multimodal signals exhibit limitations in their modality-level reliance, constraining relational reasoning over fine-grained semantics for complex intent understanding. |
| Approach: | They propose a method that harnesses the expansive knowledge of large language models to establish semantic foundations that boost smaller models’ relational reasoning performance. |
| Outcome: | The proposed method outperforms state-of-the-art methods on multimodal intent and dialogue act recognition tasks and shows consistent performance gains across diverse semantic understanding scenarios. |
Model Composition for Multimodal Large Language Models (2024.acl-long)
Copied to clipboard
Chi Chen, Yiyang Du, Zheng Fang, Ziyue Wang, Fuwen Luo, Peng Li, Ming Yan, Ji Zhang, Fei Huang, Maosong Sun, Yang Liu
| Challenge: | Existing methods for creating versatile MLLMs rely on joint training with paired instruction data, which is resource-intensive and challenging to extend to new modalities. |
| Approach: | They propose a new paradigm for multimodal large language models by reusing modality encoders and merging LLM parameters. |
| Outcome: | The proposed model retains the modal understanding capabilities of each original model. |
Differentiated Vision: Unveiling Entity-Specific Visual Modality Requirements for Multimodal Knowledge Graph (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods to extract features from images of entities overlook varying relevance of visual information across entities. |
| Approach: | a new model integrates structural and multimodal information of entities into a multimodal knowledge graph . a model evaluates the necessity of visual modality for each entity based on its attributes . |
| Outcome: | The proposed model improves on existing methods by adjusting visual data to different entity types. |
Retrieving Multimodal Information for Augmented Generation: A Survey (2023.findings-emnlp)
Copied to clipboard
Ruochen Zhao, Hailin Chen, Weishi Wang, Fangkai Jiao, Xuan Long Do, Chengwei Qin, Bosheng Ding, Xiaobao Guo, Minzhi Li, Xingxuan Li, Shafiq Joty
| Challenge: | Large Language Models (LLMs) are increasingly using multimodality to augment their generation ability, but there is no unified perception of at which stage and how to incorporate different modalities. |
| Approach: | They propose to use multimodality to augment Large Language Models (LLMs) this will provide scholars with a deeper understanding of the methods' applications and encourage them to adapt existing techniques to the fast-growing field of LLMs. |
| Outcome: | The proposed methods improve factuality, reasoning, interpretability, and robustness of the generated content. |
A Survey of Mathematical Reasoning in the Era of Multimodal Large Language Model: Benchmark, Method & Challenges (2025.findings-acl)
Copied to clipboard
Yibo Yan, Jiamin Su, Jianxiang He, Fangteng Fu, Xu Zheng, Yuanhuiyi Lyu, Kun Wang, Shen Wang, Qingsong Wen, Xuming Hu
| Challenge: | This survey provides **the first comprehensive analysis of mathematical reasoning in the era of multimodal large language models** . integrating large language model with mathematical reasoning tasks is becoming significant as AI advances . |
| Approach: | They review over 200 studies published since 2021 and examine the state-of-the-art developments in Math-LLMs . they identify five major challenges hindering the realization of AGI in this domain . |
| Outcome: | The authors examine the state-of-the-art developments in Math-LLMs with a focus on multimodal settings. |
MM-ChatAlign: A Novel Multimodal Reasoning Framework based on Large Language Models for Entity Alignment (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing MMEA methods rely on knowledge representation learning (KRL) to measure the similarity of entity embeddings. |
| Approach: | They propose a framework that utilizes the visual reasoning abilities of MLLMs for multimodal entity alignment. |
| Outcome: | The proposed framework integrates the visual reasoning abilities of MLLMs for multimodal entity alignment. |