MMNeuron: Discovering Neuron-Level Domain-Specific Interpretation in Multimodal Large Language Model (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing MLLMs have a visual question answering capability but lack domain-specific information. |
| Approach: | They propose a framework for language model modules in MLLMs when handling projected image features and verify this hypothesis using logit lens. |
| Outcome: | The proposed framework will yield a 10% change in accuracy at most, shedding light on the development of cross-domain, all-encompassing MLLMs in the future. |
Similar Papers
Probing Multimodal Large Language Models for Global and Local Semantic Representations (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing studies have focused on the ability of MLLMs to generate single tokens one by one, while lacking studies about how their representation vectors can encode global multimodal information. |
| Approach: | They propose to use image-caption corpus to train Multimodal Large Language Models (MLLMs) . they find that the topmost layers encode more global semantic information . |
| Outcome: | The proposed models can encode more global semantic information, rather than the topmost layers, and perform better on visual-language entailment tasks. |
Attribution and Application of Multiple Neurons in Multimodal Large Language Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods to identify multimodal neurons in MLLMs are insufficiently understood . previous studies focused on identifying neurons corresponding to single-tokens . |
| Approach: | They propose a method to identify multimodal neurons in Transformer-based MLLMs . they introduce fuzzy set theory to model the complex relationship between neurons and semantic concepts . |
| Outcome: | The proposed method improves performance on the Visual Question Answering task. |
Towards Unified Multimodal Large Language Models: A survey (2026.findings-acl)
Copied to clipboard
| Challenge: | unified multimodal large language models (MLLMs) are emerging but lack a systematic framework to connect them and situate current trends within a broader landscape. |
| Approach: | They present a systematic review of unified Multimodal Large Language Models . they outline the foundational concepts and prerequisites for understanding them . |
| Outcome: | The present review provides a systematic and systematic overview of unified MLLMs . it discusses persistent challenges and identify promising directions for future research . |
Multimodal Large Language Models for Text-rich Image Understanding: A Comprehensive Review (2025.findings-acl)
Copied to clipboard
Pei Fu, Tongkun Guan, Zining Wang, Zhentao Guo, Chen Duan, Hao Sun, Boming Chen, Qianyi Jiang, Jiayao Ma, Kai Zhou, Junfeng Luo
| Challenge: | Recent advances in vision-language models have unified perception and understanding tasks within Visual Question Answering paradigms. |
| Approach: | They propose to outline timeline, architecture, and pipeline of nearly all TIU MLLMs and review their performance on mainstream benchmarks. |
| Outcome: | The proposed models perform well on mainstream benchmarks and are compared with other models. |
Unveiling Multimodal Processing: Exploring Activation Patterns in Multimodal LLMs for Interpretability and Efficiency (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Recent advances in multimodal large language models have remained opaque. |
| Approach: | They propose a method to convert dense MLLMs into fine-grained Mixture-of-Experts architectures. |
| Outcome: | The proposed method outperforms random expert pruning and sparse activation and model pruning. |
MMEvol: Empowering Multimodal Large Language Models with Evol-Instruct (2025.findings-acl)
Copied to clipboard
Run Luo, Haonan Zhang, Longze Chen, Ting-En Lin, Xiong Liu, Yuchuan Wu, Min Yang, Yongbin Li, Minzheng Wang, Pengpeng Zeng, Lianli Gao, Heng Tao Shen, Yunshui Li, Hamid Alinejad-Rokny, Xiaobo Xia, Jingkuan Song, Fei Huang
| Challenge: | a new framework for image-text instruction data evolution improves MLLM performance . lack of high-quality instruction data remains a major bottleneck in ML modeling . |
| Approach: | They propose a multimodal instruction data evolution framework that iteratively enhances data quality through fine-grained perception, cognitive reasoning, and interaction evolution. |
| Outcome: | The proposed approach improves MLLM performance in nine vision-language tasks while using significantly less data. |
From Multimodal LLM to Human-level AI: Modality, Instruction, Reasoning, Efficiency and beyond (2024.lrec-tutorials)
Copied to clipboard
| Challenge: | This tutorial aims to deliver a comprehensive review of cutting-edge research in MLLMs. |
| Approach: | This tutorial will review cutting-edge research in MLLMs and examine the impact of ML in learning and reasoning. |
| Outcome: | This course will review cutting-edge research in MLLMs and examine the impact of ML models on learning, learning, and multimodal reasoning. |
Language-Specific Neurons: The Key to Multilingual Capabilities in Large Language Models (2024.acl-long)
Copied to clipboard
Tianyi Tang, Wenyang Luo, Haoyang Huang, Dongdong Zhang, Xiaolei Wang, Xin Zhao, Furu Wei, Ji-Rong Wen
| Challenge: | Despite the impressive multilingual capabilities demonstrated by LLMs, the understanding of how these abilities develop and function remains nascent. |
| Approach: | They propose a novel detection method to pinpoint language-specific neurons within LLMs by selectively activating or deactivating these neurons. |
| Outcome: | The proposed method can “steer” the output language of LLMs by selectively activating or deactivating language-specific neurons. |
MM-LLMs: Recent Advances in MultiModal Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | MultiModal Large Language Models (MM-LLMs) have undergone significant advances in the past year . traditional MM models incur substantial computational costs, especially when trained from scratch . |
| Approach: | They propose a taxonomy encompassing 126 MM-LLMs and summarize key training recipes to enhance their potency. |
| Outcome: | The proposed models preserve the reasoning and decision-making capabilities of LLMs and empower diverse range of MM tasks. |
Cross-Modal Projection in Multimodal LLMs Doesn’t Really Project Visual Attributes to Textual Space (2024.acl-short)
Copied to clipboard
| Challenge: | Existing multimodal large language models are limited to general-purpose multimodal tasks like question-answering on natural images. |
| Approach: | They propose to use cross-modal projection networks and a large language model to model domain-specific visual attributes of MLLMs. |
| Outcome: | The proposed models gain domain-specific visual capabilities when the projection is fine-tuned, but the updates do not extract relevant domain-specific visual attributes. |