Probing Multimodal Large Language Models for Global and Local Semantic Representations (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing studies have focused on the ability of MLLMs to generate single tokens one by one, while lacking studies about how their representation vectors can encode global multimodal information. |
| Approach: | They propose to use image-caption corpus to train Multimodal Large Language Models (MLLMs) . they find that the topmost layers encode more global semantic information . |
| Outcome: | The proposed models can encode more global semantic information, rather than the topmost layers, and perform better on visual-language entailment tasks. |
Similar Papers
Towards Unified Multimodal Large Language Models: A survey (2026.findings-acl)
Copied to clipboard
| Challenge: | unified multimodal large language models (MLLMs) are emerging but lack a systematic framework to connect them and situate current trends within a broader landscape. |
| Approach: | They present a systematic review of unified Multimodal Large Language Models . they outline the foundational concepts and prerequisites for understanding them . |
| Outcome: | The present review provides a systematic and systematic overview of unified MLLMs . it discusses persistent challenges and identify promising directions for future research . |
Multimodal Large Language Models for Text-rich Image Understanding: A Comprehensive Review (2025.findings-acl)
Copied to clipboard
Pei Fu, Tongkun Guan, Zining Wang, Zhentao Guo, Chen Duan, Hao Sun, Boming Chen, Qianyi Jiang, Jiayao Ma, Kai Zhou, Junfeng Luo
| Challenge: | Recent advances in vision-language models have unified perception and understanding tasks within Visual Question Answering paradigms. |
| Approach: | They propose to outline timeline, architecture, and pipeline of nearly all TIU MLLMs and review their performance on mainstream benchmarks. |
| Outcome: | The proposed models perform well on mainstream benchmarks and are compared with other models. |
The Revolution of Multimodal Large Language Models: A Survey (2024.findings-acl)
Copied to clipboard
Davide Caffagni, Federico Cocchi, Luca Barsellotti, Nicholas Moratelli, Sara Sarto, Lorenzo Baraldi, Lorenzo Baraldi, Marcella Cornia, Rita Cucchiara
| Challenge: | Recent advances in large language models have led to the development of multimodal large language model. |
| Approach: | They present a review of recent visual-based Large Language Models and analyze their architectures and alignment strategies. |
| Outcome: | The proposed models can integrate visual and textual modalities while providing a dialogue-based interface and instruction-following capabilities. |
Generative Giants, Retrieval Weaklings: Why do Multimodal Large Language Models Fail at Multimodal Retrieval? (2026.findings-acl)
Copied to clipboard
| Challenge: | Rapid advances in multimodal large language models have revolutionized cross-modality understanding. |
| Approach: | They propose a method that uses whitening transformations to adjust MLLM representation spaces . they propose ML models that are dominated by textual semantics and visual semantics . |
| Outcome: | The proposed approach improves zero-shot multimodal retrieval performance without fine-tuning efforts. |
Are Multimodal Large Language Models Pragmatically Competent Listeners in Simple Reference Resolution Tasks? (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing models are unable to resolve references to abstract visual stimuli, such as color patches and color grids, but their pragmatic capabilities are still a challenge for state-of-the-art MLLMs. |
| Approach: | They investigate whether multimodal large language models are able to resolve references to abstract visual stimuli, such as color patches and color grids, in a well-known reference resolution paradigm. |
| Outcome: | The proposed model can resolve references to abstract visual stimuli in dyadic reference games. |
From Multimodal LLM to Human-level AI: Modality, Instruction, Reasoning, Efficiency and beyond (2024.lrec-tutorials)
Copied to clipboard
| Challenge: | This tutorial aims to deliver a comprehensive review of cutting-edge research in MLLMs. |
| Approach: | This tutorial will review cutting-edge research in MLLMs and examine the impact of ML in learning and reasoning. |
| Outcome: | This course will review cutting-edge research in MLLMs and examine the impact of ML models on learning, learning, and multimodal reasoning. |
On Domain-Adaptive Post-Training for Multimodal Large Language Models (2025.findings-emnlp)
Copied to clipboard
Daixuan Cheng, Shaohan Huang, Ziyu Zhu, Xintong Zhang, Xin Zhao, Zhongzhi Luan, Bo Dai, Zhenliang Zhang
| Challenge: | Adapting general multimodal large language models to specific domains is important for practical applications. |
| Approach: | They investigate domain adaptation of multimodal large language models via post-training . they develop a generate-then-filter pipeline that curates diverse visual instruction tasks . |
| Outcome: | The proposed model outperforms existing models in domain adaptation by combining data from open-source models with training pipelines. |
Probing Logical Reasoning of MLLMs in Scientific Diagrams (2025.emnlp-main)
Copied to clipboard
| Challenge: | logical reasoning is key to real-world applications like science education, environmental monitoring, and medical diagnostics. |
| Approach: | They construct visual questions that follow seven structured templates with progressively more complex reasoning involved. |
| Outcome: | The proposed models perform logical inferences based on visual information. |
Multimodal Language Models See Better When They Look Shallower (2025.emnlp-main)
Copied to clipboard
Haoran Chen, Junyan Lin, Xinghao Chen, Yue Fan, Jianfeng Dong, Xin Jin, Hui Su, Jinlan Fu, Xiaoyu Shen
| Challenge: | Existing studies show that multimodal large language models extract visual features from the final layers of a pretrained Vision Transformer. |
| Approach: | They propose a feature fusion method that strategically incorporates shallower layers . they propose MLLMs that extract visual features from the final layers of a pretrained Vision Transformer . |
| Outcome: | The proposed method outperforms deep layers on fine-grained visual tasks . it is the first comprehensive study of visual layer selection for MLLMs . |
Explainability and Interpretability of Multilingual Large Language Models: A Survey (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing literature on multilingual large language models lacks transparency in their internal processes. |
| Approach: | They propose to use multilingual large language models to examine their explainability and interpretability methods. |
| Outcome: | The present study examines the explainability and interpretability of multilingual large language models. |