Math-LLaVA: Bootstrapping Mathematical Reasoning for Multimodal Large Language Models (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing image instruction fine-tuning datasets do not fully exploit visual information to enhance multimodal reasoning capabilities of Large language models (LLMs). |
| Approach: | They propose a LLaVA-based model fine-tuned with MathV360K to bridge this gap by collecting 40K high-quality images with question-answer pairs from 24 existing datasets and synthesizing 320K new pairs. |
| Outcome: | The proposed model improves the multimodal reasoning capabilities of LLaVA-1.5 and demonstrates enhanced generalizability on the MMMU benchmark. |
Similar Papers
A Survey of Mathematical Reasoning in the Era of Multimodal Large Language Model: Benchmark, Method & Challenges (2025.findings-acl)
Copied to clipboard
Yibo Yan, Jiamin Su, Jianxiang He, Fangteng Fu, Xu Zheng, Yuanhuiyi Lyu, Kun Wang, Shen Wang, Qingsong Wen, Xuming Hu
| Challenge: | This survey provides **the first comprehensive analysis of mathematical reasoning in the era of multimodal large language models** . integrating large language model with mathematical reasoning tasks is becoming significant as AI advances . |
| Approach: | They review over 200 studies published since 2021 and examine the state-of-the-art developments in Math-LLMs . they identify five major challenges hindering the realization of AGI in this domain . |
| Outcome: | The authors examine the state-of-the-art developments in Math-LLMs with a focus on multimodal settings. |
MathCoder-VL: Bridging Vision and Code for Enhanced Multimodal Mathematical Reasoning (2025.findings-acl)
Copied to clipboard
Ke Wang, Junting Pan, Linda Wei, Aojun Zhou, Weikang Shi, Zimu Lu, Han Xiao, Yunqiao Yang, Houxing Ren, Mingjie Zhan, Hongsheng Li
| Challenge: | Large Language Models (LMMs) struggle with simple tasks such as geometry, e.g., arithmetic, and reasoning. |
| Approach: | They propose to leverage code as supervision for cross-modal alignment . they propose to use FigCodifier and ImgCode-8.6M to synthesize novel mathematical figures . |
| Outcome: | The proposed model surpasses GPT-4o and Claude 3.5 Sonnet in the geometry problem-solving subset of MathVista, achieving improvements of 8.9% and 9.2%. |
Breaking Language Barriers in Multilingual Mathematical Reasoning: Insights and Observations (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing research focuses on developing powerful large language models for mathematical reasoning within monolingual languages. |
| Approach: | They propose to use translation to build powerful multilingual math reasoning models . they propose different training strategies to build xMR LLMs that outperform open-source LLM . |
| Outcome: | The proposed model outperforms open-source LLMs and surpasses ChatGPT in few-shot scenarios. |
MARIO: MAth Reasoning with code Interpreter Output - A Reproducible Pipeline (2024.findings-acl)
Copied to clipboard
| Challenge: | Large language models lack mathematical reasoning, a hurdle on the path to true artificial general intelligence. |
| Approach: | They propose a protocol for fine-tuning large language models with a Python code interpreter to enhance the text analysis of the LLMs. |
| Outcome: | The proposed protocol improves the performance of a 7B-parameter LLM on the GSM8K and MATH datasets while allowing for an outlier-free value model-based inference method. |
ErrorRadar: Benchmarking Complex Mathematical Reasoning of Multimodal Large Language Models Via Error Detection (2026.findings-acl)
Copied to clipboard
Yibo Yan, Shen Wang, Jiahao Huo, Hang Li, Boyan Li, Jiamin Su, Xiong Gao, YiFan Zhang, Tianlong Xu, Zhendong Chu, Aoxiao Zhong, Kun Wang, Hui Xiong, Philip S. Yu, Xuming Hu, Qingsong Wen
| Challenge: | Current mathematical benchmarks focus on evaluating MLLMs’ problem-solving ability, yet there is a crucial gap in addressing more complex scenarios such as error detection. |
| Approach: | They propose to evaluate multimodal error detection by evaluating two sub-tasks error step identification and error categorization. |
| Outcome: | The proposed task evaluates MLLMs' ability to handle multimodal questions compared to text-only models. |
Are Multimodal LLMs Robust Against Adversarial Perturbations? RoMMath: A Systematic Evaluation on Multimodal Math Reasoning (2025.naacl-long)
Copied to clipboard
| Challenge: | Recent-released MLLMs have shown remarkable performance on various multimodal math reasoning benchmarks. |
| Approach: | They introduce RoMMath, the first benchmark designed to evaluate the capabilities and robustness of multimodal large language models in handling multimodal math reasoning. |
| Outcome: | The proposed model performs well on a broad spectrum of 17 MLLMs and demonstrates that they are robust to adversarial perturbations. |
InfiMM-WebMath-40B: Advancing Multimodal Pre-Training for Enhanced Mathematical Reasoning (2025.findings-emnlp)
Copied to clipboard
Xiaotian Han, Yiren Jian, Xuefeng Hu, Haogeng Liu, Yiqi Wang, Qihang Fan, Yuang Ai, Huaibo Huang, Ran He, Zhenheng Yang, Quanzeng You
| Challenge: | InfiMM-WebMath-40B is a dataset of interleaved image-text documents . it consists of 24 million web pages, 85 million image URLs, and 40 billion text tokens . |
| Approach: | InfiMM-WebMath-40B is a high-quality dataset of interleaved image-text documents . it contains 24 million web pages, 85 million image URLs, and 40 billion text tokens . |
| Outcome: | InfiMM-WebMath-40B is a high-quality dataset of interleaved image-text documents . it consists of 24 million web pages, 85 million image URLs, and 40 billion text tokens . |
LLaVA Steering: Visual Instruction Tuning with 500x Fewer Parameters through Modality Linear Representation-Steering (2025.acl-long)
Copied to clipboard
| Challenge: | Multimodal Large Language Models (MLLMs) enhance visual tasks by integrating visual representations into large language models. |
| Approach: | They propose a method to re-balance modalities by steering visual representations . they propose LLaVA Steering, a platform that enables rapid customization of MLLMs a component-based architecture . |
| Outcome: | The proposed model re-balances the modalities of visual representations in large language models . the model requires 500 times fewer trainable parameters than LoRA while maintaining comparable performance . |
MathCanvas: Intrinsic Visual Chain-of-Thought for Multimodal Mathematical Reasoning (2026.acl-long)
Copied to clipboard
Weikang Shi, Aldrich Yu, Rongyao Fang, Houxing Ren, Ke Wang, Aojun Zhou, Changyao Tian, Xinyu Fu, Yuxuan Hu, Zimu Lu, Linjiang Huang, Si Liu, Rui Liu, Hongsheng Li
| Challenge: | Existing approaches to visual chain-of-thought are limited by external tools or fail to generate high-fidelity diagrams. |
| Approach: | They propose a framework to enable large multimodal models with VCoT capabilities . they pre-train a model on a 15.2M-pair corpus and teach it how to leverage visual aids . |
| Outcome: | The proposed framework unlocks complex, human-like visual reasoning in large language models . it pre-trains the model on a 15.2M-pair corpus and fine-tunes it on MathCanvas-Instruct . |
X-LLaVA: Optimizing Bilingual Large Vision-Language Alignment (2024.findings-naacl)
Copied to clipboard
DongJae Shin, HyeonSeok Lim, Inho Won, ChangSu Choi, Minjun Kim, SeungWoo Song, HanGyeol Yoo, SangMin Kim, KyungTae Lim
| Challenge: | constructing multilingual data for large multimodal models presents its own set of challenges due to language diversity and complexity. |
| Approach: | They propose to use GPT4-V to construct multimodal training datasets using a text-only version of GPT4. |
| Outcome: | The proposed method performs well in Korean and English, surpassing existing methods. |