ChartInstruct: Instruction Tuning for Chart Comprehension and Reasoning (2024.findings-acl)
Copied to clipboard
| Challenge: | Charts provide visual representations of data and are used for analyzing information, addressing queries, and conveying insights to others. |
| Approach: | They propose a chart-specific vision-language Instruction-following dataset with 191K instructions and a pipeline model that extracts chart data tables and inputs them into a LLM. |
| Outcome: | The proposed model can solve a wide range of chart-related tasks, achieving state-of-the-art results on four tasks. |
Similar Papers
ChartGemma: Visual Instruction-tuning for Chart Reasoning in the Wild (2025.coling-industry)
Copied to clipboard
| Challenge: | Existing methods for chart understanding and reasoning are weakly aligned and rely on underlying data tables. |
| Approach: | They propose a chart-based understanding and reasoning model that is trained on instruction-tuning data generated directly from chart images. |
| Outcome: | The proposed model achieves state-of-the-art results across 5 benchmarks spanning chart summarization, question answering, and fact-checking. |
Text2Chart31: Instruction Tuning for Chart Generation with Automatic Feedback (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing datasets do not cover full range of chart types, such as 3D, volumetric, and gridded charts. |
| Approach: | They propose a hierarchical pipeline and a new dataset for chart generation that leverages the relationships within rich datasets. |
| Outcome: | The proposed method outperforms open-source models and is comparable to state-of-the-art proprietary models in data visualization tasks. |
ChartM3: A Multi-Stage Code-Driven Pipeline for Constructing Multi-Dimensional and Multi-Step Visual Reasoning Data in Chart Comprehension (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Currently, research on complex chart understanding tasks is limited . a pipeline for visual reasoning datasets addresses these limitations . |
| Approach: | They propose a code-driven pipeline for generating visual reasoning datasets . pipeline integrates retrieval-augmented generation to retrieve professional chart templates . |
| Outcome: | The proposed pipeline enhances chart diversity and data quality through model-based evaluation. |
Omni-Chart-600K: A Comprehensive Dataset of Chart Types for Chart Understanding (2025.findings-naacl)
Copied to clipboard
Shulei Wang, Shuai Yang, Wang Lin, Zirun Guo, Sihang Cai, Hai Huang, Ye Wang, Jingyuan Chen, Tao Jin
| Challenge: | Existing chart-related training methods lack capabilities in information extraction, mathematical reasoning, and understanding of multiple chart types. |
| Approach: | They propose a two-stage training strategy and method for jointly training a vision encoder tailored for multi-type charts to address the deficiencies in chart types and limited scope of chart tasks in existing datasets. |
| Outcome: | The proposed dataset includes 21 diverse chart types and tasks, including data retrieval and mathematical reasoning. |
ChartAssistant: A Universal Chart Multimodal Language Model via Chart-to-Table Pre-training and Multitask Instruction Tuning (2024.findings-acl)
Copied to clipboard
| Challenge: | Charts are an effective tool for understanding data patterns, but their combination of graphical elements and textual components poses challenges for general-purpose multimodal models. |
| Approach: | They propose a chart-based vision-language model for universal chart comprehension and reasoning that leverages a dataset of chart-related tasks. |
| Outcome: | The proposed model outperforms the state-of-the-art charts with zero-shot setting on various chart tasks. |
UniChart: A Universal Vision-language Pretrained Model for Chart Comprehension and Reasoning (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for chart-based data analysis neglect explicit modeling of chart structures. |
| Approach: | They propose a pretrained model for chart comprehension and reasoning that encodes relevant text, data, and visual elements of charts and uses a chart-grounded text decoder for text generation. |
| Outcome: | The proposed model outperforms existing methods that lack explicit modeling of chart structures and lacks explicit modeling. |
Vision-Flan: Scaling Human-Labeled Tasks in Visual Instruction Tuning (2024.findings-acl)
Copied to clipboard
Zhiyang Xu, Chao Feng, Rulin Shao, Trevor Ashby, Ying Shen, Di Jin, Yu Cheng, Qifan Wang, Lifu Huang
| Challenge: | Recent vision-language models (VLMs) have shown impressive capabilities as general visual assistants, but there are two challenges to their performance: (1) lacking task diversity in pretraining and visual instruction tuning; (2) annotation error and bias in GPT-4 synthesized instruction tuning data. |
| Approach: | They propose a two-stage instruction tuning framework that fine tunes VLMs firstly and further tuned on GPT-4 synthesized data. |
| Outcome: | The proposed framework outperforms the traditional single-stage visual instruction tuning framework and achieves state-of-the-art performance across a wide range of multi-modal evaluation benchmarks. |
ChartThinker: A Contextual Chain-of-Thought Approach to Optimized Chart Summarization (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing methods for chart summarization lack visual-language matching and reasoning ability. |
| Approach: | They propose a method which synthesizes deep analysis based on chains of thought and strategies of context retrieval to improve the logical coherence and accuracy of the generated summaries. |
| Outcome: | The proposed method outperforms 8 state-of-the-art models over 7 evaluation metrics and can significantly reduce time and cognitive resources required. |
CharTide: Data-Centric Chart-to-Code Generation via Tri-Perspective Tuning and Inquiry-Driven Evolution (2026.acl-long)
Copied to clipboard
Xiangxi Zheng, Kuang He, Jiayi Hu, Ping Yu, Rui Yan, Yuan Yao, Peng Hou, Anxiang Zeng, Alex Jinpeng Wang
| Challenge: | Existing approaches to chart-to-code generation are constrained by data-centric limitations . authors present a new framework that redesigns both training and alignment data . |
| Approach: | They propose a data-centric framework that redesigns both training and alignment data for chart-to-code generation. |
| Outcome: | The proposed framework outperforms open-source baselines and is competitive with GPT-5. |
On the Perception Bottleneck of VLMs for Chart Understanding (2025.findings-emnlp)
Copied to clipboard
| Challenge: | a perception bottleneck in large vision-language models is critical for chart understanding . instruction tuning improves the extraction capability of LVLMs, but the vision encoder remains a bottleneck . |
| Approach: | They propose to decompose the perception bottleneck into two components . the vision encoder bottleneck is where visual representation fails to encapsulate the correct information . |
| Outcome: | The proposed approach significantly mitigates the vision encoder bottleneck and improves the ability of LVLMs to comprehend charts. |