Flow2Code: Evaluating Large Language Models for Flowchart-based Code Generation Capability (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing code generation benchmarks neglect flowchart-based code generation . existing benchmarks lack flowcharting-based evaluation, limiting the potential of large language models and minimizing human error. |
| Approach: | They propose to use flowcharts to evaluate existing LLMs' code generation capabilities. |
| Outcome: | The proposed benchmarks show that the supervised fine-tuning technique contributes greatly to the models’ performance. |
Similar Papers
Large Language Models Meet NL2Code: A Survey (2023.acl-long)
Copied to clipboard
Daoguang Zan, Bei Chen, Fengji Zhang, Dianjie Lu, Bingchao Wu, Bei Guan, Wang Yongji, Jian-Guang Lou
| Challenge: | generating code from a natural language description is a pressing and significant challenge in code intelligence. |
| Approach: | They propose to survey 27 existing large language models for NL2Code and compare them to humanEval benchmarks. |
| Outcome: | The proposed model is compared with existing models on the HumanEval benchmark. |
Code Generation From Flowcharts with Texts: A Benchmark Dataset and An Approach (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Currently, researchers focus on generating codes from requirement documents. |
| Approach: | They propose to generate source code from flowcharts with texts instead of directly translating requirements into codes. |
| Outcome: | The proposed model improves on the baselines by transforming flowcharts into pseudo-code . the proposed model is based on 320 flowchartes with their corresponding source codes . |
Plot2Code: A Comprehensive Benchmark for Evaluating Multi-modal Large Language Models in Code Generation from Scientific Plots (2025.findings-naacl)
Copied to clipboard
| Challenge: | Multi-modal Large Language Models have shown remarkable progress in visual contexts, yet their ability to convert visual figures into executable code remains underexplored. |
| Approach: | They propose to use a set of visual coding metrics to assess MLLMs' visual . pass rate, text-match ratio, and GPT-4V rating judgement to assess the quality of generated code and rendered images. |
| Outcome: | The proposed benchmark includes 132 high-quality matplotlib plots across six plot types, as well as 150 and 86 plots from Python’s and R’s plotly libraries respectively, totaling 368 plots. |
RealChart2Code: Bridging the Gap in Real-World Chart-to-Code Generation via Multi-Task Evaluation (2026.acl-long)
Copied to clipboard
Jiajun Zhang, Yuying Li, Zhixun Li, Xingyu Guo, Jingzhuo Wu, Leqi Zheng, Yiran Yang, Jianke Zhang, Qingbin Li, Shannan Yan, Changguo Jia, Junfei Wu, Zilei Wang, Qiang Liu, Liang Wang
| Challenge: | Vision-Language Models (VLMs) have demonstrated impressive capabilities in code generation across various domains, but their ability to replicate complex, multi-panel visualizations remains largely unassessed. |
| Approach: | They propose a large-scale benchmark to evaluate chart generation from large- scale raw data and assess iterative code refinement in a multi-turn conversational setting. |
| Outcome: | The new benchmark evaluates 14 leading VLMs on real-world data and shows they struggle with complex plot structures and authentic data. |
CodeScope: An Execution-based Multilingual Multitask Multidimensional Benchmark for Evaluating LLMs on Code Understanding and Generation (2024.acl-long)
Copied to clipboard
Weixiang Yan, Haitian Liu, Yunkun Wang, Yunzhe Li, Qian Chen, Wen Wang, Tingyu Lin, Weishan Zhao, Li Zhu, Hari Sundaram, Shuiguang Deng
| Challenge: | Existing benchmarks for evaluating the code understanding and generation capacities of Large Language Models are insufficient . existing benchmarks focus on a narrow range of popular programming languages and specific tasks . |
| Approach: | They propose an execution-based, multilingual, multitask evaluation benchmark for LLMs . they evaluate coding performance from three dimensions: length, difficulty, efficiency . |
| Outcome: | The proposed benchmark covers 43 programming languages and eight coding tasks. |
ChartCoder: Advancing Multimodal Large Language Model for Chart-to-Code Generation (2025.acl-long)
Copied to clipboard
| Challenge: | Existing open-source MLLMs fail to fully capture dense information embedded in charts . current models still face significant challenges in understanding and analyzing visual tasks such as captioning and question answering. |
| Approach: | They propose a chart-to-code MLLM which leverages Code LLMs as the language backbone to enhance the executability of the generated code. |
| Outcome: | The proposed model surpasses existing open-source models on chart-to-code benchmarks with only 7B parameters and provides lossless representations that contain all critical details. |
CodeJudge: Evaluating Code Generation with Large Language Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have shown promising performance in code generation, but how to reliably evaluate code generated by LLMs remains a challenging problem. |
| Approach: | They propose a framework that leverages Large Language Models to evaluate the semantic correctness of generated code without the need for test cases. |
| Outcome: | The proposed framework outperforms existing methods on four code generation datasets and five programming languages. |
DynaCode: A Dynamic Complexity-Aware Code Benchmark for Evaluating Large Language Models in Code Generation (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing code benchmarks for large language models remain static, resulting in data contamination and unreliable evaluation results. |
| Approach: | They propose a dynamic, complexity-aware benchmark that overcomes the limitations of static datasets and provides a memorization-advantaged benchmark. |
| Outcome: | DynaCode generates 189 million unique nested code problems across 4 units of code complexity and 16 types of call graphs. |
From Charts to Code: A Hierarchical Benchmark for Multimodal Models (2026.acl-long)
Copied to clipboard
Jiahao Tang, Henry Hengyuan Zhao, Lijian Wu, Zijian Zhang, Yifei Tao, Dongxing Mao, Yang Wan, Jingru Tan, Min Zeng, Min Li, Alex Jinpeng Wang
| Challenge: | Chart2Code is a new benchmark for evaluating the natural language to chart code generation capabilities of large multimodal models. |
| Approach: | They introduce Chart2Code, a new benchmark for evaluating the natural language to chart code generation capabilities of large multimodal models. |
| Outcome: | The proposed benchmark is the first to scale task complexity while capturing diverse scenarios. |
Chart2Code53: A Large-Scale Diverse and Complex Dataset for Enhancing Chart-to-Code Generation (2025.emnlp-main)
Copied to clipboard
Tianhao Niu, Yiming Cui, Baoxin Wang, Xiao Xu, Xin Yao, Qingfu Zhu, Dayong Wu, Shijin Wang, Wanxiang Che
| Challenge: | Existing Chart2code-related training datasets suffer from limited scale, limited type coverage, and inadequate complexity. |
| Approach: | They propose to synthesize chart2code-related training datasets using web plotting code and chart images to address these challenges. |
| Outcome: | The proposed dataset exhibits the greatest diversity and higher complexity compared to other open-source Chart2code related datasets. |