V-GameGym: Visual Game Generation for Code Large Language Models

Challenge:	Existing code-related benchmarks focus on single modality rather than visual game development.
Approach:	They propose a multimodal benchmark for evaluating code large language models in visual game generation that integrates a clustering-based curation methodology and a pipeline for visual code synthesis.
Outcome:	The proposed framework assesses code generation and visual game generation using a sandbox environment.

Similar Papers

Overview

Challenge

Approach

Outcome

CodeScope: An Execution-based Multilingual Multitask Multidimensional Benchmark for Evaluating LLMs on Code Understanding and Generation (2024.acl-long)

Copied to clipboard

Weixiang Yan, Haitian Liu, Yunkun Wang, Yunzhe Li, Qian Chen, Wen Wang, Tingyu Lin, Weishan Zhao, Li Zhu, Hari Sundaram, Shuiguang Deng

Multimodality

Challenge:	Existing benchmarks for evaluating the code understanding and generation capacities of Large Language Models are insufficient . existing benchmarks focus on a narrow range of popular programming languages and specific tasks .
Approach:	They propose an execution-based, multilingual, multitask evaluation benchmark for LLMs . they evaluate coding performance from three dimensions: length, difficulty, efficiency .
Outcome:	The proposed benchmark covers 43 programming languages and eight coding tasks.

PlotGen-Bench: Evaluating VLMs on Generating Visualization Code from Diverse Plots across Multiple Libraries (2026.findings-acl)

Copied to clipboard

Yi Zhao, Zhen Yang, Shuaiqi Duan, Wenmeng Yu, Zhe Su, Jibing Gong, Jie Tang

Multimodality

Challenge:	PlotGen-Bench evaluates vision-language models' ability to generate executable visualization code from plots under realistic and complex visualization requirements.
Approach:	They propose a benchmark to evaluate plot-to-code generation in vision-language models . they use Matplot, Matplos, Mat3D, Mat4D, and Mat4E to evaluate their performance .
Outcome:	The proposed benchmark covers 9 major categories, 30 subcategories, and 3 core tasks . it covers 2D, 3D and animated plots across 5 widely used visualization libraries.

VGBench: A Comprehensive Benchmark of Vector Graphics Understanding and Generation for Large Language Models (2024.emnlp-main)

Copied to clipboard

Bocheng Zou, Mu Cai, Jianrui Zhang, Yong Jae Lee

Semantic Text Processing

Challenge:	Current vision models use pixels to rasterize the visual world, but vector graphics are not the best or unique way to represent visual content.
Approach:	They propose a benchmark for vector graphics processing with capable Large Language Models . they use a set of questions to evaluate vector graphics formats and a wide range of prompting techniques .
Outcome:	The proposed benchmark compares LLMs on rasterized representations with vector graphics . it shows that LLM models show strong capability on both aspects .

XCodeEval: An Execution-based Large Scale Multilingual Multitask Benchmark for Code Understanding, Generation, Translation and Retrieval (2024.acl-long)

Copied to clipboard

Mohammad Abdullah Matin Khan, M Saiful Bari, Do Long, Weishi Wang, Md Rizwan Parvez, Shafiq Joty

Multilinguality

Challenge:	Recent advances in large language models have shown impressive abilities in generating codes from natural language descriptions, repairing buggy codes, translating codes between languages, and retrieving relevant code segments.
Approach:	They propose to use a multilingual multitask benchmark to evaluate large language models that can generate codes from natural language descriptions, repair buggy codes, and translate between languages.
Outcome:	The proposed model performs 7 tasks covering up to 11 languages with execution-level parallelism and 25 M document-level coding examples (16.5 B tokens)

TURTLEAI: Benchmarking Multimodal Models for Visual Programming in Turtle Graphics (2026.findings-acl)

Copied to clipboard

Chao Wen, Jacqueline Staub, Adish Singla

Multimodality

Challenge:	Vision-language models have been explored for visual programming, but performance is unclear . most prior work focuses on visual programming for productivity .
Approach:	They propose a visual programming benchmark that uses visual programming to evaluate VLMs.
Outcome:	The proposed model improves on GPT-5, GPT-4o, and Qwen2-VL-72B on real-world tasks by 20% . the proposed model is based on 823 visual programming tasks in the Turtle Graphics domain .

mHumanEval - A Multilingual Benchmark to Evaluate Large Language Models for Code Generation (2025.naacl-long)

Copied to clipboard

Md Nishat Raihan, Antonios Anastasopoulos, Marcos Zampieri

Text Generation

Challenge:	Current evaluations focus on English-to-Python conversion tasks with limited test cases . code generation from low-resource language prompts remains largely unexplored .
Approach:	They propose a benchmark that supports prompts in over 200 natural languages . they provide expert human translations for 15 diverse natural languages (NLs)
Outcome:	The HumanEval Benchmark is the most widely used code generation benchmark . it provides expert human translations for 15 diverse natural languages .

MMCode: Benchmarking Multimodal Large Language Models for Code Generation with Visually Rich Programming Problems (2024.findings-emnlp)

Copied to clipboard

Kaixin Li, Yuchen Tian, Qisheng Hu, Ziyang Luo, Zhiyong Huang, Jing Ma

Multimodality

Challenge:	Programming often involves translating detailed and complex specifications into code . current state-of-the-art models struggle to solve these problems, a new study shows .
Approach:	They propose a multi-modal coding dataset to evaluate algorithmic problem-solving skills in visually rich contexts.
Outcome:	The proposed model lacks powerful vision-code models due to the extreme demand for reasoning abilities.

Turning the Tide: Repository-based Code Reflection (2025.findings-emnlp)

Copied to clipboard

Wei Zhang, Jian Yang, Jiaxi Yang, Ya Wang, Zhoujun Li, Zeyu Cui, Binyuan Hui, Junyang Lin

Programming Languages in NLP

Challenge:	Code large language models (LLMs) enhance programming by understanding and generating code across languages.
Approach:	a new benchmark evaluates code understanding and generation in repositories using code large language models.
Outcome:	The proposed model improves code understanding and generation in repositories by evaluating 1,888 test cases across 6 programming languages.

Cardiverse: Harnessing LLMs for Novel Card Game Prototyping (2025.emnlp-main)

Copied to clipboard

Danrui Li, Sen Zhang, Samuel S. Sohn, Kaidong Hu, Muhammad Usman, Mubbasir Kapadia

Challenge:	Recent advances in Large Language Models (LLMs) have enhanced the possibilities for game prototyping, but they face significant challenges.
Approach:	They propose a graph-based indexing method for generating novel game variations and an LLM-driven system for consistent game code generation validated by gameplay records.
Outcome:	The proposed framework accelerates card game prototyping, reduces human labor, and lowers barriers to entry for game developers.

DynaCode: A Dynamic Complexity-Aware Code Benchmark for Evaluating Large Language Models in Code Generation (2025.findings-acl)

Copied to clipboard

Wenhao Hu, Jinhao Duan, Chunchen Wei, Li Zhang, Yue Zhang, Kaidi Xu

Multimodality

Challenge:	Existing code benchmarks for large language models remain static, resulting in data contamination and unreliable evaluation results.
Approach:	They propose a dynamic, complexity-aware benchmark that overcomes the limitations of static datasets and provides a memorization-advantaged benchmark.
Outcome:	DynaCode generates 189 million unique nested code problems across 4 units of code complexity and 16 types of call graphs.