VGBench: A Comprehensive Benchmark of Vector Graphics Understanding and Generation for Large Language Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | Current vision models use pixels to rasterize the visual world, but vector graphics are not the best or unique way to represent visual content. |
| Approach: | They propose a benchmark for vector graphics processing with capable Large Language Models . they use a set of questions to evaluate vector graphics formats and a wide range of prompting techniques . |
| Outcome: | The proposed benchmark compares LLMs on rasterized representations with vector graphics . it shows that LLM models show strong capability on both aspects . |
Similar Papers
PlotGen-Bench: Evaluating VLMs on Generating Visualization Code from Diverse Plots across Multiple Libraries (2026.findings-acl)
Copied to clipboard
| Challenge: | PlotGen-Bench evaluates vision-language models' ability to generate executable visualization code from plots under realistic and complex visualization requirements. |
| Approach: | They propose a benchmark to evaluate plot-to-code generation in vision-language models . they use Matplot, Matplos, Mat3D, Mat4D, and Mat4E to evaluate their performance . |
| Outcome: | The proposed benchmark covers 9 major categories, 30 subcategories, and 3 core tasks . it covers 2D, 3D and animated plots across 5 widely used visualization libraries. |
V-GameGym: Visual Game Generation for Code Large Language Models (2026.findings-acl)
Copied to clipboard
Wei Zhang, Jian Yang, Renshuai Tao, Linzheng Chai, Shuyue Guo, Jiajun Wu, Xiaoming Chen, Ganqu Cui, Ning Ding, Xander Xu, HU Wei, Bowen Zhou
| Challenge: | Existing code-related benchmarks focus on single modality rather than visual game development. |
| Approach: | They propose a multimodal benchmark for evaluating code large language models in visual game generation that integrates a clustering-based curation methodology and a pipeline for visual code synthesis. |
| Outcome: | The proposed framework assesses code generation and visual game generation using a sandbox environment. |
TURTLEAI: Benchmarking Multimodal Models for Visual Programming in Turtle Graphics (2026.findings-acl)
Copied to clipboard
| Challenge: | Vision-language models have been explored for visual programming, but performance is unclear . most prior work focuses on visual programming for productivity . |
| Approach: | They propose a visual programming benchmark that uses visual programming to evaluate VLMs. |
| Outcome: | The proposed model improves on GPT-5, GPT-4o, and Qwen2-VL-72B on real-world tasks by 20% . the proposed model is based on 823 visual programming tasks in the Turtle Graphics domain . |
TritonBench: Benchmarking Large Language Model Capabilities for Generating Triton Operators (2025.findings-acl)
Copied to clipboard
Jianling Li, ShangZhan Li, Zhenye Gao, Qi Shi, Yuxuan Li, Zefan Wang, Jiacheng Huang, WangHaojie WangHaojie, Jianrong Wang, Xu Han, Zhiyuan Liu, Maosong Sun
| Challenge: | Triton is a high-level Python-like programming language for building efficient GPU kernels. |
| Approach: | They propose a TritonBench benchmark that provides a comprehensive evaluation of Tritonic operators on widely deployed GPUs. |
| Outcome: | The proposed benchmarks show that current LLMs struggle to generate efficient Triton operators on widely deployed GPUs aligned with industry applications. |
NLEBench+NorGLM: A Comprehensive Empirical Analysis and Benchmark Dataset for Generative Language Models in Norwegian (2024.emnlp-main)
Copied to clipboard
Peng Liu, Lemei Zhang, Terje Farup, Even Lauvrak, Jon Ingvaldsen, Simen Eide, Jon Atle Gulla, Zhirong Yang
| Challenge: | Norwegian is under-represented within the most impressive breakthroughs in NLP tasks. |
| Approach: | they investigate the impact of existing Norwegian language models on Norwegian generation tasks . they pre-trained 4 Norwegian Open Language Models from parameter scales and architectures . |
| Outcome: | The proposed benchmark evaluates the performance of language models on Norwegian generation tasks. |
LLM Meets Scene Graph: Can Large Language Models Understand and Generate Scene Graphs? A Benchmark and Empirical Study (2025.acl-long)
Copied to clipboard
Dongil Yang, Minjin Kim, Sunghwan Kim, Beong-woo Kwak, Minjun Park, Jinseok Hong, Woontack Woo, Jinyoung Yeo
| Challenge: | Large language models (LLMs) have demonstrated impressive progress in various text-based tasks, such as question-answering and content generation. |
| Approach: | They propose a benchmark to evaluate Large Language Models’ ability to understand scene graphs and generate them from textual narratives. |
| Outcome: | The proposed model performs well on scene graph understanding but struggles with scene graph generation, particularly for complex narratives. |
Glyph: Scaling Context Windows via Visual-Text Compression (2026.acl-long)
Copied to clipboard
Jiale Cheng, Yusen Liu, Xinyu Zhang, Yulin Fei, Wenyi Hong, Ruiliang Lyu, Weihan Wang, Zhe Su, Xiaotao Gu, Xiao Liu, Yushi Bai, Jie Tang, Hongning Wang, Minlie Huang
| Challenge: | Large language models (LLMs) traditionally represent text as sequences of discrete tokens . a long-context scaling problem requires processing more tokens more efficiently . |
| Approach: | They propose a framework that renders long texts into compact visual pages and processes them with a vision-language model. |
| Outcome: | The proposed framework renders long texts into compact visual pages and processes them with a vision-language model. |
VLind-Bench: Measuring Language Priors in Large Vision-Language Models (2025.findings-naacl)
Copied to clipboard
| Challenge: | Large Vision-Language Models suffer from a problem known as language prior . such language priors can lead to undesirable biases and hallucinations when dealing with images that are out of distribution. |
| Approach: | They propose a benchmark to measure the language priors of Large Vision-Language Models. |
| Outcome: | The proposed benchmark is the first specifically designed to measure the language priors, or blindness, of LVLMs. |
TurtleBench: A Visual Programming Benchmark in Turtle Geometry (2025.naacl-long)
Copied to clipboard
| Challenge: | Large Multimodal Models (LMMs) are capable of reasoning about geometric patterns, but they are still a challenge to evaluate. |
| Approach: | They propose a benchmark to evaluate LMMs’ ability to interpret geometric patterns and generate precise code outputs. |
| Outcome: | The benchmark compared LMMs with turtle geometry and GPT-4V, and found that few-shot prompting improved their performance (2%). |
Retrieving Multimodal Information for Augmented Generation: A Survey (2023.findings-emnlp)
Copied to clipboard
Ruochen Zhao, Hailin Chen, Weishi Wang, Fangkai Jiao, Xuan Long Do, Chengwei Qin, Bosheng Ding, Xiaobao Guo, Minzhi Li, Xingxuan Li, Shafiq Joty
| Challenge: | Large Language Models (LLMs) are increasingly using multimodality to augment their generation ability, but there is no unified perception of at which stage and how to incorporate different modalities. |
| Approach: | They propose to use multimodality to augment Large Language Models (LLMs) this will provide scholars with a deeper understanding of the methods' applications and encourage them to adapt existing techniques to the fast-growing field of LLMs. |
| Outcome: | The proposed methods improve factuality, reasoning, interpretability, and robustness of the generated content. |