TechING: Towards Real World Technical Image Understanding via VLMs (2026.findings-eacl)
Copied to clipboard
| Challenge: | Modern day vision language models struggle when it comes to understanding technical diagrams . a large synthetically generated corpus is needed to train and evaluate VLMs on hand-drawn images . |
| Approach: | They propose a large synthetically generated corpus for training VLMs and evaluate them on hand-drawn images. |
| Outcome: | The proposed model improves ROUGE-L performance of Llama 3.2 11B-instruct by 2.14x on synthetic images on real-world images. |
Similar Papers
Scaling Text-Rich Image Understanding via Code-Guided Synthetic Multimodal Data Generation (2025.acl-long)
Copied to clipboard
Yue Yang, Ajay Patel, Matt Deitke, Tanmay Gupta, Luca Weihs, Andrew Head, Mark Yatskar, Chris Callison-Burch, Ranjay Krishna, Aniruddha Kembhavi, Christopher Clark
| Challenge: | Vision-language models struggle to understand text-rich images due to the scarcity of diverse text-only large language data. |
| Approach: | They propose a framework that leverages the coding capabilities of text-only large language models to create synthetic text-rich multimodal data. |
| Outcome: | The proposed framework can generate high-quality instruction-tuning data using Python, HTML, LaTeX and other languages. |
Finer: Investigating and Enhancing Fine-Grained Visual Concept Recognition in Large Vision Language Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in instruction-tuned Large Vision-Language Models (LVLMs) have imbued the models with the ability to generate high-level, image-grounded explanations with ease. |
| Approach: | They propose to use a multiple granularity attribute-centric benchmark and training mixture to evaluate LVLMs’ fine-grained visual comprehension ability. |
| Outcome: | The proposed model improves on LLaVa-1.5, InstructBLIP and GPT-4V and demonstrates that they struggle to generate descriptive visual attributes based on a concept that appears within an input image despite their prominent zero-shot image captioning ability. |
RealChart2Code: Bridging the Gap in Real-World Chart-to-Code Generation via Multi-Task Evaluation (2026.acl-long)
Copied to clipboard
Jiajun Zhang, Yuying Li, Zhixun Li, Xingyu Guo, Jingzhuo Wu, Leqi Zheng, Yiran Yang, Jianke Zhang, Qingbin Li, Shannan Yan, Changguo Jia, Junfei Wu, Zilei Wang, Qiang Liu, Liang Wang
| Challenge: | Vision-Language Models (VLMs) have demonstrated impressive capabilities in code generation across various domains, but their ability to replicate complex, multi-panel visualizations remains largely unassessed. |
| Approach: | They propose a large-scale benchmark to evaluate chart generation from large- scale raw data and assess iterative code refinement in a multi-turn conversational setting. |
| Outcome: | The new benchmark evaluates 14 leading VLMs on real-world data and shows they struggle with complex plot structures and authentic data. |
DrawEduMath: Evaluating Vision Language Models with Expert-Annotated Students’ Hand-Drawn Math Images (2025.naacl-long)
Copied to clipboard
| Challenge: | DrawEduMath examines the ability of vision language models to handle real-world math problems, such as those encountered in classrooms and tutoring sessions. |
| Approach: | They present DrawEduMath, an English-language dataset of 2,030 images of students’ handwritten responses to math problems. |
| Outcome: | The proposed model can be used to evaluate teachers' QA pairs and 44,362 synthetic QAs derived from teachers' descriptions. |
VIT-Pro: Visual Instruction Tuning for Product Images (2025.naacl-industry)
Copied to clipboard
Vishnu Prabhakaran, Purav Aggarwal, Vishruit Kulshreshtha, Arunita Das, Sahini Venkata Sitaram Sruti, Anoop Saladi
| Challenge: | general-purpose vision-language models struggle to understand and converse about real-world e-commerce product images. |
| Approach: | a new approach is proposed to use large-scale image-text pairs to train a generative VLM for e-commerce product images. |
| Outcome: | The proposed model outperforms general-purpose VLMs on multiple vision tasks in the e-commerce domain. |
On Advances in Text Generation from Images Beyond Captioning: A Case Study in Self-Rationalization (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Combining visual modality with pretrained language models has been effective for descriptive tasks such as image captioning. |
| Approach: | They ask: do multimodal models combine visual and visual adapted language models? they find that CLIP image representations and scaling of language models do not consistently improve self-rationalization in multimodal tasks. |
| Outcome: | The proposed model types do not consistently improve self-rationalization in multimodal tasks. |
TURTLEAI: Benchmarking Multimodal Models for Visual Programming in Turtle Graphics (2026.findings-acl)
Copied to clipboard
| Challenge: | Vision-language models have been explored for visual programming, but performance is unclear . most prior work focuses on visual programming for productivity . |
| Approach: | They propose a visual programming benchmark that uses visual programming to evaluate VLMs. |
| Outcome: | The proposed model improves on GPT-5, GPT-4o, and Qwen2-VL-72B on real-world tasks by 20% . the proposed model is based on 823 visual programming tasks in the Turtle Graphics domain . |
PlotGen-Bench: Evaluating VLMs on Generating Visualization Code from Diverse Plots across Multiple Libraries (2026.findings-acl)
Copied to clipboard
| Challenge: | PlotGen-Bench evaluates vision-language models' ability to generate executable visualization code from plots under realistic and complex visualization requirements. |
| Approach: | They propose a benchmark to evaluate plot-to-code generation in vision-language models . they use Matplot, Matplos, Mat3D, Mat4D, and Mat4E to evaluate their performance . |
| Outcome: | The proposed benchmark covers 9 major categories, 30 subcategories, and 3 core tasks . it covers 2D, 3D and animated plots across 5 widely used visualization libraries. |
On LLMs-Driven Synthetic Data Generation, Curation, and Evaluation: A Survey (2024.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) provide a data-centric solution to alleviate limitations of real-world data with synthetic data generation. |
| Approach: | They propose a generic workflow for LLM-driven synthetic data generation. |
| Outcome: | The proposed workflows highlight gaps in existing research and outline avenues for future studies. |
Visually-augmented pretrained language models for NLP tasks without images (2023.acl-long)
Copied to clipboard
| Challenge: | Existing approaches to improve pre-trained language models lack visual commonsense and semantics. |
| Approach: | They propose a visual-augmented approach to fine-tune pre-trained language models by using retrieved or generated images instead of relying on explicit images. |
| Outcome: | The proposed approach outperforms baselines on ten tasks and consistently outperformed other approaches. |