Visual Program Distillation with Template-Based Augmentation (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Adapting visual programming to specialized tasks or domains remains challenging due to high annotation and inference costs. |
| Approach: | They propose a low-cost visual program distillation method that can be used for models with at most 1 billion parameters and requires no human-generated program annotations. |
| Outcome: | The proposed method can generate high-quality visual programs with no human-generated annotations with a relatively small amount of question/answer data. |
Similar Papers
Modular Visual Question Answering via Code Generation (2023.acl-short)
Copied to clipboard
Sanjay Subramanian, Medhini Narasimhan, Kushal Khangaonkar, Kevin Yang, Arsha Nagrani, Cordelia Schmid, Andy Zeng, Trevor Darrell, Dan Klein
| Challenge: | a framework for visual question answering is based on modular code generation . the scope of reasoning needed for visual questions is vast, and requires many skills . |
| Approach: | They propose a framework that formulates visual question answering as modular code generation. |
| Outcome: | The proposed framework improves accuracy on COVR and GQA datasets by 3% and 2% compared to the few-shot baseline that does not employ code generation. |
From the Least to the Most: Building a Plug-and-Play Visual Reasoner via Data Synthesis (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing models consisting of multiple steps of visual and language processing are limited in the visual and visual processing community . a visual reasoner is a plug-and-play approach that can be used to improve VLMs' reasoning abilities. |
| Approach: | They propose a least-to-most visual reasoning paradigm that divides a question into sub-questions and invokes external tools for resolving sub-questions. |
| Outcome: | The proposed method can improve four VLMs on four VQA benchmarks. |
Enhancing Code Generation Performance of Smaller Models by Distilling the Reasoning Ability of LLMs (2024.lrec-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have made significant advances in code generation through the ‘Chain-of-Thought’ prompting technique. |
| Approach: | They propose a framework which aims to transfer LLMs’ reasoning capabilities to smaller models through distillation. |
| Outcome: | The proposed framework improves the smaller model's code generation performance by over 130% on the APPS benchmark. |
Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes (2023.findings-acl)
Copied to clipboard
Cheng-Yu Hsieh, Chun-Liang Li, Chih-kuan Yeh, Hootan Nakhost, Yasuhisa Fujii, Alex Ratner, Ranjay Krishna, Chen-Yu Lee, Tomas Pfister
| Challenge: | Deploying large language models (LLMs) is difficult because they are memory inefficient and compute-intensive for practical applications. |
| Approach: | They propose a mechanism that fine tunes or distills small models that outperform LLMs . they use human labels to fine tune models or LLM-generated labels to train models . |
| Outcome: | The proposed method outperforms LLMs by using fewer training examples compared to few-shot prompted models using substantially smaller model sizes. |
Making VLMs More Robot-Friendly: Self-Critical Distillation of Low-Level Procedural Reasoning (2025.emnlp-main)
Copied to clipboard
Chan Young Park, Jillian Fisher, Marius Memmel, Dipika Khullar, Seoho Yun, Abhishek Gupta, Yejin Choi
| Challenge: | Large language models (LLMs) are human-centric, but omit low-level, spatially grounded details needed for robotic execution. |
| Approach: | They propose a lightweight framework for vision-language procedural planning that enables iteratively critique, revise and verify their own plans without external supervision or teacher models. |
| Outcome: | a new framework outperforms weaker models 100X the size in vision-language procedural planning . the framework generates higher-quality, execution-ready plans that can be used at inference and fine-tuning . |
Retrieval-Augmented Fine-Tuning With Preference Optimization For Visual Program Generation (2025.acl-long)
Copied to clipboard
| Challenge: | Visual programming languages (VPLs) allow users to create programs through graphical interfaces, which results in easier accessibility and widespread usage in various domains. |
| Approach: | They propose to train VPLs from user instructions using large language models . they propose to use retrieval-augmented fine-tuning to leverage repetitive use of subroutines . |
| Outcome: | The proposed method outperforms prompting-based methods for LD generation accuracy even with smaller backbone models. |
Effective Distillation of Table-based Reasoning Ability from LLMs (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing work on table-based reasoning distillation has focused on smaller models with limited performance. |
| Approach: | They propose a table-based reasoning distillation approach to distill LLMs into smaller models . their results show that a 220 million parameter model fine-tuned using distilled data improves performance . |
| Outcome: | The proposed model improves on a scientific table-to-text generation dataset and surpasses specific LLMs. |
Distill Visual Chart Reasoning Ability from LLMs to MLLMs (2025.findings-emnlp)
Copied to clipboard
Wei He, Zhiheng Xi, Wanxu Zhao, Xiaoran Fan, Yiwen Ding, Zifei Shan, Tao Gui, Qi Zhang, Xuanjing Huang
| Challenge: | a new method for generating chart annotations is proposed to improve visual reasoning in multimodal large language models. |
| Approach: | They propose a code-as-intermediary translation method for distilling visual reasoning abilities from LLMs to MLLMs. |
| Outcome: | The proposed method is cost-effective, efficient and scalable. |
AQuilt: Weaving Logic and Self-Inspection into Low-Cost, High-Relevance Data Synthesis for Specialist LLMs (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to synthesis large language models often suffer from performance limitations and high computational costs. |
| Approach: | They propose a framework for constructing instruction-tuning data from unlabeled data for any specialized domains from corresponding unlabed data. |
| Outcome: | The proposed framework is comparable to DeepSeek-V3 while utilizing just 17% of the production cost. |
World to Code: Multi-modal Data Generation via Self-Instructed Compositional Captioning and Filtering (2024.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in Vision-Language Models and the scarcity of high-quality multi-modal alignment data have inspired numerous researches on synthetic VLM data generation. |
| Approach: | They propose a multi-modal data construction pipeline that organizes the final output into a Python code format. |
| Outcome: | The proposed pipeline improves visual question answering and visual grounding benchmarks across different VLMs. |