Retrieval-Augmented Fine-Tuning With Preference Optimization For Visual Program Generation (2025.acl-long)
Copied to clipboard
| Challenge: | Visual programming languages (VPLs) allow users to create programs through graphical interfaces, which results in easier accessibility and widespread usage in various domains. |
| Approach: | They propose to train VPLs from user instructions using large language models . they propose to use retrieval-augmented fine-tuning to leverage repetitive use of subroutines . |
| Outcome: | The proposed method outperforms prompting-based methods for LD generation accuracy even with smaller backbone models. |
Similar Papers
Integrating Symbolic Execution into the Fine-Tuning of Code-Generating LLMs (2025.naacl-srw)
Copied to clipboard
| Challenge: | Code-generating Large Language Models (LLMs) have become essential tools in modern software development, enhancing productivity and accelerating development. |
| Approach: | They propose to use Reinforcement Learning and Direct Preference Optimization to fine-tune code-generating Large Language Models (LLMs) by enhancing the training data with symbolic execution techniques. |
| Outcome: | The proposed model improves on the CodeRL benchmark and shows that it is more accurate and objective than the baseline model. |
Visually-augmented pretrained language models for NLP tasks without images (2023.acl-long)
Copied to clipboard
| Challenge: | Existing approaches to improve pre-trained language models lack visual commonsense and semantics. |
| Approach: | They propose a visual-augmented approach to fine-tune pre-trained language models by using retrieved or generated images instead of relying on explicit images. |
| Outcome: | The proposed approach outperforms baselines on ten tasks and consistently outperformed other approaches. |
Finer: Investigating and Enhancing Fine-Grained Visual Concept Recognition in Large Vision Language Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in instruction-tuned Large Vision-Language Models (LVLMs) have imbued the models with the ability to generate high-level, image-grounded explanations with ease. |
| Approach: | They propose to use a multiple granularity attribute-centric benchmark and training mixture to evaluate LVLMs’ fine-grained visual comprehension ability. |
| Outcome: | The proposed model improves on LLaVa-1.5, InstructBLIP and GPT-4V and demonstrates that they struggle to generate descriptive visual attributes based on a concept that appears within an input image despite their prominent zero-shot image captioning ability. |
A Rigorous Evaluation of LLM Data Generation Strategies for Low-Resource Languages (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are increasingly used to generate synthetic textual data for training smaller specialized models. |
| Approach: | They evaluate the performance of large language models and their generation strategies in 11 different languages using 3 NLP tasks and 4 open-source LLMs. |
| Outcome: | The proposed generation strategies and their combinations yield strong results across 11 languages, including several extremely low-resource ones. |
Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMs (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) encapsulate a vast amount of factual information within their pre-trained weights. |
| Approach: | They compare unsupervised fine-tuning and retrieval-augmented generation approaches to learning new factual information. |
| Outcome: | The proposed models outperform unsupervised fine-tuning and retrieval-augmented generation (RAG) on knowledge-intensive tasks across different topics. |
Modular and Parameter-Efficient Fine-Tuning for NLP Models (2022.emnlp-tutorials)
Copied to clipboard
| Challenge: | State-of-the-art language models in NLP perform best when fine-tuned even on small datasets. |
| Approach: | They provide an overview of parameter-efficient fine-tuning methods and highlight similarities and differences . they highlight benefits and usage scenarios of a neglected property of parameter efficient models . |
| Outcome: | This paper provides an overview of parameter-efficient fine-tuning methods . it highlights similarities and differences by presenting them in a unified view . |
CoDa: Constrained Generation based Data Augmentation for Low-Resource NLP (2024.findings-naacl)
Copied to clipboard
| Challenge: | a low-resource dataset is limited in training data, so generating task-specific data is challenging. |
| Approach: | They propose a data augmentation technique that prompts off-the-shelf instruction-following Large Language Models to generate augmentations. |
| Outcome: | The proposed technique outperforms baselines on 11 datasets spanning 3 tasks and 3 low-resource settings. |
ViFT: Towards Visual Instruction-Free Fine-tuning for Large Vision-Language Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Visual instruction tuning is the predominant technology in eliciting multimodal task-solving capabilities of large vision-language models. |
| Approach: | They propose a visual instruction-free fine-tuning framework for large vision-language models . they require only text-only instructions and image caption data during training . |
| Outcome: | The proposed framework is based on visual instruction tuning, but requires images as input . it can achieve state-of-the-art performance on several downstream benchmarks with less training data. |
Visual Program Distillation with Template-Based Augmentation (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Adapting visual programming to specialized tasks or domains remains challenging due to high annotation and inference costs. |
| Approach: | They propose a low-cost visual program distillation method that can be used for models with at most 1 billion parameters and requires no human-generated program annotations. |
| Outcome: | The proposed method can generate high-quality visual programs with no human-generated annotations with a relatively small amount of question/answer data. |
Fast Adaptation via Prompted Data: An Efficient Cross-Domain Fine-tuning Method for Large Language Models (2024.lrec-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) have been successful in a variety of natural language understanding tasks, but domain discrepancies between the downstream task and the pre-training corpora may have hindered LLMs to excel further in the vertical applications. |
| Approach: | They propose a Fast Adaptation method for LLMs via Prompted Data that integrates downstream text corpora, gold labels and external knowledge sources into a highly controllable prompt. |
| Outcome: | The proposed method bridges the gap between the downstream task and the pre-training corpora and integrates downstream text corpors, gold labels and external knowledge sources into a highly controllable prompt. |