CollagePrompt: A Benchmark for Budget-Friendly Visual Recognition with GPT-4V (2025.findings-naacl)
Copied to clipboard
| Challenge: | Recent advances in generative AI have suggested that by taking visual prompts, GPT-4V can demonstrate significant proficiency in visual recognition tasks. |
| Approach: | They propose a collage prompting task that collages multiple images into a single visual prompt and makes GPT-4V perform visual recognition on several images simultaneously. |
| Outcome: | The proposed task reduces the cost associated with GPT-4V's visual recognition . the proposed task group images of the same category together leads to better visual recognition results . |
Similar Papers
BeautifulPrompt: Towards Automatic Prompt Engineering for Text-to-Image Synthesis (2023.emnlp-industry)
Copied to clipboard
| Challenge: | Recent text-to-image models require multiple passes of prompt engineering by humans to produce satisfactory results for real-world applications. |
| Approach: | They propose a deep generative model to generate high-quality prompts from raw descriptions using visual feedback. |
| Outcome: | The proposed model produces high-quality prompts from simple raw descriptions . it can be integrated to a cloud-native AI platform to provide better image generation service in the cloud. |
TurtleBench: A Visual Programming Benchmark in Turtle Geometry (2025.naacl-long)
Copied to clipboard
| Challenge: | Large Multimodal Models (LMMs) are capable of reasoning about geometric patterns, but they are still a challenge to evaluate. |
| Approach: | They propose a benchmark to evaluate LMMs’ ability to interpret geometric patterns and generate precise code outputs. |
| Outcome: | The benchmark compared LMMs with turtle geometry and GPT-4V, and found that few-shot prompting improved their performance (2%). |
TURTLEAI: Benchmarking Multimodal Models for Visual Programming in Turtle Graphics (2026.findings-acl)
Copied to clipboard
| Challenge: | Vision-language models have been explored for visual programming, but performance is unclear . most prior work focuses on visual programming for productivity . |
| Approach: | They propose a visual programming benchmark that uses visual programming to evaluate VLMs. |
| Outcome: | The proposed model improves on GPT-5, GPT-4o, and Qwen2-VL-72B on real-world tasks by 20% . the proposed model is based on 823 visual programming tasks in the Turtle Graphics domain . |
VERITAS: Leveraging Vision Priors and Expert Fusion to Improve Multimodal Data (2025.emnlp-main)
Copied to clipboard
| Challenge: | supervised fine-tuning (SFT) data is critical for large multimodal models . current methods suffer from factual errors and hallucinations due to inadequate visual perception . |
| Approach: | They propose a pipeline that integrates vision priors and state-of-the-art LMMs with statistical methods to enhance SFT data quality. |
| Outcome: | The proposed pipeline outperforms existing models in text-rich reasoning tasks while enhancing reasoning efficiency. |
VisAidMath: Benchmarking Visual-Aided Mathematical Reasoning (2026.acl-long)
Copied to clipboard
| Challenge: | Existing Large Multi-modal Models lack a robust visual processing capability that is often masked by evaluation metrics that prioritize final-answer accuracy. |
| Approach: | They propose a three-layer evaluation framework that scrutinizes the generation of valid visual aids and the soundness of subsequent reasoning steps. |
| Outcome: | The proposed framework examines the generation of valid visual aids and the soundness of subsequent reasoning steps on state-of-the-art models. |
EffiVLM-BENCH: A Comprehensive Benchmark for Evaluating Training-Free Acceleration in Large Vision-Language Models (2025.acl-long)
Copied to clipboard
| Challenge: | Existing methods for accelerating Large Vision-Language Models lack comprehensive evaluation across diverse backbones, benchmarks, and metrics. |
| Approach: | They propose EffiVLM-BENCH framework for evaluating absolute performance and generalization and loyalty. |
| Outcome: | The proposed framework offers insights into optimal strategies for accelerating LVLMs. |
MultiCapCLIP: Auto-Encoding Prompts for Zero-Shot Multilingual Visual Captioning (2023.acl-long)
Copied to clipboard
| Challenge: | Existing methods for supervised visual captioning require large scale of images or videos paired with descriptions in a specific language. |
| Approach: | They propose a zero-shot approach that generates captions for different scenarios without labeling . they use concept prompts to retrieve concepts and auto-encode them to learn writing styles . |
| Outcome: | The proposed approach generates captions for different scenarios and languages without labeled vision-caption pairs. |
GPT-4V Cannot Generate Radiology Reports Yet (2025.findings-naacl)
Copied to clipboard
| Challenge: | Large language models (LLMs) are becoming multimodal, and GPT-4 models are supposed to possess advanced skills across a wide range of domains, including high-stakes scenarios such as medicine. |
| Approach: | They perform a systematic evaluation of GPT-4 in generating radiology reports across three chest X-ray report benchmarks: MIMIC-CXR, CheXpert Plus, and IU X ray. |
| Outcome: | The proposed model fails in lexical and clinical efficacy metrics . the distributions of model-predicted labels remain constant regardless of groundtruth conditions on the image, suggesting that the model is not interpreting chest X-rays meaningfully. |
GreaterPrompt: A Unified, Customizable, and High-Performing Open-Source Toolkit for Prompt Optimization (2025.acl-demo)
Copied to clipboard
| Challenge: | Recent advances in prompt optimization have introduced diverse techniques that automatically enhance prompts to better align model outputs with user expectations. |
| Approach: | They propose a framework that unifies different methods under a unified, customizable API while delivering highly effective prompts for different tasks. |
| Outcome: | The proposed framework unifies multiple methods under a unified, customizable API while delivering highly effective prompts for different tasks. |
Benchmarking GPT-4 on Algorithmic Problems: A Systematic Evaluation of Prompting Strategies (2024.lrec-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have revolutionized the field of natural language processing . however, it has been shown that they lack systematic generalization, which allows to extrapolate the learned statistical regularities outside the training distribution. |
| Approach: | They propose to benchmark a LLM with two parameters to find out its performance . they compare it to a variant of the Transformer-Encoder architecture to find the same problem . |
| Outcome: | The proposed model outperforms the previous model on three algorithmic tasks with two parameters. |