Papers by Jianrui Zhang
CounterCurate: Enhancing Physical and Semantic Visio-Linguistic Compositional Reasoning via Counterfactual Examples (2024.findings-acl)
Copied to clipboard
| Challenge: | Using high-performing text and image generation models, we can improve compositional reasoning capabilities for large multimodal models. |
| Approach: | They propose a framework to improve visio-linguistic compositional reasoning capability for multimodal models by leveraging physical grounding and using image-based models for semantic counterfactual fine-tuning. |
| Outcome: | The proposed framework outperforms GPT-4V and DALLE-3 on a new dataset and checkpoints. |
VGBench: A Comprehensive Benchmark of Vector Graphics Understanding and Generation for Large Language Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | Current vision models use pixels to rasterize the visual world, but vector graphics are not the best or unique way to represent visual content. |
| Approach: | They propose a benchmark for vector graphics processing with capable Large Language Models . they use a set of questions to evaluate vector graphics formats and a wide range of prompting techniques . |
| Outcome: | The proposed benchmark compares LLMs on rasterized representations with vector graphics . it shows that LLM models show strong capability on both aspects . |