CLIP Models are Few-Shot Learners: Empirical Studies on VQA and Visual Entailment (2022.acl-long)
Copied to clipboard
| Challenge: | Previously, CLIP was only regarded as a powerful visual encoder. |
| Approach: | They propose a parameter-efficient fine-tuning strategy to boost CLIP's few-shot performance on a visual entailment task without introducing any additional pre-training procedure. |
| Outcome: | The proposed strategy achieves competitive zero/few-shot results on visual question answering and visual entailment tasks without introducing any additional pre-training procedure. |
Similar Papers
CLIPText: A New Paradigm for Zero-shot Text Classification (2023.findings-acl)
Copied to clipboard
| Challenge: | Experimental results show that CLIP can be applied to zero-shot text classification tasks. |
| Approach: | They propose a CLIP model for zero-shot text classification that integrates prompt into CLIPText to better derive knowledge from CLIP. |
| Outcome: | The proposed model can be applied to a text-image matching problem and show that it can be used for language tasks. |
UniFine: A Unified and Fine-grained Approach for Zero-shot Vision-Language Understanding (2023.findings-acl)
Copied to clipboard
| Challenge: | supervised methods for vision-language tasks have been well-studied, but they lack the fine-grained information needed for semantics understanding. |
| Approach: | They propose a framework to take advantage of fine-grained information for zero-shot vision-language learning, covering multiple tasks such as VQA, SNLI-VE, and VCR. |
| Outcome: | The proposed framework outperforms previous zero-shot methods on VQA and achieves substantial improvement on SNLI-VE and VCR. |
Preserving Multi-Modal Capabilities of Pre-trained VLMs for Improving Vision-Linguistic Compositionality (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing fine-tuning approaches for compositional understanding compromise performance in zero-shot multi-modal tasks. |
| Approach: | They propose a method to enhance compositional understanding in pre-trained vision and language models without sacrificing performance in zero-shot multi-modal tasks. |
| Outcome: | The proposed method achieves compositionality on par with state-of-the-art models and retains strong multi-modal capabilities. |
Exploring How Generative MLLMs Perceive More Than CLIP with the Same Vision Encoder (2025.acl-long)
Copied to clipboard
| Challenge: | Recent studies show that CLIP models struggle with visual reasoning tasks . despite the success of Contrastive Language-Image Pretraining, there are still limitations . |
| Approach: | They propose to use a visual encoder to train CLIP-like models for fine-grained visual reasoning tasks. |
| Outcome: | The proposed models outperform CLIP-like encoders in visual reasoning tasks . the study highlights the importance of VLM architectural choices . |
PubMedCLIP: How Much Does CLIP Benefit Visual Question Answering in the Medical Domain? (2023.findings-eacl)
Copied to clipboard
| Challenge: | Medical visual question answering is a multimodal task that requires a system to understand both medical images and textual questions and infer associations between them. |
| Approach: | They propose a fine-tuned version of CLIP for the medical domain based on PubMed articles. |
| Outcome: | The proposed model improves accuracy up to 3% on two MedVQA benchmark datasets. |
Z-LaVI: Zero-Shot Language Solver Fueled by Visual Imagination (2022.emnlp-main)
Copied to clipboard
| Challenge: | Large-scale pretrained language models suffer from reporting bias, describing the lack of explicit commonsense knowledge in written text. |
| Approach: | They propose to endow language models with visual imagination capabilities by recalling existing images and synthesizing nonexistent images via text-to-image generation. |
| Outcome: | The proposed model improves the performance of existing language models across a diverse set of language tasks. |
FewshotQA: A simple framework for few-shot learning of question answering tasks using pre-trained text-to-text models (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing pre-trained models need fine-tuning on tens of thousands of examples to achieve good results. |
| Approach: | They propose a framework that leverages pre-trained text-to-text models and aligns them with their pre-training framework. |
| Outcome: | The proposed framework outperforms the XLM-Roberta-large on multiple QA benchmarks and is applicable to multilingual situations. |
Multilingual Multimodal Pre-training for Zero-Shot Cross-Lingual Transfer of Vision-Language Models (2021.naacl-main)
Copied to clipboard
| Challenge: | a new study examines zero-shot cross-lingual transfer of vision-language models . we study multilingual text-to-video search in non-English languages without annotations . |
| Approach: | They propose a Transformer-based model that learns contextual multilingual multimodal embeddings . they propose 'zero-shot cross-lingual transfer' to improve multilingual search . |
| Outcome: | The proposed model outperforms baselines on multilingual text-to-video search and multilingual image search on VTT and VATEX. |
Few-shot Learning with Multilingual Generative Language Models (2022.emnlp-main)
Copied to clipboard
Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu Wang, Shuohui Chen, Daniel Simig, Myle Ott, Naman Goyal, Shruti Bhosale, Jingfei Du, Ramakanth Pasunuru, Sam Shleifer, Punit Singh Koura, Vishrav Chaudhary, Brian O’Horo, Jeff Wang, Luke Zettlemoyer, Zornitsa Kozareva, Mona Diab, Veselin Stoyanov, Xian Li
| Challenge: | Large-scale generative language models such as GPT-3 are competitive few-shot learners. |
| Approach: | They train multilingual generative language models on a corpus covering a diverse set of languages and study their few- and zero-shot learning capabilities. |
| Outcome: | The proposed model outperforms GPT-3 on 171 out of 182 directions with 32 training examples and surpasses the official supervised baseline in 45 directions. |
SERVAL: Surprisingly Effective Zero-Shot Visual Document Retrieval Powered by Large Vision and Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Visual Document Retrieval (VDR) relies on text-to-image retrieval using specialized bi-encoders . et al., 2022, 2024, 2021, 2023, 2026, 2030, 2040, 2050, 2060) document retrieval bridges human or artificial agents to the most relevant information, authors say . |
| Approach: | They propose a zero-shot visual document retrieval method that uses bi-encoders . they propose 63.4% nDCG@5 for visual document capture and a reusable semantic proxy . |
| Outcome: | The proposed method surpasses the strongest specialised multi-vector visual document encoder on the ViDoRe-v2 benchmark and scales similarly on MIRACL-VISION with broader multilingual coverage. |