microCLIP: Unsupervised CLIP Adaptation via Coarse-Fine Token Fusion for Fine-Grained Image Classification (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing UA methods for fine-grained image classification rely on coarse-grain visual tokens, which misses fine spatial details. |
| Approach: | They propose a label-free self-training framework that adapts visual features and LLMderived text prototypes using fine-grained cues. |
| Outcome: | The proposed framework improves alignment between finegrained visual regions and rich textual descriptions while updating only layer norms and a tiny head. |
Similar Papers
ComCLIP: Training-Free Compositional Image and Text Matching (2024.naacl-long)
Copied to clipboard
| Challenge: | erroneous semantics of individual entities are essentially confounders that cause the matching failure. |
| Approach: | They propose a training-free compositional CLIP model which disentangles input images into subjects, objects, and action subimages and composes CLIP’s vision encoder and text encoder to perform evolving matching over compositional text embedding and subimage embeddments. |
| Outcome: | The proposed model mitigates spurious correlations introduced by the pretrained CLIP models and dynamically evaluates the importance of each component. |
FineLAP: Taming Heterogeneous Supervision for Fine-grained Language-Audio Pretraining (2026.acl-long)
Copied to clipboard
| Challenge: | Existing audio-language models excel at clip-level understanding but struggle with frame-level tasks. |
| Approach: | They propose a novel training paradigm that advances both clip- and frame-level alignment in CLAP with heterogeneous data. |
| Outcome: | The proposed training paradigm improves both clip- and frame-level alignment in CLAP with heterogeneous data. |
Enhancing Fine-Grained Image Classifications via Cascaded Vision Language Models (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Fine-grained image classification is a challenge for vision-language models (VLMs) such as CLIP, which struggle to distinguish between semantically similar classes due to insufficient supervision for fine-grain tasks. |
| Approach: | They propose a framework that harnesses the complementary strengths of both CLIP-like and LVLMs to tackle these challenges. |
| Outcome: | The proposed framework outperforms existing models on multiple fine-grained datasets, particularly the Stanford Cars dataset. |
CLIP-MoE: Towards Building Mixture of Experts for CLIP with Diversified Multiplet Upcycling (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies found that CLIP can only encode one aspect of the feature space, leading to substantial information loss and indistinctive features. |
| Approach: | They propose a model-agnostic approach that fine-tunes complementary CLIP models and transforms them into a CLIP-MoE. |
| Outcome: | The proposed framework fine-tunes a series of complementary CLIP models and transforms them into a CLIP-MoE. |
AltCLIP: Altering the Language Encoder in CLIP for Extended Language Capabilities (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to build a strong multilingual multimodal representation model are lacking in good-quality text-image pairs. |
| Approach: | They propose a method to build a strong multilingual multimodal representation model using English text-image pairs instead of a model from scratch. |
| Outcome: | The proposed model outperforms the original CLIP model on multilingual multimodal benchmarks. |
XtremeCLIP: Extremely Parameter-efficient Tuning for Low-resource Vision Language Understanding (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to fine-tune visual-language understanding (VLU) require tasks-specific designs and sufficient training data. |
| Approach: | They propose a simple yet efficient paradigm for low-resource Visual Language Understanding (VLU) they reformulate a series of VLU tasks as an open-book affinity-matching problem. |
| Outcome: | The proposed framework outperforms baselines in low-resource settings. |
African or European Swallow? Benchmarking Large Vision-Language Models for Fine-Grained Object Classification (2024.emnlp-main)
Copied to clipboard
| Challenge: | Recent Large Vision Language Models demonstrate impressive abilities on image understanding and reasoning tasks. |
| Approach: | They propose a benchmark for fine-grained object classification that is difficult to evaluate . they benchmark 12 public LVLMs on and show CLIP models exhibit better performance . |
| Outcome: | The proposed model improves on 12 public LVLMs on image understanding and reasoning tasks. |
Fine-grained Image Captioning with CLIP Reward (2022.findings-naacl)
Copied to clipboard
| Challenge: | Modern image captioning models are usually trained with text similarity objectives . reference captions often describe only the most salient objects in images . |
| Approach: | They propose to use CLIP to calculate multi-modal similarity and use it as a reward function . they propose a simple finetuning strategy to improve grammar that does not require extra text annotation. |
| Outcome: | The proposed model generates more distinctive captions than the CIDEroptimized model on text-to-image retrieval and fineCapEval. |
VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understanding (2021.emnlp-main)
Copied to clipboard
Hu Xu, Gargi Ghosh, Po-Yao Huang, Dmytro Okhonko, Armen Aghajanyan, Florian Metze, Luke Zettlemoyer, Christoph Feichtenhofer
| Challenge: | Recent work adopts a "pre-training + fine-tuning" approach for zero-shot transfer to end tasks without fine- tuning. |
| Approach: | They propose a contrastive approach to pre-train a transformer model for zero-shot video and text understanding without using any labels on downstream tasks. |
| Outcome: | The proposed model outperforms supervised approaches on downstream tasks and outperformed previous approaches. |
RWKV-CLIP: A Robust Vision-Language Representation Learner (2024.emnlp-main)
Copied to clipboard
| Challenge: | Using large image-text datasets, large-scale image-data sets have been used for visionlanguage pre-training. |
| Approach: | They propose a framework that leverages Large Language Models to combine and refine information from web-based image-text pairs, synthetic captions, and detection tags. |
| Outcome: | The proposed framework can combine and refine information from web-based image-text pairs, synthetic captions, and detection tags. |