Getting More Juice Out of Your Data: Hard Pair Refinement Enhances Visual-Language Models Without Extra Data (2025.naacl-long)
Copied to clipboard
Haonan Wang, Minbin Huang, Runhui Huang, Lanqing Hong, Hang Xu, Tianyang Hu, Xiaodan Liang, Zhenguo Li, Hong Cheng, Kenji Kawaguchi
| Challenge: | Contrastive Language-Image Pre-training (CLIP) is a standard for cross-modal image-text representation learning. |
| Approach: | They propose a framework that enhances pre-trained CLIP models by exploiting challenging text-image pairs within existing datasets. |
| Outcome: | The proposed framework improves CLIP models by exploiting text-image pairs in training. |
Similar Papers
Pre-training CLIP against Data Poisoning with Optimal Transport-based Matching and Alignment (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies have shown that Contrastive Language-Image Pre-training (CLIP) models are vulnerable to data poisoning and backdoor attacks due to massive training image-caption pairs crawled from the Internet. |
| Approach: | They propose an Optimal Transport-based framework to reconstruct image-caption pairs and propose an optimal transport-based distance measure to re-assign new captions based on the proposed optimal transport distance. |
| Outcome: | The proposed framework reduces the attack success rates of poisoning attacks to 0% in most cases. |
AltCLIP: Altering the Language Encoder in CLIP for Extended Language Capabilities (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to build a strong multilingual multimodal representation model are lacking in good-quality text-image pairs. |
| Approach: | They propose a method to build a strong multilingual multimodal representation model using English text-image pairs instead of a model from scratch. |
| Outcome: | The proposed model outperforms the original CLIP model on multilingual multimodal benchmarks. |
ComCLIP: Training-Free Compositional Image and Text Matching (2024.naacl-long)
Copied to clipboard
| Challenge: | erroneous semantics of individual entities are essentially confounders that cause the matching failure. |
| Approach: | They propose a training-free compositional CLIP model which disentangles input images into subjects, objects, and action subimages and composes CLIP’s vision encoder and text encoder to perform evolving matching over compositional text embedding and subimage embeddments. |
| Outcome: | The proposed model mitigates spurious correlations introduced by the pretrained CLIP models and dynamically evaluates the importance of each component. |
mCLIP: Multilingual CLIP via Cross-lingual Transfer (2023.acl-long)
Copied to clipboard
Guanhua Chen, Lu Hou, Yun Chen, Wenliang Dai, Lifeng Shang, Xin Jiang, Qun Liu, Jia Pan, Wenping Wang
| Challenge: | Existing multilingual vision-language pretrained models are biased towards English due to the lack of sufficient non-English image-text pairs. |
| Approach: | They propose to train a retrieval-efficient dual-stream multilingual VLP model by aligning CLIP model and a multilingual text encoder through a novel Triangle Cross-modal Knowledge Distillation method. |
| Outcome: | Empirical results show that mCLIP achieves new state-of-the-art performance for both zero-shot and finetuned multilingual image-text retrieval tasks. |
RWKV-CLIP: A Robust Vision-Language Representation Learner (2024.emnlp-main)
Copied to clipboard
| Challenge: | Using large image-text datasets, large-scale image-data sets have been used for visionlanguage pre-training. |
| Approach: | They propose a framework that leverages Large Language Models to combine and refine information from web-based image-text pairs, synthetic captions, and detection tags. |
| Outcome: | The proposed framework can combine and refine information from web-based image-text pairs, synthetic captions, and detection tags. |
Cross-lingual and Multilingual CLIP (2022.lrec-1)
Copied to clipboard
| Challenge: | OpenAI released CLIP, a model that relates the textual and visual domains with unprecedented accuracy. |
| Approach: | They propose to use cross-lingual teacher learning to re-train an English textual encoder using a large dataset of images and captions. |
| Outcome: | The proposed method outperforms baselines on multilingual image-text retrieval while retaining low cost. |
RelCLIP: Adapting Language-Image Pretraining for Visual Relationship Detection via Relational Contrastive Learning (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing visual relationship detection models only use numeric ids of relation labels for training, but ignore semantic correlation between labels. |
| Approach: | They propose a visual Relationship prediction framework that transfers natural language knowledge from Contrastive Language-Image Pre-training models to enhance the relationship prediction. |
| Outcome: | The proposed framework improves visual relationship prediction by matching semantic correlations with relation triplets. |
Preserving Multi-Modal Capabilities of Pre-trained VLMs for Improving Vision-Linguistic Compositionality (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing fine-tuning approaches for compositional understanding compromise performance in zero-shot multi-modal tasks. |
| Approach: | They propose a method to enhance compositional understanding in pre-trained vision and language models without sacrificing performance in zero-shot multi-modal tasks. |
| Outcome: | The proposed method achieves compositionality on par with state-of-the-art models and retains strong multi-modal capabilities. |
Delving into the Openness of CLIP (2023.findings-acl)
Copied to clipboard
| Challenge: | Contrastive Language-Image Pre-training (CLIP) allows for open-vocabulary visual recognition, where the model can recognize images from an open class set in a zero-shot manner. |
| Approach: | They propose to use image classification as an image-to-text matching task instead of discrete category IDs to achieve open-vocabulary visual recognition. |
| Outcome: | The proposed model can recognize images from an open vocabulary in a zero-shot manner, but its performance deteriorates as the vocabulary expands. |
CLIP-UP: A Simple and Efficient Mixture-of-Experts CLIP Training Recipe with Sparse Upcycling (2025.findings-emnlp)
Copied to clipboard
Xinze Wang, Chen Chen, Yinfei Yang, Hong-You Chen, Bowen Zhang, Aditya Pal, Xiangxin Zhu, Xianzhi Du
| Challenge: | Mixture-of-Experts (MoE) models are crucial for scaling model capacity while controlling inference costs. |
| Approach: | They propose an alternative training strategy that converts a dense CLIP model into a sparse MoE architecture. |
| Outcome: | The proposed training strategy outperforms dense models on COCO and Flickr30k benchmarks. |