SignCLIP: Connecting Text and Sign Language by Contrastive Learning (2024.emnlp-main)
Copied to clipboard
| Challenge: | SignCLIP is an efficient method of learning useful visual representations for sign language processing from large-scale, multilingual video-text pairs without optimizing for a specific task or sign language of limited size. |
| Approach: | They propose a method for learning visual representations for sign language processing from large-scale video-text pairs without directly optimizing for a specific task or sign language. |
| Outcome: | The proposed model can learn from multilingual video-text pairs without optimizing for a specific task or sign language of limited size. |
Similar Papers
VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understanding (2021.emnlp-main)
Copied to clipboard
Hu Xu, Gargi Ghosh, Po-Yao Huang, Dmytro Okhonko, Armen Aghajanyan, Florian Metze, Luke Zettlemoyer, Christoph Feichtenhofer
| Challenge: | Recent work adopts a "pre-training + fine-tuning" approach for zero-shot transfer to end tasks without fine- tuning. |
| Approach: | They propose a contrastive approach to pre-train a transformer model for zero-shot video and text understanding without using any labels on downstream tasks. |
| Outcome: | The proposed model outperforms supervised approaches on downstream tasks and outperformed previous approaches. |
AltCLIP: Altering the Language Encoder in CLIP for Extended Language Capabilities (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to build a strong multilingual multimodal representation model are lacking in good-quality text-image pairs. |
| Approach: | They propose a method to build a strong multilingual multimodal representation model using English text-image pairs instead of a model from scratch. |
| Outcome: | The proposed model outperforms the original CLIP model on multilingual multimodal benchmarks. |
MedCLIP: Contrastive Learning from Unpaired Medical Images and Text (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing vision-text contrastive learning methods encounter many false negatives, i.e., images and reports from separate patients probably carry the same semantics but are wrongly treated as negatives. |
| Approach: | They propose to decouple medical image-text contrastive learning and replace it with semantic matching loss based on medical knowledge to eliminate false negatives in contrastive training. |
| Outcome: | The proposed framework outperforms state-of-the-art methods on zero-shot prediction, supervised classification, and image-text retrieval with only 20K pre-training data. |
XtremeCLIP: Extremely Parameter-efficient Tuning for Low-resource Vision Language Understanding (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to fine-tune visual-language understanding (VLU) require tasks-specific designs and sufficient training data. |
| Approach: | They propose a simple yet efficient paradigm for low-resource Visual Language Understanding (VLU) they reformulate a series of VLU tasks as an open-book affinity-matching problem. |
| Outcome: | The proposed framework outperforms baselines in low-resource settings. |
Segment, Embed, and Align: A Universal Recipe for Aligning Subtitles to Signing (2026.acl-long)
Copied to clipboard
| Challenge: | Existing approaches for aligning spoken language text to sign language videos rely on end-to-end training tied to a specific language or dataset. |
| Approach: | They propose a universal approach for aligning spoken language text with corresponding timestamps to sign language videos using a lightweight dynamic programming procedure. |
| Outcome: | The proposed method can be used on four sign language datasets and is highly efficient on CPU. |
OpenHands: Making Sign Language Recognition Accessible with Pose-based Pretrained Models across Languages (2022.acl-long)
Copied to clipboard
| Challenge: | a new study examines the performance of pretraining for sign language recognition in low-resource settings. |
| Approach: | They propose using pose extracted through pretrained models as the standard modality of data to reduce training time and enable efficient inference. |
| Outcome: | The proposed model reduces training time and allows efficient inference in sign languages. |
Cross-lingual and Multilingual CLIP (2022.lrec-1)
Copied to clipboard
| Challenge: | OpenAI released CLIP, a model that relates the textual and visual domains with unprecedented accuracy. |
| Approach: | They propose to use cross-lingual teacher learning to re-train an English textual encoder using a large dataset of images and captions. |
| Outcome: | The proposed method outperforms baselines on multilingual image-text retrieval while retaining low cost. |
Selective Contrastive Learning For Gloss Free Sign Language Translation (2026.acl-long)
Copied to clipboard
| Challenge: | Recent SLT systems adopt CLIP-like Vision-Language pretraining, but the random in-batch contrast provides few, batch-dependent negatives. |
| Approach: | They propose a method to train sign video-text similarity over a time period of 3 months . they use a random in-batch contrast strategy to track negative video- text similarity . |
| Outcome: | The proposed system improves sign language translation by focusing on challenging negatives . the results show that the random in-batch contrast provides few negatives and noisy supervision . |
Sign-Language Datasets at Scale: A Comprehensive Survey on Resources, Benchmarks, and Annotation Standards (2026.acl-long)
Copied to clipboard
| Challenge: | Existing benchmarks fail to reflect real-world communication needs and are limited in their coverage. |
| Approach: | They present a comprehensive index of sign-language datasets, covering 120 resources across 35 sign languages. |
| Outcome: | The proposed index covers 120 resources across 35 sign languages. |
TaCL: Improving BERT Pre-training with Token-aware Contrastive Learning (2022.findings-naacl)
Copied to clipboard
| Challenge: | Existing pre-trained MLMs produce an anisotropic distribution of token representations . this is not ideal for tasks that require discriminative semantic meanings of distinct tokens - a problem that exists in pre-training models . |
| Approach: | They propose a continual pre-training approach that encourages BERT to learn an isotropic distribution of token representations. |
| Outcome: | The proposed approach improves on a wide range of English and Chinese benchmarks. |