| Challenge: | Existing approaches to learn discriminative features using contrastive objective are lacking. |
| Approach: | They propose a self-supervised framework that leverages a contrastive loss directly at the level of self-attention. |
| Outcome: | The proposed framework outperforms all comparable unsupervised approaches while occasionally surpassing supervised ones. |
Similar Papers
Contrastive Self-Supervised Learning for Commonsense Reasoning (2020.acl-main)
Copied to clipboard
| Challenge: | Existing methods for commonsense reasoning are limited by current methods . empirical results show that our method alleviates the limitation of current supervised approaches . |
| Approach: | They propose a self-supervised method to solve pronoun disambiguation problems . they leverage a mutual exclusive loss regularized by a contrastive margin to achieve commonsense reasoning . |
| Outcome: | The proposed method performs well on many NLP benchmarks. |
An End-to-End Contrastive Self-Supervised Learning Framework for Language Understanding (2022.tacl-1)
Copied to clipboard
| Challenge: | Existing approaches to learning data representations using contrastive learning perform data augmentation and contrastive training separately. |
| Approach: | They propose a framework that performs data augmentation and contrastive learning end-to-end . they propose to combine data augmented with text encoders to optimize for contrastive training . |
| Outcome: | Experiments on GLUE and Gururangan datasets show the proposed framework is effective in NLP. |
Progressive Self-Supervised Attention Learning for Aspect-Level Sentiment Analysis (P19-1)
Copied to clipboard
| Challenge: | Experimental results show that our proposed approach yields better attention mechanisms . dominant ASC models are mostly discriminative classifiers based on manual feature engineering . |
| Approach: | They propose a self-supervised approach to aspect-level sentiment classification that mines useful attention supervision information from a training corpus to refine attention mechanisms. |
| Outcome: | The proposed approach yields better attention mechanisms on multiple datasets. |
Contrastive Data and Learning for Natural Language Processing (2022.naacl-tutorials)
Copied to clipboard
| Challenge: | Current NLP models heavily rely on effective representation learning algorithms. |
| Approach: | This tutorial introduces contrastive learning and provides an introduction to the techniques. |
| Outcome: | This tutorial provides an introduction to the fundamentals of contrastive learning approaches and the theory behind them. |
Don’t Judge a Language Model by Its Last Layer: Contrastive Learning with Layer-Wise Attention Pooling (2022.coling-1)
Copied to clipboard
| Challenge: | Recent pre-trained language models (PLMs) have shown competitive performance on many natural language processing tasks. |
| Approach: | They propose a pooling strategy which preserves layer-wise signals captured in each layer and learns digested linguistic features for downstream tasks. |
| Outcome: | The proposed method improves on standard semantic textual similarity and semantic search tasks. |
Adversarial Self-Supervised Learning for Out-of-Domain Detection (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing methods for detecting out-of-domain (OOD) intents are unsupervised and require extensive labeled data. |
| Approach: | They propose a self-supervised contrastive learning framework to model discriminative semantic features from unlabeled data. |
| Outcome: | The proposed framework outperforms baseline methods on two public benchmark datasets with a statistically significant margin. |
SCD: Self-Contrastive Decorrelation of Sentence Embeddings (2022.acl-short)
Copied to clipboard
| Challenge: | Existing methods for self-supervised learning of representations are based on contrastive learning. |
| Approach: | They propose a self-supervised approach that optimizes a joint decorrelation and self-contrastive objective by leveraging the contrast arising from standard dropout at different rates. |
| Outcome: | The proposed method achieves comparable results with state-of-the-art methods on multiple benchmarks without using contrastive pairs. |
Guiding Attention for Self-Supervised Learning with Transformers (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Recent studies show that self-attention patterns in trained models contain a majority of non-linguistic regularities. |
| Approach: | They propose a technique to allow efficient self-supervised learning with bi-directional Transformers by using an auxiliary loss function to guide attention heads to conform to such patterns. |
| Outcome: | The proposed method achieves state-of-the-art in low-resource settings and is agnostic to pre-training objectives. |
A Simple Angle-based Approach for Contrastive Learning of Unsupervised Sentence Representation (2024.findings-emnlp)
Copied to clipboard
| Challenge: | a promising baseline SimCSE has made notable breakthroughs in unsupervised SRL . however, there is still room for designing a novel contrastive framework specifically targeted for SRL. |
| Approach: | They propose an angle-based similarity function for a contrastive objective and propose a new approach for SRL. |
| Outcome: | The proposed approach shows better training dynamics on SRL than the standard cosine similarity function. |
Domain Confused Contrastive Learning for Unsupervised Domain Adaptation (2022.naacl-main)
Copied to clipboard
| Challenge: | Existing studies on domain-shifting adaptations have focused on domain . |
| Approach: | They propose a self-supervised approach to unsupervised domain adduction using domain puzzles to bridge the source and target domains and retain discriminative representations after adaptation. |
| Outcome: | The proposed approach outperforms baselines and further ablation studies show that it is more stable and effective when performing other data augmentations. |