Neighborhood Contrastive Learning for Scientific Document Representations with Citation Embeddings (2022.emnlp-main)
Copied to clipboard
| Challenge: | Prior work relies on discrete citation relations to generate contrast samples, but discrete ones enforce a hard cut-off to similarity. |
| Approach: | They propose to use nearest neighbor sampling to learn continuous similarity and to sample hard-to-learn negatives and positives by controlling the sampling margin between them. |
| Outcome: | The proposed method outperforms the state-of-the-art on the SciDocs benchmark and can train (or tune) language models sample-efficiently. |
Similar Papers
Contrastive Data and Learning for Natural Language Processing (2022.naacl-tutorials)
Copied to clipboard
| Challenge: | Current NLP models heavily rely on effective representation learning algorithms. |
| Approach: | This tutorial introduces contrastive learning and provides an introduction to the techniques. |
| Outcome: | This tutorial provides an introduction to the fundamentals of contrastive learning approaches and the theory behind them. |
Composition-contrastive Learning for Sentence Embeddings (2023.acl-long)
Copied to clipboard
| Challenge: | Recent work shows potential to learn vector representations from unlabelled data without task-specific fine-tuning. |
| Approach: | They propose to maximize alignment between textual embeddings and a composition of their phrasal constituents. |
| Outcome: | The proposed approach improves on similarity tasks comparable to state-of-the-art approaches. |
Virtual Augmentation Supported Contrastive Learning of Sentence Representations (2022.findings-acl)
Copied to clipboard
| Challenge: | Despite profound successes, contrastive representation learning relies on carefully designed data augmentations using domain-specific knowledge. |
| Approach: | They propose a virtual augmentation supported Contrastive Learning of sentence representations . they approximate the neighborhood of an instance via its K-nearest in-batch neighbors . |
| Outcome: | The proposed model outperforms existing methods on a wide range of downstream tasks. |
Debiased Contrastive Learning of Unsupervised Sentence Representations (2022.acl-long)
Copied to clipboard
| Challenge: | Recent studies have shown that contrastive learning improves pre-trained language models to derive high-quality sentence representations. |
| Approach: | They propose a framework to punish false negatives and generate noise-based negatives to guarantee the uniformity of the representation space. |
| Outcome: | The proposed framework improves pre-trained language models while pushing apart irrelevant negatives to guarantee the uniformity of the representation space. |
Fine-grained Contrastive Learning for Definition Generation (2022.aacl-main)
Copied to clipboard
| Challenge: | Recent pre-trained transformer-based definition generation models lack effective representation learning to contain full semantic components of the given word, leading to under-specific definitions. |
| Approach: | They propose a novel contrastive learning method that encourages the model to capture more detailed semantic representations from the definition sequence encoding. |
| Outcome: | The proposed method could generate more specific definitions compared with state-of-the-art models. |
Multifaceted Domain-Specific Document Embeddings (2021.naacl-demos)
Copied to clipboard
| Challenge: | Current document embeddings require large training corpora but fail to learn high-quality representations when confronted with a small number of domain-specific documents and rare terms. |
| Approach: | They propose a faceted domain encoder that transforms each document into a single embedding vector . they use a Siamese neural network architecture to leverage knowledge graphs to enhance the embeddables . |
| Outcome: | The proposed model achieves the same embedding quality as state-of-the-art models while requiring only a tiny fraction of training data. |
Contrastive Learning-Enhanced Nearest Neighbor Mechanism for Multi-Label Text Classification (2022.acl-short)
Copied to clipboard
| Challenge: | Existing methods for multi-label text classification neglect the knowledge from the existing similar instances when predicting labels of a specific text. |
| Approach: | They propose a k nearest neighbor mechanism which retrieves several neighbor instances and interpolates the model output with their labels. |
| Outcome: | Extensive experiments show that the proposed method can bring significant performance improvements to multiple MLTC models including state-of-the-art pretrained and non-pretrained ones. |
Differentiable Data Augmentation for Contrastive Sentence Representation Learning (2022.emnlp-main)
Copied to clipboard
| Challenge: | a contrastive learning framework is used to fine-tune pre-trained language models with unlabeled sentences or labeled sentences. |
| Approach: | They propose a method that makes hard positives from unlabeled sentences . they use a prefix attached to a model to allow for differentiable data augmentation . |
| Outcome: | The proposed method yields significant improvements over existing methods under semi-supervised and supervised settings. |
Self-Discriminative Learning for Unsupervised Document Embedding (N19-1)
Copied to clipboard
| Challenge: | Existing methods for document embedding learning do not consider inter-document relationships. |
| Approach: | They propose to exploit the inter-document information and directly model the relations of documents in embedding space with a discriminative network and a novel objective. |
| Outcome: | The proposed method has errors that are 5 to 13% lower than state-of-the-art models and is even more pronounced in scarce label setting. |
SimCSE: Simple Contrastive Learning of Sentence Embeddings (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for learning universal sentence embeddings are based on unsupervised approaches with only dropout as noise. |
| Approach: | They propose an unsupervised approach that takes an input sentence and predicts itself in a contrastive objective with only standard dropout used as noise. |
| Outcome: | The proposed framework performs on par with previous supervised approaches and can produce superior sentence embeddings from unlabeled or labeled data. |