Bootstrap Your Own PLM: Boosting Semantic Features of PLMs for Unsuperivsed Contrastive Learning (2024.findings-eacl)
Copied to clipboard
| Challenge: | Existing studies have shown that SimCSE significantly improves the performance of pretrained language models on the sentence representation benchmark. |
| Approach: | They propose a method called IFM which reduces the tendency of contrastive models for VRL to rely on feature-suppressing shortcut solutions. |
| Outcome: | The proposed method reduces the tendency of contrastive models for VRL to rely on feature-suppressing shortcut solutions. |
Similar Papers
Virtual Data Augmentation: A Robust and General Framework for Fine-tuning Pre-trained Models (2021.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies have shown that powerful pre-trained language models can be fooled by small perturbations or intentional attacks. |
| Approach: | They propose a framework for fine-tuning PLMs using a masked language model and Gaussian noise to augment semantically relevant examples with sufficient diversity. |
| Outcome: | The proposed framework improves the robustness of pre-trained language models and alleviates performance degradation under adversarial attacks. |
Visually-augmented pretrained language models for NLP tasks without images (2023.acl-long)
Copied to clipboard
| Challenge: | Existing approaches to improve pre-trained language models lack visual commonsense and semantics. |
| Approach: | They propose a visual-augmented approach to fine-tune pre-trained language models by using retrieved or generated images instead of relying on explicit images. |
| Outcome: | The proposed approach outperforms baselines on ten tasks and consistently outperformed other approaches. |
Differentiable Data Augmentation for Contrastive Sentence Representation Learning (2022.emnlp-main)
Copied to clipboard
| Challenge: | a contrastive learning framework is used to fine-tune pre-trained language models with unlabeled sentences or labeled sentences. |
| Approach: | They propose a method that makes hard positives from unlabeled sentences . they use a prefix attached to a model to allow for differentiable data augmentation . |
| Outcome: | The proposed method yields significant improvements over existing methods under semi-supervised and supervised settings. |
LA-UCL: LLM-Augmented Unsupervised Contrastive Learning Framework for Few-Shot Text Classification (2024.lrec-main)
Copied to clipboard
| Challenge: | Experimental results show that our model exceeds the baseline models due to the lack of cognitive ability. |
| Approach: | They propose a LLM-Augmented Unsupervised Contrastive Learning Framework which introduces a cognition-enabled Large Language Model (LLM) for efficient data augmentation and presents corresponding contrastive learning strategies. |
| Outcome: | The proposed model exceeds baseline models on six datasets. |
Improved Universal Sentence Embeddings with Prompt-based Contrastive Learning and Energy-based Learning (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing contrastive methods for learning universal sentence embeddings have limitations due to their over-parameterization and poor performance under domain shift settings. |
| Approach: | They propose to integrate an Energy-based Hinge loss to enhance the pairwise discriminative power of contrastive learning for sentence embeddings by combining PLMs with energy-based learning. |
| Outcome: | Empirical results show that the proposed method improves on seven standard semantic textual similarity tasks and a domain-shifted STS task. |
SALAD: Improving Robustness and Generalization through Contrastive Learning with Structure-Aware and LLM-Driven Augmented Data (2025.naacl-long)
Copied to clipboard
| Challenge: | In many natural language processing tasks, model training often leads to spurious correlations . shortcuts allow models to rely on irrelevant patterns in the data, leading to biased predictions. |
| Approach: | They propose a method to generate structure-aware positive and negative sentences using tagging. |
| Outcome: | The proposed method improves model robustness and generalization across different environments while minimizing spurious correlations. |
Retrofitting Light-weight Language Models for Emotions using Supervised Contrastive Learning (2023.emnlp-main)
Copied to clipboard
| Challenge: | a novel retrofitting method to induce emotion aspects into pre-trained language models is proposed . the models are computationally less expensive and open, but do not capture affective aspects of human communication well. |
| Approach: | They propose a retrofitting method to induce emotion aspects into pre-trained language models . they retrofit text fragments exhibiting similar emotions into pretrained networks . |
| Outcome: | The proposed method produces emotion-aware text representations for sentiment analysis and sarcasm detection tasks. |
Recent Advances in Pre-trained Language Models: Why Do They Work and How Do They Work (2022.aacl-tutorials)
Copied to clipboard
| Challenge: | Pre-trained language models are language models that are pre-taught on large-scaled corpora in a self-supervised fashion. |
| Approach: | This tutorial provides a broad and comprehensive introduction to pre-trained language models . it focuses on emerging methods that enable PLMs to perform diverse downstream tasks . |
| Outcome: | This tutorial focuses on the benefits of pre-trained language models and how to use them in NLP tasks. |
Paraphrase-based Contrastive Learning for Sentence Pair Modeling (2025.naacl-srw)
Copied to clipboard
| Challenge: | Existing methods to improve performance of sentence pair modeling are not available on a large-scale for non-English languages. |
| Approach: | They propose a method to apply contrastive learning to pre-trained masked language models . they use sentence embeddings of paraphrase pairs to make similar sentences . |
| Outcome: | The proposed method can be used on four sentence pair modeling tasks in English and Japanese. |
Alleviating Over-smoothing for Unsupervised Sentence Representation (2023.acl-long)
Copied to clipboard
| Challenge: | Existing approaches to learn better unsupervised sentence representations have been successful . over-smoothing problem in unsupervised sentences reduces the capacity of powerful PLMs . |
| Approach: | They propose a method to solve the over-smoothing problem in unsupervised sentence representations by combining negatives from PLMs intermediate layers. |
| Outcome: | The proposed method improves on different strong baselines on Semantic Textual Similarity and Transfer datasets. |