Representation Degeneration Problem in Prompt-based Models for Natural Language Understanding (2024.lrec-main)
Copied to clipboard
| Challenge: | Prompt-based fine-tuning (PF) models have shown improved performance on few-shot natural language understanding benchmarks. |
| Approach: | They propose a framework to alleviate anisotropy in the embedding space by aligning with pre-trained language models' training objective. |
| Outcome: | The proposed method outperforms mainstream methods on many NLU benchmarks. |
Similar Papers
Context-Tuning: Learning Contextualized Prompts for Natural Language Generation (2022.coling-1)
Copied to clipboard
| Challenge: | Recent studies have shown that pretrained language models (PLMs) lack sufficient consideration of input semantics to generate natural language. |
| Approach: | They propose a continuous prompting approach to fine-tune PLMs for natural language generation by modeling an inverse generation process from output to input. |
| Outcome: | The proposed method fine-tunes only 0.12% of the parameters while maintaining good performance. |
Rare Tokens Degenerate All Tokens: Improving Neural Text Generation via Adaptive Gradient Gating for Rare Token Embeddings (2022.acl-long)
Copied to clipboard
| Challenge: | Recent studies have determined that the learned token embeddings of large-scale neural language models are degenerated to be anisotropic with a narrow-cone shape. |
| Approach: | They propose a method to degenerate the learning gradient for rare token embeddings by gating the specific part of the gradient for all tokens during training stage. |
| Outcome: | The proposed method improves the performance of the models but lacks the training dynamics needed to solve the representation degeneration problem. |
A Cluster-based Approach for Improving Isotropy in Contextual Embedding Space (2021.acl-short)
Copied to clipboard
| Challenge: | Existing approaches to address the representation degeneration problem in contextual embedding spaces require a learning process to retrain models with additional objectives. |
| Approach: | They propose a local cluster-based method to address the representation degeneration problem in contextual embedding spaces by removing local dominant directions from verb representations. |
| Outcome: | The proposed method improves CWRs performance on semantic tasks by removing dominant directions of verb representations. |
Prompt Tuning Pushes Farther, Contrastive Learning Pulls Closer: A Two-Stage Approach to Mitigate Social Biases (2023.acl-long)
Copied to clipboard
| Challenge: | Existing debiasing techniques use Counterfactual Data Augmentation (CDA) to balance the training corpus, but this technique slightly modifies the original corpus limiting the representation distance between different demographic groups. |
| Approach: | They propose a two-stage debiasing model using Contrastive learning with Continuous Prompt Augmentation to mitigate social biases in PLMs’ encoding. |
| Outcome: | The proposed model outperforms baselines in terms of debiasing performance while maintaining the language modeling capability of PLMs. |
The Power of Prompt Tuning for Low-Resource Semantic Parsing (2022.acl-short)
Copied to clipboard
| Challenge: | Prompt tuning is an effective method for adapting pre-trained language models to downstream tasks. |
| Approach: | They propose to use prompt tuning for semantic parsing to map natural language utterances onto formal meaning representations. |
| Outcome: | The proposed method outperforms the fine-tuned model on low-resource splits of Overnight and TOPv2 on language representations with increasing model scale and target representations. |
LM-CPPF: Paraphrasing-Guided Data Augmentation for Contrastive Prompt-Based Few-Shot Fine-Tuning (2023.acl-short)
Copied to clipboard
| Challenge: | Recent advances in pre-trained language models have been limited when fine-tuned on small datasets. |
| Approach: | They propose to add contrastive learning to prompt-based fine-tuning to improve model performance. |
| Outcome: | The proposed approach outperforms other methods on multiple text classification benchmarks. |
How Does Fine-tuning Affect the Geometry of Embedding Space: A Case Study on Isotropy (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for fine-tuning pre-trained language models are ineffective, despite their potential, pre-training models suffer from important weaknesses. |
| Approach: | They analyze the extent to which the isotropy of the embedding space changes after fine-tuning. |
| Outcome: | The proposed model improves the isotropy of embedding space after fine-tuning . the model can encode linguistic properties, but lacks the social bias needed to improve it . |
Improved Universal Sentence Embeddings with Prompt-based Contrastive Learning and Energy-based Learning (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing contrastive methods for learning universal sentence embeddings have limitations due to their over-parameterization and poor performance under domain shift settings. |
| Approach: | They propose to integrate an Energy-based Hinge loss to enhance the pairwise discriminative power of contrastive learning for sentence embeddings by combining PLMs with energy-based learning. |
| Outcome: | Empirical results show that the proposed method improves on seven standard semantic textual similarity tasks and a domain-shifted STS task. |
Scaling Sentence Embeddings with Large Language Models (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Current methods based on contrastive learning have generated high-quality sentence embeddings. |
| Approach: | They propose a method to enhance LLM performance on sentence embeddings with a one-word limitation. |
| Outcome: | The proposed method outperforms contrastive learning methods on sentence embeddings without fine-tuning and with fine-untun. |
Contrastive Prompting Enhances Sentence Embeddings in LLMs through Inference-Time Steering (2025.acl-long)
Copied to clipboard
| Challenge: | Existing studies focus on prompt engineering to encode the full semantics of a sentence into the embedding of the last token. |
| Approach: | They propose a technique that introduces an extra auxiliary prompt to elicit better sentence embedding . they propose to use the hidden state of the token as the sentence embedded in LLMs . |
| Outcome: | The proposed technique can improve performance of existing prompt-based methods on STS tasks and downstream classification tasks. |