Papers with CWRs
A Cluster-based Approach for Improving Isotropy in Contextual Embedding Space (2021.acl-short)
Copied to clipboard
| Challenge: | Existing approaches to address the representation degeneration problem in contextual embedding spaces require a learning process to retrain models with additional objectives. |
| Approach: | They propose a local cluster-based method to address the representation degeneration problem in contextual embedding spaces by removing local dominant directions from verb representations. |
| Outcome: | The proposed method improves CWRs performance on semantic tasks by removing dominant directions of verb representations. |
How Does Fine-tuning Affect the Geometry of Embedding Space: A Case Study on Isotropy (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for fine-tuning pre-trained language models are ineffective, despite their potential, pre-training models suffer from important weaknesses. |
| Approach: | They analyze the extent to which the isotropy of the embedding space changes after fine-tuning. |
| Outcome: | The proposed model improves the isotropy of embedding space after fine-tuning . the model can encode linguistic properties, but lacks the social bias needed to improve it . |