| Challenge: | Contextualized pre-trained representations are widely used as input to various tasks such as information retrieval, anomaly detection and document clustering. |
| Approach: | They propose to examine the impact of different transformations on isotropy and performance to assess the true impact of anisotropi. |
| Outcome: | The proposed model is based on a clustering task and shows that it has limited impact on expressiveness and closeness. |
Similar Papers
Exploring Anisotropy and Outliers in Multilingual Language Models for Cross-Lingual Semantic Sentence Similarity (2023.findings-acl)
Copied to clipboard
| Challenge: | Recent studies have shown that contextual language models display outlier dimensions . this is true for monolingual and multilingual models, but little work has been done on multilingual contexts . |
| Approach: | They investigate outlier dimensions and their relationship to anisotropy in multilingual contexts . they focus on cross-lingual semantic similarity tasks . |
| Outcome: | The proposed model improves on cross-lingual semantic similarity tasks. |
Is anisotropy really the cause of BERT embeddings not being semantic? (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to train contextual language models for NLP use a lightweight approach called bi-encoder, which takes two sentences as input, but does not perform well with vanilla pre-trained Transformers. |
| Approach: | They conduct a set of experiments to improve our understanding of the lack of semantic isometry in contextualized word representations in BERT. |
| Outcome: | The proposed approach does not perform well with vanilla pre-trained Transformers. |
On the Effect of Isotropy on VAE Representations of Text (2022.acl-short)
Copied to clipboard
| Challenge: | Injecting desired geometric properties into text representations has attracted a lot of attention due to its better utilisation of representation space. |
| Approach: | They propose to use an isotropic Gaussian posterior instead of the ellipsoidal Gausssian priori to inject isotropy into text representations. |
| Outcome: | The proposed method improves classification performance, robustness to input perturbation, and generative behavior compared to the ellipsoidal Gaussian posterior. |
A Cluster-based Approach for Improving Isotropy in Contextual Embedding Space (2021.acl-short)
Copied to clipboard
| Challenge: | Existing approaches to address the representation degeneration problem in contextual embedding spaces require a learning process to retrain models with additional objectives. |
| Approach: | They propose a local cluster-based method to address the representation degeneration problem in contextual embedding spaces by removing local dominant directions from verb representations. |
| Outcome: | The proposed method improves CWRs performance on semantic tasks by removing dominant directions of verb representations. |
Isotropy, Clusters, and Classifiers (2024.acl-short)
Copied to clipboard
| Challenge: | Existing evidence supports and challenges the use of isotropy in embedding spaces. |
| Approach: | They propose to formalize this connection mathematically and empirically and prove it's true . they argue that isotropy imposes requirements on embedding space that are not compatible with clusters . |
| Outcome: | The proposed method sheds light on previous studies focusing on anisotropy in embedding spaces. |
Better Embeddings with Coupled Adam (2025.acl-long)
Copied to clipboard
| Challenge: | Anisotropic Embeddings Large Language Models exhibit undesirable yet poorly understood feature of anisotropy. |
| Approach: | They propose an algorithm that uses the second moment in Adam to mitigate anisotropic embeddings . they propose an embeddable matrix and unembedding matrix to map the input and output tokens based on weight tying . |
| Outcome: | The proposed model improves quality and performance on large datasets. |
Is Neural Topic Modelling Better than Clustering? An Empirical Study on Clustering with Contextual Embeddings for Topics (2022.naacl-main)
Copied to clipboard
| Challenge: | Recent work incorporates pre-trained word embeddings into Neural Topic Models (NTMs), generating highly coherent topics. |
| Approach: | They conduct thorough experiments to investigate whether embeddings directly with an appropriate word selection method can generate more coherent and diverse topics than NTMs. |
| Outcome: | The proposed model generates more coherent and diverse topics than traditional NTMs, achieving higher efficiency and simplicity. |
Too Much in Common: Shifting of Embeddings in Transformer Language Models and its Implications (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing studies have shown that word embeddings do not occupy a narrow cone, but rather drift in common directions. |
| Approach: | They show that anisotropy can be restored using a simple transformation of word embeddings. |
| Outcome: | The proposed model can restore anisotropy using a simple transformation. |
Analyzing Text Representations by Measuring Task Alignment (2023.acl-short)
Copied to clipboard
| Challenge: | Recent advances in text classification have shown that pre-trained representations are key for text classification. |
| Approach: | They propose a task alignment score that measures alignment at different levels of granularity. |
| Outcome: | The proposed score shows that task alignment can explain the performance of a given representation. |
Understanding Cross-Lingual Alignment—A Survey (2024.findings-acl)
Copied to clipboard
| Challenge: | Cross-lingual alignment is the meaningful similarity of representations across languages in multilingual language models. |
| Approach: | They propose a taxonomy of methods to improve cross-lingual alignment . they argue that an effective trade-off between language-neutral and language-specific information is key . |
| Outcome: | The proposed methods can be applied to encoder models and encoder-decoder-only models . they show that language-neutral and language-specific information is key . |