Exploring Interpretability of Independent Components of Word Embeddings with Automated Word Intruder Test (2024.lrec-main)
Copied to clipboard
| Challenge: | Independent Component Analysis (ICA) is an algorithm for finding separate sources in a mixed signal. |
| Approach: | They propose to use ICA to analyze word embeddings to quantify interpretability . they propose to automate word intruder test to quantify the components . |
| Outcome: | The proposed algorithm can be used to find semantic features of words . it can be combined to find words that have features associated with the components . |
Similar Papers
Discovering Universal Geometry in Embeddings with ICA (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies have focused on achieving sparse embeddings or acquiring semantic axes, but this study focuses on the intrinsic independence present within embeddables. |
| Approach: | They propose to use independent component analysis to extract independent semantic components from pre-trained embeddings by leveraging anisotropic information that remains after the whitening process in Principal Component Analysis. |
| Outcome: | The proposed method reveals that embeddings can be expressed as a composition of a few interpretable axes and that these axe axe are consistent across languages, algorithms, and modalities. |
Understanding Higher-Order Correlations Among Semantic Components in Embeddings (2024.emnlp-main)
Copied to clipboard
| Challenge: | Independent Component Analysis (ICA) is an effective method for visualizing and interpreting the geometric structure of embeddings. |
| Approach: | They quantified embeddings' non-independencies using higher-order correlations and a maximum spanning tree of semantic components. |
| Outcome: | The results provide deeper insights into embeddings through ICA. |
Exploring Intra and Inter-language Consistency in Embeddings with ICA (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies have shown that ICA can reveal universal semantic axes across languages but lack verification of consistency of independent components within and across languages. |
| Approach: | They propose to use independent component analysis to identify independent components that are more interpretable than PCA to find universal semantic axes. |
| Outcome: | The proposed framework ensures the reliability and universality of semantic axes. |
Revisiting Cosine Similarity via Normalized ICA-transformed Embeddings (2025.coling-main)
Copied to clipboard
| Challenge: | Existing studies on cosine similarity focus on the angle or correlation coefficient, but this study proposes a novel interpretation of the term word similarity. |
| Approach: | They propose a method for selecting statistically significant axes by deriving the probability distributions that govern each component and the product of components. |
| Outcome: | The proposed interpretation of cosine similarity is demonstrated through intuitive numerical examples and thorough numerical experiments. |
Simplifying Outcomes of Language Model Component Analyses with ELIA (2026.eacl-demo)
Copied to clipboard
| Challenge: | ELIA is an interactive web application that simplifies the outputs of various language model component analyses for a broader audience. |
| Approach: | They propose to use a vision-language model to automatically generate natural language explanations for the complex visualizations produced by these methods. |
| Outcome: | The proposed system integrates three key techniques and generates natural language explanations for complex visualizations. |
Are Embedding Spaces Interpretable? Results of an Intrusion Detection Evaluation on a Large French Corpus (2022.lrec-1)
Copied to clipboard
| Challenge: | Word embedding methods use word co-occurrences to encode, syntactic and semantic information to describe vocabulary in a low-dimensional space. |
| Approach: | They evaluate word embedding interpretability using two methods . they use a word-in-space vector encoder and graph-based method SPINE . |
| Outcome: | The proposed methods show that they can be interpretable on a large French corpus. |
Axis Tour: Word Tour Determines the Order of Axes in ICA-transformed Embeddings (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Embedding is an important component in natural language processing, but interpreting high-dimensional embeddings remains challenging. |
| Approach: | They propose a method which optimizes the order of axes in word embedding space by maximizing semantic continuity. |
| Outcome: | The proposed method improves the clarity of the word embedding space by maximizing the semantic continuity of the axes. |
Interpretable Text Embeddings and Text Similarity Explanation: A Survey (2025.emnlp-main)
Copied to clipboard
| Challenge: | Text embeddings are a fundamental component in many NLP tasks, but their interpretation and explanation remain challenging. |
| Approach: | They propose a framework for interpretable text embeddings and text similarity explanation . they characterize the main ideas, approaches, and trade-offs and discuss lessons learned . |
| Outcome: | The proposed methods are compared with existing models and compare them with existing ones. |
SensePOLAR: Word sense aware interpretability for pre-trained contextual word embeddings (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing word embedding models lack interpretability for words . |
| Approach: | They propose to add interpretability to word embeddings by using a POLAR framework that enables wordsense aware interpretations for pre-trained contextual word embeds. |
| Outcome: | The proposed framework achieves comparable performance to existing embeddings across GLUE and SQuAD benchmarks. |
What’s in Your Embedding, And How It Predicts Task Performance (C18-1)
Copied to clipboard
| Challenge: | Attempts to find a single technique for general-purpose intrinsic evaluation of word embeddings have so far not been successful. |
| Approach: | They propose a method that quantifies interpretable characteristics of word vector neighborhoods and shows how they correlate with performance on 14 extrinsic and intrinsic task datasets. |
| Outcome: | The proposed approach enables multi-faceted evaluation, parameter search, and generally – a more principled, hypothesis-driven approach to development of distributional semantic representations. |