Papers by Hiroaki Yamagiwa
Understanding Higher-Order Correlations Among Semantic Components in Embeddings (2024.emnlp-main)
Copied to clipboard
| Challenge: | Independent Component Analysis (ICA) is an effective method for visualizing and interpreting the geometric structure of embeddings. |
| Approach: | They quantified embeddings' non-independencies using higher-order correlations and a maximum spanning tree of semantic components. |
| Outcome: | The results provide deeper insights into embeddings through ICA. |
Mapping 1,000+ Language Models via the Log-Likelihood Vector (2025.acl-long)
Copied to clipboard
| Challenge: | Existing methods to compare autoregressive language models are based on log-likelihoods . a model map is constructed using coordinates that capture the geometric structure of probability distributions based upon text-generation probabilities. |
| Approach: | They propose to use log-likelihood vectors to compare autoregressive language models . when treated as model features, their squared Euclidean distance approximates KL divergence . |
| Outcome: | The proposed method is highly scalable and easy to implement. |
Axis Tour: Word Tour Determines the Order of Axes in ICA-transformed Embeddings (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Embedding is an important component in natural language processing, but interpreting high-dimensional embeddings remains challenging. |
| Approach: | They propose a method which optimizes the order of axes in word embedding space by maximizing semantic continuity. |
| Outcome: | The proposed method improves the clarity of the word embedding space by maximizing the semantic continuity of the axes. |
Norm of Mean Contextualized Embeddings Determines their Variance (2025.coling-main)
Copied to clipboard
| Challenge: | Contextualized embeddings vary by context, even for the same token . a recent study shows a trade-off between the norm and the variance of the embedded word . |
| Approach: | They show that contextualized embeddings vary by context, even for the same token . they focus on the norm of the mean embeddment and the variance of the embeddables . |
| Outcome: | The proposed method is efficient and efficient for embeddings in sentences. |
Improving word mover’s distance by leveraging self-attention matrix (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Word mover's distance (WMD) computes semantic similarity via the optimal alignment between word embeddings, but does not consider word order, making it challenging to distinguish sentences with significant overlaps of similar words. |
| Approach: | They propose to incorporate the sentence structure represented by BERT’s self-attention matrix (SAM) to improve WMD and its variants in paraphrase identification. |
| Outcome: | The proposed method improves WMD and its variants in paraphrase identification with near-equivalent performance in semantic textual similarity. |
Quantifying Lexical Semantic Shift via Unbalanced Optimal Transport (2025.acl-long)
Copied to clipboard
| Challenge: | Existing methods for detecting semantic change only measure the level of individual usage instances. |
| Approach: | They propose to use unbalanced optimal transport to capture semantic change through excess and deficit in the alignment between usage instances. |
| Outcome: | The proposed method captures semantic change through excess and deficit in the alignment between usage instances. |
Likelihood Variance as Text Importance for Resampling Texts to Map Language Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | a language model map requires large text sets to be constructed . a resampling method reduces the number of texts needed while preserving accuracy of KL divergence estimates. |
| Approach: | They propose a method that selects important texts with weights proportional to log-likelihoods across models for each text. |
| Outcome: | The proposed method reduces the number of required texts while preserving the accuracy of KL divergence estimates. |
Revisiting Cosine Similarity via Normalized ICA-transformed Embeddings (2025.coling-main)
Copied to clipboard
| Challenge: | Existing studies on cosine similarity focus on the angle or correlation coefficient, but this study proposes a novel interpretation of the term word similarity. |
| Approach: | They propose a method for selecting statistically significant axes by deriving the probability distributions that govern each component and the product of components. |
| Outcome: | The proposed interpretation of cosine similarity is demonstrated through intuitive numerical examples and thorough numerical experiments. |
Establishing a Scale for Kullback-Leibler Divergence in Language Models Across Various Settings (2026.findings-acl)
Copied to clipboard
| Challenge: | Fig. 1 and 2 shows that log-likelihood vectors provide a consistent representation for language models . weight permutation symmetries and architectural dependencies hinder direct comparisons between models with different learning methods or designs. |
| Approach: | They propose a log-likelihood vector for comparing language models as probability distributions . they establish a consistent scale for KL divergence across various settings . |
| Outcome: | The proposed model comparisons show that the log-likelihood space is smaller than the weight space . the proposed model compares language models across checkpoints, model sizes, quantization, fine-tuning, and layers . |
Discovering Universal Geometry in Embeddings with ICA (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies have focused on achieving sparse embeddings or acquiring semantic axes, but this study focuses on the intrinsic independence present within embeddables. |
| Approach: | They propose to use independent component analysis to extract independent semantic components from pre-trained embeddings by leveraging anisotropic information that remains after the whitening process in Principal Component Analysis. |
| Outcome: | The proposed method reveals that embeddings can be expressed as a composition of a few interpretable axes and that these axe axe are consistent across languages, algorithms, and modalities. |