Papers by Hidetoshi Shimodaira
Memorization, Emergence, and Explaining Reversal Failures: A Controlled Study of Relational Semantics in LLMs (2026.acl-long)
Copied to clipboard
Yihua Zhu, Qianying Liu, Jiaxin Wang, Fei Cheng, Chaoran Liu, Akiko Aizawa, Sadao Kurohashi, Hidetoshi Shimodaira
| Challenge: | Autoregressive LLMs perform well on relational tasks that require linking entities via relational words, but it is unclear whether they learn the logical semantics of such relations or whether left-to-right order bias is involved. |
| Approach: | They propose a framework that generates text from symmetric/inverse triples and trains autoregressive models from scratch. |
| Outcome: | The proposed framework generates text from symmetric/inverse triples, trains autoregressive models from scratch, and evaluates memorization, logical inference, and in-context generalization to unseen entities. |
Understanding Higher-Order Correlations Among Semantic Components in Embeddings (2024.emnlp-main)
Copied to clipboard
| Challenge: | Independent Component Analysis (ICA) is an effective method for visualizing and interpreting the geometric structure of embeddings. |
| Approach: | They quantified embeddings' non-independencies using higher-order correlations and a maximum spanning tree of semantic components. |
| Outcome: | The results provide deeper insights into embeddings through ICA. |
Mapping 1,000+ Language Models via the Log-Likelihood Vector (2025.acl-long)
Copied to clipboard
| Challenge: | Existing methods to compare autoregressive language models are based on log-likelihoods . a model map is constructed using coordinates that capture the geometric structure of probability distributions based upon text-generation probabilities. |
| Approach: | They propose to use log-likelihood vectors to compare autoregressive language models . when treated as model features, their squared Euclidean distance approximates KL divergence . |
| Outcome: | The proposed method is highly scalable and easy to implement. |
Segmentation-free compositional n-gram embedding (N19-1)
Copied to clipboard
| Challenge: | Existing word embedding models depend on word segmentation, but this method is difficult when corpora written in noisy or unsegmented languages. |
| Approach: | They propose a new method that models words, phrases and sentences seamlessly without word segmentation. |
| Outcome: | The proposed method is very effective for noisy corpora written in unsegmented languages such as Chinese and Japanese. |
Axis Tour: Word Tour Determines the Order of Axes in ICA-transformed Embeddings (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Embedding is an important component in natural language processing, but interpreting high-dimensional embeddings remains challenging. |
| Approach: | They propose a method which optimizes the order of axes in word embedding space by maximizing semantic continuity. |
| Outcome: | The proposed method improves the clarity of the word embedding space by maximizing the semantic continuity of the axes. |
Norm of Word Embedding Encodes Information Gain (2023.emnlp-main)
Copied to clipboard
| Challenge: | Distributed representations of words encode lexical semantic information, but what type of information is encoded and how? |
| Approach: | They propose to use the squared norm of static word embedding to encode the information gain conveyed by the word. |
| Outcome: | The proposed model can encode the information gain of a word in a language model or neural network. |
Norm of Mean Contextualized Embeddings Determines their Variance (2025.coling-main)
Copied to clipboard
| Challenge: | Contextualized embeddings vary by context, even for the same token . a recent study shows a trade-off between the norm and the variance of the embedded word . |
| Approach: | They show that contextualized embeddings vary by context, even for the same token . they focus on the norm of the mean embeddment and the variance of the embeddables . |
| Outcome: | The proposed method is efficient and efficient for embeddings in sentences. |
Improving word mover’s distance by leveraging self-attention matrix (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Word mover's distance (WMD) computes semantic similarity via the optimal alignment between word embeddings, but does not consider word order, making it challenging to distinguish sentences with significant overlaps of similar words. |
| Approach: | They propose to incorporate the sentence structure represented by BERT’s self-attention matrix (SAM) to improve WMD and its variants in paraphrase identification. |
| Outcome: | The proposed method improves WMD and its variants in paraphrase identification with near-equivalent performance in semantic textual similarity. |
Quantifying Lexical Semantic Shift via Unbalanced Optimal Transport (2025.acl-long)
Copied to clipboard
| Challenge: | Existing methods for detecting semantic change only measure the level of individual usage instances. |
| Approach: | They propose to use unbalanced optimal transport to capture semantic change through excess and deficit in the alignment between usage instances. |
| Outcome: | The proposed method captures semantic change through excess and deficit in the alignment between usage instances. |
Block-Diagonal Orthogonal Relation and Matrix Entity for Knowledge Graph Embedding (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing knowledge graph embeddings (KGs) are limited in their flexibility and difficulties in generalizing them for higher-dimensional rotations. |
| Approach: | They propose a KGE model employing matrices for entities and block-diagonal orthogonal matrics with Riemannian optimization for relations that captures several relation patterns that rotation-based methods can identify. |
| Outcome: | The proposed model outperforms state-of-the-art models while reducing the number of relation parameters. |
Likelihood Variance as Text Importance for Resampling Texts to Map Language Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | a language model map requires large text sets to be constructed . a resampling method reduces the number of texts needed while preserving accuracy of KL divergence estimates. |
| Approach: | They propose a method that selects important texts with weights proportional to log-likelihoods across models for each text. |
| Outcome: | The proposed method reduces the number of required texts while preserving the accuracy of KL divergence estimates. |
Revisiting Cosine Similarity via Normalized ICA-transformed Embeddings (2025.coling-main)
Copied to clipboard
| Challenge: | Existing studies on cosine similarity focus on the angle or correlation coefficient, but this study proposes a novel interpretation of the term word similarity. |
| Approach: | They propose a method for selecting statistically significant axes by deriving the probability distributions that govern each component and the product of components. |
| Outcome: | The proposed interpretation of cosine similarity is demonstrated through intuitive numerical examples and thorough numerical experiments. |
Establishing a Scale for Kullback-Leibler Divergence in Language Models Across Various Settings (2026.findings-acl)
Copied to clipboard
| Challenge: | Fig. 1 and 2 shows that log-likelihood vectors provide a consistent representation for language models . weight permutation symmetries and architectural dependencies hinder direct comparisons between models with different learning methods or designs. |
| Approach: | They propose a log-likelihood vector for comparing language models as probability distributions . they establish a consistent scale for KL divergence across various settings . |
| Outcome: | The proposed model comparisons show that the log-likelihood space is smaller than the weight space . the proposed model compares language models across checkpoints, model sizes, quantization, fine-tuning, and layers . |
Discovering Universal Geometry in Embeddings with ICA (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies have focused on achieving sparse embeddings or acquiring semantic axes, but this study focuses on the intrinsic independence present within embeddables. |
| Approach: | They propose to use independent component analysis to extract independent semantic components from pre-trained embeddings by leveraging anisotropic information that remains after the whitening process in Principal Component Analysis. |
| Outcome: | The proposed method reveals that embeddings can be expressed as a composition of a few interpretable axes and that these axe axe are consistent across languages, algorithms, and modalities. |
3D Rotation and Translation for Hyperbolic Knowledge Graph Embedding (2024.eacl-long)
Copied to clipboard
| Challenge: | Existing knowledge graph embeddings do not capture relation patterns, but they capture symmetry, antisymmetry, inversion, commutative composition, non-commutable composition, hierarchy, and multiplicity. |
| Approach: | They propose a 3D Rotation and Translation in Hyperbolic space model that captures relation patterns simultaneously. |
| Outcome: | The proposed model outperforms state-of-the-art models in terms of accuracy, hierarchy property, and other relation patterns in low-dimensional space, while performing similarly in high-dimensional spaces. |