Papers by Tatsuro Inaba
Weight-based Analysis of Detokenization in Language Models: Understanding the First Stage of Inference Without Inference (2025.findings-naacl)
Copied to clipboard
| Challenge: | Existing evidence for the stages-of-inference hypothesis is that early layers of language models map their subword tokenized input to more meaningful representations that form the model’s “inner vocabulary”. |
| Approach: | They propose an analytical decomposition of first-layer attention in language models that quantifies the relative contributions of position-related, token-related and mixed effects. |
| Outcome: | The proposed analysis yields interpretable terms that quantify the relative contributions of position-related, token-related and mixed effects. |
MultiTool-CoT: GPT-3 Can Use Multiple External Tools with Chain of Thought Prompting (2023.acl-short)
Copied to clipboard
| Challenge: | Recent studies have focused on using a single external tool to solve a problem with large language models and have not addressed different problems together. |
| Approach: | They propose a framework that leverages chain-of-thought prompting to incorporate multiple external tools into the reasoning process. |
| Outcome: | The proposed framework outperforms baselines and achieves state-of-the-art performance on a task that requires both numerical reasoning and domain-specific knowledge. |
How a Bilingual LM Becomes Bilingual: Tracing Internal Representations with Sparse Autoencoders (2025.findings-emnlp)
Copied to clipboard
Tatsuro Inaba, Go Kamoda, Kentaro Inui, Masaru Isonuma, Yusuke Miyao, Yohei Oseki, Yu Takagi, Benjamin Heinzerling
| Challenge: | Using sparse autoencoders, we explore how bilingual language models develop complex internal representations. |
| Approach: | They employ sparse autoencoders to analyze bilingual language models' internal representations. |
| Outcome: | The proposed method integrates decomposed representations from a fully trained model into a mid-training model. |