Papers by Yohei Oseki
Language Acquisition Device in Large Language Models (2026.acl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are less data-efficient than humans, and pre-pretraining on synthetic languages has been proposed to close this gap. |
| Approach: | They propose to pre-pretrain on MP-STRUCT, a formal language whose strings encode hierarchical composition, feature-based dependencies, and long-distance displacement via MERGE, AGREE, and MOVE. |
| Outcome: | The proposed model outperforms k-Shuffle Dyck despite not being definable in C-RASP despite being hierarchically expressive and circuit-theoretically learnable . |
Context Limitations Make Neural Language Models More Human-Like (2022.emnlp-main)
Copied to clipboard
| Challenge: | Language models (LMs) have been used in cognitive modeling and engineering studies to simulate human cognitive load during reading. |
| Approach: | They propose to constrain LMs' context access to improve their simulation of human reading behavior by incorporating syntactic biases into their context access. |
| Outcome: | The proposed model improves the simulation of human reading behavior by incorporating syntactic biases into their context access. |
Modeling Human Sentence Processing with Left-Corner Recurrent Neural Network Grammars (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing literature is agnostic about a parsing strategy of hierarchical models . a recent study showed that hierarchically model hierarchic structures capture grammatical dependencies much better than RNNs in targeted syntactic evaluations. |
| Approach: | They evaluated three LMs with head-final left-branching structures and Recurrent Neural Network Grammars with top-down and left-corner parsing strategies as hierarchical models. |
| Outcome: | The proposed model outperforms top-down and left-corner models against human reading times in Japanese. |
Developmentally-plausible Working Memory Shapes a Critical Period for Language Acquisition (2025.acl-long)
Copied to clipboard
| Challenge: | Large language models possess general linguistic abilities comparable to humans but their efficiency in language acquisition remains far inferior. |
| Approach: | They propose a method that initially constrains working memory during the early stages of training and gradually relaxes this constraint as learning progresses. |
| Outcome: | The proposed method outperforms conventional methods without memory constraints or with static memory constraints. |
Targeted Syntactic Evaluation on the Chomsky Hierarchy (2024.lrec-main)
Copied to clipboard
| Challenge: | a novel evaluation paradigm for targeted syntactic evaluations is proposed . we create formal languages that abstract four syntaktic phenomena in natural languages . |
| Approach: | They propose a new evaluation paradigm for Targeted Syntactic Evaluations . they create formal languages that abstract syntactical phenomena in natural languages . |
| Outcome: | The proposed evaluation paradigm evaluates language models on language modeling tasks . it shows that they can capture the structural patterns of the (Adj)n NP type formal language . |
Design of BCCWJ-EEG: Balanced Corpus with Human Electroencephalography (2020.lrec-1)
Copied to clipboard
| Challenge: | Recent research has focused on the fusion of NLP and neuroscience of language. |
| Approach: | They propose to use a balanced corpus of written Japanese (BCCWJ) annotated with human electroencephalography to improve annotations and annotations. |
| Outcome: | The proposed language resource is annotated with human electroencephalography (EEG) and can improve on annotations, genres, languages, etc. |
Massive Supervised Fine-tuning Experiments Reveal How Data, Layer, and Training Factors Shape LLM Alignment Quality (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in large language models (LLMs) have greatly improved natural language understanding and generation. |
| Approach: | They train a wide range of base models on a variety of datasets including code generation, mathematical reasoning, and general-domain tasks. |
| Outcome: | The results show that training–task synergies persist across all models while others vary substantially, emphasizing the importance of model-specific strategies. |
An Existence Proof for Neural Language Models That Can Explain Garden-Path Effects via Surprisal (2026.acl-long)
Copied to clipboard
| Challenge: | Surprisal theory claims that difficulty of sentences increases linearly with surprise . a neural LM that can explain garden-path effects cannot be built, says a new study . |
| Approach: | They propose to fine-tune neural LMs to better align surprisal-based reading-time estimates with actual reading times. |
| Outcome: | a new study shows that fine-tuned neural LMs do not overfit on held-out items . the results show that they improve predictive power for human reading times . |
How Much Syntactic Supervision is “Good Enough”? (2023.findings-eacl)
Copied to clipboard
| Challenge: | RNNGs with syntactic supervision underperformed RNNs with some syntaktic supervision, whereas RNNS with mild supervision achieved the best performance comparable to the state-of-the-art GPT-2-XL. |
| Approach: | They propose a method where syntactic LMs are gradually ablated from full syntatic supervision to zero syntastic supervision by preserving NP, VP, PP, SBAR nonterminal symbols. |
| Outcome: | The proposed method underperforms the RNNGs with zero syntactic supervision, and the LMs with mild syntaktic supervision perform better than the state-of-the-art GPT-2-XL. |
Tree-Planted Transformers: Unidirectional Transformer Language Models with Implicit Syntactic Supervision (2024.findings-acl)
Copied to clipboard
| Challenge: | Syntactic Language Models (SLMs) have difficulty with inference efficiency due to explicit generation of syntactical structures. |
| Approach: | They propose a method to "plant" trees into attention weights of unidirectional Transformer LMs to implicitly reflect syntactic structures of natural language. |
| Outcome: | The proposed method outperforms SLMs on the SyntaxGym benchmark. |
If Attention Serves as a Cognitive Model of Human Memory Retrieval, What is the Plausible Memory Representation? (2025.acl-long)
Copied to clipboard
| Challenge: | Recent work in computational psycholinguistics has revealed intriguing parallels between attention mechanisms and human memory retrieval, focusing primarily on vanilla Transformers that operate on token-level representations. |
| Approach: | They propose that the attention mechanism of Transformer Grammar (TG) can serve as a cognitive model of human memory retrieval using Normalized Attention Entropy (NAE) they propose that TG's attention can implement a human memory-retrieval theory known as cue-based retrieval . |
| Outcome: | The attention mechanism of Transformer Grammar (TG) achieves superior predictive power for self-paced reading times compared to vanilla Transformer’s, with further analyses revealing independent contributions from both models. |
Cognitive Information Bottleneck: Extracting Minimal Sufficient Cognitive Language Processing Signals (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing methods to extract only task-relevant information from cognitive processing signals are lacking in the field of NLP. |
| Approach: | They propose a method that extracts only task-relevant information from cognitive processing signals. |
| Outcome: | The proposed method outperforms existing methods in compressing cognitive signals and enhances performance on downstream tasks. |
Can Language Models Learn Typologically Implausible Languages? (2026.tacl-1)
Copied to clipboard
| Challenge: | Language models provide a naturalistic framework for studying artificial language learning . authors: typological universals and tendencies are thought to be caused by a learning bias . |
| Approach: | They propose to train LMs on highly naturalistic counterfactual versions of English and Japanese . they show that LM learn subtly implausible languages more slowly . |
| Outcome: | The proposed language models learn subtly implausible languages more slowly compared to human models . the findings suggest that LMs exhibit typologically aligned learning preferences . |
Modeling Overregularization in Children with Small Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing research has analyzed regularization in language acquisition only by modeling word inflection directly, which is unnatural in light of human language acquisition. |
| Approach: | They hypothesize that language models that imitate errors children make during language acquisition have a learning process more similar to humans. |
| Outcome: | The proposed model shows child-like U-shaped learning curves clearly for certain verbs, but the preferences for types of overgeneralization did not fully match the observations in children. |
Effective Batching for Recurrent Neural Network Grammars (2021.findings-acl)
Copied to clipboard
| Challenge: | RNNGs are hard to scale due to the difficulty of batched training. |
| Approach: | They propose to batch RNNGs where every operation is computed in parallel with tensors across multiple sentences. |
| Outcome: | The proposed RNNG scales faster than existing models and achieves x6 speedup compared to existing C++ DyNet implementation . |
JCoLA: Japanese Corpus of Linguistic Acceptability (2024.lrec-main)
Copied to clipboard
| Challenge: | Neural language models have exhibited outstanding performance in downstream tasks, yet there is limited understanding regarding the extent of their internalization of syntactic knowledge. |
| Approach: | They introduce a dataset that analyzes sentences annotated with binary acceptability judgments from linguistic textbooks and handbooks and splits them into in-domain and out-of-domain data. |
| Outcome: | The proposed datasets show that models can surpass human performance for in-domain data while no models can exceed human performance on out-of-domain datasets. |
How a Bilingual LM Becomes Bilingual: Tracing Internal Representations with Sparse Autoencoders (2025.findings-emnlp)
Copied to clipboard
Tatsuro Inaba, Go Kamoda, Kentaro Inui, Masaru Isonuma, Yusuke Miyao, Yohei Oseki, Yu Takagi, Benjamin Heinzerling
| Challenge: | Using sparse autoencoders, we explore how bilingual language models develop complex internal representations. |
| Approach: | They employ sparse autoencoders to analyze bilingual language models' internal representations. |
| Outcome: | The proposed method integrates decomposed representations from a fully trained model into a mid-training model. |
Dual Alignment Between Language Model Layers and Human Sentence Processing (2026.acl-long)
Copied to clipboard
| Challenge: | Existing studies have demonstrated both the successes and limitations of accurate predictability estimation by modern LMs in cognitive modeling. |
| Approach: | They propose to use internal layers to better estimate human cognitive effort observed in syntactic ambiguity processing in English. |
| Outcome: | The proposed models can be modeled using surprisal from early layers of large language models (LLMs) this raises the question whether such advantages extend to more syntactically challenging constructions, where surprised estimates underestimate human cognitive effort. |
Lower Perplexity is Not Always Human-Like (2021.acl-long)
Copied to clipboard
| Challenge: | Existing efforts to build human-like computational models have focused on English . a cross-lingual evaluation is needed to build such models, but current research has focused on Japanese . |
| Approach: | They re-examine an established generalization that lower perplexity is not always human-like in Japanese . they propose a cross-lingual evaluation to build human-type computational models . |
| Outcome: | The proposed model lacks universality and lower perplexity is not always human-like . the results suggest a cross-lingual evaluation will be necessary to build human-type models . |
Learning Bidirectional Morphological Inflection like Humans (2024.lrec-main)
Copied to clipboard
| Challenge: | Recent research has focused on whether neural models can acquire morphological inflection like humans. |
| Approach: | They propose to use a recurrent neural network with attention and the transformer to train a symbolic model under a human-like learning environment to evaluate their models. |
| Outcome: | The proposed models did not accurately inflect verbs in the same manner as humans in terms of morphological inflection direction. |
Composition, Attention, or Both? (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing work suggests that language models implicitly learn syntactic structures of natural language, even though they do not receive explicit syntatic supervision. |
| Approach: | They propose a novel architecture that recursively compose subtrees with a composition function and selectively attend to previous structural information with sc-attention mechanisms. |
| Outcome: | The proposed architecture can induce human-like syntactic generalization by recursive composition and selective attention to previous structural information. |
Psychometric Predictive Power of Large Language Models (2024.findings-naacl)
Copied to clipboard
| Challenge: | despite efforts to align large language models with human preferences, instruction tuning does not always make LLMs human-like from a cognitive modeling perspective. |
| Approach: | They find that instruction tuning does not always make large language models human-like from a cognitive perspective. |
| Outcome: | The proposed prompts improve predictive power but are still inferior to small base models. |
JBLiMP: Japanese Benchmark of Linguistic Minimal Pairs (2023.findings-eacl)
Copied to clipboard
| Challenge: | In this paper, we compare syntactic knowledge of language models across different languages. |
| Approach: | They introduce a dataset for targeted syntactic evaluations of language models in Japanese. |
| Outcome: | The proposed dataset compares the syntactic knowledge of language models across languages. |
Can Language Models Induce Grammatical Knowledge from Indirect Evidence? (2024.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in language models have shown remarkable progress in various tasks. |
| Approach: | They introduce a dataset that incorporates wug words and inject them into pretraining data and evaluate them on evaluation data. |
| Outcome: | The proposed model does not induce grammatical knowledge even after repeated exposure to instances with the same structure but differing only in lexical items from evaluation instances in certain language phenomena. |