Papers by Takateru Yamakoshi
Evaluating distillation methods for data-efficient syntax learning (2025.findings-emnlp)
Copied to clipboard
| Challenge: | knowledge distillation (KD) targeting attention should selectively accelerate syntax acquisition, a study finds . logit-based KD dramatically improves data-efficiency, attention-based one provides minimal benefit even for syntactic tasks. |
| Approach: | a study predicts that knowledge distillation targeting attention should selectively accelerate syntax acquisition . a systolic analysis of student models compared to logit-based knowledge distillations . |
| Outcome: | a new study shows that knowledge distillation (KD) targeting attention accelerates syntax acquisition . the hypothesis is tested on syntactic benchmarks and perplexity. |
Probing BERT’s priors with serial reproduction chains (2022.findings-acl)
Copied to clipboard
| Challenge: | Large neural language models have induced surprisingly human-like linguistic knowledge, from syntactic structure and subtle lexical biases to more insidious social biase and stereotypes. |
| Approach: | They propose to use serial reproduction chains to generate representative samples from popular masked language models like BERT to test their hypothesis. |
| Outcome: | The proposed method is based on theories of iterated learning in cognitive science and can be used to probe masked language models. |
Investigating representations of verb bias in neural language models (2020.emnlp-main)
Copied to clipboard
| Challenge: | Languages typically provide more than one grammatical construction to express certain types of messages. |
| Approach: | They propose a large benchmark dataset containing 50K human judgments for 5K distinct sentence pairs in the English dative alternation. |
| Outcome: | The proposed model outperforms recurrent architectures even under comparable parameter and training settings. |
Causal interventions expose implicit situation models for commonsense language understanding (2023.findings-acl)
Copied to clipboard
| Challenge: | Classical psycholinguistic accounts have suggested that world knowledge enters into language understanding through structured schemas called situation models. |
| Approach: | They apply causal intervention techniques to transformer models to analyze performance on the Winograd Schema Challenge . |
| Outcome: | The proposed model performs well on the Winograd Schema Challenge . |