Papers by Kazuki Irie
Practical Computational Power of Linear Transformers and Their Recurrent and Self-Referential Extensions (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies of the computational power of recurrent neural networks reveal a hierarchy of RNN architectures, given finite-precision assumptions. |
| Approach: | They propose to use auto-regressive Transformers with linearised attention to build RNNs . they show that many well-known results for the standard Transformer directly transfer to LTs - a new approach is proposed . |
| Outcome: | The proposed extensions overcome limitations of the LT and self-referential weight matrices. |
The Devil is in the Detail: Simple Tricks Improve Systematic Generalization of Transformers (2021.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies show that basic configurations can improve the performance of neural networks on systematic generalization. |
| Approach: | They propose to revisit basic configurations to improve the performance of Transformers on systematic generalization by revisiting scaling of embeddings, early stopping, relative positional embeddment, and Universal Transformer variants. |
| Outcome: | The proposed models improve accuracy from 50% to 85% on the PCFG productivity split and from 35% to 81% on COGS. |
CTL++: Evaluating Generalization on Never-Seen Compositional Patterns of Known Functions, and Compatibility of Neural Representations (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing neural nets fail to generalize systematically due to superficial differences in training data. |
| Approach: | They propose a new diagnostic dataset based on compositions of unary symbolic functions that tests systematicity of NNs. |
| Outcome: | The proposed dataset shows that recent CTL-solving Transformer variants fail on CTL++. |
Why Are Positional Encodings Nonessential for Deep Autoregressive Transformers? A Petroglyph Revisited (2025.findings-acl)
Copied to clipboard
| Challenge: | Autoregressive Transformer language models do not require explicit positional encodings (PEs) this is because a cascade of (permutation invariant) set processors collectively exhibit sequence-sensitive behavior in the autoregressively setting. |
| Approach: | They propose to explain why autoregressive Transformers require explicit positional encodings (PEs) this property has been known since early efforts adopting the Transformer for language modeling . |
| Outcome: | The proposed model can distinguish sequences with permuted tokens without the need for explicit PEs. |
Approximating Two-Layer Feedforward Networks for Efficient Transformers (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent work uses sparse Mixtures of Experts (MoEs) to build resource-efficient large language models. |
| Approach: | They propose a general framework that unifies various methods to build two-layer NNs . they propose methods to improve both MoEs and PKMs based on their results . |
| Outcome: | The proposed framework improves both MoEs and product-key memories (PKMs) it shows that MoE's are competitive with dense Transformer-XL on two different scales while being much more resource efficient. |