Papers by Juergen Schmidhuber
The Devil is in the Detail: Simple Tricks Improve Systematic Generalization of Transformers (2021.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies show that basic configurations can improve the performance of neural networks on systematic generalization. |
| Approach: | They propose to revisit basic configurations to improve the performance of Transformers on systematic generalization by revisiting scaling of embeddings, early stopping, relative positional embeddment, and Universal Transformer variants. |
| Outcome: | The proposed models improve accuracy from 50% to 85% on the PCFG productivity split and from 35% to 81% on COGS. |
CTL++: Evaluating Generalization on Never-Seen Compositional Patterns of Known Functions, and Compatibility of Neural Representations (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing neural nets fail to generalize systematically due to superficial differences in training data. |
| Approach: | They propose a new diagnostic dataset based on compositions of unary symbolic functions that tests systematicity of NNs. |
| Outcome: | The proposed dataset shows that recent CTL-solving Transformer variants fail on CTL++. |