Papers by James Michaelov
Emergent Inabilities? Inverse Scaling Over the Course of Pretraining (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent research has found that increased number of model parameters and increased size of the training dataset positively influence model performance. |
| Approach: | They investigate whether language models' performance on specific tasks can decrease over the course of training. |
| Outcome: | The proposed model size-based scaling is found on 8 tasks on which Pythia 12B shows decreased performance over the course of training. |
Structural Priming Demonstrates Abstract Grammatical Representations in Multilingual Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Abstract grammatical knowledge is key to linguistic generalization in humans . strong evidence for grammatikal abstraction in humans comes from structural priming . |
| Approach: | They compare human models of crosslingual structural priming to human models . they find evidence for abstract monolingual and crosslingual grammatical representations . |
| Outcome: | The results show that grammatical representations in multilingual models are similar to humans . the strongest evidence for grammatikal abstraction in humans comes from structural priming . |
Rarely a problem? Language models exhibit inverse scaling in their predictions following few-type quantifiers (2023.findings-acl)
Copied to clipboard
| Challenge: | Current work suggests that language models deal poorly with quantifiers-they struggle to predict which quantifier is used in a given context and also perform poorly at generating appropriate continuations following logical quantifier. |
| Approach: | They propose to use 960 English sentence stimuli to build 22 autoregressive transformer models of different sizes to test their performance on ‘few’-type quantifiers. |
| Outcome: | The proposed models perform poorly on ‘few’-type quantifiers, and the larger the model, the worse its performance. |