Papers by Yair Lakretz
Language acquisition: do children and language models follow similar learning stages? (2023.findings-acl)
Copied to clipboard
| Challenge: | During language acquisition, children follow a typical sequence of learning stages, whereby they first learn to categorize phonemes before they develop their lexicon and eventually master complex syntactic structures. |
| Approach: | They train 48 GPT-2 models from scratch and evaluate their syntactic and semantic abilities at each training step using 96 probes curated from the BLiMP, Zorro and BIG-Bench benchmarks. |
| Outcome: | The proposed model exhibits similar learning trajectories to human children aged between 18 months and 6 years. |
Meta-Learning Neural Mechanisms rather than Bayesian Priors (2025.acl-long)
Copied to clipboard
| Challenge: | a meta-learning approach has been proposed to integrate human-like learning biases into neural networks . a recent study suggests that meta-training on a single formal language can improve a model . |
| Approach: | They propose to integrate human-like learning biases into neural-network architectures . they use symbolic models to capture aspects of humans' basic generalisations from small data . |
| Outcome: | The proposed model can learn from a single language as much as 5000 different languages . the model can be scaled to a larger model and training datasets . |
Can Transformers Process Recursive Nested Constructions, Like Humans? (2022.coling-1)
Copied to clipboard
| Challenge: | A recent study evaluated recursive processing in recurrent neural language models (RNN-LMs) and showed that such models perform below chance level on embedded dependencies within nested constructions. |
| Approach: | They evaluated recursive processing in recurrent neural language models and found that Transformers perform below chance level on embedded dependencies within nested constructions. |
| Outcome: | The proposed models perform below chance level on embedded dependencies within nested constructions, compared to humans. |
The emergence of number and syntax units in LSTM language models (N19-1)
Copied to clipboard
| Challenge: | a recent study shows that LSTMs can capture syntax-sensitive generalizations such as long-distance number agreement. |
| Approach: | They investigate the inner mechanics of number tracking in LSTMs at the single neuron level . they find that long-distance number information is largely managed by two "number units" importantly, the behaviour of these units is partially controlled by other units to track syntactic structure . |
| Outcome: | The proposed model is based on a language model with a long-distance number agreement task. |