Papers by Stanislas Dehaene
Can Transformers Process Recursive Nested Constructions, Like Humans? (2022.coling-1)
Copied to clipboard
| Challenge: | A recent study evaluated recursive processing in recurrent neural language models (RNN-LMs) and showed that such models perform below chance level on embedded dependencies within nested constructions. |
| Approach: | They evaluated recursive processing in recurrent neural language models and found that Transformers perform below chance level on embedded dependencies within nested constructions. |
| Outcome: | The proposed models perform below chance level on embedded dependencies within nested constructions, compared to humans. |
The emergence of number and syntax units in LSTM language models (N19-1)
Copied to clipboard
| Challenge: | a recent study shows that LSTMs can capture syntax-sensitive generalizations such as long-distance number agreement. |
| Approach: | They investigate the inner mechanics of number tracking in LSTMs at the single neuron level . they find that long-distance number information is largely managed by two "number units" importantly, the behaviour of these units is partially controlled by other units to track syntactic structure . |
| Outcome: | The proposed model is based on a language model with a long-distance number agreement task. |