Papers by Theo Desbordes
The emergence of number and syntax units in LSTM language models (N19-1)
Copied to clipboard
| Challenge: | a recent study shows that LSTMs can capture syntax-sensitive generalizations such as long-distance number agreement. |
| Approach: | They investigate the inner mechanics of number tracking in LSTMs at the single neuron level . they find that long-distance number information is largely managed by two "number units" importantly, the behaviour of these units is partially controlled by other units to track syntactic structure . |
| Outcome: | The proposed model is based on a language model with a long-distance number agreement task. |