Papers by Clement Delangue
Continuous Learning in a Hierarchical Multiscale Neural Network (P18-2)
Copied to clipboard
| Challenge: | Language models are a major class of natural language processing (NLP) models whose development has led to major progress in many areas like translation, speech recognition or summarization. |
| Approach: | They propose a hierarchical multi-scale language model where short time-scale dependencies are encoded in the hidden state of a lower-level recurrent neural network while longer time- scale dependencies can be encoded into the dynamic of the lower- level network. |
| Outcome: | The proposed model uses a meta-learner to update the weights of the lower-level neural network in an online meta-learning fashion to prevent catastrophic forgetting in the continuous learning framework. |
Transformers: State-of-the-Art Natural Language Processing (2020.emnlp-demos)
Copied to clipboard
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, Alexander Rush
| Challenge: | Transformers is an open-source library that aims to open up advances in natural language processing to the wider machine learning community. |
| Approach: | they propose an open-source library that aims to open up advances in machine learning to the wider community. |
| Outcome: | Transformers is an open-source library with the goal of opening up these advances to the wider machine learning community. |