Papers by Fabio Fehr
HyperMixer: An MLP-based Low Cost Alternative to Transformers (2023.acl-long)
Copied to clipboard
Florian Mai, Arnaud Pannatier, Fabio Fehr, Haolin Chen, Francois Marelli, Francois Fleuret, James Henderson
| Challenge: | Existing MLP-based architectures that combine multiple features are expensive and require a lot of training data. |
| Approach: | They propose a simple MLP-based model which allows token mixing by dynamically applying hypernetworks to each feature independently. |
| Outcome: | The proposed model performs better than Transformers and lowers costs in terms of processing time, training data, and hyperparameter tuning. |
Learning to Abstract with Nonparametric Variational Information Bottleneck (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Learning textual embeddings is costly as they are tokenization specific and require different models to be trained for each level of abstraction. |
| Approach: | They propose a character-level representation model which learns different levels of abstraction in different layers of the same model. |
| Outcome: | The proposed model is more robust to adversarial perturbations and can learn to compress representations to different levels of abstraction. |