Papers by Damian Pascual
pNLP-Mixer: an Efficient all-MLP Architecture for Language (2023.acl-industry)
Copied to clipboard
| Challenge: | large pre-trained language models are impractical for on-device applications due to their size and inference cost. |
| Approach: | They propose an embedding-free MLP-Mixer model for on-device NLP that achieves high weight-efficiency thanks to a novel projection layer. |
| Outcome: | The proposed model beats state-of-the-art of tiny models by 97.8% on two datasets . it beats mBERT on MTOP and multiATIS, while using 170x less parameters . |
A Plug-and-Play Method for Controlled Text Generation (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for controlling language generation are not able to produce fluent text . current methods require additional models or fine-tuning to ensure specific words are included . |
| Approach: | They propose a plug-and-play decoding method that allows for controlled language generation . they add a shift in the probability distribution over our vocabulary towards semantically similar words . |
| Outcome: | The proposed method outperforms competing methods in human evaluations and does not impact fluency. |
Telling BERT’s Full Story: from Local Attention to Global Aggregation (2021.eacl-main)
Copied to clipboard
| Challenge: | Recent work discouraging the use of attention distributions for explaining a model’s behaviour suggests that attention distribution can provide insights into local behaviour of attention heads. |
| Approach: | They propose a distinction between local patterns revealed by attention and global patterns that refer back to the input and analyze BERT from both angles. |
| Outcome: | The proposed model can explain local behaviour of attention heads by comparing local and global patterns from both angles. |