Papers by Vijay Shanker
ArabicTransformer: Efficient Large Arabic Language Model with Funnel Transformer and ELECTRA Objective (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Existing solutions to reduce the cost of pretraining Transformer-based models are expensive especially for large-scale models. |
| Approach: | They propose to reduce the cost of pre-training Transformer-based models by compressing the sequence of hidden states inside Transformer architecture. |
| Outcome: | The proposed model achieves state-of-the-art on several Arabic downstream tasks despite using less computational resources compared to other BERT-based models. |