Papers by Amjad Almahairi
Residual Prompt Tuning: improving prompt tuning with residual reparameterization (2023.findings-acl)
Copied to clipboard
| Challenge: | Prompt tuning is one of the most parameter-efficient approaches for parameter-effective tuning of pre-trained language models. |
| Approach: | They propose to reparameterize soft prompt embeddings using a shallow network with a residual connection and use it to tune prompt embeds P. |
| Outcome: | The proposed method outperforms prompt tuning on SuperGLUE, T5-Base and BERT-Bass models and can reduce the prompt length by 10 times without hurting performance. |
Learning Easily Updated General Purpose Text Representations with Adaptable Task-Specific Prefix (2023.findings-emnlp)
Copied to clipboard
| Challenge: | a large pre-trained language model can cause computational burdens in inference time due to multiple forward passes. |
| Approach: | They propose a method to learn fixed text representations with source tasks . they learn a task-specific prefix for each source task independently and combine them . |
| Outcome: | The proposed method improves generalizability of representations with source tasks. |
UniPELT: A Unified Framework for Parameter-Efficient Language Model Tuning (2022.acl-long)
Copied to clipboard
| Challenge: | Existing methods for parameter-efficient language model tuning (PELT) match the performance of fine-tuning with fewer trainable parameters. |
| Approach: | They propose a framework which integrates different PELT methods as submodules and learns to activate the ones that best suit the current data or task setup via gating mechanism. |
| Outcome: | The proposed framework outperforms fine-tuning methods on the GLUE benchmark and achieves 14% gains over the best individual PELT method. |