Papers by Mostafa Dehghani
Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworks (2021.acl-long)
Copied to clipboard
| Challenge: | State-of-the-art parameter-efficient fine-tuning methods rely on introducing adapter modules between the layers of a pretrained language model. |
| Approach: | They propose a framework that can learn adapter parameters for all layers and tasks by generating them using shared hypernetworks. |
| Outcome: | The proposed framework improves performance on the well-known GLUE benchmark while adding only 0.29% parameters per task. |
Scaling Laws vs Model Architectures: How does Inductive Bias Influence Scaling? (2023.findings-emnlp)
Copied to clipboard
Yi Tay, Mostafa Dehghani, Samira Abnar, Hyung Chung, William Fedus, Jinfeng Rao, Sharan Narang, Vinh Tran, Dani Yogatama, Donald Metzler
| Challenge: | Existing studies on the scaling properties of model architectures have not explored the impact of inductive biases on scaling behaviour. |
| Approach: | They conduct extensive experiments to understand scaling behaviour of ten different model architectures. |
| Outcome: | The results show that the best performing model can fluctuate at different scales. |
DSI++: Updating Transformer Memory with New Documents (2023.emnlp-main)
Copied to clipboard
Sanket Mehta, Jai Gupta, Yi Tay, Mostafa Dehghani, Vinh Tran, Jinfeng Rao, Marc Najork, Emma Strubell, Donald Metzler
| Challenge: | Differentiable Search Indices (DSIs) encode a corpus of documents and use the same model to map queries directly to relevant document identifiers. |
| Approach: | They propose a continual learning challenge for Differentiable Search Indices (DSIs) they propose to continuously index new documents while answering queries related to previously and newly indexed documents. |
| Outcome: | The proposed model stably memorizes more documents and improves the average Hits@10 by +21.1% over baselines. |
Transcending Scaling Laws with 0.1% Extra Compute (2023.emnlp-main)
Copied to clipboard
Yi Tay, Jason Wei, Hyung Chung, Vinh Tran, David So, Siamak Shakeri, Xavier Garcia, Steven Zheng, Jinfeng Rao, Aakanksha Chowdhery, Denny Zhou, Donald Metzler, Slav Petrov, Neil Houlsby, Quoc Le, Mostafa Dehghani
| Challenge: | Existing scaling of language models is expensive and requires significant computational costs. |
| Approach: | They propose a method that substantially improves existing language models and their scaling curves with a relatively tiny amount of extra compute. |
| Outcome: | The proposed method significantly improves existing language models and their scaling curves with a relatively tiny amount of extra compute. |
Are Pretrained Convolutions Better than Pretrained Transformers? (2021.acl-long)
Copied to clipboard
| Challenge: | Recent research has shown promise in entirely convolutional, or CNN, architectures, but they have not been explored using the pre-train-fine-tune paradigm. |
| Approach: | They propose to use the pre-train-fine-tune paradigm to study convolutional models. |
| Outcome: | The proposed architectures outperform Transformers in certain scenarios, but with caveats. |
Low-Rank Adaptation for Multilingual Summarization: An Empirical Study (2024.findings-naacl)
Copied to clipboard
| Challenge: | Pre-trained Large Language Models have significantly advanced NLP, but their ever-increasing size poses significant challenges for conventional fine-tuning. |
| Approach: | They investigate the potential of Low-Rank Adaptation (LoRA) in multilingual summarization, a task that is challenging and relatively unexplored. |
| Outcome: | The proposed method outperforms full fine-tuning and cross-lingual transfer strategies in multilingual summarization tasks. |