Papers by Samira Abnar
Scaling Laws vs Model Architectures: How does Inductive Bias Influence Scaling? (2023.findings-emnlp)
Copied to clipboard
Yi Tay, Mostafa Dehghani, Samira Abnar, Hyung Chung, William Fedus, Jinfeng Rao, Sharan Narang, Vinh Tran, Dani Yogatama, Donald Metzler
| Challenge: | Existing studies on the scaling properties of model architectures have not explored the impact of inductive biases on scaling behaviour. |
| Approach: | They conduct extensive experiments to understand scaling behaviour of ten different model architectures. |
| Outcome: | The results show that the best performing model can fluctuate at different scales. |
Quantifying Attention Flow in Transformers (2020.acl-main)
Copied to clipboard
| Challenge: | In the Transformer model, “self-attention” combines information from attended embeddings into the representation of the focal embeddable in the next layer. |
| Approach: | They propose two methods to quantify flow of information through self-attention using attention weights as relative relevance of input tokens. |
| Outcome: | The proposed methods give complementary views on the flow of information and yield higher correlations with importance scores of input tokens. |