Papers with dot-product
Tensorized Self-Attention: Efficiently Modeling Pairwise and Global Dependencies Together (N19-1)
Copied to clipboard
| Challenge: | Neural networks equipped with self-attention have parallelizable computation and the ability to capture both long-range and local dependencies. |
| Approach: | They propose a novel attention mechanism called "Multi-mask Tensorized Self-Attention" it captures pairwise and global dependencies by a compatibility function composed of dot-product and additive attentions . |
| Outcome: | The proposed model outperforms CNN-/RNN-/attention-based models on nine NLP benchmarks with compelling memory- and time-efficiency. |
Large Dual Encoders Are Generalizable Retrievers (2022.emnlp-main)
Copied to clipboard
Jianmo Ni, Chen Qu, Jing Lu, Zhuyun Dai, Gustavo Hernandez Abrego, Ji Ma, Vincent Zhao, Yi Luan, Keith Hall, Ming-Wei Chang, Yinfei Yang
| Challenge: | Experimental results show that dual encoders outperform sparse and dense retrievers on the BEIR dataset significantly. |
| Approach: | They challenge belief that bottleneck layer is too limited for out-of-domain generalization . they scale up the model while keeping bottleneck as a single dot-product with a fixed size . |
| Outcome: | The proposed model outperforms sparse and dense retrievers on the BEIR dataset significantly. |