DAPE V2: Process Attention Score as Feature Map for Length Extrapolation

Challenge:	Extensive experiments demonstrate that treating attention as a feature map and applying convolution as . a processing method significantly enhances Transformer performance.
Approach:	They propose to use the convolution operator to mimic the processing methods in computer vision to treat attention as a feature map and apply it to neighboring attention scores across different heads.
Outcome:	The proposed model can be adapted to various attention-related models and achieves high performance.

Similar Papers

Overview

Challenge

Approach

Outcome

DA-Transformer: Distance-aware Transformer (2021.naacl-main)

Copied to clipboard

Chuhan Wu, Fangzhao Wu, Yongfeng Huang

Language Models

Challenge:	Existing models that capture token distances are not optimal for modeling the orders and relations of contexts.
Approach:	They propose a distance-aware Transformer that can exploit the real distances between tokens to re-scale the raw self-attention weights.
Outcome:	The proposed model outperforms the existing Transformer and its variants on five benchmark datasets and can improve the performance of many tasks.

On the Distribution, Sparsity, and Inference-time Quantization of Attention Values in Transformers (2021.findings-acl)

Copied to clipboard

Tianchu Ji, Shraddhan Jain, Michael Ferdman, Peter Milder, H. Andrew Schwartz, Niranjan Balasubramanian

Language Models

Challenge:	Recent work shows that attention can be pruned to zeros with minimal loss in accuracy.
Approach:	They propose a pruning technique which quantizes attention to a 3-bit format without retraining . they find that 80% of attention values can be pruned to zeros with minimal loss in accuracy .
Outcome:	The proposed approach produces only a few unique attention values with minimal loss in accuracy.

How Much Does Attention Actually Attend? Questioning the Importance of Attention in Pretrained Transformers (2022.findings-emnlp)

Copied to clipboard

Michael Hassid, Hao Peng, Daniel Rotem, Jungo Kasai, Ivan Montero, Noah A. Smith, Roy Schwartz

Language Models

Challenge:	Pretrained language models use the attention mechanism to contextualize input inputs . but, we find that it is not as important as thought for pretrained models .
Approach:	They propose a probing method that replaces input-dependent attention matrices with constant ones.
Outcome:	The proposed method improves performance of pretrained language models without input-dependent attention.

A Multiscale Visualization of Attention in the Transformer Model (P19-3)

Copied to clipboard

Jesse Vig

Language Models

Challenge:	Various tools have been developed to visualize attention in NLP models, ranging from attention-matrix heatmaps to bipartite graph representations.
Approach:	They propose an open-source tool that visualizes attention at multiple scales and provides a unique perspective on the attention mechanism.
Outcome:	The proposed model outperforms OpenAI GPT-2 and BERT on several language modeling benchmarks.

ETC: Encoding Long and Structured Inputs in Transformers (2020.emnlp-main)

Copied to clipboard

Joshua Ainslie, Santiago Ontanon, Chris Alberti, Vaclav Cvicek, Zachary Fisher, Philip Pham, Anirudh Ravula, Sumit Sanghai, Qifan Wang, Li Yang

Language Models

Challenge:	Existing models for natural language processing (NLP) have been challenging to scale attention to longer inputs.
Approach:	They propose an extended Transformer construction architecture that scales attention to longer inputs by combining global-local attention with relative position encodings and a "Contrastive Predictive Coding" objective.
Outcome:	The proposed architecture scales attention to longer inputs and encodes structured inputs.

The NLP Task Effectiveness of Long-Range Transformers (2023.eacl-main)

Copied to clipboard

Guanghui Qin, Yukun Feng, Benjamin Van Durme

Language Models

Challenge:	Existing benchmarks on long-range attention models have not been sufficient to develop efficient Transformers and their practical application on complex NLP tasks.
Approach:	They propose to benchmark 7 Transformer variants on 5 difficult NLP tasks and 7 datasets to examine their capacity for long-range attention.
Outcome:	The proposed models have advantages on content selection and query-guided decoding, but they come with previously unrecognized drawbacks such as insufficient attention to distant tokens and accumulated approximation error.

Attention Alignment and Flexible Positional Embeddings Improve Transformer Length Extrapolation (2024.findings-naacl)

Copied to clipboard

Ta-Chung Chi, Ting-Han Fan, Alexander Rudnicky

Semantic Text Processing

Challenge:	Existing methods for length extrapolation are tailored for natural language modeling, a task known to have strong recency bias.
Approach:	They propose two attention alignment strategies to improve T5's long-context utilization capability without fine-tuning.
Outcome:	The proposed methods improve the long-context utilization capability of T5 on language modeling, retrieval, multi-document question answering, and code completion tasks without any fine-tuning.

Adaptive Attention Span in Transformers (P19-1)

Copied to clipboard

Sainbayar Sukhbaatar, Edouard Grave, Piotr Bojanowski, Armand Joulin

Language Models

Challenge:	We extend the maximum context size of a neural network called Transformer to 8k characters.
Approach:	They propose a self-attention mechanism that can learn its optimal attention span . this allows for models with longer context and the capability to catch longer dependencies.
Outcome:	The proposed model achieves state-of-the-art performance on text8 and enwiki8 using 8k characters with no loss of performance, and maintains control over memory footprint and computational time.

Length Extrapolation of Transformers: A Survey from the Perspective of Positional Encoding (2024.findings-emnlp)

Copied to clipboard

Liang Zhao, Xiachong Feng, Xiaocheng Feng, Weihong Zhong, Dongliang Xu, Qing Yang, Hongtao Liu, Bing Qin, Ting Liu

Language Models

Challenge:	Existing methods to enhance length extrapolation of large language models have been developed, but a systematic survey is lacking.
Approach:	They propose to examine the effects of positional encoding on length extrapolation.
Outcome:	The proposed methods improve the extrapolation of large language models, but they are still lacking a systematic survey.

Quantifying Attention Flow in Transformers (2020.acl-main)

Copied to clipboard

Samira Abnar, Willem Zuidema

Semantic Text Processing

Challenge:	In the Transformer model, “self-attention” combines information from attended embeddings into the representation of the focal embeddable in the next layer.
Approach:	They propose two methods to quantify flow of information through self-attention using attention weights as relative relevance of input tokens.
Outcome:	The proposed methods give complementary views on the flow of information and yield higher correlations with importance scores of input tokens.