Papers by Ting-Han Fan
Transformer Working Memory Enables Regular Language Reasoning And Natural Language Length Extrapolation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing models of working memory are limited by self-attention, but we propose a new variant of regular programming that can model regular languages. |
| Approach: | They propose a Transformer variant that constructs working memory along the depth dimension . they find it rediscovers the local windowed attention effect needed for length extrapolation . |
| Outcome: | The proposed model rediscovers the local windowed attention effect needed for length extrapolation. |
Attention Alignment and Flexible Positional Embeddings Improve Transformer Length Extrapolation (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing methods for length extrapolation are tailored for natural language modeling, a task known to have strong recency bias. |
| Approach: | They propose two attention alignment strategies to improve T5's long-context utilization capability without fine-tuning. |
| Outcome: | The proposed methods improve the long-context utilization capability of T5 on language modeling, retrieval, multi-document question answering, and code completion tasks without any fine-tuning. |
Latent Positional Information is in the Self-Attention Variance of Transformer Language Models Without Positional Embeddings (2023.acl-short)
Copied to clipboard
| Challenge: | Recent research has called into question the necessity of positional embeddings in transformer language models. |
| Approach: | They propose to discard positional embeddings in transformer language models to facilitate more efficient pretraining. |
| Outcome: | The proposed model encodes strong positional information through shrinkage of self-attention variance. |
Dissecting Transformer Length Extrapolation via the Lens of Receptive Field Analysis (2023.acl-long)
Copied to clipboard
| Challenge: | Length extrapolation allows training a transformer language model on short sequences that preserves perplexities when tested on substantially longer sequences. |
| Approach: | They propose a relative positional embedding design that uses longer than the training sequence to create sandwich. |
| Outcome: | The proposed model can extrapolate to L ex L tr much better than other models. |
Advancing Regular Language Reasoning in Linear Recurrent Neural Networks (2024.naacl-short)
Copied to clipboard
| Challenge: | Existing linear recurrent neural networks have been used for natural language and long-range modeling for decades. |
| Approach: | They propose a linear recurrent neural network with a block-diagonal transition matrix and a transition matrix for LRNNs. |
| Outcome: | The proposed model is the only one capable of performing length extrapolation on regular language tasks such as Sum, Even Pair, and Modular Arithmetic. |