Papers by Alexander Rudnicky
Transformer Working Memory Enables Regular Language Reasoning And Natural Language Length Extrapolation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing models of working memory are limited by self-attention, but we propose a new variant of regular programming that can model regular languages. |
| Approach: | They propose a Transformer variant that constructs working memory along the depth dimension . they find it rediscovers the local windowed attention effect needed for length extrapolation . |
| Outcome: | The proposed model rediscovers the local windowed attention effect needed for length extrapolation. |
Attention Alignment and Flexible Positional Embeddings Improve Transformer Length Extrapolation (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing methods for length extrapolation are tailored for natural language modeling, a task known to have strong recency bias. |
| Approach: | They propose two attention alignment strategies to improve T5's long-context utilization capability without fine-tuning. |
| Outcome: | The proposed methods improve the long-context utilization capability of T5 on language modeling, retrieval, multi-document question answering, and code completion tasks without any fine-tuning. |
Latent Positional Information is in the Self-Attention Variance of Transformer Language Models Without Positional Embeddings (2023.acl-short)
Copied to clipboard
| Challenge: | Recent research has called into question the necessity of positional embeddings in transformer language models. |
| Approach: | They propose to discard positional embeddings in transformer language models to facilitate more efficient pretraining. |
| Outcome: | The proposed model encodes strong positional information through shrinkage of self-attention variance. |
Adjusting Image Attributes of Localized Regions with Low-level Dialogue (2020.lrec-1)
Copied to clipboard
| Challenge: | Image editing is time-consuming and requires a wide assortment of features and combinations of these features to achieve a desired effect. |
| Approach: | They propose a task-oriented dialogue system to investigate low-level instructions for NLIE . 25% of users found the system easy-to-use, resonating with their motivation . |
| Outcome: | The proposed system is easy-to-use and user-friendly. |
Dissecting Transformer Length Extrapolation via the Lens of Receptive Field Analysis (2023.acl-long)
Copied to clipboard
| Challenge: | Length extrapolation allows training a transformer language model on short sequences that preserves perplexities when tested on substantially longer sequences. |
| Approach: | They propose a relative positional embedding design that uses longer than the training sequence to create sandwich. |
| Outcome: | The proposed model can extrapolate to L ex L tr much better than other models. |
Advancing Regular Language Reasoning in Linear Recurrent Neural Networks (2024.naacl-short)
Copied to clipboard
| Challenge: | Existing linear recurrent neural networks have been used for natural language and long-range modeling for decades. |
| Approach: | They propose a linear recurrent neural network with a block-diagonal transition matrix and a transition matrix for LRNNs. |
| Outcome: | The proposed model is the only one capable of performing length extrapolation on regular language tasks such as Sum, Even Pair, and Modular Arithmetic. |
Language Models Can be Efficiently Steered via Minimal Embedding Layer Transformations (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for fine-tuning Large Language Models (LLMs) neglect the embedding layer. |
| Approach: | They propose a PEFT approach that modifies input embeddings without altering hidden layers. |
| Outcome: | Experiments show that TinyTE modifies embeddings without altering hidden layers . the proposed approach achieves competitive performance while requiring 0.0001% of parameters . |
Zero-Shot Dialogue Disentanglement by Self-Supervised Entangled Response Selection (2021.emnlp-main)
Copied to clipboard
| Challenge: | a zero-shot dialogue disentanglement solution is difficult due to the need for manual annotation. |
| Approach: | They propose a zero-shot dialogue disentanglement solution using a web dataset . they train a model on the data and fine-tune the model using labeled data . |
| Outcome: | The proposed model achieves a cluster F1 score of 25 without labeling data . it can be used to analyze discourses and to perform response selection . |