| Challenge: | In natural language, it is not absolute position that matters, but relative position . et al., 2017) language models incorporate positional encodings that encode absolute (linear) word order. |
| Approach: | They find that Transformer language models encode word order using positional information . they also find that models that use absolute position embeddings over-rely on positional data . |
| Outcome: | The results raise questions about the efficacy of APEs to model the relativity of position information. |
Similar Papers
What Do Position Embeddings Learn? An Empirical Study of Pre-Trained Language Model Positional Encoding (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing work on pre-trained Transformers has focused on learning the meaning of positions . Embedding the position information in the self-attention mechanism is also an indispensable factor in NLP . |
| Approach: | They propose to use feature-level analysis to examine pre-trained Transformers' position embeddings . they also use empirical experiments to determine the appropriate positional encoding function . |
| Outcome: | The results of the empirical study can guide future work to choose the appropriate positional encoding function for specific tasks. |
Word Order Does Matter and Shuffled Language Models Know It (2022.acl-long)
Copied to clipboard
| Challenge: | Recent studies have shown that language models pretrained and/or fine-tuned on randomly permuted sentences exhibit competitive performance on GLUE, putting into question the importance of word order information. |
| Approach: | They propose a transformer-based BERT architecture that uses a fixed, sinusoidal position embedding added to each token embeddable to compensate for this absence of linear order. |
| Outcome: | The proposed model retains word order information because of the dependencies between sentence length and unigram probabilities. |
Improve Transformer Models with Better Relative Position Embeddings (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for generating position embeddings are not fully utilized in NLP tasks. |
| Approach: | They propose to generalize the absolute position embedding to a generalized relative position embedded method . they also propose to use the relative embeddable method to improve the accuracy of large models . |
| Outcome: | The proposed method improves accuracy on the SQuAD1.1 dataset compared to previous methods . it can be easily adopted as a drop-in replacement for improving accuracy of large models . |
Dwell in the Beginning: How Language Models Embed Long Documents for Dense Retrieval (2024.acl-short)
Copied to clipboard
| Challenge: | Existing studies have shown that Transformer-based language models lose information in the middle of input sequences, especially in the context of web document retrieval. |
| Approach: | They examine position biases at multiple stages of the training pipeline for an encoder-decoder neural retrieval model, namely language model pre-training, contrastive pre- training, and contrastive fine-tuning. |
| Outcome: | The proposed model generates embeddings that better capture the beginning of the input content, with fine-tuning further aggravating this effect. |
Word Order Matters When You Increase Masking (2022.emnlp-main)
Copied to clipboard
| Challenge: | Recent work shows that word order information is not always useful for pre-trained models to solve downstream tasks. |
| Approach: | They examine the effect of removing position encodings on the pre-training objective itself to see whether models can reconstruct position information from co-occurrences. |
| Outcome: | The proposed model can reconstruct position information from co-occurrences without explicit encoding, but not on some tasks. |
The Impact of Positional Encodings on Multilingual Compression (2021.emnlp-main)
Copied to clipboard
| Challenge: | Several modifications have been proposed to improve monolingual language models, but none of them result in better multilingual models. |
| Approach: | They propose to add positional encodings to token embeddings to preserve word-order information in a non-autoregressive setting. |
| Outcome: | The proposed modifications tend to improve monolingual models, but none improve multilingual models. |
Latent Positional Information is in the Self-Attention Variance of Transformer Language Models Without Positional Embeddings (2023.acl-short)
Copied to clipboard
| Challenge: | Recent research has called into question the necessity of positional embeddings in transformer language models. |
| Approach: | They propose to discard positional embeddings in transformer language models to facilitate more efficient pretraining. |
| Outcome: | The proposed model encodes strong positional information through shrinkage of self-attention variance. |
Revisiting Zero-Shot Abstractive Summarization in the Era of Large Language Models from the Perspective of Position Bias (2024.naacl-short)
Copied to clipboard
| Challenge: | Position bias is a tendency of a model unfairly prioritizing information from certain parts of the input text over others, leading to undesirable behavior. |
| Approach: | They propose to measure position bias in large language models for zero-shot summarization tasks by measuring position bias. |
| Outcome: | The proposed model performance and position biases lead to new insights and discussion on zero-shot summarization tasks. |
Information Representation Fairness in Long-Document Embeddings: The Peculiar Interaction of Positional and Language Bias (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing studies show that embedding models exhibit systematic positional and language biases when documents are longer and consist of multiple segments. |
| Approach: | They propose a permutation-based evaluation framework to quantify embedding biases . they propose an inference-time attention calibration method that redistributes attention more evenly across document positions . |
| Outcome: | The proposed framework reduces the positional and language biases in embedding models . the proposed framework improves the discoverability of later segments . |
Transformer Language Models without Positional Encodings Still Learn Positional Information (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Using positional embeddings, Causal transformer language models learn an implicit notion of absolute positions. |
| Approach: | They propose to use positional embeddings to encode positional information in transformer language models. |
| Outcome: | The proposed model learns an implicit notion of absolute positions across datasets, model sizes, and sequence lengths. |