POSITION BIAS MITIGATES POSITION BIAS: Mitigate Position Bias Through Inter-Position Knowledge Distillation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Positional bias (PB) manifests as non-uniform sensitivity across contextual locations . previous studies have addressed PB by modifying the underlying architectures or employing extensive contextual awareness training. |
| Approach: | They propose a position-to-position knowledge distillation framework that leverages position-induced disparities to counteract PB. |
| Outcome: | The proposed framework reduces positional bias and improves performance on retrieval and reasoning tasks. |
Similar Papers
Self-Consistency Falls Short! The Adverse Effects of Positional Bias on Long-Context Problems (2026.tacl-1)
Copied to clipboard
| Challenge: | Existing approaches to improve long-context understanding of large language models lack scalability and stability. |
| Approach: | They challenge the assumption that SC’s benefits generalize to long-context settings . they find that persistent position bias degrades performance on long-consistency tasks . |
| Outcome: | The proposed approach fails to improve and actively degrades performance on long-context tasks. |
Can We Instruct LLMs to Compensate for Position Bias? (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Recent studies reveal that position bias in large language models (LLMs) leads to difficulty in accessing information retrieved from the retriever. |
| Approach: | They propose to direct LLMs to allocate more attention towards a selected segment of the context through prompting. |
| Outcome: | The proposed approach improves the performance of large language models by promoting instruction with an exact document index. |
An Empirical Study of Position Bias in Modern Information Retrieval (2025.findings-emnlp)
Copied to clipboard
| Challenge: | a new evaluation framework is used to assess the extent and impact of position bias in information retrieval. |
| Approach: | They introduce a position-aware retrieval benchmark and a diagnostic metric to quantify position bias . they compare models with BM25, dense embedding models, ColBERT-style late-interaction models . |
| Outcome: | The proposed framework evaluates retrieval models for position bias from a worst-case perspective. |
Mitigate Position Bias in LLMs via Scaling a Single Hidden States Channel (2025.findings-acl)
Copied to clipboard
Yijiong Yu, Huiqiang Jiang, Xufang Luo, Qianhui Wu, Chin-Yew Lin, Dongsheng Li, Yuqing Yang, Yongfeng Huang, Lili Qiu
| Challenge: | Long-context language models exhibit position bias, also known as "lost in the middle" research shows that even long-contemporary LLMs fail to utilize all context information effectively . |
| Approach: | They propose a method to mitigate position bias by scaling positional hidden states . they propose to use a channel of hidden states to modify positional Hidden states a LCLM's positional bias . |
| Outcome: | The proposed method can improve performance by 15.2% in a "lost in the middle" benchmark. |
Do RAG Systems Really Suffer From Positional Bias? (2025.emnlp-main)
Copied to clipboard
| Challenge: | Retrieval Augmented Generation (RAG) improves the factual accuracy of LLMs on knowledgeintensive tasks by including in the prompt passages retrieved from an external corpus. |
| Approach: | They propose to use a retrieval algorithm to add passages from an external corpus to the LLM prompt to improve the factual accuracy of LLMs. |
| Outcome: | The proposed approach improves the factual accuracy of LLMs on knowledgeintensive tasks by including in the prompt passages retrieved from an external corpus. |
Where to show Demos in Your Prompt: A Positional Bias of In-Context Learning (2025.emnlp-main)
Copied to clipboard
| Challenge: | In-context learning (ICL) is a critical emerging capability of large language models (LLMs), enabling few-shot learning during inference by including a few demonstrations in the prompt. |
| Approach: | They propose to use positional bias to study ICL's performance for the first time by examining the positional variation in demos, system prompt, and user message in LLM input. |
| Outcome: | The proposed model can predict accuracy and accuracy when demos are placed at different positions in the input prompt and in the user message. |
PairDistill: Pairwise Relevance Distillation for Dense Retrieval (2024.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in dense retrieval have demonstrated remarkable efficacy compared to traditional sparse retrieval methods. |
| Approach: | They propose to use pairwise relevance distillation to leverage pairwise reranking to enrich the training of dense retrieval models. |
| Outcome: | The proposed method outperforms existing methods and achieves state-of-the-art results on multiple benchmarks. |
Can Calibration of Positional Encodings Enhance Long Context Utilization? (2026.findings-eacl)
Copied to clipboard
| Challenge: | Large language models suffer from positional biases that reduce effective utilization of long contexts. |
| Approach: | They propose a training-free framework for calibrating Positional Encodings at inference time. |
| Outcome: | The proposed framework improves on needle-in-a-haystack and cross-chunk reasoning benchmarks and provides a lightweight method for improving long-context utilization. |
Positional Overload: Positional Debiasing and Context Window Extension for Large Language Models using Set Encoding (2025.acl-long)
Copied to clipboard
| Challenge: | Large Language Models typically track the order of tokens using positional encoding, which causes two significant limitations: 1. Positional Bias: When processing long text sequences, the number of token can exceed the range the model was trained on. |
| Approach: | They propose a method that allows multiple pieces of text to be encoded in the same position, eliminating positional bias entirely. |
| Outcome: | The proposed method eliminates positional bias entirely and increases the size of the input an LLM can handle. |
Self-Supervised Position Debiasing for Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for debiasing large language models require external bias knowledge or annotated non-biased samples, which is lacking for position debiases. |
| Approach: | They propose a self-supervised position debiasing framework that leverages unsupervised responses from pre-trained LLMs for debiazing without external bias knowledge. |
| Outcome: | The proposed framework outperforms existing methods in mitigating three types of position biases on eight datasets and five tasks. |