| Challenge: | *context length probing* is a new explanation technique for large language models . it tracks the predictions of a model as a function of the length of available context . |
| Approach: | They propose a method that tracks the predictions of a model as a function of the length of available context and allows to assign *differential importance scores* to different contexts. |
| Outcome: | The proposed method is model-agnostic and does not rely on access to model internals beyond token-level probabilities. |
Similar Papers
How Large Language Models Encode Context Knowledge? A Layer-Wise Probing Study (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing studies have focused on enhancing the factualness of large language models using context knowledge. |
| Approach: | They propose to use ChatGPT to construct probing datasets that provide diverse and coherent evidence corresponding to various facts. |
| Outcome: | The proposed model can encode knowledge across different layers, and it is compared with existing models. |
Explanation in the Era of Large Language Models (2024.naacl-tutorials)
Copied to clipboard
| Challenge: | Explanation has long been a part of communication, where humans use language to elucidate each other and transmit information about mechanisms of events. |
| Approach: | They review the opportunities and challenges of explanations in the era of large language models and examine how they can be used to generate explanations. |
| Outcome: | The proposed methods are based on the models of large language models (LLMs) and their opaque nature. |
Lost in the Prompt Order: Revealing the Limitations of Causal Attention in Language Models (2026.findings-acl)
Copied to clipboard
| Challenge: | Large language models exhibit surprising sensitivity to structure of the prompt, but mechanisms underlying this sensitivity remain poorly understood. |
| Approach: | They conduct an in-depth investigation on placing context before the questions and options in MCQA prompts. |
| Outcome: | The proposed model outperforms the reverse order (QOC) by over 14%p over a wide range of models and datasets. |
Logic Haystacks: Probing LLMs’ Long-Context Logical Reasoning (Without Easily Identifiable Unrelated Padding) (2026.eacl-short)
Copied to clipboard
| Challenge: | Recent large language models claim long context windows, but evaluations often involve simple retrieval tasks or synthetic tasks padded with irrelevant text. |
| Approach: | They use grammars to generate simplified English with logical representations to create long input text while controlling its semantics. |
| Outcome: | The proposed model performs better with realistic distractors than with standard models. |
In-Context Learning (and Unlearning) of Length Biases (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing work has demonstrated the ability of large language models to learn lexical and label biases in-context negatively impacts performance and robustness of models. |
| Approach: | They investigate the impact of length biases on in-context learning by analyzing model length information in-constext. |
| Outcome: | The proposed model learns length biases in the context window without parameter updates. |
On the data requirements of probing (2022.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to probe neural networks are expensive and require large datasets. |
| Approach: | They propose a method to estimate the required number of data samples in probing datasets . they use a classification task to encode a text with a deep neural network . |
| Outcome: | The proposed method estimates the required number of data samples in two probing configurations and proves it is statistically powerful. |
How to Train Long-Context Language Models (Effectively) (2025.acl-long)
Copied to clipboard
| Challenge: | a new study shows that language models can process extremely long contexts with minimal training. |
| Approach: | They use supervised fine-tuning and continued training to evaluate a language model's long-context capabilities. |
| Outcome: | The proposed model outperforms Llama-3.1-8B-Instruct on most long-context tasks . the model can process 512K tokens, one of the longest context windows of LMs . |
NeedleChain: Measuring Intact Context Comprehension Capability of Large Language Models (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing benchmarks for context understanding embed query-irrelevant content . this shifts evaluation toward retrieving relevant snippets rather than fully integrating all provided information. |
| Approach: | They propose a benchmark to evaluate whether models can faithfully incorporate all given evidence . they propose 'needlechain' benchmark to test whether models incorporate all available information . |
| Outcome: | The proposed benchmarks overestimate the ability of large language models to integrate all given evidence when the context is entirely query-relevant. |
Larger Probes Tell a Different Story: Extending Psycholinguistic Datasets Via In-Context Learning (2023.emnlp-main)
Copied to clipboard
| Challenge: | Language model probing is often used to test specific capabilities of models, but results are limited when benchmarks are small and lack statistical power. |
| Approach: | They extend existing NEG-136 and ROLE-88 benchmarks to 750 sentence pairs and create an extended negation dataset using template-based generation. |
| Outcome: | The proposed datasets show that models perform better on the extended datasets than on the smaller benchmarks. |
Explaining Language Model Predictions with High-Impact Concepts (2024.findings-eacl)
Copied to clipboard
| Challenge: | Existing methods to explain large language models (LLMs) are mostly correlational and lack causal features due to compositional nature of languages. |
| Approach: | They propose a framework to provide impact-aware explanations for large language models that are robust to feature changes and influential to the model’s predictions. |
| Outcome: | The proposed explanations improve on real and synthetic tasks and are robust to feature changes and influential to the model’s predictions. |