Can’t Remember Details in Long Documents? You Need Some R&R (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Long-context large language models miss important information in the middle of context documents . a recent study shows that LLMs can be used for document-based QA tasks . |
| Approach: | They propose a prompt-based method called *reprompting* and *in-context retrieval* to alleviate this effect in document-based QA. |
| Outcome: | The proposed method improves QA accuracy on documents up to 80k tokens in length. |
Similar Papers
Leave No Document Behind: Benchmarking Long-Context LLMs with Extended Multi-Doc QA (2024.emnlp-main)
Copied to clipboard
Minzheng Wang, Longze Chen, Fu Cheng, Shengyi Liao, Xinghua Zhang, Bingli Wu, Haiyang Yu, Nan Xu, Lei Zhang, Run Luo, Yunshui Li, Min Yang, Fei Huang, Yongbin Li
| Challenge: | Existing benchmarks for evaluating long-context language models employ irrelevant noise texts to artificially extend the length of test cases, diverging from the real-world scenarios of long-constituency applications. |
| Approach: | They propose a long-context benchmark, Loong, aligning with realistic scenarios through extended multi-document question answering (QA) . |
| Outcome: | The proposed model can scale up the context window of large language models to perform in-depth analysis of multiple long documents. |
More Bang for your Context: Virtual Documents for Question Answering over Long Documents (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models struggle to utilize long contexts efficiently, resulting in a question answering problem. |
| Approach: | They propose a method to generate a short document that contains the most relevant parts for a given context window. |
| Outcome: | The proposed method improves the QA task by providing a short and focused VDoc to the LLM while keeping the context window full. |
KV Cache Compression, But What Must We Give in Return? A Comprehensive Benchmark of Long Context Capable Approaches (2024.findings-emnlp)
Copied to clipboard
Jiayi Yuan, Hongyi Liu, Shaochen Zhong, Yu-Neng Chuang, Songchen Li, Guanchu Wang, Duy Le, Hongye Jin, Vipin Chaudhary, Zhaozhuo Xu, Zirui Liu, Xia Hu
| Challenge: | Long context capability is a crucial competency for large language models as it mitigates the human struggle to digest long-form texts. |
| Approach: | They propose to evaluate 10+ state-of-the-art approaches for long context-capable LLMs. |
| Outcome: | The proposed methods are compared against 10+ state-of-the-art approaches across seven categories of long context tasks. |
Stronger Baselines for Retrieval-Augmented Generation with Long-Context Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing long-context language models (LMs) can handle tens of thousands of tokens in a single context window. |
| Approach: | They compare two recent multi-stage pipelines, ReadAgent and RAPTOR, against three baselines. |
| Outcome: | The proposed pipelines outperform more complex methods on multiple long-context QA benchmarks. |
LongCite: Enabling LLMs to Generate Fine-grained Citations in Long-Context QA (2025.findings-acl)
Copied to clipboard
Jiajie Zhang, Yushi Bai, Xin Lv, Wanjun Gu, Danqing Liu, Minhao Zou, Shulin Cao, Lei Hou, Yuxiao Dong, Ling Feng, Juanzi Li
| Challenge: | Current long-context large language models lack citations to support their responses, making verification difficult due to potential hallucinations. |
| Approach: | They propose to use off-the-shelf LLMs to automatically construct long-context QA instances with precise sentence-level citations and leverage this pipeline to construct a large-scale SFT dataset for LQAC. |
| Outcome: | The proposed pipeline can generate responses with fine-grained citations on the fly, surpassing existing models including GPT-4o. |
Tagging-Augmented Generation: Assisting Language Models in Finding Intricate Knowledge In Long Contexts (2025.emnlp-industry)
Copied to clipboard
Anwesan Pal, Karen Hovsepian, Tinghao Guo, Mengnan Zhao, Somendra Tripathi, Nikos Kanakaris, George Mihaila, Sumit Nigam
| Challenge: | Recent studies into effective context lengths of flagship large language models (LLMs) have revealed major limitations in effective question answering (QA) and reasoning over long and complex contexts for even the largest and most impressive cadre of models. |
| Approach: | They propose a lightweight data augmentation strategy that boosts LLM performance in long-context scenarios without degrading and altering the integrity and composition of retrieved documents. |
| Outcome: | The proposed strategy boosts performance in long-context scenarios without degrading and altering the integrity and composition of retrieved documents. |
Exploring Fine-Tuning for In-Context Retrieval and Efficient KV-Caching in Long-Context Language Models (2026.eacl-short)
Copied to clipboard
| Challenge: | Long-Context Language Models (LCLMs) can encode entire document collections, offering a strong alternative to retrieval-augmented generation (RAG). |
| Approach: | They propose to use LCLMs to encode documents with context windows of millions of tokens to improve their performance. |
| Outcome: | The proposed training strategies improve long-context performance and their robustness under compression techniques. |
ELITR-Bench: A Meeting Assistant Benchmark for Long-Context Language Models (2025.coling-main)
Copied to clipboard
| Challenge: | Existing benchmarks for long-context LLMs focus on generic tasks that are not necessarily aligned with real-world applications. |
| Approach: | They propose to augment existing ELITR corpus by adding 271 manually crafted questions with their ground-truth answers and noisy versions of meeting transcripts altered to target different Word Error Rate levels. |
| Outcome: | The proposed benchmark augments the existing ELITR corpus by adding 271 manually crafted questions with ground-truth answers, as well as noisy versions of meeting transcripts altered to target different Word Error Rate levels. |
LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression (2024.acl-long)
Copied to clipboard
| Challenge: | Longer prompts introduce irrelevant and redundant information, which can weaken LLMs' performance. |
| Approach: | They propose a prompt compression tool that improves LLMs' perception of key information in input prompts by up to 21.4% with around 4x fewer tokens in GPT-3.5-Turbo. |
| Outcome: | The proposed solution improves performance and reduces costs and latency by up to 21.4% with around 4x fewer tokens in the NaturalQuestions benchmark. |
Context Length Alone Hurts LLM Performance Despite Perfect Retrieval (2025.findings-emnlp)
Copied to clipboard
Yufeng Du, Minyang Tian, Srikanth Ronanki, Subendhu Rongali, Sravan Babu Bodapati, Aram Galstyan, Azton Wells, Roy Schwartz, Eliu A Huerta, Hao Peng
| Challenge: | Large language models (LLMs) often fail to scale their performance on long-context tasks performance in line with the context lengths they support. |
| Approach: | They propose a model-agnostic mitigation strategy that transforms a long-context task into a short-concept one by prompting the model to recite the retrieved evidence before attempting to solve the problem. |
| Outcome: | The proposed model improves on a long-context task up to 4% on RULER. |