Leave No Document Behind: Benchmarking Long-Context LLMs with Extended Multi-Doc QA

Challenge:	Existing benchmarks for evaluating long-context language models employ irrelevant noise texts to artificially extend the length of test cases, diverging from the real-world scenarios of long-constituency applications.
Approach:	They propose a long-context benchmark, Loong, aligning with realistic scenarios through extended multi-document question answering (QA) .
Outcome:	The proposed model can scale up the context window of large language models to perform in-depth analysis of multiple long documents.

Similar Papers

Overview

Challenge

Approach

Outcome

CLongEval: A Chinese Benchmark for Evaluating Long-Context Large Language Models (2024.findings-emnlp)

Copied to clipboard

Zexuan Qiu, Jingjing Li, Shijue Huang, Xiaoqi Jiao, Wanjun Zhong, Irwin King

Language Models

Challenge:	Developing long-context LLMs with robust long-text capabilities is underdeveloped due to a lack of benchmarks.
Approach:	They propose a Chinese benchmark for evaluating long-context LLMs with Chinese capabilities.
Outcome:	The proposed model is based on 6 open-source LLMs and 2 commercial ones.

Can LLMs reason over extended multilingual contexts? Towards long-context evaluation beyond retrieval over haystacks (2026.eacl-long)

Copied to clipboard

Amey Hengle, Prasoon Bajpai, Soham Dan, Tanmoy Chakraborty

Reasoning

Challenge:	Existing multilingual long-context benchmarks are myopic and inherently limited, as successful recall alone does not indicate a model’s capacity to reason over extended contexts.
Approach:	They propose a new synthetic benchmark for multilingual long-context reasoning that includes bAbI-style tasks that test multi-hop inference, aggregation, and epistemic reasoning.
Outcome:	The proposed benchmarks are based on a multilingual long-context model and span seven languages.

TAIL: A Toolkit for Automatic and Realistic Long-Context Large Language Model Evaluation (2024.emnlp-demo)

Copied to clipboard

Gefei Gu, Yilun Zhao, Ruoxi Ning, Yanan Zheng, Arman Cohan

Language Models

Challenge:	Existing evaluation methods for long-context large language models are overly simplistic and require extensive human annotations.
Approach:	They propose an automatic toolkit to create realistic evaluation benchmarks . they use a document-grounded benchmark to generate question-answer pairs .
Outcome:	The proposed toolkit provides a way to create realistic evaluation benchmarks and visualize performance metrics of evaluated models.

BAMBOO: A Comprehensive Benchmark for Evaluating Long Text Modeling Capacities of Large Language Models (2024.lrec-main)

Copied to clipboard

Zican Dong, Tianyi Tang, Junyi Li, Wayne Xin Zhao, Ji-Rong Wen

Language Models

Challenge:	Existing long context models suffer from performance decline when the input text exceeds their length limit.
Approach:	They propose a multi-task long context benchmark to evaluate LLMs' long context ability using 10 datasets from 5 different NLP tasks.
Outcome:	The proposed model covers 5 domains and core capacities of large language models.

ELITR-Bench: A Meeting Assistant Benchmark for Long-Context Language Models (2025.coling-main)

Copied to clipboard

Thibaut Thonet, Laurent Besacier, Jos Rozen

Language Models

Challenge:	Existing benchmarks for long-context LLMs focus on generic tasks that are not necessarily aligned with real-world applications.
Approach:	They propose to augment existing ELITR corpus by adding 271 manually crafted questions with their ground-truth answers and noisy versions of meeting transcripts altered to target different Word Error Rate levels.
Outcome:	The proposed benchmark augments the existing ELITR corpus by adding 271 manually crafted questions with ground-truth answers, as well as noisy versions of meeting transcripts altered to target different Word Error Rate levels.

LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding (2024.acl-long)

Copied to clipboard

Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, Juanzi Li

Multilinguality

Challenge:	Large language models (LLMs) can only handle texts a few thousand tokens long, limiting their applications on longer sequence inputs, such as books, reports, and codebases.
Approach:	They propose a bilingual, multi-task benchmark for long context understanding that extends context windows and more sophisticated memory mechanisms to improve models' long context capabilities.
Outcome:	The proposed model outperforms open-source models but struggles on longer contexts.

Ada-LEval: Evaluating long-context LLMs with length-adaptable benchmarks (2024.naacl-long)

Copied to clipboard

Chonghua Wang, Haodong Duan, Songyang Zhang, Dahua Lin, Kai Chen

Challenge:	Existing long-text evaluation benchmarks, such as L-Eval and LongBench, focus on QA and summarization tasks.
Approach:	They propose a length-adaptable benchmark for evaluating the long-context understanding of large language models.
Outcome:	The proposed benchmarks do not cover ultralong settings (100k+ tokens) and are difficult to evaluate across different length ranges.

Marathon: A Race Through the Realm of Long Context with Large Language Models (2024.acl-long)

Copied to clipboard

Lei Zhang, Yunshui Li, Ziqiang Liu, Jiaxi Yang, Junhao Liu, Longze Chen, Run Luo, Min Yang

Language Models

Challenge:	Existing long-context benchmarks do not accurately evaluate large language models’ comprehension and reasoning abilities in extended texts.
Approach:	They propose a new evaluation benchmark that adopts a multiple-choice question format and uses a multi-choke question format to assess the comprehension and reasoning skills of large language models.
Outcome:	The proposed benchmark provides a rapid, precise, and unbiased appraisal of the long-context comprehension skills of large language models.

LongGenBench: Long-context Generation Benchmark (2024.findings-emnlp)

Copied to clipboard

Xiang Liu, Peijie Dong, Xuming Hu, Xiaowen Chu

Language Models

Challenge:	Current long-context benchmarks focus on retrieval-based tests, requiring Large Language Models to locate specific information within extensive input contexts.
Approach:	They propose a long-context generation benchmark that allows for flexible configurations of customized generation context lengths.
Outcome:	The proposed benchmark improves performance on NIAH and other retrieval-based tests.

LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitasks (2025.acl-long)

Copied to clipboard

Yushi Bai, Shangqing Tu, Jiajie Zhang, Hao Peng, Xiaozhi Wang, Xin Lv, Shulin Cao, Jiazheng Xu, Lei Hou, Yuxiao Dong, Jie Tang, Juanzi Li

Reasoning

Challenge:	Existing benchmarks on longcontext large language models fail to reflect their deep understanding capabilities across diverse tasks.
Approach:	They propose a benchmark to assess the ability of long-context large language models to handle long-text problems.
Outcome:	The proposed model achieves 50.1% accuracy when directly answering the questions . human experts achieve only 53.7% accuracy under a 15-minute time constraint .