| Challenge: | Renewed interest in understanding long texts has sparked interest in benchmarks based on length of input text . |
| Approach: | They propose a new metric that determines the shortest average text length that needs to be preserved to execute the task with limited performance degradation. |
| Outcome: | The proposed benchmarks show that models outperform the previous generation on the QuALITY task due to their limited understanding of long-range dependencies. |
Similar Papers
ZeroSCROLLS: A Zero-Shot Benchmark for Long Text Understanding (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing benchmarks for long text understanding focus on short sequences, such as BigBench and HELM. |
| Approach: | They propose a zero-shot benchmark for natural language understanding over long texts . they adapt six tasks from the SCROLLS benchmark and add four new datasets . |
| Outcome: | The proposed benchmark outperforms ChatGPT and GPT-4 in a number of open tasks. |
SCROLLS: Standardized CompaRison Over Long Language Sequences (2022.emnlp-main)
Copied to clipboard
Uri Shaham, Elad Segal, Maor Ivgi, Avia Efrat, Ori Yoran, Adi Haviv, Ankit Gupta, Wenhan Xiong, Mor Geva, Jonathan Berant, Omer Levy
| Challenge: | Standard NLP benchmarks focus on short texts, but long texts are produced in the context of longer discourses. |
| Approach: | They propose a new benchmark that places models in context of long texts that require reasoning over long texts. |
| Outcome: | The proposed task sets are based on a set of long-text datasets and host a live leaderboard to facilitate research on model architecture and pretraining methods. |
Every Token Counts: Generalizing 16M Ultra-Long Context in Large Language Models (2026.acl-long)
Copied to clipboard
| Challenge: | a recent study explores efficient ultra-long context modeling. |
| Approach: | They propose to use Hierarchical Sparse Attention to achieve efficient ultra-long context modeling. |
| Outcome: | The proposed model performs comparable to full-attention baselines on in-domain and out-of-domain tasks. |
Ada-LEval: Evaluating long-context LLMs with length-adaptable benchmarks (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing long-text evaluation benchmarks, such as L-Eval and LongBench, focus on QA and summarization tasks. |
| Approach: | They propose a length-adaptable benchmark for evaluating the long-context understanding of large language models. |
| Outcome: | The proposed benchmarks do not cover ultralong settings (100k+ tokens) and are difficult to evaluate across different length ranges. |
MiniLongBench: The Low-cost Long Context Understanding Benchmark for Large Language Models (2025.acl-long)
Copied to clipboard
| Challenge: | Existing LCU benchmarks for large language models often result in prohibitively high evaluation costs . existing benchmarks exhibit significant redundancy, which means inefficiency in evaluation . |
| Approach: | They propose a data compression method tailored for long-text data with sparse information characteristics. |
| Outcome: | The proposed method reduces evaluation costs to 4.5% of the long-text benchmark LongBench . the proposed method is based on a long-term LCU benchmark with sparse information characteristics . |
LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding (2024.acl-long)
Copied to clipboard
Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, Juanzi Li
| Challenge: | Large language models (LLMs) can only handle texts a few thousand tokens long, limiting their applications on longer sequence inputs, such as books, reports, and codebases. |
| Approach: | They propose a bilingual, multi-task benchmark for long context understanding that extends context windows and more sophisticated memory mechanisms to improve models' long context capabilities. |
| Outcome: | The proposed model outperforms open-source models but struggles on longer contexts. |
QuALITY: Question Answering with Long Input Texts, Yes! (2022.naacl-main)
Copied to clipboard
Richard Yuanzhe Pang, Alicia Parrish, Nitish Joshi, Nikita Nangia, Jason Phang, Angelica Chen, Vishakh Padmakumar, Johnny Ma, Jana Thompson, He He, Samuel Bowman
| Challenge: | Existing models for natural language understanding are limited to processing only a few hundred words at a time. |
| Approach: | They propose a dataset with context passages in English that have an average length of 5,000 tokens. |
| Outcome: | a new dataset with long-text comprehension questions is used to test models on long-document comprehension . the questions are validated by contributors who have read the entire passage, not just excerpts . only half of the questions can be answered by annotators working under tight time constraints . |
MuLD: The Multitask Long Document Benchmark (2022.lrec-1)
Copied to clipboard
| Challenge: | Existing benchmarks for NLP focus on tasks for one or two sentences, but efficient techniques are needed for processing much longer sequences. |
| Approach: | They propose to modify existing NLP tasks to create a long document benchmark which requires models to successfully model long-term dependencies in the text. |
| Outcome: | The proposed benchmark is much more challenging than its ‘short document’ equivalents. |
Marathon: A Race Through the Realm of Long Context with Large Language Models (2024.acl-long)
Copied to clipboard
| Challenge: | Existing long-context benchmarks do not accurately evaluate large language models’ comprehension and reasoning abilities in extended texts. |
| Approach: | They propose a new evaluation benchmark that adopts a multiple-choice question format and uses a multi-choke question format to assess the comprehension and reasoning skills of large language models. |
| Outcome: | The proposed benchmark provides a rapid, precise, and unbiased appraisal of the long-context comprehension skills of large language models. |
LongGenBench: Long-context Generation Benchmark (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Current long-context benchmarks focus on retrieval-based tests, requiring Large Language Models to locate specific information within extensive input contexts. |
| Approach: | They propose a long-context generation benchmark that allows for flexible configurations of customized generation context lengths. |
| Outcome: | The proposed benchmark improves performance on NIAH and other retrieval-based tests. |