LCFO: Long Context and Long Form Output Dataset and Benchmarking

Challenge:	Using long text outputs to evaluate progress in summarization and summary expansion tasks is challenging.
Approach:	They propose a framework for assessing gradual summarization and summary expansion capabilities across diverse domains.
Outcome:	The proposed framework provides alignments between specific QA pairs and corresponding summaries in 7 domains.

Similar Papers

Overview

Challenge

Approach

Outcome

LC-Eval: A Bilingual Multi-Task Evaluation Benchmark for Long-Context Understanding (2025.findings-emnlp)

Copied to clipboard

Sheikh Jubair, Arwa Omayrah, Amal Alshammari, Alhanoof Althnian, Abdulhamed Alothaimen, Norah A. Alzahrani, Shahad D. Alzaidi, Nora Al-Twairesh, Abdulmohsen Al-Thubaity

Language Models

Challenge:	Recent advances in Large Language Models (LLMs) have demonstrated sophisticated capabilities, including the ability to process and comprehend extended contexts.
Approach:	They propose a bilingual, multi-task evaluation benchmark designed to evaluate long-context understanding in English and Arabic.
Outcome:	The proposed benchmark targets context lengths ranging from 4k to over 128k tokens.

Ref-Long: Benchmarking the Long-context Referencing Capability of Long-context Language Models (2025.acl-long)

Copied to clipboard

Junjie Wu, Gefei Gu, Yanan Zheng, Dit-Yan Yeung, Arman Cohan

Language Models

Challenge:	Long-context language models have impressive capabilities in long-contrast understanding tasks, but long-text referencing remains underexplored.
Approach:	They propose a benchmark to assess long-context referencing capability of LCLMs . they use three subsets to test the model's ability to identify key indexes based on contextual relationships .
Outcome:	The proposed benchmark assesses the long-context referencing capability of LCLMs.

LongTableBench: Benchmarking Long-Context Table Reasoning across Real-World Formats and Domains (2025.findings-emnlp)

Copied to clipboard

Liyao Li, Jiaming Tian, Hao Chen, Wentao Ye, Chao Ye, Haobo Wang, Ningtao Wang, Xing Fu, Gang Chen, Junbo Zhao

Multimodality

Challenge:	Evaluating 52 LLMs reveals that only the strongest models maintain robust performance under increasing context lengths and format diversity.
Approach:	They propose a benchmark for evaluating long-context reasoning over semi-structured tables across diverse formats, tasks, and domains.
Outcome:	The proposed model outperforms compression-based approaches on tasks requiring semantic integration.

Systematic Evaluation of Long-Context LLMs on Financial Concepts (2024.emnlp-industry)

Copied to clipboard

Lavanya Gupta, Saket Sharma, Yiyun Zhao

Language Models

Challenge:	Long-context large language models (LC LLMs) are promising for tasks with long context windows . however, their ability to reliably utilize their growing context windows remains under investigation .
Approach:	They evaluate the performance of long-context large language models using a real-world financial news dataset.
Outcome:	The proposed models exhibit brittleness at longer context lengths even for simple tasks, the authors show . they advocate for more rigorous evaluation of LC LLMs by employing holistic metrics such as F1 (rather than recall)

LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding (2024.acl-long)

Copied to clipboard

Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, Juanzi Li

Multilinguality

Challenge:	Large language models (LLMs) can only handle texts a few thousand tokens long, limiting their applications on longer sequence inputs, such as books, reports, and codebases.
Approach:	They propose a bilingual, multi-task benchmark for long context understanding that extends context windows and more sophisticated memory mechanisms to improve models' long context capabilities.
Outcome:	The proposed model outperforms open-source models but struggles on longer contexts.

QuALITY: Question Answering with Long Input Texts, Yes! (2022.naacl-main)

Copied to clipboard

Richard Yuanzhe Pang, Alicia Parrish, Nitish Joshi, Nikita Nangia, Jason Phang, Angelica Chen, Vishakh Padmakumar, Johnny Ma, Jana Thompson, He He, Samuel Bowman

Question Answering

Challenge:	Existing models for natural language understanding are limited to processing only a few hundred words at a time.
Approach:	They propose a dataset with context passages in English that have an average length of 5,000 tokens.
Outcome:	a new dataset with long-text comprehension questions is used to test models on long-document comprehension . the questions are validated by contributors who have read the entire passage, not just excerpts . only half of the questions can be answered by annotators working under tight time constraints .

On Context Utilization in Summarization with Large Language Models (2024.acl-long)

Copied to clipboard

Mathieu Ravaut, Aixin Sun, Nancy Chen, Shafiq Joty

Language Models

Challenge:	Large language models excel in abstractive summarization tasks, delivering fluent and pertinent summaries.
Approach:	They conduct the first comprehensive study on context utilization and position bias in summarization.
Outcome:	The proposed benchmark compares two methods to alleviate position bias in summarization tasks.

MiniLongBench: The Low-cost Long Context Understanding Benchmark for Large Language Models (2025.acl-long)

Copied to clipboard

Zhongzhan Huang, Guoming Ling, Shanshan Zhong, Hefeng Wu, Liang Lin

Semantic Text Processing

Challenge:	Existing LCU benchmarks for large language models often result in prohibitively high evaluation costs . existing benchmarks exhibit significant redundancy, which means inefficiency in evaluation .
Approach:	They propose a data compression method tailored for long-text data with sparse information characteristics.
Outcome:	The proposed method reduces evaluation costs to 4.5% of the long-text benchmark LongBench . the proposed method is based on a long-term LCU benchmark with sparse information characteristics .

CNNSum: Exploring Long-Context Summarization with Large Language Models in Chinese Novels (2025.findings-acl)

Copied to clipboard

Lingxiao Wei, He Yan, Lu Xiangju, Junmin Zhu, Jun Wang, Wei Zhang

Information Extraction & Text Mining

Challenge:	Currently, long-context summarization mainly relies on memory ability.
Approach:	They propose a multi-scale long-context summarization benchmark based on Chinese novels . they use human-driven annotations to analyze long-constituency models .
Outcome:	The proposed benchmark features human-driven annotations across four subsets with lengths ranging from 16k to 128k.

SummN: A Multi-Stage Summarization Framework for Long Input Dialogues and Documents (2022.acl-long)

Copied to clipboard

Yusen Zhang, Ansong Ni, Ziming Mao, Chen Henry Wu, Chenguang Zhu, Budhaditya Deb, Ahmed Awadallah, Dragomir Radev, Rui Zhang

Language Models

Challenge:	Existing methods to handle long text are limited due to time and memory complexity and limited input lengths.
Approach:	They propose a multi-stage split-then-summarize framework for long input summarization . their framework can process input text of arbitrary length by adjusting the number of stages .
Outcome:	The proposed framework outperforms existing methods on three long meeting summarization datasets and on a long document summarizing dataset.