LC-Eval: A Bilingual Multi-Task Evaluation Benchmark for Long-Context Understanding

Challenge:	Recent advances in Large Language Models (LLMs) have demonstrated sophisticated capabilities, including the ability to process and comprehend extended contexts.
Approach:	They propose a bilingual, multi-task evaluation benchmark designed to evaluate long-context understanding in English and Arabic.
Outcome:	The proposed benchmark targets context lengths ranging from 4k to over 128k tokens.

Similar Papers

Overview

Challenge

Approach

Outcome

LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding (2024.acl-long)

Copied to clipboard

Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, Juanzi Li

Multilinguality

Challenge:	Large language models (LLMs) can only handle texts a few thousand tokens long, limiting their applications on longer sequence inputs, such as books, reports, and codebases.
Approach:	They propose a bilingual, multi-task benchmark for long context understanding that extends context windows and more sophisticated memory mechanisms to improve models' long context capabilities.
Outcome:	The proposed model outperforms open-source models but struggles on longer contexts.

Marathon: A Race Through the Realm of Long Context with Large Language Models (2024.acl-long)

Copied to clipboard

Lei Zhang, Yunshui Li, Ziqiang Liu, Jiaxi Yang, Junhao Liu, Longze Chen, Run Luo, Min Yang

Language Models

Challenge:	Existing long-context benchmarks do not accurately evaluate large language models’ comprehension and reasoning abilities in extended texts.
Approach:	They propose a new evaluation benchmark that adopts a multiple-choice question format and uses a multi-choke question format to assess the comprehension and reasoning skills of large language models.
Outcome:	The proposed benchmark provides a rapid, precise, and unbiased appraisal of the long-context comprehension skills of large language models.

BenchMAX: A Comprehensive Multilingual Evaluation Suite for Large Language Models (2025.findings-emnlp)

Copied to clipboard

Xu Huang, Wenhao Zhu, Hanxu Hu, Conghui He, Lei Li, Shujian Huang, Fei Yuan

Language Models

Challenge:	Existing multilingual benchmarks focus primarily on language understanding tasks.
Approach:	They develop a multi-way multilingual benchmark that measures critical capabilities of large language models across languages.
Outcome:	Extensive experiments on BenchMAX reveal uneven utilization of core capabilities across languages, emphasizing the performance gaps that scaling model size alone does not resolve.

F-Eval: Asssessing Fundamental Abilities with Refined Evaluation Methods (2024.acl-long)

Copied to clipboard

Yu Sun, Keyuchen Keyuchen, Shujie Wang, Peiji Li, Qipeng Guo, Hang Yan, Xipeng Qiu, Xuanjing Huang, Dahua Lin

Challenge:	Large language models (LLMs) have been evaluated for their instruction-following capabilities but lack references to their fundamental abilities.
Approach:	They propose a bilingual evaluation benchmark to evaluate the fundamental abilities of large language models including expression, commonsense and logic.
Outcome:	The proposed evaluation methods show higher correlation coefficients and larger distinction than other evaluators.

Ada-LEval: Evaluating long-context LLMs with length-adaptable benchmarks (2024.naacl-long)

Copied to clipboard

Chonghua Wang, Haodong Duan, Songyang Zhang, Dahua Lin, Kai Chen

Challenge:	Existing long-text evaluation benchmarks, such as L-Eval and LongBench, focus on QA and summarization tasks.
Approach:	They propose a length-adaptable benchmark for evaluating the long-context understanding of large language models.
Outcome:	The proposed benchmarks do not cover ultralong settings (100k+ tokens) and are difficult to evaluate across different length ranges.

Ref-Long: Benchmarking the Long-context Referencing Capability of Long-context Language Models (2025.acl-long)

Copied to clipboard

Junjie Wu, Gefei Gu, Yanan Zheng, Dit-Yan Yeung, Arman Cohan

Language Models

Challenge:	Long-context language models have impressive capabilities in long-contrast understanding tasks, but long-text referencing remains underexplored.
Approach:	They propose a benchmark to assess long-context referencing capability of LCLMs . they use three subsets to test the model's ability to identify key indexes based on contextual relationships .
Outcome:	The proposed benchmark assesses the long-context referencing capability of LCLMs.

MMLU-ProX: A Multilingual Benchmark for Advanced Large Language Model Evaluation (2025.emnlp-main)

Copied to clipboard

Weihao Xuan, Rui Yang, Heli Qi, Qingcheng Zeng, Yunze Xiao, Aosong Feng, Dairui Liu, Yun Xing, Junjue Wang, Fan Gao, Jinghui Lu, Yuang Jiang, Huitao Li, Xin Li, Kunyu Yu, Ruihai Dong, Shangding Gu, Yuekang Li, Xiaofei Xie, Felix Juefei-Xu, Foutse Khomh, Osamu Yoshie, Qingyu Chen, Douglas Teodoro, Nan Liu, Randy Goebel, Lei Ma, Edison Marrese-Taylor, Shijian Lu, Yusuke Iwasawa, Yutaka Matsuo, Irene Li

Multilinguality

Challenge:	Existing large language model evaluation benchmarks focus on English, while current multilingual tasks lack parallel questions that specifically assess cross-lingual reasoning abilities.
Approach:	They propose a comprehensive benchmark covering 29 languages, built on an English benchmark.
Outcome:	The MMLU-ProX is a comprehensive benchmark covering 29 languages, built on an English benchmark.

Are Large Language Model-based Evaluators the Solution to Scaling Up Multilingual Evaluation? (2024.findings-eacl)

Copied to clipboard

Rishav Hada, Varun Gumma, Adrian Wynter, Harshita Diddee, Mohamed Ahmed, Monojit Choudhury, Kalika Bali, Sunayana Sitaram

Semantic Text Processing

Challenge:	Large Language Models (LLMs) excel in various tasks, but their evaluation, especially in languages beyond the top 20, remains inadequate due to existing benchmarks and metrics limitations.
Approach:	They propose to use Large Language Models as evaluators to rank or score other models’ outputs by calibrating them against 20K human judgments across three text-generation tasks, five metrics, and eight languages.
Outcome:	The proposed evaluation methods can be used to improve multilingual evaluation by calibrating them against 20K human judgments across three text-generation tasks, five metrics, and eight languages.

LCFO: Long Context and Long Form Output Dataset and Benchmarking (2025.findings-acl)

Copied to clipboard

Marta R. Costa-jussà, Pierre Andrews, Mariano Coria Meglioli, Joy Chen, Joe Chuang, David Dale, Christophe Ropers, Alexandre Mourachko, Eduardo Sánchez, Holger Schwenk, Tuan A. Tran, Arina Turkatenko, Carleigh Wood

Summarization

Challenge:	Using long text outputs to evaluate progress in summarization and summary expansion tasks is challenging.
Approach:	They propose a framework for assessing gradual summarization and summary expansion capabilities across diverse domains.
Outcome:	The proposed framework provides alignments between specific QA pairs and corresponding summaries in 7 domains.

Systematic Evaluation of Long-Context LLMs on Financial Concepts (2024.emnlp-industry)

Copied to clipboard

Lavanya Gupta, Saket Sharma, Yiyun Zhao

Language Models

Challenge:	Long-context large language models (LC LLMs) are promising for tasks with long context windows . however, their ability to reliably utilize their growing context windows remains under investigation .
Approach:	They evaluate the performance of long-context large language models using a real-world financial news dataset.
Outcome:	The proposed models exhibit brittleness at longer context lengths even for simple tasks, the authors show . they advocate for more rigorous evaluation of LC LLMs by employing holistic metrics such as F1 (rather than recall)