BAMBOO: A Comprehensive Benchmark for Evaluating Long Text Modeling Capacities of Large Language Models (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing long context models suffer from performance decline when the input text exceeds their length limit. |
| Approach: | They propose a multi-task long context benchmark to evaluate LLMs' long context ability using 10 datasets from 5 different NLP tasks. |
| Outcome: | The proposed model covers 5 domains and core capacities of large language models. |
Similar Papers
LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding (2024.acl-long)
Copied to clipboard
Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, Juanzi Li
| Challenge: | Large language models (LLMs) can only handle texts a few thousand tokens long, limiting their applications on longer sequence inputs, such as books, reports, and codebases. |
| Approach: | They propose a bilingual, multi-task benchmark for long context understanding that extends context windows and more sophisticated memory mechanisms to improve models' long context capabilities. |
| Outcome: | The proposed model outperforms open-source models but struggles on longer contexts. |
Ada-LEval: Evaluating long-context LLMs with length-adaptable benchmarks (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing long-text evaluation benchmarks, such as L-Eval and LongBench, focus on QA and summarization tasks. |
| Approach: | They propose a length-adaptable benchmark for evaluating the long-context understanding of large language models. |
| Outcome: | The proposed benchmarks do not cover ultralong settings (100k+ tokens) and are difficult to evaluate across different length ranges. |
CLongEval: A Chinese Benchmark for Evaluating Long-Context Large Language Models (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Developing long-context LLMs with robust long-text capabilities is underdeveloped due to a lack of benchmarks. |
| Approach: | They propose a Chinese benchmark for evaluating long-context LLMs with Chinese capabilities. |
| Outcome: | The proposed model is based on 6 open-source LLMs and 2 commercial ones. |
Leave No Document Behind: Benchmarking Long-Context LLMs with Extended Multi-Doc QA (2024.emnlp-main)
Copied to clipboard
Minzheng Wang, Longze Chen, Fu Cheng, Shengyi Liao, Xinghua Zhang, Bingli Wu, Haiyang Yu, Nan Xu, Lei Zhang, Run Luo, Yunshui Li, Min Yang, Fei Huang, Yongbin Li
| Challenge: | Existing benchmarks for evaluating long-context language models employ irrelevant noise texts to artificially extend the length of test cases, diverging from the real-world scenarios of long-constituency applications. |
| Approach: | They propose a long-context benchmark, Loong, aligning with realistic scenarios through extended multi-document question answering (QA) . |
| Outcome: | The proposed model can scale up the context window of large language models to perform in-depth analysis of multiple long documents. |
ETHIC: Evaluating Large Language Models on Long-Context Tasks with High Information Coverage (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing evaluation methods do not assess whether large language models fully utilize contextual information. |
| Approach: | They introduce a new metric to assess LLMs' ability to fully utilize contextual information. |
| Outcome: | The proposed benchmark comprises 1,986 test instances spanning four long-context tasks with high IC scores in the domains of books, debates, medicine, and law. |
CNNSum: Exploring Long-Context Summarization with Large Language Models in Chinese Novels (2025.findings-acl)
Copied to clipboard
| Challenge: | Currently, long-context summarization mainly relies on memory ability. |
| Approach: | They propose a multi-scale long-context summarization benchmark based on Chinese novels . they use human-driven annotations to analyze long-constituency models . |
| Outcome: | The proposed benchmark features human-driven annotations across four subsets with lengths ranging from 16k to 128k. |
LongLeader: A Comprehensive Leaderboard for Large Language Models in Long-context Scenarios (2025.naacl-long)
Copied to clipboard
Pei Chen, Hongye Jin, Cheng-Che Lee, Rulin Shao, Jingfeng Yang, Mingyu Zhao, Zhaoyu Zhang, Qin Lu, Kaiwen Men, Ning Xie, Huasheng Li, Bing Yin, Han Li, Lingyun Wang
| Challenge: | LongLeader aims to assess different LLMs' long-context comprehension abilities . long-constext comprehension is a key bottleneck for many use cases . |
| Approach: | They propose a leaderboard to assess different LLMs' long-context comprehension abilities . they offer open-source access to the benchmarks and maintain a dedicated website . |
| Outcome: | The proposed model assesses different LLMs on selected benchmarks and provides open-source access to the benchmarks. |
LongSafety: Evaluating Long-Context Safety of Large Language Models (2025.acl-long)
Copied to clipboard
Yida Lu, Jiale Cheng, Zhexin Zhang, Shiyao Cui, Cunxiang Wang, Xiaotao Gu, Yuxiao Dong, Jie Tang, Hongning Wang, Minlie Huang
| Challenge: | Large Language Models (LLMs) have demonstrated remarkable capabilities in understanding and generating long sequences. |
| Approach: | They propose a benchmark to evaluate LLM safety in open-ended long-context tasks . they find that relevant context and extended input sequences can exacerbate safety risks . |
| Outcome: | The proposed benchmark identifies significant safety vulnerabilities in 16 LLMs . strong safety performance in short-context scenarios does not correlate with safety in long-contact tasks . |
KV Cache Compression, But What Must We Give in Return? A Comprehensive Benchmark of Long Context Capable Approaches (2024.findings-emnlp)
Copied to clipboard
Jiayi Yuan, Hongyi Liu, Shaochen Zhong, Yu-Neng Chuang, Songchen Li, Guanchu Wang, Duy Le, Hongye Jin, Vipin Chaudhary, Zhaozhuo Xu, Zirui Liu, Xia Hu
| Challenge: | Long context capability is a crucial competency for large language models as it mitigates the human struggle to digest long-form texts. |
| Approach: | They propose to evaluate 10+ state-of-the-art approaches for long context-capable LLMs. |
| Outcome: | The proposed methods are compared against 10+ state-of-the-art approaches across seven categories of long context tasks. |
Benchmarking and Improving Long-Text Translation with Large Language Models (2024.findings-acl)
Copied to clipboard
Longyue Wang, Zefeng Du, Wenxiang Jiao, Chenyang Lyu, Jianhui Pang, Leyang Cui, Kaiqiang Song, Derek Wong, Shuming Shi, Zhaopeng Tu
| Challenge: | Recent studies have illuminated the promising capabilities of large language models (LLMs) in handling long texts. |
| Approach: | They construct a benchmark dataset specifically designed for the finetuning and evaluation of large language models (LLMs) they compare LLMs with MT models and find they exhibit shortcomings in long-text domains . |
| Outcome: | The proposed model performs better in long-text translation, and its performance diminishes as document size increases. |