CLongEval: A Chinese Benchmark for Evaluating Long-Context Large Language Models (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Developing long-context LLMs with robust long-text capabilities is underdeveloped due to a lack of benchmarks. |
| Approach: | They propose a Chinese benchmark for evaluating long-context LLMs with Chinese capabilities. |
| Outcome: | The proposed model is based on 6 open-source LLMs and 2 commercial ones. |
Similar Papers
Leave No Document Behind: Benchmarking Long-Context LLMs with Extended Multi-Doc QA (2024.emnlp-main)
Copied to clipboard
Minzheng Wang, Longze Chen, Fu Cheng, Shengyi Liao, Xinghua Zhang, Bingli Wu, Haiyang Yu, Nan Xu, Lei Zhang, Run Luo, Yunshui Li, Min Yang, Fei Huang, Yongbin Li
| Challenge: | Existing benchmarks for evaluating long-context language models employ irrelevant noise texts to artificially extend the length of test cases, diverging from the real-world scenarios of long-constituency applications. |
| Approach: | They propose a long-context benchmark, Loong, aligning with realistic scenarios through extended multi-document question answering (QA) . |
| Outcome: | The proposed model can scale up the context window of large language models to perform in-depth analysis of multiple long documents. |
BAMBOO: A Comprehensive Benchmark for Evaluating Long Text Modeling Capacities of Large Language Models (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing long context models suffer from performance decline when the input text exceeds their length limit. |
| Approach: | They propose a multi-task long context benchmark to evaluate LLMs' long context ability using 10 datasets from 5 different NLP tasks. |
| Outcome: | The proposed model covers 5 domains and core capacities of large language models. |
Ada-LEval: Evaluating long-context LLMs with length-adaptable benchmarks (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing long-text evaluation benchmarks, such as L-Eval and LongBench, focus on QA and summarization tasks. |
| Approach: | They propose a length-adaptable benchmark for evaluating the long-context understanding of large language models. |
| Outcome: | The proposed benchmarks do not cover ultralong settings (100k+ tokens) and are difficult to evaluate across different length ranges. |
CLEVA: Chinese Language Models EVAluation Platform (2023.emnlp-demo)
Copied to clipboard
Yanyang Li, Jianqiao Zhao, Duo Zheng, Zi-Yuan Hu, Zhi Chen, Xiaohui Su, Yongfeng Huang, Shijia Huang, Dahua Lin, Michael Lyu, Liwei Wang
| Challenge: | Large language models (LLMs) have revolutionized natural language processing. |
| Approach: | They propose a Chinese-based platform that assesses Chinese LLMs using a standardized workflow and a unique sampling strategy. |
| Outcome: | CLEVA evaluates Chinese LLMs on a standardized workflow and a competitive leaderboard with minimal coding. |
LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding (2024.acl-long)
Copied to clipboard
Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, Juanzi Li
| Challenge: | Large language models (LLMs) can only handle texts a few thousand tokens long, limiting their applications on longer sequence inputs, such as books, reports, and codebases. |
| Approach: | They propose a bilingual, multi-task benchmark for long context understanding that extends context windows and more sophisticated memory mechanisms to improve models' long context capabilities. |
| Outcome: | The proposed model outperforms open-source models but struggles on longer contexts. |
OpenEval: Benchmarking Chinese LLMs across Capability, Alignment and Safety (2024.acl-demos)
Copied to clipboard
Chuang Liu, Linhao Yu, Jiaxuan Li, Renren Jin, Yufei Huang, Ling Shi, Junhui Zhang, Xinmeng Ji, Tingting Cui, Liutao Liutao, Jinwang Song, Hongying Zan, Sun Li, Deyi Xiong
| Challenge: | a rapid development of Chinese large language models poses big challenges for efficient LLM evaluation. |
| Approach: | They propose an evaluation testbed that benchmarks Chinese LLMs across capability, alignment and safety. |
| Outcome: | The evaluation platform OpenEval benchmarks Chinese LLMs across capability, alignment and safety. |
CNNSum: Exploring Long-Context Summarization with Large Language Models in Chinese Novels (2025.findings-acl)
Copied to clipboard
| Challenge: | Currently, long-context summarization mainly relies on memory ability. |
| Approach: | They propose a multi-scale long-context summarization benchmark based on Chinese novels . they use human-driven annotations to analyze long-constituency models . |
| Outcome: | The proposed benchmark features human-driven annotations across four subsets with lengths ranging from 16k to 128k. |
100-LongBench: Are de facto Long-Context Benchmarks Literally Evaluating Long-Context Ability? (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing benchmarks for long-context capability are too synthetic and do not represent the real world usage of LLMs. |
| Approach: | They propose a length-controllable, real-life reflective benchmark that disentangles baseline knowledge from long-context capabilities. |
| Outcome: | Experiments show that the proposed benchmarks disentangle baseline knowledge from long-context capabilities. |
ELITR-Bench: A Meeting Assistant Benchmark for Long-Context Language Models (2025.coling-main)
Copied to clipboard
| Challenge: | Existing benchmarks for long-context LLMs focus on generic tasks that are not necessarily aligned with real-world applications. |
| Approach: | They propose to augment existing ELITR corpus by adding 271 manually crafted questions with their ground-truth answers and noisy versions of meeting transcripts altered to target different Word Error Rate levels. |
| Outcome: | The proposed benchmark augments the existing ELITR corpus by adding 271 manually crafted questions with ground-truth answers, as well as noisy versions of meeting transcripts altered to target different Word Error Rate levels. |
CToolEval: A Chinese Benchmark for LLM-Powered Agent Evaluation in Real-World API Interactions (2024.findings-acl)
Copied to clipboard
| Challenge: | a benchmark is designed to evaluate the capabilities of large language models (LLMs) as agents in decision making and operational tasks. |
| Approach: | They propose a benchmark to evaluate LLMs in the context of Chinese societal applications . they propose he benchmark will evaluate tool invocation ability of LLM and task completion ability . |
| Outcome: | The proposed benchmark features 398 APIs across 27 widely-used Apps across 14 domains. |