LongWeave: A Long-Form Generation Benchmark Bridging Real-World Relevance and Verifiability (2025.findings-emnlp)
Copied to clipboard
Zikai Xiao, Fei Huang, Jianhong Tu, Jianhui Wei, Wen Ma, Yuxuan Zhou, Jian Wu, Bowen Yu, Zuozhu Liu, Junyang Lin
| Challenge: | Existing benchmarks for long-form generation assess real-world queries with hard-to-verify metrics or use synthetic setups that overlook real-life intricacies. |
| Approach: | They propose a new approach that balances verifiable and real-world assessment with Target-Anchored Evaluation. |
| Outcome: | The proposed model balances real-world and verifiable assessment with Target-Anchored Evaluation (TAE) it generates queries, textual materials, and anchors based on verifier targets within real-life scenarios . |
Similar Papers
LongGenBench: Long-context Generation Benchmark (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Current long-context benchmarks focus on retrieval-based tests, requiring Large Language Models to locate specific information within extensive input contexts. |
| Approach: | They propose a long-context generation benchmark that allows for flexible configurations of customized generation context lengths. |
| Outcome: | The proposed benchmark improves performance on NIAH and other retrieval-based tests. |
FaStFact: Faster, Stronger Long-Form Factuality Evaluations in LLMs (2025.findings-emnlp)
Copied to clipboard
Yingjia Wan, Haochen Tan, Xiao Zhu, Xinyu Zhou, Zhiwei Li, Qingsong Lv, Changxuan Sun, Jiaqi Zeng, Yi Xu, Jianqiao Lu, Yinhong Liu, Zhijiang Guo
| Challenge: | Prior evaluation pipelines fail to evaluate factuality of long-form LLMs due to inefficiency and costly human assessment. |
| Approach: | They propose a fast and strong evaluation pipeline that can evaluate factuality of long-form LLMs . they propose 'faStFact' to reduce cost of web searching and inference calling . |
| Outcome: | The proposed evaluation pipeline achieves highest alignment with human evaluation and efficiency among existing baselines. |
AnalystBench: Benchmarking professional long-form report generation with web-mined multimodal tasks (2026.findings-acl)
Copied to clipboard
Chau Minh Pham, Zichao Wang, Puneet Mathur, Alexa Siu, Akriti Jain, Aparna Garimella, Ananya B. Sai, Nedim Lipka, Mohit Iyyer, Varun Manjunatha
| Challenge: | Existing benchmarks decompose the end-to-end professional report generation into individual components. |
| Approach: | They propose a benchmarking tool that evaluates 20 real-world professional report generation tasks grounded in multimodal document collections. |
| Outcome: | The proposed model outperforms closed-source models on executive summarization tasks but drops significantly on long-horizon synthesis tasks. |
ProxyQA: An Alternative Framework for Evaluating Long-Form Text Generation with Large Language Models (2024.acl-long)
Copied to clipboard
Haochen Tan, Zhijiang Guo, Zhan Shi, Lu Xu, Zhili Liu, Yunlong Feng, Xiaoguang Li, Yasheng Wang, Lifeng Shang, Qun Liu, Linqi Song
| Challenge: | Existing evaluation methods for large language models are labor-intensive and lack efficiency. |
| Approach: | They propose a framework dedicated to assessing long-text generation that includes in-depth human-curated meta-questions spanning various domains . they use a set of proxy-quests with pre-annotated answers to assess the content's quality by incorporating the generated texts as contextual background. |
| Outcome: | The proposed framework assesses the quality of long-text content by matching it with references through human evaluation or automated metrics. |
LongDPO: Unlock Better Long-form Generation Abilities for LLMs via Critique-augmented Stepwise Information (2025.findings-acl)
Copied to clipboard
| Challenge: | Recent advances in large language models have improved their capacity to handle long text inputs, but current models still exhibit unsatisfactory performance in long-form generation. |
| Approach: | They propose a method to enhance long-form text generation through step-level supervision by leveraging Monte Carlo Tree Search to collect stepwise preference pairs and employ a global memory pool to maintain factual accuracy. |
| Outcome: | The proposed method improves performance on long-form generation benchmarks while maintaining lossless performance on several general benchmarks. |
LongTutor: Benchmarking Large Language Models for Long-term Personalized Tutoring (2026.acl-long)
Copied to clipboard
| Challenge: | Existing evaluations focus on isolated, short-term interactions, overlooking the inherently long-term nature of learning. |
| Approach: | They propose a benchmark for long-term personalized tutoring based on an annotated learning log . they propose an automated generator–verifier pipeline to enable benchmark expansion . |
| Outcome: | The proposed benchmarks evaluate LLMs across three progressive tasks: evidence acquisition, knowledge state diagnosis, and adaptive teaching action. |
BAMBOO: A Comprehensive Benchmark for Evaluating Long Text Modeling Capacities of Large Language Models (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing long context models suffer from performance decline when the input text exceeds their length limit. |
| Approach: | They propose a multi-task long context benchmark to evaluate LLMs' long context ability using 10 datasets from 5 different NLP tasks. |
| Outcome: | The proposed model covers 5 domains and core capacities of large language models. |
VeriScore: Evaluating the factuality of verifiable claims in long-form text generation (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing metrics for evaluating the factuality of long-form text assume that every claim is verifiable. |
| Approach: | They propose a metric to evaluate factuality in diverse long-form generation tasks . they use open-weight language models to extract verifiable and unverifiably content . |
| Outcome: | The proposed metric can be implemented with either closed or fine-tuned open-weight language models. |
LongLeader: A Comprehensive Leaderboard for Large Language Models in Long-context Scenarios (2025.naacl-long)
Copied to clipboard
Pei Chen, Hongye Jin, Cheng-Che Lee, Rulin Shao, Jingfeng Yang, Mingyu Zhao, Zhaoyu Zhang, Qin Lu, Kaiwen Men, Ning Xie, Huasheng Li, Bing Yin, Han Li, Lingyun Wang
| Challenge: | LongLeader aims to assess different LLMs' long-context comprehension abilities . long-constext comprehension is a key bottleneck for many use cases . |
| Approach: | They propose a leaderboard to assess different LLMs' long-context comprehension abilities . they offer open-source access to the benchmarks and maintain a dedicated website . |
| Outcome: | The proposed model assesses different LLMs on selected benchmarks and provides open-source access to the benchmarks. |
LongTableBench: Benchmarking Long-Context Table Reasoning across Real-World Formats and Domains (2025.findings-emnlp)
Copied to clipboard
Liyao Li, Jiaming Tian, Hao Chen, Wentao Ye, Chao Ye, Haobo Wang, Ningtao Wang, Xing Fu, Gang Chen, Junbo Zhao
| Challenge: | Evaluating 52 LLMs reveals that only the strongest models maintain robust performance under increasing context lengths and format diversity. |
| Approach: | They propose a benchmark for evaluating long-context reasoning over semi-structured tables across diverse formats, tasks, and domains. |
| Outcome: | The proposed model outperforms compression-based approaches on tasks requiring semantic integration. |