Papers with Claude-3.7-Sonnet
LitBench: A Benchmark and Dataset for Reliable Evaluation of Creative Writing (2026.eacl-long)
Copied to clipboard
| Challenge: | a single prompt can inspire countless valid stories, making objective verification impossible. |
| Approach: | They propose a large-scale benchmark for creative writing evaluation using a reddit corpus and a 2,480-pair test set. |
| Outcome: | The proposed model outperforms existing OTS judges and generative reward models in the evaluation of creative writing. |
FinMaster: A Holistic Benchmark for Full-Pipeline Financial Management with Large Language Models (2026.findings-acl)
Copied to clipboard
Junzhe Jiang, Chang Yang, Aixin Cui, Sihan Jin, Yujing Zhang, Yilin Xiao, Ruiyu Wang, Bo Li, Xiao Huang, Danny Dongning Sun, Xinrun Wang
| Challenge: | Existing benchmarks lack domain-specific data, realistic workflow-level task design, and standardized workflow- level evaluation. |
| Approach: | a new benchmark evaluates large language models on financial management workflows . the global financial services market is projected to grow to $37 trillion by 2027 . |
| Outcome: | a new benchmark for large language models on financial management workflows reveals critical capability gaps . accuracy drops from 90% on basic tasks to 40% on complex scenarios requiring multi-step reasoning . the global financial services market reached $25.8 trillion in 2022 and is projected to grow to $37 trillion by 2027 . |
ReliableEval: A Recipe for Stochastic LLM Evaluation via Method of Moments (2025.findings-emnlp)
Copied to clipboard
| Challenge: | LLMs are highly sensitive to prompt phrasing, yet standard benchmarks typically report performance using a single prompt, raising concerns about the reliability of such evaluations. |
| Approach: | They propose a stochastic method of moments evaluation over the space of meaning-preserving prompt perturbations and propose resamplings to estimate the number of prompt re-sampleds needed to obtain meaningful results. |
| Outcome: | The proposed method is model-, task-, and metric-agnostic, offering a recipe for meaningful and robust evaluation. |
VisualEDU: A Benchmark for Assessing Coding and Visual Comprehension through Educational Problem-Solving Video Generation (2025.findings-emnlp)
Copied to clipboard
| Challenge: | VisualEDU is a benchmark to evaluate VLMs' ability to produce coherent video from text . advanced proprietary models show promise, but struggle with increasing task complexity . |
| Approach: | VisualEDU is a benchmark to evaluate VLMs' ability to produce coherent video from text . it integrates meta-prompt learning, visual and code feedback, and a drawing toolkit to enhance output quality. |
| Outcome: | VisualEDU is a benchmark to evaluate VLMs' ability to produce coherent video from text . it integrates meta-prompt learning, visual and code feedback, and a drawing toolkit to improve output quality. |
WebAggregator: Enhancing Compositional Reasoning Capabilities of Deep Research Agent Foundation Models (2026.acl-long)
Copied to clipboard
Rui Wang, Ce Zhang, Jun-Yu Ma, Jianshu Zhang, Hongru Wang, Yi Chen, Boyang Xue, Tianqing Fang, Zhisong Zhang, Hongming Zhang, Haitao Mi, Dong Yu, Kam-Fai Wong
| Challenge: | Existing agentic systems are retrieval-heavy but reasoning-light . current systems lack compositional reasoning, a key component of deep research . |
| Approach: | They propose a data synthesis pipeline WebAggregator to shift agentic paradigm . they use Proactive Explorer to collect interconnected knowledge and Compositional Logic Proposer to weave knowledge into complex questions . |
| Outcome: | The proposed pipeline surpasses GPT-4.1 and matches Claude-3.7-Sonnet on GAIA, WebWalkerQA, and XBench. |
PeerCheck: Enhancing LLM-Generated Academic Reviews Towards Human-Level Quality (2026.findings-acl)
Copied to clipboard
| Challenge: | Increasing use of large language models (LLMs) in academic review has raised concerns about quality and fairness. |
| Approach: | They propose a framework to improve the quality of LLM-generated reviews by using retrieval-augmented generation. |
| Outcome: | The proposed framework improves the human-level quality of LLM-generated reviews by adopting prompt engineering and retrieval-augmented generation. |