LRBench and Judge-R1: Principled Evaluation and Training of LLM-Based Judges for Long-Context Reasoning (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing benchmarks for evaluating large language models (LLMs) under long contexts are underexplored. |
| Approach: | They propose a large-scale benchmark for evaluating large language models (LLMs) that combines reinforcement learning with multi-turn search to enable grounded and principle-aware evaluation. |
| Outcome: | The proposed model outperforms single-turn baselines across domains and principles. |
Similar Papers
Does Context Matter? ContextualJudgeBench for Evaluating LLM-based Judges in Contextual Settings (2025.acl-long)
Copied to clipboard
| Challenge: | Contextual evaluation is challenging for state-of-the-art judge models . evaluation criteria are often conditional and dependent on practitioner priorities . |
| Approach: | They propose a judge benchmark that evaluates large language models as judges in contexts . they use human annotations and model-based perturbations to build the benchmark . |
| Outcome: | The proposed benchmark aims to evaluate large language models in contexts with 2,000 challenging response pairs. |
CodeJudgeBench: Benchmarking LLM-as-a-Judge for Coding Tasks (2026.acl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are increasingly used to judge code, but their reliability remains poorly understood. |
| Approach: | They propose a benchmark to evaluate Large Language Models as code judges . they find that small reasoning models outperform larger non-reasoning models . |
| Outcome: | The proposed benchmark evaluates LLM-as-a-Judge models across three coding tasks. |
Can LLMs reason over extended multilingual contexts? Towards long-context evaluation beyond retrieval over haystacks (2026.eacl-long)
Copied to clipboard
| Challenge: | Existing multilingual long-context benchmarks are myopic and inherently limited, as successful recall alone does not indicate a model’s capacity to reason over extended contexts. |
| Approach: | They propose a new synthetic benchmark for multilingual long-context reasoning that includes bAbI-style tasks that test multi-hop inference, aggregation, and epistemic reasoning. |
| Outcome: | The proposed benchmarks are based on a multilingual long-context model and span seven languages. |
LongTableBench: Benchmarking Long-Context Table Reasoning across Real-World Formats and Domains (2025.findings-emnlp)
Copied to clipboard
Liyao Li, Jiaming Tian, Hao Chen, Wentao Ye, Chao Ye, Haobo Wang, Ningtao Wang, Xing Fu, Gang Chen, Junbo Zhao
| Challenge: | Evaluating 52 LLMs reveals that only the strongest models maintain robust performance under increasing context lengths and format diversity. |
| Approach: | They propose a benchmark for evaluating long-context reasoning over semi-structured tables across diverse formats, tasks, and domains. |
| Outcome: | The proposed model outperforms compression-based approaches on tasks requiring semantic integration. |
LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Tasks (2025.acl-short)
Copied to clipboard
Anna Bavaresco, Raffaella Bernardi, Leonardo Bertolazzi, Desmond Elliott, Raquel Fernández, Albert Gatt, Esam Ghaleb, Mario Giulianelli, Michael Hanna, Alexander Koller, Andre Martins, Philipp Mondorf, Vera Neplenbroek, Sandro Pezzelle, Barbara Plank, David Schlangen, Alessandro Suglia, Aditya K Surikuchi, Ece Takmaz, Alberto Testoni
| Challenge: | Existing evaluations of NLP models with LLMs are based on human judgments . however, there are concerns about their validity and reproducibility in proprietary models . |
| Approach: | They evaluate 11 current LLMs for their ability to replicate annotations. they show substantial variance across models and datasets. |
| Outcome: | The proposed model can replicate human annotations on 20 NLP datasets and show substantial variance across models and datasets. |
LIFBench: Evaluating the Instruction Following Performance and Stability of Large Language Models in Long-Context Scenarios (2025.acl-long)
Copied to clipboard
| Challenge: | Existing benchmarks rarely focus on instruction-following in long-context scenarios or stability on different inputs. |
| Approach: | They propose a scalable dataset to evaluate LLMs’ instruction-following capabilities and stability across long contexts. |
| Outcome: | The proposed method evaluates LLMs’ instruction-following capabilities and stability across long contexts. |
JudgeAgent: Beyond Static Benchmarks for Knowledge-Driven and Dynamic LLM Evaluation (2026.findings-acl)
Copied to clipboard
Zhichao Shi, Xuhui Jiang, Chengjin Xu, Cangli Yao, Shengjie Ma, Yinghan Shen, Zixuan Li, Jian Guo, Yuanzhuo Wang
| Challenge: | Current evaluation methods for large language models rely on static benchmarks . limited knowledge coverage and fixed difficulties hinder the targeted optimizations resulting in superficial evaluations of LLMs - a problem that has been addressed by JudgeAgent . |
| Approach: | They propose a knowledge-driven and dynamic evaluation framework for large language models . judgeAgent leverages LLM agents equipped with context graphs to traverse knowledge structures . |
| Outcome: | The proposed framework can achieve comprehensive evaluations and facilitate effective model iterations. |
When Can We Trust LLMs in Mental Health? Large-Scale Benchmarks for Reliable LLM Evaluation (2026.eacl-long)
Copied to clipboard
Abeer Badawi, Elahe Rahimi, Md Tahmid Rahman Laskar, Sheri Grach, Lindsay Bertrand, Lames Danok, Prathiba Dhanesh, Jimmy Huang, Frank Rudzicz, Elham Dolatabadi
| Challenge: | Existing benchmarks for large language models are limited in scale, authenticity, and reliability due to the emotionally complex nature of therapeutic dialogue. |
| Approach: | They propose two benchmarks that provide a framework for evaluating large language models for mental health support. |
| Outcome: | The proposed framework provides a framework for generation and evaluation of large-scale authentic dialogue datasets and judge-reliability assessments. |
CriticBench: Benchmarking LLMs for Critique-Correct Reasoning (2024.findings-acl)
Copied to clipboard
| Challenge: | CriticBench is a benchmark designed to assess LLMs’ abilities to critique and refine their reasoning across a variety of tasks. |
| Approach: | They propose a benchmark to assess LLMs' ability to critique and correct reasoning across a variety of tasks. |
| Outcome: | The proposed benchmark examines the performance of 17 large language models in generation, critique, and correction reasoning. |
From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judge (2025.emnlp-main)
Copied to clipboard
Dawei Li, Bohan Jiang, Liangjie Huang, Alimohammad Beigi, Chengshuai Zhao, Zhen Tan, Amrita Bhattacharjee, Yuxuan Jiang, Canyu Chen, Tianhao Wu, Kai Shu, Lu Cheng, Huan Liu
| Challenge: | Recent advances in Large Language Models (LLMs) inspire the "LLM-as-a-judge" paradigm . traditional methods of assessment and evaluation fail in dynamic and open-ended scenarios . |
| Approach: | They propose a paradigm where LLMs are leveraged to perform scoring, ranking, or selection for machine learning evaluation scenarios. |
| Outcome: | The proposed model-based judgment and evaluation paradigms are based on large language models and are compared to the current model-driven evaluation paradigm. |