TurnBench-MS: A Benchmark for Evaluating Multi-Turn, Multi-Step Reasoning in Large Language Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing benchmarks focus on single-turn or single-step tasks, failing to capture iterative reasoning in real-world settings. |
| Approach: | They propose a benchmark that evaluates multi-turn, multi-step reasoning through an interactive code-breaking task inspired by the "Turing Machine Board Game" the best model achieves 84% accuracy in Classic mode, but performance drops to 18% in Nightmare mode. |
| Outcome: | The new benchmark evaluates multi-turn, multi-step reasoning through an interactive code-breaking task inspired by the "Turing Machine Board Game" the best model achieves 84% accuracy in Classic mode, but performance drops to 18% in Nightmare mode. |
Similar Papers
MTR-Bench: A Comprehensive Benchmark for Multi-Turn Reasoning Evaluation (2026.acl-long)
Copied to clipboard
Xiaoyuan Li, Keqin Bao, Yubo Ma, Moxin Li, Wenjie Wang, Rui Men, Yichang Zhang, Fuli Feng, Dayiheng Liu
| Challenge: | Recent advances in Large Language Models (LLMs) have shown promising results in complex reasoning tasks. |
| Approach: | They propose to use a multi-turn reasoning evaluation framework to cover multi-turn interactions with the environments of large language models. |
| Outcome: | The proposed framework covers diverse reasoning capabilities, fine-grained difficulty granularity, and necessitates multi-turn interactions with the environments. |
MT-Bench-101: A Fine-Grained Benchmark for Evaluating Large Language Models in Multi-Turn Dialogues (2024.acl-long)
Copied to clipboard
Ge Bai, Jie Liu, Xingyuan Bu, Yancheng He, Jiaheng Liu, Zhanhui Zhou, Zhuoran Lin, Wenbo Su, Tiezheng Ge, Bo Zheng, Wanli Ouyang
| Challenge: | Large Language Models (LLMs) have greatly enhanced dialogue systems, but evaluation of their capabilities remains a challenge. |
| Approach: | They propose a model to evaluate the fine-grained abilities of Large Language Models in multi-turn dialogues. |
| Outcome: | The proposed model evaluates 21 popular chatbots based on MT-Bench-101 . it includes 3 overarching abilities and 13 distinct tasks within multi-turn dialogue scenarios. |
RiddleBench: A New Generative Reasoning Benchmark for LLMs (2026.findings-eacl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) show remarkable capabilities, but complex reasoning skills require deeper investigation. |
| Approach: | They propose a benchmark of 1,737 puzzles to test reasoning beyond simple pattern matching. |
| Outcome: | The proposed model performs poorly when faced with reordered constraints or irrelevant information. |
MPBench: A Comprehensive Multimodal Reasoning Benchmark for Process Errors Identification (2025.findings-acl)
Copied to clipboard
xu Zhao Pan, Pengfei Zhou, Jiaxin Ai, Wangbo Zhao, Kai Wang, Xiaojiang Peng, Wenqi Shao, Hongxun Yao, Kaipeng Zhang
| Challenge: | Existing benchmarks of large language models focus on error detection, neglecting other scenarios like reasoning search. |
| Approach: | et al. propose a multi-task, multimodal benchmark to assess effectiveness of PRMs . step correctness, answers aggregation and reasoning process search are evaluated . ethical principles of MPBench are based on a set of evaluation paradigms based in a text-based benchmark . |
| Outcome: | a new benchmark assesses the effectiveness of large language models (LLMs) in multiple scenarios . it uses three evaluation paradigms to assess the effectiveness and compares them with existing models . a the proposed model improves reasoning accuracy by providing stepwise feedback for multi-step reasoning results . |
Metric Calculating Benchmark: Code-Verifiable Complicate Instruction Following Benchmark for Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent frontier-level LLMs have saturated many previously difficult benchmarks, leaving little room for further differentiation. |
| Approach: | They propose a benchmark to evaluate whether LLMs can execute string-matching NLP metrics by strictly following step-by-step instructions. |
| Outcome: | The proposed benchmarks show that they can perform step-by-step execution, instruction adherence, numerical computation, and long-range consistency in handling intermediate results. |
ACEBench: A Comprehensive Evaluation of LLM Tool Usage (2025.findings-emnlp)
Copied to clipboard
Chen Chen, Xinlong Hao, Weiwen Liu, Xu Huang, Xingshan Zeng, Shuai Yu, Dexun Li, Yuefeng Huang, Xiangcheng Liu, Wang Xinzhi, Wu Liu
| Challenge: | Existing benchmarks for evaluating LLMs’ tool usage face several limitations: limited evaluation scenarios, lacking assessments in real multi-turn dialogue contexts; narrow evaluation dimensions, with insufficient detailed assessments of how LLM use tools; and reliance on LLM or real API executions for evaluation, which introduces significant overhead. |
| Approach: | ACEBench is a benchmark for evaluating tool usage in Large Language Models . it categorizes data into three primary types based on evaluation methodology: Normal, Special, and Agent. |
| Outcome: | ACEBench categorizes data into three primary types based on evaluation methodology: Normal, Special, and Agent. |
TRUEBench: Can LLM Response Meet Real-world Constraints as Productivity Assistant? (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing benchmarks fail to evaluate large language models' instruction-following capabilities . current benchmarks lack multilinguality, implicit constraints and multi-turn dialogue . |
| Approach: | a new benchmark is designed to evaluate large language models' instruction-following capabilities . the benchmark features input prompts across 12 languages and includes inter-instance multilingual instructions . |
| Outcome: | a new benchmark for large language models (LLMs) is designed to assess their performance in real-world settings. |
MT-Eval: A Multi-Turn Capabilities Evaluation Benchmark for Large Language Models (2024.emnlp-main)
Copied to clipboard
Wai-Chung Kwan, Xingshan Zeng, Yuxin Jiang, Yufei Wang, Liangyou Li, Lifeng Shang, Xin Jiang, Qun Liu, Kam-Fai Wong
| Challenge: | Existing evaluation frameworks focus on single-turn evaluations, overlooking the models’ capabilities in multi-turn interactions. |
| Approach: | They propose a benchmark to evaluate the multi-turn conversational abilities of large language models (LLMs) by analyzing human-LLM conversations and constructing multi-turned queries for each category using GPT-4. |
| Outcome: | The proposed model outperforms open-source models in multi-turn tasks while retaining and recalling historical information. |
MDBench: A Synthetic Multi-Document Reasoning Benchmark Generated with Knowledge Guidance (2025.findings-acl)
Copied to clipboard
| Challenge: | Multi-document reasoning is an area of increasing relevance given LLM capabilities in handling longer-context inputs, but few benchmarks exist to rigorously examine model behavior in this setting. |
| Approach: | They propose a new dataset for evaluating LLMs on the task of multi-document reasoning that uses condensed structured seed knowledge to modify it through LLM-assisted edits. |
| Outcome: | The proposed method generates document sets and QA examples on a multi-document reasoning task using a synthetic generation process. |
Can LLMs Reason About Program Semantics? A Comprehensive Evaluation of LLMs on Formal Specification Inference (2025.acl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are increasingly being used to automate programming tasks. |
| Approach: | They propose a benchmark to evaluate LLMs' reasoning abilities on program semantics. |
| Outcome: | The proposed benchmark shows that LLMs perform well with simple control flows but struggle with more complex structures, especially loops, even with advanced prompting. |