Triangulating LLM Progress through Benchmarks, Games, and Cognitive Tests (2025.findings-emnlp)
Copied to clipboard
Filippo Momentè, Alessandro Suglia, Mario Giulianelli, Ambra Ferrari, Alexander Koller, Oliver Lemon, David Schlangen, Raquel Fernández, Raffaella Bernardi
| Challenge: | MMLU and BBH are three evaluation paradigms for language learning models . interactive games are superior to standard benchmarks in discriminating models based on human cognitive assessments . |
| Approach: | They examine three evaluation paradigms: standard benchmarks, interactive games and cognitive tests . they examine whether interactive games are more effective at discriminating LLMs . |
| Outcome: | The results show that interactive games are superior to standard benchmarks in discriminating models. |
Similar Papers
LLMsPark: A Benchmark for Evaluating Large Language Models in Strategic Gaming Contexts (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models (LLMs) are increasingly important for their intelligence evaluation. |
| Approach: | They propose a game theory-based evaluation platform that measures LLMs’ decision-making strategies and social behaviors in classic game-theoretic settings. |
| Outcome: | The proposed system cross-evaluates 15 leading LLMs using leaderboard rankings and scoring mechanisms. |
From Remembering to Metacognition: Do Existing Benchmarks Accurately Evaluate LLMs? (2025.findings-emnlp)
Copied to clipboard
Geng Zhang, Yizhou Ying, Sihang Jiang, Jiaqing Liang, Guanglei Yue, Yifei Fu, Hailin Hu, Yanghua Xiao
| Challenge: | Existing benchmark datasets focus on low-level cognitive tasks while providing limited coverage of higher-level reasoning skills. |
| Approach: | They analyze the cognitive depth of popular LLM benchmarks using Bloom’s Taxonomy to evaluate both the cognitive and knowledge dimensions. |
| Outcome: | The results show that incorporating higher-level cognitive instructions into the current instruction fine-tuning process improves model performance. |
AwarenessBench: Assessing Cognitive Capabilities of Language Models (2026.acl-long)
Copied to clipboard
Xiaojian Li, Rongwu Xu, Tianyun Zhang, Yue Wang, Shuo Chen, Qiner Lyu, Briana Zhang, Peiran Yang, Kyle Xue Chen, Haoyuan Shi, Yu Wang, Wei Xu
| Challenge: | Language models exhibit increasingly consciousness-like behaviors, requiring a baseline to evaluate their cognitive abilities. |
| Approach: | They propose a benchmark to assess the cognitive abilities of language models (LMs) they compare 18 state-of-the-art LMs to human models in metacognition, self-awareness, social awareness and situational awareness . |
| Outcome: | Evaluating 18 state-of-the-art LMs, they find they consistently surpass baselines . but most models fall short in metacognition and self-awareness, the study finds . |
LogicGame: Benchmarking Rule-Based Reasoning Abilities of Large Language Models (2025.findings-acl)
Copied to clipboard
Jiayi Gui, Yiming Liu, Jiale Cheng, Xiaotao Gu, Xiao Liu, Hongning Wang, Yuxiao Dong, Jie Tang, Minlie Huang
| Challenge: | Large Language Models (LLMs) have demonstrated notable capabilities across various tasks, showcasing complex problem-solving abilities. |
| Approach: | They propose a benchmark to evaluate the rule-based logical reasoning capabilities of Large Language Models (LLMs) they create simulated scenarios in which models execute or plan operations to achieve specific outcomes. |
| Outcome: | The proposed benchmark evaluates the performance of large language models on a variety of scenarios with varying difficulty levels. |
How Far Can LLMs Improve from Experience? Measuring Test-Time Learning Ability in LLMs with Human Comparison (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing benchmarks primarily assess static knowledge, while intelligence also entails the ability to rapidly learn from experience. |
| Approach: | They propose to use semantic games to evaluate test-time learning . they recruit eight human participants to complete the same task . |
| Outcome: | The proposed framework compares model performance under limited and cumulative experience settings and contains four forms of experience representation. |
SMART: Evaluating LLMs’ Mathematical Reasoning via a Human Cognitive Process-Inspired Benchmark (2026.acl-long)
Copied to clipboard
| Challenge: | Existing evaluation methods focus on the final answer or on the intermediate reasoning steps, overlooking its inherently multi-stage and multi-dimensional nature. |
| Approach: | They propose a benchmark that decomposes mathematical problem-solving into four cognitive dimensions and introduces dimension-specific tasks to measure their cognitive processes. |
| Outcome: | The proposed model decomposes mathematical problem-solving into four cognitive dimensions and introduces dimension-specific tasks to measure their cognitive processes. |
Do You Get the Hint? Benchmarking LLMs on the Board Game Concept (2026.findings-acl)
Copied to clipboard
| Challenge: | Large language models have achieved impressive progress on many benchmarks, yet they still have fundamental weaknesses. |
| Approach: | They introduce Concept, a word-guessing board game, as a benchmark for probing abductive reasoning. |
| Outcome: | The proposed game is easily solved by humans, but is still very challenging for state-of-the-art LLMs (no model exceeds 40% success rate). |
LLMs meet Bloom’s Taxonomy: A Cognitive View on Large Language Model Evaluations (2025.coling-main)
Copied to clipboard
| Challenge: | Existing evaluation approaches for Large Language Models lack a structured approach that reflects the underlying cognitive abilities required for solving the tasks. |
| Approach: | They propose a hierarchical approach to evaluation of Large Language Models that leverages Bloom’s Taxonomy to identify how well they cover the levels of Bloom’ s taxonomies. |
| Outcome: | The proposed evaluation frameworks cover the Bloom’s Taxonomy, a hierarchical framework for categorizing cognitive skills, on the most widely used benchmarks. |
Can Large Language Models Win the International Mathematical Games? (2025.emnlp-main)
Copied to clipboard
Alessio Cocchieri, Luca Ragazzi, Giuseppe Tagliavini, Lorenzo Tordi, Antonella Carbonaro, Gianluca Moro
| Challenge: | Recent advances in large language models (LLMs) have demonstrated strong mathematical reasoning abilities, even in visual contexts. |
| Approach: | They propose a benchmark of 2,183 high-quality mathematical problems in an open-ended format that enables a structured evaluation of LLMs’ mathematical and logical reasoning abilities. |
| Outcome: | The new benchmark spans seven age groups and a skill-based taxonomy and enables a structured evaluation of LLMs’ mathematical and logical reasoning abilities. |
GAMEBoT: Transparent Assessment of LLM Reasoning in Games (2025.acl-long)
Copied to clipboard
| Challenge: | Existing efforts to create benchmarks that move beyond superficial pattern recognition to delve into the profound reasoning skills required for problemsolving face challenges such as insufficient interpretability, performance saturation or data contamination. |
| Approach: | They propose a gaming arena designed for rigorous assessment of LLM reasoning capabilities. |
| Outcome: | The proposed framework decomposes complex reasoning into predefined modular subproblems and generates ground truth for these subproblem types. |