| Challenge: | Language models evolve to tackle complex, multifaceted tasks, requiring granular evaluations . recent studies have focused on leaderboard and benchmark results, but limited interpretability makes it difficult to compare strengths and weaknesses of models. |
| Approach: | They propose an unsupervised tree-structured diagnosis framework for understanding model proficiency in specific abilities with an LLM as a judge. |
| Outcome: | The proposed framework improves model in-context learning and predicts model weaknesses with a 55% success rate compared to the framework without SkillVerse. |
Similar Papers
On Evaluating LLMs’ Capabilities as Functional Approximators: A Bayesian Evaluation Framework (2025.coling-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have revolutionized the way we can formulate tasks in text-in-text-out format. |
| Approach: | They propose a new evaluation framework to comprehensively assess LLMs’ function modeling abilities by adopting a Bayesian perspective of function modeling. |
| Outcome: | The proposed evaluation framework enables LLMs to excel in utilizing prior knowledge to develop a strong understanding of the underlying function. |
FAC2E: Better Understanding Large Language Model Capabilities by Dissociating Language and Cognition (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) are evaluated by overall performance on various text understanding and generation tasks. |
| Approach: | They propose a framework for Fine-grAined and Cognition-grounded LLMs’ Capability Evaluation that dissociates the language-related capabilities from cognition-related ones. |
| Outcome: | The proposed framework dissociates the language-related capabilities from cognition-related ones and breaks down the process of applying a specific capability into three sub-steps: recalling relevant knowledge, utilizing knowledge, and solving problems. |
SCAN: Structured Capability Assessment and Navigation for LLMs (2026.acl-long)
Copied to clipboard
| Challenge: | Existing research has focused on approximating model rankings, but such benchmarks fail to provide users and developers with a comprehensive and fine-grained understanding of a specific model’s capabilities. |
| Approach: | They propose a framework that enables detailed characterization of LLM capabilities through comprehensive and fine-grained evaluation. |
| Outcome: | The proposed framework enables detailed characterization of large language models through comprehensive and fine-grained evaluation. |
HoWToBench: Holistic Evaluation for LLM’s Capability in Human-level Writing using Tree of Writing (2026.acl-long)
Copied to clipboard
Andrew Zhuoer Feng, Cunxiang Wang, Yu Luo, Lin Fan, Irene Zhou, Zikang Wang, Xiaotao Gu, Jie Tang, Hongning Wang, Minlie Huang
| Challenge: | Evaluating the writing capabilities of large language models remains a significant challenge due to the multidimensional nature of writing skills and the limitations of existing metrics. |
| Approach: | They propose to model the aggregation weights of sub-features in a tree-structured workflow and propose a Chinese writing benchmark that mitigates biases. |
| Outcome: | The proposed tree-of-writing (ToW) measures the writing capabilities of large language models (LLMs) in Chinese and shows that it mitigates biases and achieves a *0.93* Pearson correlation with human judgments. |
LLMs meet Bloom’s Taxonomy: A Cognitive View on Large Language Model Evaluations (2025.coling-main)
Copied to clipboard
| Challenge: | Existing evaluation approaches for Large Language Models lack a structured approach that reflects the underlying cognitive abilities required for solving the tasks. |
| Approach: | They propose a hierarchical approach to evaluation of Large Language Models that leverages Bloom’s Taxonomy to identify how well they cover the levels of Bloom’ s taxonomies. |
| Outcome: | The proposed evaluation frameworks cover the Bloom’s Taxonomy, a hierarchical framework for categorizing cognitive skills, on the most widely used benchmarks. |
From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judge (2025.emnlp-main)
Copied to clipboard
Dawei Li, Bohan Jiang, Liangjie Huang, Alimohammad Beigi, Chengshuai Zhao, Zhen Tan, Amrita Bhattacharjee, Yuxuan Jiang, Canyu Chen, Tianhao Wu, Kai Shu, Lu Cheng, Huan Liu
| Challenge: | Recent advances in Large Language Models (LLMs) inspire the "LLM-as-a-judge" paradigm . traditional methods of assessment and evaluation fail in dynamic and open-ended scenarios . |
| Approach: | They propose a paradigm where LLMs are leveraged to perform scoring, ranking, or selection for machine learning evaluation scenarios. |
| Outcome: | The proposed model-based judgment and evaluation paradigms are based on large language models and are compared to the current model-driven evaluation paradigm. |
Exploring the Cognitive Knowledge Structure of Large Language Models: An Educational Diagnostic Assessment Approach (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies on LLMs evaluation with exams are lacking in cognitive research on their overall knowledge structure. |
| Approach: | They conduct an evaluation using a human test dataset based on Bloom Taxonomy to reveal the knowledge structures of Large Language Models and gain insights of their cognitive capabilities. |
| Outcome: | The proposed model can pass AP, SAT, and Leetcode exams, but lacks the cognitive power to perform on human exams. |
Praetor: A Fine-Grained Generative LLM Evaluator with Instance-Level Customizable Evaluation Criteria (2025.acl-long)
Copied to clipboard
Yongqi Leng, Renren Jin, Yue Chen, Zhuowen Han, Ling Shi, Jianxiang Peng, Lei Yang, Juesi Xiao, Deyi Xiong
| Challenge: | Existing evaluation methods are inadequate to evaluate large language models (LLMs). |
| Approach: | They propose a fine-grained generative LLM evaluator with instance-level customazable evaluation criteria that can be used to evaluate large language models. |
| Outcome: | The proposed model outperforms existing LLM evaluators and instruction-tuned LLMs on multiple benchmarks and sets new SOTA results. |
Navigating the Modern Evaluation Landscape: Considerations in Benchmarks and Frameworks for Large Language Models (LLMs) (2024.lrec-tutorials)
Copied to clipboard
| Challenge: | General-purpose Language Models have changed the world of Natural Language Processing, if not the world itself. |
| Approach: | This tutorial will lay the foundations and explain the basics of evaluation and compare traditional methods to newly developed methods. |
| Outcome: | The tutorial assumes little familiarity with metrics, datasets, prompts and benchmarks . it will compare traditional methods to newly developed methods . |
CDT: A Comprehensive Capability Framework for Large Language Models Across Cognition, Domain, and Task (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing benchmarks focus on isolated abilities, lacking a holistic framework for assessing LLM capabilities. |
| Approach: | They propose a Cognition-Domain-Task framework which measures a model’s capabilities across three dimensions. |
| Outcome: | The proposed framework improves performance on dataset evaluation and data selection, while achieving higher scores on general and specific benchmarks. |