Forget What You Know about LLMs Evaluations - LLMs are Like a Chameleon (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) excel on public benchmarks, but high scores may mask overreliance on dataset-specific surface cues rather than true language understanding. |
| Approach: | They propose a meta-evaluation framework that systematically rephrases benchmark inputs to detect overfitting. |
| Outcome: | The proposed framework detects performance degradation indicative of superficial pattern reliance on dataset-specific cues and distortion levels. |
Similar Papers
From Remembering to Metacognition: Do Existing Benchmarks Accurately Evaluate LLMs? (2025.findings-emnlp)
Copied to clipboard
Geng Zhang, Yizhou Ying, Sihang Jiang, Jiaqing Liang, Guanglei Yue, Yifei Fu, Hailin Hu, Yanghua Xiao
| Challenge: | Existing benchmark datasets focus on low-level cognitive tasks while providing limited coverage of higher-level reasoning skills. |
| Approach: | They analyze the cognitive depth of popular LLM benchmarks using Bloom’s Taxonomy to evaluate both the cognitive and knowledge dimensions. |
| Outcome: | The results show that incorporating higher-level cognitive instructions into the current instruction fine-tuning process improves model performance. |
LLMEval-Fair: A Large-Scale Longitudinal Study on Robust and Fair Evaluation of Large Language Models (2026.acl-long)
Copied to clipboard
Ming Zhang, Yujiong Shen, Jingyi Deng, Yuhui Wang, Huayu Sha, Kexin Tan, Qiyuan Peng, Yue Zhang, Junzhe Wang, Shichun Liu, Yueyuan Huang, Jingqi Tong, Changhao Jiang, Yilong Wu, Zhihao Zhang, Mingqi Wu, Mingxu Chai, Zhiheng Xi, Shihan Dou, Tao Gui, Qi Zhang, Xuanjing Huang
| Challenge: | Existing evaluation of Large Language Models on static benchmarks is vulnerable to data contamination and leaderboard overfitting. |
| Approach: | LLMEval-Fair framework provides a framework for dynamic evaluation of Large Language Models . evaluators use a proprietary bank of 220k graduate-level questions to analyze model data . |
| Outcome: | LLMEval-Fair provides robust and credible evaluation framework for Large Language Models . it provides a strong empirical validation for the dynamic evaluation paradigm . |
Benchmark Profiling: Mechanistic Diagnosis of LLM Benchmarks (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models are often judged by their scores on standard benchmarks, yet such scores often overstate real capability since they mask the mix of skills a task actually demands. |
| Approach: | They propose a diagnostic framework that decomposes benchmark performance into ten cognitively grounded abilities and computes an Ability Impact Score (AIS) AIS quantifies how much each ability contributes to a model’s success on a given benchmark. |
| Outcome: | The proposed framework decomposes performance into ten cognitively grounded abilities and computes an Ability Impact Score (AIS) that quantifies how much each ability contributes to a model’s success on a given benchmark. |
Dynamic Evaluation for Oversensitivity in LLMs (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing benchmarks rely on static datasets that degrade over time as models evolve, leading to data contamination and diminished evaluative power. |
| Approach: | They construct a framework that generates model-specific challenging datasets and aggregates them across diverse LLM families. |
| Outcome: | The framework captures emerging defensive patterns and aligns with each model’s unique behavior. |
The Model Agreed, But Didn’t Learn: Diagnosing Surface Compliance in Large Language Models (2026.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models internalize vast world knowledge as parametric memory, yet inherit the staleness and errors of their source corpora. |
| Approach: | They propose a framework that subjects models to discriminative self-assessment under diverse contextual pressures to scrutinize subtle behavioral nuances induced by memory modifications. |
| Outcome: | The proposed framework achieves high benchmarks without overwriting internal beliefs, while recursive modifications accumulate representational residues, triggering cognitive instability and permanently diminishing the reversibility of the model’s memory state. |
When Benchmarks Leak: Inference-Time Decontamination for LLMs (2026.acl-long)
Copied to clipboard
| Challenge: | a large number of large language models (LLMs) are being evaluated for their performance, but their reliability is threatened by test set contamination. |
| Approach: | They propose a framework that decontaminates large language models by applying small perturbations to the input embedding space. |
| Outcome: | The proposed framework achieves strong decontamination effectiveness while incurring minimal degradation in benign utility. |
Ready to Translate, Not to Represent? Bias and Performance Gaps in Multilingual LLMs Across Language Families and Domains (2026.findings-acl)
Copied to clipboard
Md. Faiyaz Abdullah Sayeedi, Subhey Sadi Rahman, Md. Mahbub Alam, Md. Adnanul Islam, Jannatul Ferdous Deepti, Tasnim Mohiuddin, Md Mofijul Islam, Swakkhar Shatabda
| Challenge: | Large Language Models (LLMs) have redefined Machine Translation, enabling context-aware and fluent translations across hundreds of languages and textual domains. |
| Approach: | They propose a framework and dataset to evaluate the translation quality and fairness of open-source LLMs. |
| Outcome: | The proposed framework and dataset evaluates translation quality and fairness of open-source LLMs. |
LLM-Powered Benchmark Factory: Reliable, Generic, and Efficient (2026.acl-long)
Copied to clipboard
Peiwen Yuan, Shaoxiong Feng, Yiwei Li, Xinglin Wang, Yueqi Zhang, Jiayi Shi, Chuyi Tan, Boyuan Pan, Yao Hu, Kan Li
| Challenge: | Using generic and efficient benchmark generators, human annotators are limited by inefficiency . current benchmark generator methods rely on seed signals, leading to long cycles and high costs . |
| Approach: | They propose a framework to evaluate LLMs as generic benchmark generators and integrate them as BenchMaker. |
| Outcome: | The proposed framework achieves comparable performance to human-annotated benchmarks on most metrics. |
CodeJudge-Eval: Can Large Language Models be Good Judges in Code Understanding? (2025.coling-main)
Copied to clipboard
| Challenge: | Recent advances in large language models (LLMs) have showcased impressive code generation capabilities, primarily evaluated through language-to-code benchmarks. |
| Approach: | They propose a benchmark to assess LLMs’ code understanding abilities from the perspective of code judging rather than code generation. |
| Outcome: | The proposed benchmark evaluates 12 well-known large language models to determine the correctness of provided code solutions. |
CodeJudgeBench: Benchmarking LLM-as-a-Judge for Coding Tasks (2026.acl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are increasingly used to judge code, but their reliability remains poorly understood. |
| Approach: | They propose a benchmark to evaluate Large Language Models as code judges . they find that small reasoning models outperform larger non-reasoning models . |
| Outcome: | The proposed benchmark evaluates LLM-as-a-Judge models across three coding tasks. |