SCORE: Systematic COnsistency and Robustness Evaluation for Large Language Models (2025.naacl-industry)
Copied to clipboard
| Challenge: | Typical evaluations of Large Language Models (LLMs) report a single accuracy metric per dataset, often derived from an optimized setup. |
| Approach: | They propose a framework for non-adversarial evaluation of large language models that evaluates models by repeatedly testing them on the same benchmarks in various setups. |
| Outcome: | The proposed framework evaluates models by repeatedly testing them on the same benchmarks in various setups to give a realistic estimate of their accuracy and consistency. |
Similar Papers
A Novel Metric for Measuring the Robustness of Large Language Models in Non-adversarial Scenarios (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Using large language models, we evaluated their robustness on multiple datasets. |
| Approach: | They propose a new metric for assessing model robustness by empirical evaluation of several models on multiple datasets. |
| Outcome: | The proposed metric is based on a set of datasets that are constructed by introducing naturally-occurring, non-malicious perturbations or by generating semantically equivalent paraphrases of input questions or statements. |
Evaluating the Consistency of LLM Evaluators (2025.coling-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) have shown potential as general evaluators with the benefits of speed and cost. |
| Approach: | They conduct extensive studies on the two aspects of consistency in LLM evaluations, Self-Consistency (SC) and Inter-scale Consistency on different scoring scales and criterion granularity with open-source and proprietary models. |
| Outcome: | The results show that strong proprietary models are not necessarily consistent evaluators, highlighting the importance of considering consistency in assessing the capability of LLM evalueators. |
A Systematic Survey and Critical Review on Evaluating Large Language Models: Challenges, Limitations, and Recommendations (2024.emnlp-main)
Copied to clipboard
Md Tahmid Rahman Laskar, Sawsan Alqahtani, M Saiful Bari, Mizanur Rahman, Mohammad Abdullah Matin Khan, Haidar Khan, Israt Jahan, Amran Bhuiyan, Chee Wei Tan, Md Rizwan Parvez, Enamul Hoque, Shafiq Joty, Jimmy Huang
| Challenge: | Large Language Models (LLMs) have gained significant attention due to their capabilities in performing diverse tasks across domains. |
| Approach: | They review the primary challenges and limitations causing inconsistencies in evaluations . early models could generate coherent text but limited to simple tasks . |
| Outcome: | The proposed evaluations are reproducible, reliable, and robust. |
Leveraging Large Language Models for NLG Evaluation: Advances and Challenges (2024.emnlp-main)
Copied to clipboard
| Challenge: | introducing Large Language Models (LLMs) has opened new avenues for assessing generated content quality, e.g., coherence, creativity, and context relevance. |
| Approach: | They propose a taxonomy for organizing existing LLM-based evaluation metrics and a structured framework to understand and compare them. |
| Outcome: | The proposed taxonomy offers a framework to understand and compare LLM-based evaluation methods. |
Examining the robustness of LLM evaluation to the distributional assumptions of benchmarks (2024.acl-long)
Copied to clipboard
| Challenge: | Using benchmarks to evaluate Large Language Models is inconsistent with the assumption that the test prompts within a benchmark represent a random sample from some real-world distribution of interest. |
| Approach: | They propose to use a model's average performance across the test prompts of a benchmark to evaluate its performance. |
| Outcome: | The results show that the correlation between model performance across test prompts and the test prompt can change model rankings on major benchmarks. |
Can Large Language Models Always Solve Easy Problems if They Can Solve Harder Ones? (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) have impressive capabilities, but still suffer from inconsistency issues. |
| Approach: | They develop a ConsisEval benchmark to evaluate LLMs' inconsistency . they find that LLM models can paradoxically fail at easier problems . |
| Outcome: | The proposed model achieves highest consistency score but inconsistent to specific questions due to distraction by redundant information, misinterpretation of questions, etc. |
Model Consistency as a Cheap yet Predictive Proxy for LLM Elo Scores (2025.emnlp-main)
Copied to clipboard
| Challenge: | a rapid proliferation of large language models (LLMs) makes it difficult to assess which models are best suited for specific tasks. |
| Approach: | They find that the consistency of an LLM's Elo score is 91% correlated with its own human-produced Elo scores. |
| Outcome: | a new method to evaluate large language models is needed to scale with the increasing number of models released . current best way is to measure model's Elo score by comparing it to other models in contests . a simple proxy for Elo scores can be computed cheaply without human data or prior knowledge . |
How Reliable is Multilingual LLM-as-a-Judge? (2025.findings-emnlp)
Copied to clipboard
| Challenge: | LLMs are a popular evaluation strategy, but their reliability in multilingual evaluation remains uncertain. |
| Approach: | They evaluate five models from different model families across five diverse tasks involving 25 languages. |
| Outcome: | The models perform poorly across languages and average Fleiss’ Kappa is 0.3 . |
Exploring the Reliability of Large Language Models as Customized Evaluators for Diverse NLP Tasks (2025.coling-main)
Copied to clipboard
| Challenge: | Existing work uses large language models (LLMs) to evaluate natural language process tasks, but there are shortcomings in current LLMs. |
| Approach: | They examine the alignment between LLM evaluators and human annotators by comparing conventional and alignment tasks with different evaluation criteria. |
| Outcome: | The proposed models excel in general criteria, such as fluency, but face challenges with complex criteria, including numerical reasoning. |
Benchmarking and Improving LLM Robustness for Personalized Generation (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing evaluations focus on whether a model’s responses align with a user’s preferences, but factuality is an important yet overlooked dimension. |
| Approach: | They propose a scalable framework for evaluating robustness of large language models in personalization and a new dataset, PERGData. |
| Outcome: | The proposed framework improves robustness by 25% across models. |