Utility is in the Eye of the User: A Critique of NLP Leaderboards (2020.emnlp-main)
Copied to clipboard
| Challenge: | Performance-based evaluation has been at the expense of other attributes valued by the NLP community, such as compactness and energy efficiency. |
| Approach: | They propose to frame both the leaderboard and NLP practitioners as consumers and the benefit they get from a model as its utility to them. |
| Outcome: | The proposed model size and energy efficiency benchmarks have been successful in driving the creation of more accurate models, but have been at the expense of other attributes valued by the NLP community. |
Similar Papers
Evaluation Examples are not Equally Informative: How should that change NLP Leaderboards? (2021.acl-long)
Copied to clipboard
| Challenge: | Rather than replacing leaderboards, we advocate a re-imagining of the model to highlight if and where progress is made. |
| Approach: | They propose a Bayesian leaderboard model where latent subject skill and latent item difficulty predict correct responses. |
| Outcome: | The proposed model can guide what to annotate, identify annotation errors, detect overfitting, and identify informative examples. |
ExplainaBoard: An Explainable Leaderboard for NLP (2021.acl-demo)
Copied to clipboard
Pengfei Liu, Jinlan Fu, Yang Xiao, Weizhe Yuan, Shuaichen Chang, Junqi Dai, Yixin Liu, Zihuiwen Ye, Graham Neubig
| Challenge: | Using leaderboards, researchers can track the performance of various systems on various NLP tasks. |
| Approach: | They propose a new conceptualization and implementation of NLP evaluation using a leaderboard. |
| Outcome: | The ExplainaBoard is an evaluation tool for natural language processing (NLP) it covers more than 400 systems, 50 datasets, 40 languages, and 12 tasks. |
Confidence and Stability of Global and Pairwise Scores in NLP Evaluation (2025.acl-srw)
Copied to clipboard
| Challenge: | Modern natural language processing benchmarks are often represented as pairwise comparison leaderboards, such as LMSYS Arena. |
| Approach: | They investigate the strengths and weaknesses of global scores and pairwise comparisons to aid decision-making in selecting appropriate model evaluation strategies. |
| Outcome: | The proposed method underestimates strong models with rare errors or low confidence, while relying on global scores can be more effective. |
With Little Power Comes Great Responsibility (2020.emnlp-main)
Copied to clipboard
| Challenge: | Underpowered experiments make it more difficult to discern the difference between statistical noise and meaningful model improvements and increase the chances of exaggerated findings. |
| Approach: | They characterize typical statistical power for a variety of settings and characterize it by a set of existing NLP papers and datasets. |
| Outcome: | The authors characterize typical power for a variety of settings and find it common in the literature. |
Academics Can Contribute to Domain-Specialized Language Models (2024.emnlp-main)
Copied to clipboard
Mark Dredze, Genta Winata, Prabhanjan Kambadur, Shijie Wu, Ozan Irsoy, Steven Lu, Vadim Dabravolski, David Rosenberg, Sebastian Gehrmann
| Challenge: | Commercially available models dominate academic leaderboards, focusing on creating and adapting general-purpose models . however, general- purpose models often underperform in specialized domains, and domain-specific models yield superior results. |
| Approach: | They advocate for a renewed focus on developing and evaluating domain- and task-specific models . they advocate for an adapted or adapted model that can be used to improve academic leaderboard standings . |
| Outcome: | The proposed model can do well on professional and linguistic examinations, college-level knowledge questions, and collections of reasoning tasks. |
Language (Technology) is Power: A Critical Survey of “Bias” in NLP (2020.acl-main)
Copied to clipboard
| Challenge: | 146 papers analyzing "bias" in NLP systems lack normative reasoning, we find . authors propose three recommendations for work analyzing “bias” in Nlp systems . |
| Approach: | They propose three recommendations for analyzing "bias" in NLP systems . they propose to focus on what kinds of system behaviors are harmful, in what ways, to whom, and why . |
| Outcome: | The proposed methods for measuring or mitigating “bias” are poorly matched to their motivations and do not engage critically with literature outside of NLP. |
How Good Is NLP? A Sober Look at NLP Tasks through the Lens of Social Impact (2021.findings-acl)
Copied to clipboard
| Challenge: | Recent years have seen many breakthroughs in natural language processing (NLP), transitioning it from a mostly theoretical field to one with many real-world applications. |
| Approach: | They propose a moral philosophy definition of social good and a framework to evaluate the direct and indirect real-world impact of NLP tasks. |
| Outcome: | The proposed framework evaluates the direct and indirect real-world impact of NLP tasks and adopts the methodology of global priorities research to identify priority causes for NLP research. |
What happens if you treat ordinal ratings as interval data? Human evaluations in NLP are even more under-powered than you think (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies have shown that human evaluations in NLP are under-powered because of two common factors: they treat ordinal data as interval data and operate under high variance settings. |
| Approach: | They propose to use ordinal mixed effects models to detect small differences between models, especially in high variance settings common in NLP evaluations of generated texts. |
| Outcome: | The proposed models detect small differences in high variance settings, especially in high-variance evaluations of generated texts. |
When Benchmarks are Targets: Revealing the Sensitivity of Large Language Model Leaderboards (2024.acl-long)
Copied to clipboard
Norah Alzahrani, Hisham Alyahya, Yazeed Alnumay, Sultan AlRashed, Shaykhah Alsubaie, Yousef Almushayqih, Faisal Mirza, Nouf Alotaibi, Nora Al-Twairesh, Areeb Alowisheq, M Saiful Bari, Haidar Khan
| Challenge: | Existing leaderboards are often taken at face value, but this is costly . a recent study shows that minor perturbations to the benchmark result in rankings up to 8 positions. |
| Approach: | They propose to use a *hybrid* scoring method for answer selection for large language models . they find that minor perturbations to the benchmark result in rankings changes . |
| Outcome: | The proposed model is a hybrid scoring method, the authors argue . the proposed model could be used to improve the performance of large language models . |
Bidimensional Leaderboards: Generate and Evaluate Language Hand in Hand (2022.naacl-main)
Copied to clipboard
Jungo Kasai, Keisuke Sakaguchi, Ronan Le Bras, Lavinia Dunagan, Jacob Morrison, Alexander Fabbri, Yejin Choi, Noah A. Smith
| Challenge: | Recent advances on models and metrics should benefit and inform each other, authors argue . bidimensional leaderboards allow for fast, accurate evaluation of language generation models . |
| Approach: | They propose a bidimensional leaderboard that tracks progress in language generation models and metrics for their evaluation. |
| Outcome: | The proposed leaderboards track progress in language generation models and metrics for their evaluation. |