Improving Model Evaluation using SMART Filtering of Benchmark Datasets (2025.naacl-long)
Copied to clipboard
| Challenge: | Creating high quality human-annotated datasets is difficult due to dataset saturation. |
| Approach: | They propose a method to filter a subset of test examples from existing benchmarks by removing less informative and lower quality examples. |
| Outcome: | The proposed method reduces dataset size by 48% while increasing Pearson correlation with rankings from ChatBot Arena. |
Similar Papers
Fair Enough: Standardizing Evaluation and Model Selection for Fairness Research in NLP (2023.eacl-main)
Copied to clipboard
| Challenge: | Modern NLP systems exhibit a range of biases, which a growing literature on model debiasing attempts to correct. |
| Approach: | They propose to clarify the current situation and plot a course for meaningful progress in fair learning by making clear inter-relations among the current gamut of methods and their relation to fairness theory. |
| Outcome: | The proposed approach addresses the practical problem of model selection, which involves a trade-off between fairness and accuracy and has led to systemic issues in fairness research. |
Anchor Points: Benchmarking Models with Much Fewer Examples (2024.eacl-long)
Copied to clipboard
| Challenge: | Modern language models exhibit powerful but brittle behavior, leading to larger and more diverse benchmarks. |
| Approach: | They propose to use anchor points to select small subsets of a language model-prompt dataset to capture model behavior across the entire dataset. |
| Outcome: | The proposed technique outperforms standard benchmarks in language models with 1-30 anchor points . the proposed technique can be used to compare models on different regions of the dataset . |
Disentangling the Roles of Representation and Selection in Data Pruning (2025.acl-long)
Copied to clipboard
| Challenge: | Existing methods for data pruning involve many different design choices, which have not been systematically studied. |
| Approach: | They decompose data pruning into two key components: data representation and selection algorithm . theoretical and empirical results highlight crucial role of representations . |
| Outcome: | The proposed method can be used to train models with less data. |
Take the essence and discard the dross: A Rethinking on Data Selection for Fine-Tuning Large Language Models (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing studies focus on data selection but lack a clear, unified framework . variability in experimental settings complicates systematic comparisons . |
| Approach: | They propose a three-stage scheme to standardize data selection for fine-tuning large language models . they propose unified comparison approach that incorporates ratio-based efficiency and ranking-based feasibility metrics to address inconsistencies across experiments. |
| Outcome: | The proposed scheme outperforms existing methods in a dozen key studies and identifies key challenges. |
Not All Metrics Are Guilty: Improving NLG Evaluation by Diversifying References (2024.naacl-long)
Copied to clipboard
Tianyi Tang, Hongyuan Lu, Yuchen Jiang, Haoyang Huang, Dongdong Zhang, Xin Zhao, Tom Kocmi, Furu Wei
| Challenge: | Existing evaluation benchmarks with limited references may not accurately reflect the quality of the model’s hypotheses. |
| Approach: | They propose a method to enrich evaluation benchmarks by diversifying the expression of a single reference into multiple high-quality ones to cover the semantic space of the reference sentence as much as possible. |
| Outcome: | The proposed method can enhance evaluation benchmarks by diversifying the expression of reference into multiple high-quality ones to cover the semantic space of the reference sentence as much as possible. |
A guide to the dataset explosion in QA, NLI, and commonsense reasoning (2020.coling-tutorials)
Copied to clipboard
| Challenge: | a tutorial aims to provide an up-to-date guide to the recent datasets . the target audience is the NLP practitioners who are lost in dozens of the recent data sets. |
| Approach: | This tutorial provides an up-to-date guide to the recent datasets . it surveys old and new methodological issues with dataset construction . |
| Outcome: | This tutorial aims to provide an up-to-date guide to the recent datasets . it surveys the old and new methodological issues with dataset construction . |
How Good is Your Wikipedia? Auditing Data Quality for Low-resource and Multilingual NLP (2026.acl-long)
Copied to clipboard
Kushal Tatariya, Artur Kulmizev, Wessel Poelman, Esther Ploeger, Marcel Bollmann, Johannes Bjerva, Jiaming Luo, Heather Lent, Miryam de Lhoneux
| Challenge: | Wikipedia’s perceived high quality and broad language coverage have established it as a fundamental resource in NLP. |
| Approach: | They propose a data filtering procedure which removes a large percentage of Wikipedia's data and a 4-level quality ranking of the site. |
| Outcome: | The results show that the proposed filtering procedure outperforms the raw Wikipedia models in three language modelling scenarios. |
Fighting Bias With Bias: Promoting Model Robustness by Amplifying Dataset Biases (2023.findings-acl)
Copied to clipboard
| Challenge: | Recent work sought to develop robust, unbiased models by filtering biased examples from training sets. |
| Approach: | They propose to filter out biased examples from training sets to improve models' performance. |
| Outcome: | The proposed evaluation framework is more challenging than the original dataset splits and even more challenging that hand-crafted challenge sets. |
Make Every Example Count: On the Stability and Utility of Self-Influence for Learning from Noisy NLP Datasets (2023.emnlp-main)
Copied to clipboard
| Challenge: | Increasingly larger datasets have become a standard ingredient to advancing the state-of-the-art in NLP, however, data quality might have already become the bottleneck to unlock further gains. |
| Approach: | They propose a general method for improving model performance in the presence of noisy training data based on self-influence and bandit curriculum learning. |
| Outcome: | The proposed method improves model performance in machine translation, question answering and text classification, building up on approaches to self-influence calculation and automated curriculum learning. |
LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Tasks (2025.acl-short)
Copied to clipboard
Anna Bavaresco, Raffaella Bernardi, Leonardo Bertolazzi, Desmond Elliott, Raquel Fernández, Albert Gatt, Esam Ghaleb, Mario Giulianelli, Michael Hanna, Alexander Koller, Andre Martins, Philipp Mondorf, Vera Neplenbroek, Sandro Pezzelle, Barbara Plank, David Schlangen, Alessandro Suglia, Aditya K Surikuchi, Ece Takmaz, Alberto Testoni
| Challenge: | Existing evaluations of NLP models with LLMs are based on human judgments . however, there are concerns about their validity and reproducibility in proprietary models . |
| Approach: | They evaluate 11 current LLMs for their ability to replicate annotations. they show substantial variance across models and datasets. |
| Outcome: | The proposed model can replicate human annotations on 20 NLP datasets and show substantial variance across models and datasets. |