| Challenge: | Existing methods to evaluate systems with a held-out test set are insufficient for system comparison. |
| Approach: | They propose to use multiple random splits to compare performance of systems . they replicate results on standard split but fail to reproduce some rankings . |
| Outcome: | The proposed method is based on multiple random splits to replicate results with a set of part-of-speech taggers. |
Similar Papers
Is the Best Better? Bayesian Statistical Model Comparison for Natural Language Processing (2020.emnlp-main)
Copied to clipboard
| Challenge: | a recent study raises concerns about the use of standard splits to compare models . we compare the performance of six English part-of-speech taggers to those of other models based on standard split analysis . |
| Approach: | They propose a Bayesian statistical model comparison technique using k-fold cross-validation . they rank six English part-of-speech taggers across two data sets and three evaluation metrics . |
| Outcome: | The proposed method ranks English part-of-speech taggers on two data sets and three evaluation metrics. |
We Need To Talk About Random Splits (2021.eacl-main)
Copied to clipboard
| Challenge: | argued that random splits, like standard splits lead to overly optimistic performance estimates. |
| Approach: | They argue that random splits, like standard splits lead to overly optimistic performance estimates. |
| Outcome: | The proposed method leads to more realistic performance estimates than standard splits. |
We Need to Talk About train-dev-test Splits (2021.emnlp-main)
Copied to clipboard
| Challenge: | Standard train-dev-test splits used to benchmark multiple models are now used in NLP . comparing multiple versions of the same model on the test data leads to overfitting and "expiration" of test sets. |
| Approach: | They propose to use a tune-set when developing neural network methods to do model picking. |
| Outcome: | The proposed model picker is more robust against the evaluated hyperparameter ranges than the standard split split. |
How Bad are PoS Tagger in Cross-Corpora Settings? Evaluating Annotation Divergence in the UD Project. (N19-1)
Copied to clipboard
| Challenge: | Using annotation variation principles, Part-of-Speech tagging performance degrades when applied to test sentences that depart from training data. |
| Approach: | They propose to use the annotation variation principle to identify inconsistencies between annotations . they also evaluate their impact on prediction performance . |
| Outcome: | The proposed method can detect errors in gold standard annotations and improve prediction performance. |
Split and Rephrase: Better Evaluation and Stronger Baselines (P18-2)
Copied to clipboard
| Challenge: | a dataset mapping a complex sentence to a sequence of sentences conveying the same meaning is challenging in NLP. |
| Approach: | They propose a neural split and a copy-mechanism to break a complex sentence into several shorter sentences that convey the same meaning. |
| Outcome: | The proposed model outperforms the baseline model by 8.68 BLEU and further improves on the task. |
Benchmarking Long-tail Generalization with Likelihood Splits (2023.findings-eacl)
Copied to clipboard
| Challenge: | Existing methods to test out of distribution generalization have their own strengths and weaknesses. |
| Approach: | They propose a method to create challenging benchmarks that require generalizing to the tail of the distribution by re-splitting existing datasets. |
| Outcome: | The proposed approach can be customized to construct meaningful splits for a wide range of tasks. |
Please, Don’t Forget the Difference and the Confidence Interval when Seeking for the State-of-the-Art Status (2022.lrec-1)
Copied to clipboard
| Challenge: | comparing NLP systems by performance has become an essential question . comparing systems by performing performance criterion is criticized for allowing chance to determine superiority . |
| Approach: | They propose to use bootstrap confidence intervals instead of state-of-the-art status and statistical significance testing to compare NLP system performance. |
| Outcome: | The bootstrap confidence intervals are used to compare NLP system performance . the bootstrap test is more accurate than state-of-the-art status and statistical significance testing . |
Rethinking RL Evaluation: Can Benchmarks Truly Reveal Failures of RL Methods? (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing benchmarks for reinforcement learning for large language models do not accurately assess generalization. |
| Approach: | They propose three core principles for designing more faithful benchmarks: sufficient difficulty, balanced evaluation, and distributional robustness. |
| Outcome: | The proposed benchmarks do not accurately assess generalization across distribution shifts, difficulty levels, and counterfactual scenarios. |
What Will it Take to Fix Benchmarking in Natural Language Understanding? (2021.naacl-main)
Copied to clipboard
| Challenge: | Evaluation for many natural language understanding (NLU) tasks is broken due to unreliable and biased systems scoring so high on standard benchmarks. |
| Approach: | They argue that current benchmarks fail at four criteria for evaluation . they argue that adversarial data collection does not address the causes of failures . |
| Outcome: | The proposed frameworks fail at four criteria, and adversarial data collection does not address the causes of these failures, the authors argue . restoring a healthy evaluation ecosystem will require significant progress in the design of benchmark datasets, reliability with which they are annotated, their size, and the ways they handle social bias. |
Splits! Flexible Sociocultural Linguistic Investigation at Scale (2026.acl-long)
Copied to clipboard
| Challenge: | Variation in language use offers a rich lens into cultural perspectives, values, and opinions. |
| Approach: | They propose to construct a "sandbox" for systematic and flexible sociolinguistic research by splitting a reddit dataset into demographically/topically split SLPs. |
| Outcome: | The proposed method analyzes a demographically/topically split Reddit dataset validated by self-identification and replicating several known SLPs from existing literature. |