| Challenge: | Current practice is to train multiple instantiations of each, choose the best model of each type, and compare their performance on held-out test data. |
| Approach: | They propose to measure expected validation accuracy as a function of computation budget . authors find comparisons where authors would have reached different conclusions if they had used more computation . |
| Outcome: | The proposed method shows that test-set performance scores alone are insufficient for drawing accurate conclusions about which model performs best. |
Similar Papers
Showing Your Work Doesn’t Always Work (2020.acl-main)
Copied to clipboard
| Challenge: | countless experimental papers lack empirical rigor, disregarding necessities such as statistical significance tests and computational environments. |
| Approach: | They propose to report the expected validation effectiveness of the best-tuned model with respect to the computational budget. |
| Outcome: | The proposed model favors negative errors and yields poor bootstrapped confidence intervals, the authors argue . they find that the proposed model is biased and uses error-prone assumptions . |
Predicting Performance for Natural Language Processing Tasks (2020.acl-main)
Copied to clipboard
| Challenge: | Natural language processing (NLP) is a vast field, with a wide variety of tasks, languages, and domains. |
| Approach: | They build regression models to predict evaluation score of an NLP experiment . they find that their models can produce meaningful predictions over unseen languages . |
| Outcome: | The proposed model outperforms baseline models and human experts on 9 different tasks. |
Towards Realistic Practices In Low-Resource Natural Language Processing: The Development Set (D19-1)
Copied to clipboard
| Challenge: | Using development sets for low-resource training is often more effective . however, some studies show that early stopping can overestimate performance . |
| Approach: | They find that early stopping on a development set is more effective than using all available data for training. |
| Outcome: | The proposed model overestimates accuracy over languages and tasks by 1.4% compared to a more realistic set of training epochs. |
Reviewing Natural Language Processing Research (2020.acl-tutorials)
Copied to clipboard
| Challenge: | a tutorial on reviewing research in natural language processing will cover the theory and practice of reviewing research. |
| Approach: | tutorial covers the theory and practice of reviewing research in natural language processing . authors say reviewers should be more aware of "false negatives" |
| Outcome: | tutorial covers the theory and practice of reviewing research in natural language processing . authors say their reviews leave something to be desired . |
Targeting the Benchmark: On Methodology in Current Natural Language Processing Research (2021.acl-short)
Copied to clipboard
| Challenge: | a language benchmark is a task devised that is restricted enough to be managable with current methods, but is deemed challenging enough to serve as a benchmark. |
| Approach: | They propose to use a language task as a benchmark and a baseline model to argue it is challenging enough to be a good one. |
| Outcome: | The proposed language benchmarks are based on a dataset and a language task . the proposed benchmarks can be used to measure progress towards the goal of the research . |
Efficient Methods for Natural Language Processing: A Survey (2023.tacl-1)
Copied to clipboard
Marcos Treviso, Ji-Ung Lee, Tianchu Ji, Betty van Aken, Qingqing Cao, Manuel R. Ciosici, Michael Hassid, Kenneth Heafield, Sara Hooker, Colin Raffel, Pedro H. Martins, André F. T. Martins, Jessica Zosa Forde, Peter Milder, Edwin Simpson, Noam Slonim, Jesse Dodge, Emma Strubell, Niranjan Balasubramanian, Leon Derczynski, Iryna Gurevych, Roy Schwartz
| Challenge: | Recent work in natural language processing (NLP) has yielded appealing results from scaling model parameters and training data, but using only scale to improve performance means resource consumption also grows. |
| Approach: | They propose to use data, time, storage, or energy to improve model performance. |
| Outcome: | The proposed methods and findings provide guidance for conducting NLP under limited resources and point towards promising research directions for developing more efficient methods. |
Do Neural Language Models Overcome Reporting Bias? (2020.coling-main)
Copied to clipboard
| Challenge: | Recent studies show that pre-trained language models can overcome reporting bias by estimating the plausibility of rare but unspoken facts. |
| Approach: | They revisit the experiments conducted by Gordon and Van Durme (2013) . they find that pre-trained language models overestimate the very rare . |
| Outcome: | The proposed approach overestimates the rare at the expense of the rare, while minimizing reporting bias. |
Towards Reproducible Machine Learning Research in Natural Language Processing (2022.acl-tutorials)
Copied to clipboard
Ana Lucic, Maurits Bleeker, Samarth Bhargav, Jessica Forde, Koustuv Sinha, Jesse Dodge, Sasha Luccioni, Robert Stojnic
| Challenge: | a tutorial on reproducibility in ML addresses the problem of research results that are not reproducible. |
| Approach: | They propose a tutorial to ensure reproducible research in ML with an emphasis on computational linguistics and NLP. |
| Outcome: | The proposed tutorial focuses on computational linguistics and NLP . it provides a framework for using reproducibility as a teaching tool in university-level computer science programs. |
What Has Been Lost with Synthetic Evaluation? (2025.findings-emnlp)
Copied to clipboard
| Challenge: | a recent study evaluated the validity and difficulty of large language models for evaluation benchmarks . large language model evaluation benchmarking is challenging and requires specific phenomena to be addressed . |
| Approach: | They compare LLM-generated reasoning-over-text benchmarks to those generated through crowdsourcing . they find they are *less challenging for LLMs* than their human-authored counterparts . |
| Outcome: | The results show that LLMs can generate variants that are valid according to annotation guidelines, but less challenging than human-authored counterparts. |
Experimental Standards for Deep Learning in Natural Language Processing Research (2022.findings-emnlp)
Copied to clipboard
Dennis Ulmer, Elisa Bassignana, Max Müller-Eberstein, Daniel Varab, Mike Zhang, Rob van der Goot, Christian Hardmeier, Barbara Plank
| Challenge: | a lack of common experimental standards remains an open challenge to the field at large . |
| Approach: | They propose to distill discussions on experimental standards into a single, widely-applicable methodology. |
| Outcome: | Using best practices, we can strengthen experimental evidence, improve reproducibility and enable scientific progress. |