| Challenge: | Existing statistical tests to compare the test scores of different NLP models have been proposed to account for nuisance factors such as noise, randomness, or hyperparameter values. |
| Approach: | They propose a regression analysis which isolates the effect of nuisance factors from the effects of the models’ capabilities. |
| Outcome: | The proposed model is able to show that the difference between BioLinkBERT and MSR BiomedBERT is 7 times smaller than previously reported. |
Similar Papers
Exploring Variation of Results from Different Experimental Conditions (2023.findings-acl)
Copied to clipboard
| Challenge: | Recent research has shown that reproducibility of NLP experiments is not guaranteed by arbitrary factors like random seed and different data splits. |
| Approach: | They propose to run the training and testing pipeline for three neural text simplification models under varying experimental conditions using the same data and train/dev/test set splits. |
| Outcome: | The proposed pipeline yields a large number of results for three neural text simplification models under varying experimental conditions. |
Whispers of Doubt Amidst Echoes of Triumph in NLP Robustness (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing approaches to measure robustness are problematic, and out-of-domain evaluations are no longer relevant. |
| Approach: | They examine models of different sizes spanning different architectural choices and pretraining objectives. |
| Outcome: | The results show that not all out-of-domain tests provide insight into robustness . merely scaling models does not make them adequately robust . |
Measure and Improve Robustness in NLP Models: A Survey (2022.naacl-main)
Copied to clipboard
| Challenge: | Despite the performance gains, NLP models are still fragile and brittle to out-of-domain data, adversarial attacks, or small perturbation to the input. |
| Approach: | They propose a survey of how to define, measure and improve robustness in NLP by connecting multiple definitions of robustness and identifying failures. |
| Outcome: | The proposed models are robust against unseen or challenging scenarios, but are still fragile and brittle to out-of-domain data and adversarial attacks. |
Please, Don’t Forget the Difference and the Confidence Interval when Seeking for the State-of-the-Art Status (2022.lrec-1)
Copied to clipboard
| Challenge: | comparing NLP systems by performance has become an essential question . comparing systems by performing performance criterion is criticized for allowing chance to determine superiority . |
| Approach: | They propose to use bootstrap confidence intervals instead of state-of-the-art status and statistical significance testing to compare NLP system performance. |
| Outcome: | The bootstrap confidence intervals are used to compare NLP system performance . the bootstrap test is more accurate than state-of-the-art status and statistical significance testing . |
Robustness and Adversarial Examples in Natural Language Processing (2021.emnlp-tutorials)
Copied to clipboard
| Challenge: | This tutorial aims to raise awareness of practical concerns about NLP robustness . it aims at addressing the weaknesses of NLP systems when faced with adversarial inputs and data with a distribution shift . |
| Approach: | This tutorial aims to bring awareness of practical concerns about NLP robustness . it reviews recent studies on analyzing the weakness of NLP systems when facing adversarial inputs . |
| Outcome: | This tutorial aims to bring awareness of practical concerns about NLP robustness . it will examine the weaknesses of NLP systems when faced with adversarial inputs and data with a distribution shift . |
On Measures of Biases and Harms in NLP (2022.findings-aacl)
Copied to clipboard
Sunipa Dev, Emily Sheng, Jieyu Zhao, Aubrie Amstutz, Jiao Sun, Yu Hou, Mattie Sanseverino, Jiin Kim, Akihiro Nishi, Nanyun Peng, Kai-Wei Chang
| Challenge: | Recent studies show that natural language processing (NLP) technologies propagate societal biases about demographic groups associated with attributes such as gender, race, and nationality. |
| Approach: | They propose a framework for harms and questions to help practitioners understand biases . they propose measurable measures to detect and mitigate biased groups . |
| Outcome: | The proposed framework provides a framework for harms and questions for practitioners to answer to guide the development of bias measures. |
NLPStatTest: A Toolkit for Comparing NLP System Performance (2020.aacl-demo)
Copied to clipboard
| Challenge: | Statistical significance testing is used to compare NLP system performance, but p-values alone are insufficient because statistical significance differs from practical significance. |
| Approach: | They propose a three-stage procedure for comparing NLP system performance and a toolkit that automates the process. |
| Outcome: | The proposed procedure is based on a three-stage procedure and compares it with existing statistical testing toolkits. |
Fighting Bias With Bias: Promoting Model Robustness by Amplifying Dataset Biases (2023.findings-acl)
Copied to clipboard
| Challenge: | Recent work sought to develop robust, unbiased models by filtering biased examples from training sets. |
| Approach: | They propose to filter out biased examples from training sets to improve models' performance. |
| Outcome: | The proposed evaluation framework is more challenging than the original dataset splits and even more challenging that hand-crafted challenge sets. |
What happens if you treat ordinal ratings as interval data? Human evaluations in NLP are even more under-powered than you think (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies have shown that human evaluations in NLP are under-powered because of two common factors: they treat ordinal data as interval data and operate under high variance settings. |
| Approach: | They propose to use ordinal mixed effects models to detect small differences between models, especially in high variance settings common in NLP evaluations of generated texts. |
| Outcome: | The proposed models detect small differences in high variance settings, especially in high-variance evaluations of generated texts. |
Principles from Clinical Research for NLP Model Generalization (2024.naacl-long)
Copied to clipboard
| Challenge: | In clinical research, generalizability depends on (a) internal validity of experiments and (b) external validity or transportability of the results to the wider population. |
| Approach: | They propose to ensure internal validity when building machine learning models in NLP by incorporating learning spurious correlations into their models. |
| Outcome: | The proposed model can perform well on data unseen during training, but drawn from the same distribution or population. |