Exploring Variation of Results from Different Experimental Conditions (2023.findings-acl)
Copied to clipboard
| Challenge: | Recent research has shown that reproducibility of NLP experiments is not guaranteed by arbitrary factors like random seed and different data splits. |
| Approach: | They propose to run the training and testing pipeline for three neural text simplification models under varying experimental conditions using the same data and train/dev/test set splits. |
| Outcome: | The proposed pipeline yields a large number of results for three neural text simplification models under varying experimental conditions. |
Similar Papers
Towards Robust Comparisons of NLP Models: A Case Study (2025.coling-main)
Copied to clipboard
| Challenge: | Existing statistical tests to compare the test scores of different NLP models have been proposed to account for nuisance factors such as noise, randomness, or hyperparameter values. |
| Approach: | They propose a regression analysis which isolates the effect of nuisance factors from the effects of the models’ capabilities. |
| Outcome: | The proposed model is able to show that the difference between BioLinkBERT and MSR BiomedBERT is 7 times smaller than previously reported. |
Unveiling the Multi-Annotation Process: Examining the Influence of Annotation Quantity and Instance Difficulty on Model Performance (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies have shown that multi-annotator datasets can improve performance when they expand from a single annotation per instance to multiple annotations. |
| Approach: | They propose a multi-annotator simulation process to generate datasets with varying annotation budgets and compare them to a single annotation per instance. |
| Outcome: | The proposed model can generate datasets with varying annotation budgets and show that similar datasets can lead to varying performance gains. |
We Need to Talk About train-dev-test Splits (2021.emnlp-main)
Copied to clipboard
| Challenge: | Standard train-dev-test splits used to benchmark multiple models are now used in NLP . comparing multiple versions of the same model on the test data leads to overfitting and "expiration" of test sets. |
| Approach: | They propose to use a tune-set when developing neural network methods to do model picking. |
| Outcome: | The proposed model picker is more robust against the evaluated hyperparameter ranges than the standard split split. |
Robust Estimation of Population-Level Effects in Repeated-Measures NLP Experimental Designs (2025.acl-long)
Copied to clipboard
| Challenge: | a study shows that linear mixed-effects models can capture language-dependent differences in a population . a long-standing concern in experimental linguistics is the multilevel nature of language data . significance testing is inconsistent or improperly applied, authors say . |
| Approach: | a case study shows that linear mixed-effects models can capture language-dependent differences . authors encourage NLP community to embrace variability as feature, not nuisance . a long-standing concern in experimental linguistics is multilevel, hierarchical nature of language data . |
| Outcome: | a case study shows that mixed-effects models can capture language-dependent differences . authors encourage the NLP community to embrace variability as a feature, not a nuisance . |
Predicting Performance for Natural Language Processing Tasks (2020.acl-main)
Copied to clipboard
| Challenge: | Natural language processing (NLP) is a vast field, with a wide variety of tasks, languages, and domains. |
| Approach: | They build regression models to predict evaluation score of an NLP experiment . they find that their models can produce meaningful predictions over unseen languages . |
| Outcome: | The proposed model outperforms baseline models and human experts on 9 different tasks. |
Investigating Text Simplification Evaluation (2021.findings-acl)
Copied to clipboard
| Challenge: | Existing studies show that parallel TS corpora contain inaccurate simplifications and incorrect alignments. |
| Approach: | They propose to improve the distribution of parallel text simplification corpora to build more robust TS models. |
| Outcome: | The proposed models can be improved by improving the distribution of TS datasets. |
Principles from Clinical Research for NLP Model Generalization (2024.naacl-long)
Copied to clipboard
| Challenge: | In clinical research, generalizability depends on (a) internal validity of experiments and (b) external validity or transportability of the results to the wider population. |
| Approach: | They propose to ensure internal validity when building machine learning models in NLP by incorporating learning spurious correlations into their models. |
| Outcome: | The proposed model can perform well on data unseen during training, but drawn from the same distribution or population. |
Gauging the Gap Between Human and Machine Text Simplification Through Analytical Evaluation of Simplification Strategies and Errors (2023.findings-eacl)
Copied to clipboard
| Challenge: | Recent studies on text simplification have focused on lexical and syntactic simplification, but few studies have attempted to assess what kind of editing operations are performed by the systems in concrete terms. |
| Approach: | They develop an analytical evaluation framework for neural text simplification systems that includes fine-grained taxonomies of simplification strategies and errors. |
| Outcome: | The framework was used to evaluate TS models produced by human editors and multiple neural TS systems and found that human editors perform deletions and local substitutions while excessively omitting important information. |
Revisiting the Effects of Leakage on Dependency Parsing (2022.findings-acl)
Copied to clipboard
| Challenge: | Recent work shows that treebank size and linguistic variation are important factors that explain the variation in dependency parsing performance. |
| Approach: | They propose a measure of leakage that explains and correlates with observed performance variation. |
| Outcome: | The proposed measure explains and correlates with observed performance variation. |
Measuring the Instability of Fine-Tuning (2023.acl-long)
Copied to clipboard
| Challenge: | Many previous studies have investigated fine-tuning pre-trained language models on downstream tasks with varying random seeds, but they only used the standard deviation of performance scores (SD) as their measure, which is a narrow characterization of instability. |
| Approach: | They propose a systematic evaluation framework for the standard deviation of performance scores (SD) and six other measures quantifying instability of different granularity levels. |
| Outcome: | The proposed framework will be used to evaluate the validity of these measures and to improve them. |