Principles from Clinical Research for NLP Model Generalization (2024.naacl-long)
Copied to clipboard
| Challenge: | In clinical research, generalizability depends on (a) internal validity of experiments and (b) external validity or transportability of the results to the wider population. |
| Approach: | They propose to ensure internal validity when building machine learning models in NLP by incorporating learning spurious correlations into their models. |
| Outcome: | The proposed model can perform well on data unseen during training, but drawn from the same distribution or population. |
Similar Papers
Out-of-Distribution Generalization in Natural Language Processing: Past, Present, and Future (2023.emnlp-main)
Copied to clipboard
Linyi Yang, Yaoxian Song, Xuan Ren, Chenyang Lyu, Yidong Wang, Jingming Zhuo, Lingqiao Liu, Jindong Wang, Jennifer Foster, Yue Zhang
| Challenge: | Existing literature on the generalization of machine learning models to out-of-distribution data is lacking. |
| Approach: | They propose to present the first comprehensive review of recent progress, methods, and evaluations on the generalization challenge from an OOD perspective in natural language understanding. |
| Outcome: | The proposed survey provides the first comprehensive review of recent progress, methods, and evaluations on the generalization challenge from an OOD perspective in natural language understanding. |
Memorization vs. Generalization : Quantifying Data Leakage in NLP Performance Evaluation (2021.eacl-main)
Copied to clipboard
| Challenge: | Public datasets are often used to evaluate the efficacy and generalizability of state-of-the-art methods for many tasks in natural language processing (NLP). |
| Approach: | They identify leakage of training data into test data on several publicly available datasets used to evaluate NLP tasks, including named entity recognition and relation extraction. |
| Outcome: | The proposed model can memorize and generalize data on several publicly available datasets and is compared against previously unseen data. |
Data-driven Model Generalizability in Crosslinguistic Low-resource Morphological Segmentation (2022.tacl-1)
Copied to clipboard
| Challenge: | morphological segmentation is a common method of evaluation for multilingual tasks . authors often examine models with one data set that is representative of all possible data . |
| Approach: | They compare three broad classes of models with different parameterizations using morphological segmentation as the test case. |
| Outcome: | The results show that the extent of model generalization depends on the characteristics of the data set, and does not necessarily rely heavily on the data sets size. |
How Do Neural Sequence Models Generalize? Local and Global Cues for Out-of-Distribution Prediction (2021.emnlp-main)
Copied to clipboard
| Challenge: | Using RNN and transformer language models, we show consistent generalization in out-of-distribution contexts. |
| Approach: | They propose two idealized models of generalization in next-word prediction . they show that neural language models interpolate between these two forms of generalisation . |
| Outcome: | The proposed models exhibit consistent generalization in out-of-distribution contexts. |
Data Factors for Better Compositional Generalization (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent diagnostic datasets on compositional generalization expose severe problems . state-of-the-art models trained on larger and more general datasets show better generalization ability . |
| Approach: | They conduct an empirical analysis by training Transformer models on a variety of training sets with different data factors including dataset scale, pattern complexity, example difficulty, etc. |
| Outcome: | The proposed model training on larger datasets improves on compositional generalization tasks. |
On Evaluating Multilingual Compositional Generalization with Translated Datasets (2023.acl-long)
Copied to clipboard
| Challenge: | a growing amount of research investigating compositional generalization in NLP is done on English . a critical semantic distortion is a limitation of the translation of datasets . |
| Approach: | They propose to translate a dataset for evaluating compositional generalization in semantic parsing. |
| Outcome: | The proposed benchmarks show that the translation of the MCWQ dataset suffers from semantic distortion. |
Do Generalisation Results Generalise? (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing studies evaluating generalisation performance on large language models focuses on a single out-of-distribution dataset . |
| Approach: | They examine whether OOD generalisation results generalise across multiple OOD testsets throughout a finetuning run and then evaluate the partial correlation of results . |
| Outcome: | The proposed model achieves high scores on multiple OOD testsets, regressing out in-domain performance. |
A Systematic Assessment of Syntactic Generalization in Neural Language Models (2020.acl-main)
Copied to clipboard
| Challenge: | Existing work on syntactic knowledge models has not provided a clear picture of the properties required to produce proper syntaktic generalizations. |
| Approach: | They propose to evaluate syntactic knowledge of language models by varying model architectures . they find substantial differences in syntaktic generalization performance by model architecture . |
| Outcome: | The proposed model architectures outperform other architectures on a set of 34 English-language syntactic test suites. |
Generalizing over Long Tail Concepts for Medical Term Normalization (2022.emnlp-main)
Copied to clipboard
Beatrice Portelli, Simone Scaboro, Enrico Santus, Hooman Sedghamiz, Emmanuele Chersoni, Giuseppe Serra
| Challenge: | Medical term normalization is a task of mapping a text to a large number of output classes. |
| Approach: | They propose a learning strategy that leverages hierarchical information to enhance generalizability of models. |
| Outcome: | The proposed strategy produces state-of-the-art performance on seen concepts and consistent improvements on unseen ones, allowing efficient zero-shot knowledge transfer across text typologies and datasets. |
Methods for Estimating and Improving Robustness of Language Models (2022.naacl-srw)
Copied to clipboard
| Challenge: | Large language models suffer from weak generalisation ability due to shallow textual relations over full semantic complexity of the problem. |
| Approach: | They propose to incorporate some of these measures into training objectives to enhance distributional robustness of LLMs. |
| Outcome: | The proposed models outperform human models on complex tasks and outperformed other models on deep networks. |