Data-driven Model Generalizability in Crosslinguistic Low-resource Morphological Segmentation (2022.tacl-1)
Copied to clipboard
| Challenge: | morphological segmentation is a common method of evaluation for multilingual tasks . authors often examine models with one data set that is representative of all possible data . |
| Approach: | They compare three broad classes of models with different parameterizations using morphological segmentation as the test case. |
| Outcome: | The results show that the extent of model generalization depends on the characteristics of the data set, and does not necessarily rely heavily on the data sets size. |
Similar Papers
The Effect of Data Partitioning Strategy on Model Generalizability: A Case Study of Morphological Segmentation (2024.naacl-long)
Copied to clipboard
| Challenge: | Recent work to enhance data partitioning strategies for more realistic model evaluations faces challenges in providing a clear optimal choice. |
| Approach: | They analyze morphological segmentation and morphology of ten languages from 19 languages . they use multiple datasets and splits to evaluate models . |
| Outcome: | The proposed model training and evaluation sets and new test data show that models trained from random splits can achieve higher numerical scores and model rankings tend to generalize more consistently. |
Evaluating Large Language Models along Dimensions of Language Variation: A Systematik Invesdigatiom uv Cross-lingual Generalization (2024.emnlp-main)
Copied to clipboard
| Challenge: | Xue et al., 2021) show that large language models suffer from performance degradation on unseen closely-related languages and dialects relative to their high-resource language neighbour (HRLN). |
| Approach: | They propose to model phonological, morphological, and lexical distance as Bayesian noise processes to synthesize artificial languages that are controllably distant from the HRLN. |
| Outcome: | The proposed model offers insights on model robustness to isolated and composed linguistic phenomena and the impact of task and HRL characteristics on PD. |
Tackling the Low-resource Challenge for Canonical Segmentation (2020.emnlp-main)
Copied to clipboard
| Challenge: | morphological segmentation is a task of dividing words into their constituting morphemes . we compare two new approaches for the task when training data is limited . |
| Approach: | They propose to use an LSTM pointer-generator and a sequence-to-sequence model to perform canonical segmentation when training data is limited. |
| Outcome: | The proposed models outperform existing models on German, English, and Indonesian in low-resource scenarios by 11.4% accuracy. |
Principles from Clinical Research for NLP Model Generalization (2024.naacl-long)
Copied to clipboard
| Challenge: | In clinical research, generalizability depends on (a) internal validity of experiments and (b) external validity or transportability of the results to the wider population. |
| Approach: | They propose to ensure internal validity when building machine learning models in NLP by incorporating learning spurious correlations into their models. |
| Outcome: | The proposed model can perform well on data unseen during training, but drawn from the same distribution or population. |
Morphological Inflection: A Reality Check (2023.acl-long)
Copied to clipboard
| Challenge: | Morphological inflection is a popular task in sub-word NLP with practical and cognitive applications. |
| Approach: | They propose new methods to analyze data sets and evaluate their generalization abilities to better reflect likely use-cases. |
| Outcome: | The proposed methods improve generalizability and reliability of results and improve generalization abilities. |
Evaluating Morphological Compositional Generalization in Large Language Models (2025.naacl-long)
Copied to clipboard
Mete Ismayilzada, Defne Circi, Jonne Sälevä, Hale Sirin, Abdullatif Köksal, Bhuwan Dhingra, Antoine Bosselut, Duygu Ataman, Lonneke Van Der Plas
| Challenge: | Large language models (LLMs) have demonstrated significant progress in various natural language generation and understanding tasks. |
| Approach: | They define morphemes as compositional primitives and design a suite of generative and discriminative tasks to assess morphological productivity and systematicity. |
| Outcome: | The proposed models can identify individual morphological combinations better than chance, but their performance lacks systematicity, leading to significant accuracy gaps compared to humans. |
Language Models Lack Temporal Generalization and Bigger is Not Better (2025.findings-acl)
Copied to clipboard
| Challenge: | 450 encoder models are fine-tuned on 15 data splits on a task to detect events in Early Modern Dutch archival texts. |
| Approach: | They propose to fine tune six encoder models that have been pretrained with very different data on a task in Early Modern Dutch archival texts. |
| Outcome: | The proposed model is fine tuned with 5 seeds on 15 different data splits and reaches highest F1 performance. |
A Multi-dimensional Evaluation of Tokenizer-free Multilingual Pretrained Models (2023.findings-eacl)
Copied to clipboard
| Challenge: | Recent work on tokenizer-free models shows promising results in cross-lingual transfer . previous work focused on reporting accuracy on a limited set of tasks and data settings . |
| Approach: | They compare tokenizer-free and subword-based models using various dimensions . they find subword models are still the most practical choice in many settings . |
| Outcome: | The proposed model improves cross-lingual transfer and reduces engineering overhead. |
LLMSegm: Surface-level Morphological Segmentation Using Large Language Model (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing approaches to morphological segmentation split word into its morphemes . LLMSegm is applicable in low-data settings and low-resourced languages . |
| Approach: | They propose a novel approach to surface-level morphological segmentation leveraging large language models. |
| Outcome: | The proposed method is applicable in low-data settings and low-resource languages. |
A Systematic Assessment of Syntactic Generalization in Neural Language Models (2020.acl-main)
Copied to clipboard
| Challenge: | Existing work on syntactic knowledge models has not provided a clear picture of the properties required to produce proper syntaktic generalizations. |
| Approach: | They propose to evaluate syntactic knowledge of language models by varying model architectures . they find substantial differences in syntaktic generalization performance by model architecture . |
| Outcome: | The proposed model architectures outperform other architectures on a set of 34 English-language syntactic test suites. |