Empowering Cross-lingual Behavioral Testing of NLP Models with Typological Features (2023.acl-long)
Copied to clipboard
| Challenge: | Existing datasets do not allow for a fine-grained cross-lingual evaluation and mainly permit comparisons on a language level. |
| Approach: | They propose a morphologically-aware framework for behavioral testing of NLP models that generates tests in light of specific linguistic features in 12 typologically diverse languages. |
| Outcome: | The proposed framework evaluates state-of-the-art language models on the generated tests. |
Similar Papers
SynthEval: Hybrid Behavioral Testing of NLP Models with Synthetic Evaluation (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing frameworks for benchmarking in NLP often overestimate performance . however, manually creating a variety of test types requires significant human labor . |
| Approach: | They propose a framework that leverages large language models to generate a wide range of test types . they first generate sentences via LLMs and then identifies challenging examples . |
| Outcome: | The proposed framework overestimates performance on two classification tasks. |
On the Relation between Linguistic Typology and (Limitations of) Multilingual Language Modeling (D18-1)
Copied to clipboard
| Challenge: | a key challenge in cross-lingual NLP is developing general language-independent architectures that are equally applicable to any language. |
| Approach: | They propose to use a full-vocabulary setup to test the performance of language modeling (LM) on 50 typologically diverse languages. |
| Outcome: | The proposed language modeling task is based on a full vocabulary setup focused on word-level prediction on 50 typologically diverse languages. |
What is ”Typological Diversity” in NLP? (2024.emnlp-main)
Copied to clipboard
| Challenge: | linguistic typology is commonly used to motivate language selections, but there are no set definitions or criteria for such claims. |
| Approach: | They propose to use linguistic typology to motivate language selections on the basis that a broad typological sample ought to imply generalization across a wide range of languages. |
| Outcome: | The proposed measures show that skewed language selection can lead to overestimated multilingual performance. |
Studying the Inductive Biases of RNNs with Synthetic Variations of Natural Languages (N19-1)
Copied to clipboard
| Challenge: | Recent studies have identified both strengths and limitations of recurrent neural networks (RNNs) in applied natural language processing tasks. |
| Approach: | They propose a paradigm that addresses typological differences between languages . they create synthetic versions of English and train them to predict agreement features . |
| Outcome: | The proposed model improves on predicting agreement with subject and object, suggesting that RNNs have a recency bias. |
Beyond Accuracy: Behavioral Testing of NLP Models with CheckList (2020.acl-main)
Copied to clipboard
| Challenge: | In a recent study, we show that holding-out data can overestimate performance of NLP models. |
| Approach: | They propose a task-agnostic methodology for testing NLP models using a matrix of general linguistic capabilities and test types that facilitate comprehensive test ideation. |
| Outcome: | The proposed method identifies critical failures in commercial and state-of-the-art models. |
Are All Languages Equally Hard to Language-Model? (N18-2)
Copied to clipboard
| Challenge: | a fair comparison of language models is tricky because of the size of the corpora and the variability of orthographic systems. |
| Approach: | They propose a framework for fair cross-linguistic comparison of language models . they show that in some languages, textual expression is harder to predict with n-gram models compared to LSTM models based on translated text . |
| Outcome: | The proposed framework is based on translated text and language models on 21 languages. |
Exploring Linguistic Probes for Morphological Inflection (2023.emnlp-main)
Copied to clipboard
| Challenge: | morphological inflection models typically employ language-independent data splitting algorithms. |
| Approach: | They propose language-specific probes to test aspects of morphological generalization . they use three morphology-distinct languages to test their generalization abilities . |
| Outcome: | The proposed language-specific probes are used to test morphological generalization abilities on three distinct languages. |
Morphological Inflection: A Reality Check (2023.acl-long)
Copied to clipboard
| Challenge: | Morphological inflection is a popular task in sub-word NLP with practical and cognitive applications. |
| Approach: | They propose new methods to analyze data sets and evaluate their generalization abilities to better reflect likely use-cases. |
| Outcome: | The proposed methods improve generalizability and reliability of results and improve generalization abilities. |
Typology-Guided Adaptation in Multilingual Models (2025.acl-long)
Copied to clipboard
| Challenge: | Multilingual models often treat language diversity as a problem of data imbalance, overlooking structural variation. |
| Approach: | They propose a typologically grounded metric that quantifies how strongly a language relies on morphology for noun classification. |
| Outcome: | The proposed model outperforms baseline models on 10 Bantu languages . it improves Swahili accuracy by 14 points while maintaining performance on morphology-rich languages like Zulu . |
Assessing the Impact of Typological Features on Multilingual Machine Translation in the Age of Large Language Models (2026.eacl-long)
Copied to clipboard
| Challenge: | Existing evidence on the intrinsic difficulty of multilingual modeling is limited to small monolingual models or bilingual models trained from scratch. |
| Approach: | They propose to use typological properties to determine the difficulty of modeling a language . they analyze two large pre-trained multilingual translation models . |
| Outcome: | The proposed models are based on two large pre-trained models of encoder-decoder and decoder-only machine translation. |