Vocabulary Shapes Cross-Lingual Variation of Word-Order Learnability in Language Models (2026.acl-long)
Copied to clipboard
| Challenge: | a coarse distinction between free- and fixed-word-order languages does not explain cross-lingual variation . aaron e. smith: human languages have emerged over millennia through communicative and cognitive constraints . but, within those universally shared bounds, languages exhibit striking typological diversity . |
| Approach: | They propose to train transformer language models on a spectrum of synthetic word-order variants of natural languages. |
| Outcome: | a new study shows that word order irregularities raise model surprisal, but only weakly affects learnability . the study also shows that vocabulary structure emerges as a key driver of word-order learnability across languages. |
Similar Papers
Understanding Cross-Lingual Alignment—A Survey (2024.findings-acl)
Copied to clipboard
| Challenge: | Cross-lingual alignment is the meaningful similarity of representations across languages in multilingual language models. |
| Approach: | They propose a taxonomy of methods to improve cross-lingual alignment . they argue that an effective trade-off between language-neutral and language-specific information is key . |
| Outcome: | The proposed methods can be applied to encoder models and encoder-decoder-only models . they show that language-neutral and language-specific information is key . |
Unsupervised Cross-Lingual Representation Learning (P19-4)
Copied to clipboard
| Challenge: | a comprehensive survey of cutting-edge weakly-supervised and unsupervised cross-lingual word representations is presented . |
| Approach: | This tutorial provides a comprehensive survey of recent work on weakly-supervised and unsupervised cross-lingual word representations. |
| Outcome: | This tutorial provides a comprehensive survey of cutting-edge weakly-supervised and unsupervised word representations. |
Cross-Lingual Ability of Multilingual Masked Language Models: A Study of Language Structure (2022.acl-long)
Copied to clipboard
| Challenge: | Multilingual pre-trained language models have shown impressive cross-lingual ability. |
| Approach: | They argue that cross-language ability comes from commonality between languages . they create an artificial language by modifying property in source language . |
| Outcome: | The proposed model can be implemented in multilingual and low-resource language scenarios without cross-lingual supervision or aligned data. |
A Massively Multilingual Analysis of Cross-linguality in Shared Embedding Space (2021.emnlp-main)
Copied to clipboard
| Challenge: | Cross-lingual language models house representations for many different languages in the same space. |
| Approach: | They investigate linguistic and non-linguistic factors affecting sentence-level alignment in cross-lingual pretrained language models for 101 languages and 5,050 language pairs. |
| Outcome: | The results show that word order agreement and agreement in morphological complexity are strongest predictors of cross-linguality. |
Improving Cross-lingual Transfer through Subtree-aware Word Reordering (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent studies show that multilingual language models are not effective when dealing with less-represented languages. |
| Approach: | They propose a powerful reordering method that learns word-order patterns conditioned on the syntactic context from a small amount of annotated data. |
| Outcome: | The proposed method outperforms baselines on a variety of tasks and is effective in both zero-shot and few-shot scenarios. |
Can Transformers Learn n-gram Language Models? (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing work has tested transformers' ability to represent formal languages, but language models are not classifiers of strings but rather distributions over them. |
| Approach: | They relate transformers' ability to learn random n-gram language models to ngram language model (LM) they find add- smoothing outperforms transformers on the former, while transformers perform better on the latter . |
| Outcome: | The proposed models outperform classical methods designed to learn n-gram LMs, while transformers perform better on the latter. |
Investigating Cross-Linguistic Adjective Ordering Tendencies with a Latent-Variable Model (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing models of crosslinguistic adjective ordering have relied on native speakers' intuitive judgment, not corpus data. |
| Approach: | They propose a latent-variable model that can order adjectives across 24 languages . they use tools and techniques to find universal, cross-linguistic, hierarchical ordering tendencies . |
| Outcome: | The proposed model can order adjectives across 24 languages even when languages are different . similar ordering preferences have been found to apply universally across languages . |
Are Knowledge and Reference in Multilingual Language Models Cross-Lingually Consistent? (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Cross-lingual consistency should be considered to assess cross-lingual transferability, maintain factuality of model knowledge across languages, and preserve parity of language model performance. |
| Approach: | They examine pretrained and tuned models with code-mixed coreferential statements that convey identical knowledge across languages. |
| Outcome: | The proposed model shows different levels of consistency in multilingual models, subject to language families, linguistic factors, scripts, and bottlenecks on a particular layer. |
Unlike “Likely”, “Unlike” is Unlikely: BPE-based Segmentation hurts Morphological Derivations in LLMs (2025.coling-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) use subword vocabularies to process and generate text. |
| Approach: | They find that Large Language Models (LLMs) perform poorly at handling some types of affixations because subwords are marked as initial- or intra-word . |
| Outcome: | The largest models trained on enough data can mitigate this tendency because initial- and intra-word embeddings are aligned; in-context learning also helps when all examples are selected in a consistent way; but only morphological segmentation can achieve a near-perfect accuracy. |
What Kind of Language Is Hard to Language-Model? (P19-1)
Copied to clipboard
| Challenge: | a recent study suggests that language models perform poorly across languages. |
| Approach: | They propose a model that fits a paired-sample multiplicative mixed-effects model to obtain language difficulty coefficients from at least-pairwise parallel corpora. |
| Outcome: | The proposed model is able to handle missing data and is aware of inter-sentence variation. |