Does the Language Matter? Curriculum Learning over Neo-Latin Languages (2024.lrec-main)
Copied to clipboard
| Challenge: | Curriculum Learning (CL) is emerging as a useful technique to reduce the cost of pre-training Large Language Models. |
| Approach: | They propose to organize training examples from the simplest to the most complex . they then test the approach to Italian and French to determine the complexity of examples . |
| Outcome: | The proposed method can be exported to other languages without adaptation. |
Similar Papers
Ling-CL: Understanding NLP Models through Linguistic Curricula (2023.emnlp-main)
Copied to clipboard
| Challenge: | linguistic complexity is a key component of learning in NLP, according to a new study . linguistic complex is based on lexical diversity, word sophistication, and readability . |
| Approach: | They employ a characterization of linguistic complexity from psycholinguistic and language acquisition research to develop data-driven curricula. |
| Outcome: | The proposed approach will inform future research in all NLP areas . it uses linguistic metrics (indices) that inform the challenges and reasoning required to address each task . |
What Makes a Good Curriculum? Disentangling the Effects of Data Ordering on LLM Mathematical Reasoning (2026.acl-long)
Copied to clipboard
| Challenge: | Curriculum learning (CL) orders data corpus by difficulty, but prior work employs disparate difficulty metrics and training setups. |
| Approach: | They propose a framework that decomposes curriculum difficulty into five dimensions: Problem Difficulty, Model Surprisal, Confidence Margin, Predictive Uncertainty and Decision Variability. |
| Outcome: | The proposed framework decomposes curriculum difficulty into five dimensions . the results show that no curriculum strategy dominates universally . |
How to Improve LLMs’ Performance on Specific Languages: A Perspective on LLM-Derived Language Similarity (2026.acl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) exhibit uneven performance across languages. |
| Approach: | They propose to use a framework to quantify the similarity within each language pair through both the lenses of language-specific performance patterns and cross-lingual transferability. |
| Outcome: | The proposed approach outperforms traditional linguistic typology and cross-lingual transferability measures on multilingual LLMs. |
Beyond Random Sampling: Efficient Language Model Pretraining via Curriculum Learning (2026.eacl-long)
Copied to clipboard
| Challenge: | Curriculum learning has improved efficiency across machine learning domains, but remains underexplored for language model pretraining. |
| Approach: | They present a systematic investigation of curriculum learning in LLM pretraining . they use vanilla curriculum learning, pacing-based sampling, and interleaved curricula . |
| Outcome: | The proposed framework accelerates convergence in early and mid-training phases, reducing training steps by 18-45% to reach baseline performance. |
Don’t Trust ChatGPT when your Question is not in English: A Study of Multilingual Abilities and Types of LLMs (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies have shown that large language models can perform a wide variety of language tasks when presented in English. |
| Approach: | They propose a method to evaluate the multilingual capabilities of large language models using a prompt back-translation method to find out how LLMs acquire their multilingual abilities. |
| Outcome: | The proposed method shows that large language models can transfer learned knowledge across different languages, but struggle to provide accurate results in translation-variant tasks. |
Training Dynamics for Curriculum Learning: A Study on Monolingual and Cross-lingual NLU (2022.emnlp-main)
Copied to clipboard
| Challenge: | Current approaches for NLU use CL to improve in-distribution data performance via heuristic-oriented or task-agnostic difficulties. |
| Approach: | They propose to use CL to improve in-distribution data performance by taking advantage of training dynamics as difficulty metrics instead of heuristic-oriented or task-agnostic difficulties. |
| Outcome: | The proposed model schedulers improve on in-distribution, out-of-distortion and zero-shot cross-lingual transfer datasets while being 20% faster on average. |
“Vorbești Românește?” A Recipe to Train Powerful Romanian LLMs with English Instructions (2024.findings-emnlp)
Copied to clipboard
Mihai Masala, Denis Ilie-Ablachim, Alexandru Dima, Dragos Georgian Corlatescu, Miruna-Andreea Zavelca, Ovio Olaru, Simina-Maria Terian, Andrei Terian, Marius Leordeanu, Horia Velicu, Marius Popescu, Mihai Dascalu, Traian Rebedea
| Challenge: | Large Language Models (LLMs) have achieved almost human-like performance on various tasks. |
| Approach: | They are the first to collect and translate a large collection of texts, instructions, and benchmarks and train, evaluate and release open-source LLMs tailored for Romanian. |
| Outcome: | The proposed model trains, evaluates and releases open-source models tailored for Romanian. |
Anything Goes? A Crosslinguistic Study of (Im)possible Language Learning in LMs (2025.acl-long)
Copied to clipboard
| Challenge: | LMs are highly flexible learners, capable of acquiring linguistic patterns beyond those learnable by humans. |
| Approach: | They train LMs to model impossible and typologically unattested languages . they find that the model does not achieve perfect separation between attested and unattest languages - suggesting some human-like inductive biases . |
| Outcome: | The proposed model can largely distinguish attested from impossible languages, but does not achieve perfect separation between them and their impossible counterparts. |
Measuring Cross-lingual Transfer in Bytes (2024.naacl-long)
Copied to clipboard
| Challenge: | Multilingual pretraining models can transfer knowledge to target languages with minimal or no examples . underlying mechanisms for this transfer remain unclear, with hypotheses ranging from language contamination to syntactic similarity. |
| Approach: | They conducted an experiment to investigate whether multilingual models transfer knowledge to target languages . they found that models initialized from diverse languages perform similarly to a target language . |
| Outcome: | a new study shows that models initialized from diverse languages perform similarly to a target language in a cross-lingual setting. |
LexComSpaL2: A Lexical Complexity Corpus for Spanish as a Foreign Language (2024.lrec-main)
Copied to clipboard
| Challenge: | 58,240 annotations are available for learners of Spanish as a foreign/second language (L2). |
| Approach: | They propose a corpus which can be employed to train personalised word-level difficulty classifiers for learners of Spanish as a foreign/second language (L2). |
| Outcome: | The proposed model can train personalised word-level difficulty classifiers for learners of Spanish as a foreign/second language (L2) using a customised version of the 5-point lexical complexity prediction scale. |