| Challenge: | a recent study examined the cross-lingual transferability of neural language models . previous studies focused on their first language acquisition . |
| Approach: | They propose to pretrain bilingual LMs with a scenario similar to human L2 acquisition . they find that pretraining accelerated their linguistic generalization in L2 . |
| Outcome: | The results show that pretraining bilingual LMs accelerates their linguistic generalizations . the results clarify their (non-)human-like L2 acquisition in particular aspects . |
Similar Papers
The Grammar-Learning Trajectories of Neural Language Models (2022.acl-long)
Copied to clipboard
| Challenge: | In this paper, we show that neural language models with different initialization, architecture, and training data acquire linguistic phenomena in a similar order, despite their different end performance. |
| Approach: | They propose to use mutual inductive bias to study linguistic representations implicit in NLMs. |
| Outcome: | The proposed approach shows that NLMs with different initialization, architecture, and training data acquire linguistic phenomena in a similar order, despite their different end performance. |
On the Multilingual Ability of Decoder-based Pre-trained Language Models: Finding and Controlling Language-Specific Neurons (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing decoder-based pre-trained language models demonstrate excellent multilingual capabilities, but it is unclear how they handle multilingualism. |
| Approach: | They propose to examine the neuron-level internal behavior of decoder-based PLMs by finding neurons that fire “uniquely for each language” within decoded PLM models. |
| Outcome: | The proposed models fire “uniquely for each language” and show that language-specific neurons are unique, with a slight overlap (5%) between languages. |
Does Vision Accelerate Hierarchical Generalization in Neural Language Learners? (2025.coling-main)
Copied to clipboard
| Challenge: | Neural language models (LMs) are arguably less data-efficient than humans from a language acquisition perspective. |
| Approach: | They investigate the advantage of grounded language acquisition over visual input to improve syntactic generalization. |
| Outcome: | The proposed model is less efficient than humans in language acquisition . it shows that visual input helps syntactic generalization, but not vision . |
Improving Language Model Integration for Neural Machine Translation (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to integrate external language models into machine translation systems have been based on the assumption that the external model learns an implicit target-side language model at decoding time. |
| Approach: | They transfer this concept to the task of machine translation and compare it with the most prominent way of including additional monolingual data - namely back-translation. |
| Outcome: | The proposed approach outperforms the most prominent way of including additional monolingual data, namely back-translation. |
Language Adaptation of Large Language Models: An Empirical Study on LLaMA2 (2025.coling-main)
Copied to clipboard
| Challenge: | Popularity of Large Language Models (LLMs) has seen a skyrocketing increase in recent years. |
| Approach: | They present a systematic review of the language adaptation process for Large Language Models including vocabulary expansion, continued pre-training, and instruction fine-tuning. |
| Outcome: | The proposed model is based on empirical studies conducted on LLaMA2 and discussions on various settings affecting the model's capabilities. |
Modeling Nonnative Sentence Processing with L2 Language Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | Experimental results show that while all of the LMs’ word surprisals improve prediction of L2 reading times, there is no reliable effect of the choice of L1’s L1. |
| Approach: | They pretrain GPT2 on 6 different first languages, followed by English as the second language (L2). |
| Outcome: | The pretraining of L1 improves prediction of L2 reading times, but there is no reliable effect of the pretraining L1 on the model's performance on English speakers. |
Code-switched Language Models Using Dual RNNs and Same-Source Pretraining (D18-1)
Copied to clipboard
| Challenge: | Using recurrent neural networks to build language models for code-switched text is an important problem with implications to downstream applications such as speech recognition and machine translation. |
| Approach: | They propose a novel recurrent neural network unit with dual components that focus on each language in the code-switched text separately and a generative model estimated using the training data. |
| Outcome: | The proposed techniques yield significant reductions in perplexity on Mandarin-English task and improve on baseline models. |
Pre-trained language model representations for language generation (N19-1)
Copied to clipboard
| Challenge: | Pre-trained language model representations have been successful in a wide range of language understanding tasks. |
| Approach: | They propose to use pre-trained language model representations to integrate them into sequence to sequence models and apply it to machine translation and abstractive summarization. |
| Outcome: | The proposed model is able to perform 5.3 BLEU in machine translation and 5.3 on the full text version of CNN/DailyMail. |
Cross-lingual Transfer of Monolingual Models (2022.lrec-1)
Copied to clipboard
| Challenge: | Existing studies on cross-lingual learning using multilingual models cast doubt on shared vocabulary and joint pre-training . et al. (2005) show that model knowledge learned in the source language enhances the learning of the target language independently of language proximity. |
| Approach: | They propose a method for transferring monolingual models to other languages through continuous pre-training and investigate their results in English. |
| Outcome: | The proposed method outperforms a model trained from scratch in the GLUE benchmark for English . it shows that model knowledge from the source language enhances the learning of syntactic and semantic knowledge in english. |
How Do Language Models Acquire Character-Level Information? (2026.eacl-long)
Copied to clipboard
| Challenge: | Language models (LMs) implicitly encode character-level information, despite not being explicitly provided during training. |
| Approach: | They analyze how language models acquire character-level knowledge by comparing them to standard settings. |
| Outcome: | The results show that LMs do not treat words as opaque tokens, but instead treat them as tokens. |