Parameter-Efficient Finetuning for Robust Continual Multilingual Learning (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to Continual Multilingual Learning (CML) are based on updating models using new data in stages. |
| Approach: | They propose a parameter-efficient finetuning strategy to increase the number of languages on which the model improves after an update while reducing the magnitude of loss for the remaining languages. |
| Outcome: | The proposed model improves on the languages included in the latest update while reducing the loss of performance on the remaining languages. |
Similar Papers
Fine-tuned Language Models are Continual Learners (2022.emnlp-main)
Copied to clipboard
| Challenge: | Recent work on large language models relies on intuition that most tasks can be described via natural language instructions. |
| Approach: | They propose that a model should be able to keep extending its knowledge without forgetting previous skills. |
| Outcome: | The proposed model can learn 8 new diverse language generation tasks while maintaining good performance on previous tasks, spanning in total of 70 datasets. |
TL-CL: Task And Language Incremental Continual Learning (2024.emnlp-main)
Copied to clipboard
| Challenge: | a multilingual model is periodically updated to accommodate new tasks in previously learned languages or new languages for established tasks. |
| Approach: | They propose an adapter-based parameter-efficient fine-tuning strategy for continual learning in multilingual models. |
| Outcome: | The proposed approach outperforms other parameter-efficient approaches without access to historical data for replay. |
HFT: Half Fine-Tuning for Large Language Models (2025.acl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) with one or more fine-tuning phases can unlock various capabilities, but can be catastrophic forgetting during sequential training. |
| Approach: | They propose a method to regularly reset partial parameters to mitigate forgetting issues by using half fine-tuning instead of full fine-uning. |
| Outcome: | The proposed approach reduces the risk of catastrophic forgetting during training and the parametric knowledge lost during training may be overwhelmed by incoming training data. |
Exploring Two-Phase Continual Instruction Fine-tuning for Multilingual Adaptation in Large Language Models (2026.findings-acl)
Copied to clipboard
| Challenge: | A key challenge for Large Language Models (LLMs) is improving their Multilingual instruction-following ability over time without deteriorating their ability in languages they already excel at, typically English. |
| Approach: | They propose a two-phase Continual Fine-tuning setup to improve a model's Multilingual adaptability by comparing an English-only LLM with a multilingual instruction dataset. |
| Outcome: | The proposed model improves on two-phase Continual Fine-tuning (CFT) setups on a multilingual instruction dataset. |
Unlocking Continual Learning Abilities in Language Models (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to learning models (LMs) incorporate old task data or task-wise inductive bias into LMs, but old data and accurate task information are often unavailable or costly to collect. |
| Approach: | They propose a rehearsal-free method that updates model parameters with large magnitudes . they found that the L1-normalized magnitude distribution is different when different task data is used . |
| Outcome: | The proposed method improves accuracy and performance on four CL benchmarks. |
MAPLE: Multilingual Evaluation of Parameter Efficient Finetuning of Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Prior work on multilingual evaluation has shown that there is a large gap between the performance of Large Language Models on English and other languages. |
| Approach: | They propose to finetune Llama-2 and Mistral models on two datasets to determine their effect on model performance on six downstream tasks covering forty one languages. |
| Outcome: | The proposed model can improve on six multilingual tasks while degrading on high-resource languages. |
Bridging the Language Gap: Dynamic Learning Strategies for Improving Multilingual Performance in LLMs (2025.coling-main)
Copied to clipboard
Somnath Kumar, Vaibhav Balloli, Mercy Ranjit, Kabir Ahuja, Sunayana Sitaram, Kalika Bali, Tanuja Ganu, Akshay Nambi
| Challenge: | Large language models (LLMs) excel in diverse applications but still struggle with non-Latin scripts and low-resource languages. |
| Approach: | They propose a dynamic learning approach that optimizes prompt strategy, embedding model, and LLM per query at runtime. |
| Outcome: | The proposed approach achieves 10-15% improvements in multilingual performance over pre-trained models and 4x gains compared to fine-tuned, language-specific models. |
Learning to Route for Dynamic Adapter Composition in Continual Learning with Language Models (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Recent work shows that PEFT methods can be competitive with, or even superior to, full fine-tuning of PLMs. |
| Approach: | They propose a method that isolates the training of new PEFT modules to ensure their task specialization and learns to compose them by training a network of routers that leverages a small memory containing examples of previously seen tasks. |
| Outcome: | The proposed method improves generalization and performance in two CL setups. |
Cross-lingual Continual Learning (2023.acl-long)
Copied to clipboard
| Challenge: | Existing multi-lingual representations such as the one-hop transfer learning pipeline are difficult to adapt to new languages. |
| Approach: | They propose a cross-lingual continuum learning paradigm that evaluates continuous learning approaches that adapt to emerging data from different languages. |
| Outcome: | The proposed model can be used to adapt to new languages in a sequential manner. |
A Semantic-Aware Layer-Freezing Approach to Computation-Efficient Fine-Tuning of Language Models (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing work on how to finetune but neglects the issue of where to fine-tune language models is expensive. |
| Approach: | They propose to use transition traces of latent representation to compute deviations (or loss) and then estimate the gain of each layer in reducing deviation (or gain). |
| Outcome: | The proposed approach outperforms baseline methods and is cost-benefit balanced. |