MergeDistill: Merging Language Models using Pre-trained Distillation (2021.findings-acl)
Copied to clipboard
| Challenge: | Existing pre-trained multilingual language models often lack capacity and skewed data . this leads to inequitable representation of languages due to limited capacity and sub-optimal vocabularies. |
| Approach: | They propose a framework to merge pre-trained multilingual language models to maximize their assets with minimal dependencies. |
| Outcome: | The proposed framework outperforms teacher-trained models on multiple datasets and with a fixed model capacity. |
Similar Papers
Merge to Learn: Efficiently Adding Skills to Language Models with Model Merging (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Adapting general-purpose language models to new skills is currently expensive . Adaptation to new skill sets requires repeated training or models forget older skills . |
| Approach: | They propose a parallel-train-then-merge procedure that adds new skills to preexisting models in isolation and later merges with the general model. |
| Outcome: | The proposed method is cheaper than retraining models on updated datasets . it improves model compliance with safe prompts while preserving model's ability to refuse dangerous or harmful prompts. |
LM-Cocktail: Resilient Tuning of Language Models via Model Merging (2024.findings-acl)
Copied to clipboard
| Challenge: | Pre-trained language models are continually fine-tuned to better support downstream applications. however, this operation may result in significant performance degeneration on general perspectives. |
| Approach: | They propose a method which enables pre-trained language models to stay resilient in general perspectives. |
| Outcome: | The proposed model achieves strong empirical performance in the whole scope of general tasks while preserving a superior capacity in its targeted domain. |
Generation-Distillation for Efficient Natural Language Understanding in Low-Data Settings (D19-61)
Copied to clipboard
| Challenge: | Recent research points to knowledge distillation as a potential solution for NLU tasks. |
| Approach: | They propose a training approach that distills large finetuned LMs into a small network using unlabeled training examples. |
| Outcome: | The proposed approach outperforms BERT training approaches while using 300 times fewer parameters. |
Cross-lingual Transfer of Monolingual Models (2022.lrec-1)
Copied to clipboard
| Challenge: | Existing studies on cross-lingual learning using multilingual models cast doubt on shared vocabulary and joint pre-training . et al. (2005) show that model knowledge learned in the source language enhances the learning of the target language independently of language proximity. |
| Approach: | They propose a method for transferring monolingual models to other languages through continuous pre-training and investigate their results in English. |
| Outcome: | The proposed method outperforms a model trained from scratch in the GLUE benchmark for English . it shows that model knowledge from the source language enhances the learning of syntactic and semantic knowledge in english. |
Multi-level Distillation of Semantic Knowledge for Pre-training Multilingual Language Model (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for improving multilingual models did not focus on learning the semantic structure of representation. |
| Approach: | They propose a method to improve multilingual language models by aligning parallel sentences . they propose token-, word-, sentence- and structure-level alignment objectives . |
| Outcome: | The proposed method outperforms baseline models on XNLI, PAWS-X, and XQuAD . it obtains comparable performance on low-resource languages, the authors show . |
Improving Training Efficiency and Reducing Maintenance Costs via Language Specific Model Merging (2026.eacl-industry)
Copied to clipboard
| Challenge: | Recent research on merging multilingual multitask models has shown promise in terms of improved quality, but its computational and maintenance efficiency remains unstudied. |
| Approach: | They propose a method for fine-tuning a multilingual large language model . they compare a "retrain-all" approach and a 'train-once, merge-as-needed' approach . |
| Outcome: | The proposed approach reduces training time by up to 50% while maintaining parity in terms of quality. |
Combining Deep Generative Models and Multi-lingual Pretraining for Semi-supervised Document Classification (2021.eacl-main)
Copied to clipboard
| Challenge: | Semi-supervised learning and multilingual pretraining have been shown to be effective for task-specific labelled data shortages. |
| Approach: | They propose to combine semi-supervised deep generative models and multi-lingual pretraining to form a pipeline for document classification task. |
| Outcome: | The proposed method outperforms state-of-the-art models in low-resource settings across several languages and outperformed existing models in English. |
Training-free LLM Merging for Multi-task Learning (2025.acl-long)
Copied to clipboard
Zichuan Fu, Xian Wu, Yejing Wang, Wanyu Wang, Shanshan Ye, Hongzhi Yin, Yi Chang, Yefeng Zheng, Xiangyu Zhao
| Challenge: | Large Language Models (LLMs) have demonstrated exceptional capabilities across diverse natural language processing tasks. |
| Approach: | They propose a training-free method for unifying different specialized LLMs into a single model using model-wise and layer-wise pruning and scaling. |
| Outcome: | The proposed method outperforms existing merging techniques and surpasses models fine-tuned on combined datasets in most scenarios. |
Emerging Cross-lingual Structure in Pretrained Language Models (2020.acl-main)
Copied to clipboard
| Challenge: | Recent work has shown that multilingual pretraining works, but is unable to measure these effects. |
| Approach: | They propose to use multilingual masked language modeling to train a model on concatenated text from multiple languages to find universal latent symmetries in embedding spaces. |
| Outcome: | The proposed models can be trained on concatenated text from multiple languages without shared vocabulary or domain similarity. |
Adapt-and-Distill: Developing Small, Fast and Effective Pretrained Language Models for Domains (2021.findings-acl)
Copied to clipboard
| Challenge: | Large pre-trained models suffer from domain shift and are not optimal for specific domains. |
| Approach: | They propose a general approach to developing small, fast and effective pretrained models for specific domains by adapting off-the-shelf general pretrained model and performing task-agnostic knowledge distillation in target domains. |
| Outcome: | The proposed approach achieves better performance over the BERT BASE model in domain-specific tasks while 3.3 smaller and 5.1 faster than the BRT BASE. |