| Challenge: | In language model training, it is difficult to obtain the right data mixtures for various tasks as the relationship between data and tasks is difficult. |
| Approach: | They propose to identify checkpoint models based on their respective capabilities and leverage them as data mixers by using their aggregated first-order influence approximation over source data. |
| Outcome: | The proposed framework shows significant improvements on eight reasoning benchmarks, with accuracy increases of up to 1.93%. |
Similar Papers
Practical Guidelines for Model Merging in LLMs Pre-Training (2026.acl-industry)
Copied to clipboard
| Challenge: | Existing studies on model merging have focused on stable learning rate regimes, but its effectiveness during LLM pre-training remains underexplored. |
| Approach: | They systematically investigate model merging across training phases, focusing on the transition from stable to decaying learning rates. |
| Outcome: | The proposed methods improve performance during stable learning rate regimes but diminish under decay, a phe-nomenon that is linked to reduced checkpoint diversity and lower parameter-space variability. |
Leveraging Pre-trained Checkpoints for Sequence Generation Tasks (2020.tacl-1)
Copied to clipboard
| Challenge: | Unsupervised pre-training of large neural models has revolutionized Natural Language Processing. |
| Approach: | They propose to use pre-trained checkpoints for Sequence Generation to initialize a Transformer-based sequence-to-sequence model that is compatible with these checkpoint. |
| Outcome: | The proposed model is compatible with pre-trained BERT, GPT-2, and RoBERTa checkpoints and achieves state-of-the-art results on Machine Translation, Text Summarization, Sentence Splitting, and Sentance Fusion. |
MixtureKit: A General Framework for Composing, Training, and Visualizing Mixture-of-Experts Models (2026.acl-demo)
Copied to clipboard
| Challenge: | MixtureKit is a modular open-source framework for constructing, training, and analyzing Mixture-of-Experts (MoE) models from arbitrary pre-trained or fine-tuned checkpoints. |
| Approach: | They propose a modular open-source framework for constructing, training, and analyzing Mixture-of-Experts (MoE) models from arbitrary pre-trained or fine-tuned checkpoints. |
| Outcome: | Experiments on multilingual code-switched (Arabic–Latin) show that BTX models built with MixtureKit outperform dense baselines across multiple benchmarks. |
Enhancing Language Generation with Effective Checkpoints of Pre-trained Language Model (2021.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to exploit PrLMs for NLG tasks do not get as much performance gain as in the NLU task. |
| Approach: | They propose a method to integrate public checkpoints of PrLMs for the most convenience. |
| Outcome: | The proposed method significantly improves the quality of the language generation tasks on 6 different kinds of PrLMs. |
Free Lunch: Robust Cross-Lingual Transfer via Model Checkpoint Averaging (2023.acl-long)
Copied to clipboard
| Challenge: | Massively multilingual language models have shown strong performance in zero-shot (ZS-XLT) and few-shot cross-lingual transfer setups where models are fine-tuned on task data in a source language are transferred without any or with only a few annotated instances to the target language(s). |
| Approach: | They propose a method that averages different checkpoints during task fine-tuning to improve model robustness. |
| Outcome: | The proposed method overestimates model performance in cross-lingual transfer setups where models are evaluated at checkpoints that generalize best to validation instances in the target languages. |
Revisiting Checkpoint Averaging for Neural Machine Translation (2022.findings-aacl)
Copied to clipboard
| Challenge: | Checkpoint averaging is a simple and effective method to boost the performance of converged neural machine translation models. |
| Approach: | They propose to use checkpoint averaging to increase model performance . they also propose to calculate weighted average instead of simple mean . |
| Outcome: | The proposed method is widely adopted in neural machine translation research. |
How Can We Know What Language Models Know? (2020.tacl-1)
Copied to clipboard
| Challenge: | Recent work examines knowledge contained in language models by having the LM fill in the blanks of prompts such as “Obama is a __ by profession”. |
| Approach: | They propose mining-based and paraphrasing-based methods to automatically generate high-quality and diverse prompts, as well as ensemble methods to combine answers from different prompts. |
| Outcome: | The proposed methods improve accuracy from 31.1% to 39.6% on the LAMA benchmark for extracting relational knowledge from LMs. |
MergeME: Model Merging Techniques for Homogeneous and Heterogeneous MoEs (2025.naacl-long)
Copied to clipboard
Yuhang Zhou, Giannis Karamanolakis, Victor Soto, Anna Rumshisky, Mayank Kulkarni, Furong Huang, Wei Ai, Jianhua Lu
| Challenge: | State-of-the-art methods for merging expert models with different architectures do not address parameter interference and require extensive fine-tuning to restore performance. |
| Approach: | They propose a method for merging experts with different architectures into a unified Mixture-of-Experts model with a goal of enhancing performance in each domain while retaining effectiveness on general tasks. |
| Outcome: | Experiments across multiple domains show that the proposed methods reduce fine-tuning costs and improve performance over state-of-the-art methods. |
AMRs Assemble! Learning to Ensemble with Autoregressive Models for AMR Parsing (2023.acl-short)
Copied to clipboard
| Challenge: | et al., 2013) examines the current state-of-the-art in AMR parsing . current models violate structural constraints, but they can corrupt graphs . |
| Approach: | They propose two new ensemble strategies to improve AMR parsing robustness and reduce computational time. |
| Outcome: | The proposed methods improve robustness to structural constraints while reducing computational time. |
Online Distilling from Checkpoints for Neural Machine Translation (N19-1)
Copied to clipboard
| Challenge: | Existing neural machine translation models have a deep structure with large amounts of parameters, making them hard to train. |
| Approach: | They propose an online method to generate a teacher model from checkpoints . they show steady improvement over a strong self-attention-based baseline system . |
| Outcome: | The proposed method improves on-the-fly on several datasets and language pairs. |