ColD Fusion: Collaborative Descent for Distributed Multitask Finetuning (2023.acl-long)
Copied to clipboard
| Challenge: | Pretraining has been shown to scale well with compute, data size and data diversity. |
| Approach: | They propose a method that provides benefits of multitask learning but leverages distributed computation . they propose 'coldfusion' can create synergistic loop where finetuned models can be "recycled" |
| Outcome: | The proposed method outperforms RoBERTa and previous multitask models on 35 datasets. |
Similar Papers
Cool-Fusion: Fuse Large Language Models without Training (2025.acl-long)
Copied to clipboard
| Challenge: | Cool-Fusion is a simple yet effective approach to combine two or more heterogeneous large language models . |
| Approach: | They propose a method that fuses the knowledge of two or more heterogeneous large language models to leverage complementary strengths. |
| Outcome: | The proposed method increases accuracy from three strong source LLMs on GSM8K by 17.4%. |
Attention Fusion: a light yet efficient late fusion mechanism for task adaptation in NLU (2022.findings-naacl)
Copied to clipboard
| Challenge: | a recent study has shown that fine-tuning pre-trained models is parameter-inefficient and expensive. |
| Approach: | They propose a task-attuned token module which integrates pre-trained network representations into a pre-trainer. |
| Outcome: | The proposed model trains only 0.0009% of the parameters and is efficient during computation and scalable during deployment. |
Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworks (2021.acl-long)
Copied to clipboard
| Challenge: | State-of-the-art parameter-efficient fine-tuning methods rely on introducing adapter modules between the layers of a pretrained language model. |
| Approach: | They propose a framework that can learn adapter parameters for all layers and tasks by generating them using shared hypernetworks. |
| Outcome: | The proposed framework improves performance on the well-known GLUE benchmark while adding only 0.29% parameters per task. |
Multitask Semi-Supervised Learning for Class-Imbalanced Discourse Classification (2021.emnlp-main)
Copied to clipboard
| Challenge: | Discourse learning is a complex task, and schemas evolve across annotation efforts preventing compilation of smaller datasets into larger ones. |
| Approach: | They propose a multitask learning approach that can combine discourse datasets from similar and diverse domains to improve discourse classification. |
| Outcome: | The proposed approach improves on the NewsDiscourse dataset by 4.9% over current state-of-the-art benchmarks on one of the largest discourse datasets. |
Jointly Reparametrized Multi-Layer Adaptation for Efficient and Private Tuning (2023.findings-acl)
Copied to clipboard
| Challenge: | Efficient finetuning of pretrained language transformers requires a large number of tunable parameters. |
| Approach: | They propose a language transformer finetuning strategy that introduces task-specific parameters in multiple transformer layers. |
| Outcome: | The proposed method outperforms other methods with 4,100 parameters on GLUE tasks with 5% of full finetuning performance. |
Tricks for Training Sparse Translation Models (2022.naacl-main)
Copied to clipboard
| Challenge: | Multitask learning with an unbalanced data distribution skews model learning towards high resource tasks. |
| Approach: | They propose to use a temperature heating mechanism and dense pre-training to mitigate this by training models with a fixed model capacity. |
| Outcome: | The proposed techniques improve performance on two multilingual translation benchmarks compared to BASELayers and Dense scaling baselines and in combination, more than 2x model convergence speed. |
Muppet: Massive Multi-task Representations with Pre-Finetuning (2021.emnlp-main)
Copied to clipboard
| Challenge: | Recent work shows gains from pre-training and fine-tuning that are multi-task . but it can be difficult to know which intermediate tasks will best transfer . |
| Approach: | They propose a large-scale learning stage for pre-finetuning between pre-training and fine-tun. |
| Outcome: | The proposed model improves performance on pretrained discriminators and generation models on a wide range of tasks while improving sample efficiency during fine-tuning. |
CoLA: Collaborative Low-Rank Adaptation (2025.findings-acl)
Copied to clipboard
| Challenge: | The scaling law of Large Language Models (LLMs) reveals diminishing return on performance as model scale increases. |
| Approach: | They propose a more flexible LoRA architecture with an efficient initialization scheme . they propose combining three collaborative strategies to enhance performance . |
| Outcome: | The proposed model outperforms existing methods in low-sample scenarios. |
Combining Deep Generative Models and Multi-lingual Pretraining for Semi-supervised Document Classification (2021.eacl-main)
Copied to clipboard
| Challenge: | Semi-supervised learning and multilingual pretraining have been shown to be effective for task-specific labelled data shortages. |
| Approach: | They propose to combine semi-supervised deep generative models and multi-lingual pretraining to form a pipeline for document classification task. |
| Outcome: | The proposed method outperforms state-of-the-art models in low-resource settings across several languages and outperformed existing models in English. |
Fuse to Forget: Bias Reduction and Selective Memorization through Model Fusion (2024.emnlp-main)
Copied to clipboard
| Challenge: | a recent study examined the effects of model fusion on learning of shortcuts and social biases in fine-tuned language models. |
| Approach: | They investigate whether model fusion can be used to reduce unwanted knowledge . they examine classification tasks with artificially fusioned models . |
| Outcome: | The proposed model fusion can reduce unshared knowledge, the authors show . their results show that the model merged models can improve performance and generalize . |