Fisher Information-based Efficient Curriculum Federated Learning with Large Language Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing frameworks for learning Large Language Models (LLMs) require adaptive data processing and low-rank adjustment to improve accuracy and fine-tuning speed. |
| Approach: | They propose a fisher information-based adaptive federated curriculum learning framework with two novel methods to improve FL fine-tuning process. |
| Outcome: | The proposed framework improves performance and fine-tuning speed compared with baseline approaches. |
Similar Papers
EcoLoRA: Communication-Efficient Federated Fine-Tuning of Large Language Models (2025.emnlp-main)
Copied to clipboard
Han Liu, Ruoyao Wen, Srijith Nair, Jia Liu, Wenjing Lou, Chongjie Zhang, William Yeoh, Yevgeniy Vorobeychik, Ning Zhang
| Challenge: | Recurrent exchange of model updates in FL can result in prohibitively high communication costs, hindering the distributed learning process. |
| Approach: | They propose a federated fine-tuning framework that uses a round-robin segment sharing scheme to reduce network bandwidth and adaptive sparsification methods tailored to LoRA’s training dynamics. |
| Outcome: | The proposed framework reduces communication overhead without compromising performance on question-answering and value-alignment tasks. |
Communication-Efficient and Tensorized Federated Fine-Tuning of Large Language Models (2025.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) excel in translation and summarization due to the capabilities of transformer architectures. |
| Approach: | They propose to integrate tensorized adapters into model encoder/decoder blocks to improve model adaptability against data heterogeneity. |
| Outcome: | Experiments on large-scale cross-device FL and large-silo FL show that the proposed methods perform on par or even better than existing federated PEFT approaches while reducing communication cost. |
Federated Data-Efficient Instruction Tuning for Large Language Models (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing federated learning (FL) uses all local data, causing excessive computational overhead and overfitting to local data. |
| Approach: | They propose a federated data-efficient instruction tuning approach which utilizes a representative subset of edge-side data to tune LLMs. |
| Outcome: | The proposed method improves Rouge-L on unseen tasks by 10.72% over the SOTA full-data instruction tuning methods while using less than 1.5% of the data samples. |
FedPETuning: When Federated Learning Meets the Parameter-Efficient Tuning Methods of Pre-trained Language Models (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing research on federated learning (FL) for pre-trained language models (PLMs) with increasing concerns about data privacy, enterprises or institutions are not allowed to collect data from end devices or local clients to a centralized server for fine-tuning PLMs. |
| Approach: | They investigate the parameter-efficient tuning of pre-trained language models (PLMs) and develop a federated benchmark for four representative PETuning methods . |
| Outcome: | The proposed method can defend against privacy attacks and maintain acceptable performance with reducing heavy resource consumption. |
Federated Learning of Large Language Models with Parameter-Efficient Prompt Tuning and Adaptive Optimization (2023.emnlp-main)
Copied to clipboard
| Challenge: | Prompt tuning of Large Language Models (LLMs) can incur performance degradation or low training efficiency. |
| Approach: | They propose a prompt tuning approach with Adaptive Optimization to enable efficient FL of LLMs. |
| Outcome: | The proposed approach improves performance and efficiency simultaneously and addresses client drift problems on both the device and server sides. |
pFedGPT: Hierarchically Optimizing LoRA Aggregation Weights for Personalized Federated GPT Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for fine-tuning Large Language Models (LLMs) struggle with data heterogeneity and adapt shared global knowledge to individual client needs. |
| Approach: | They propose a framework that leverages Hierarchical Bayesian Optimization (HBO) for fine-grained, personalized LoRA aggregation. |
| Outcome: | The proposed framework achieves state-of-the-art (SOTA) performance on personalized FL benchmarks while introducing only minimal (approx. 4%) additional optimization overhead. |
FedDQC: Data Quality Control in Federated Instruction-tuning of Large Language Models (2025.findings-acl)
Copied to clipboard
| Challenge: | Federated Learning (FL) enables privacy-preserving collaborative instruction tuning of large language models. |
| Approach: | They propose a federated instruction tuning framework with dynamic data quality control to solve this problem. |
| Outcome: | The proposed framework improves performance on mixed-quality datasets on synthetic and real-world datasets. |
Towards Robust and Efficient Federated Low-Rank Adaptation with Heterogeneous Clients (2025.acl-long)
Copied to clipboard
| Challenge: | Existing methods for federated fine-tuning for Large Language Models suffer from performance degradation at low ranks in heterogeneous data settings. |
| Approach: | They propose a low-rank adaptive model with Alternating freeze and Adaptive rank selection which reduces the number of uploaded parameters by 99.8% . |
| Outcome: | The proposed low-rank Adaptation maintains robustness even under extreme heterogeneity and low rank conditions while preserving communication efficiency. |
FedLFC: Towards Efficient Federated Multilingual Modeling with LoRA-based Language Family Clustering (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing frameworks for multilingual modeling face communication costs and parameter interference conflicts. |
| Approach: | They propose a communication-efficient federated learning framework with low-rank adaptation and language family clustering for Multilingual Modeling (MM) they maintain the weights of the base model, updating the lightweight Low-rank adapt parameters to minimize communication costs. |
| Outcome: | The proposed model outperforms the baseline models in performance and reduces communication overhead. |
FedProxy: Federated Fine-Tuning of LLMs via Proxy SLMs and Heterogeneity-Aware Fusion (2026.acl-long)
Copied to clipboard
| Challenge: | Existing methods for fine-tuning Large Language Models (LLMs) suffer from a performance bottleneck . Existing approaches like Offsite-Tuning (OT) secure the LLMs IP . |
| Approach: | They propose a framework that replaces weak adapters with a unified, powerful Proxy Small Language Model (SLM) they propose 'resource-friendly' compression and 'robust optimization' to handle data heterogeneity. |
| Outcome: | Experiments show that FedProxy outperforms OT and centralized fine-tuning methods. |