| Challenge: | Low-Rank Adaptation (LoRA) has emerged as a prominent solution to mitigate the communication and computation costs in federated fine-tuning of Large Language Models (LLMs). |
| Approach: | They propose a plug-and-play layer freezing mechanism to integrate with existing federated fine-tuning frameworks. |
| Outcome: | The proposed solution reduces communication overhead and lowers computational costs while preserving the performance of the underlying federated fine-tuning methods. |
Similar Papers
Towards Robust and Efficient Federated Low-Rank Adaptation with Heterogeneous Clients (2025.acl-long)
Copied to clipboard
| Challenge: | Existing methods for federated fine-tuning for Large Language Models suffer from performance degradation at low ranks in heterogeneous data settings. |
| Approach: | They propose a low-rank adaptive model with Alternating freeze and Adaptive rank selection which reduces the number of uploaded parameters by 99.8% . |
| Outcome: | The proposed low-rank Adaptation maintains robustness even under extreme heterogeneity and low rank conditions while preserving communication efficiency. |
Federated LoRA Fine-Tuning with Pipelined Error-Mitigated Aggregation and Matrix-Wise Freezing (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for fine-tuning large language models often suffer from biased model aggregation and are hindered by significant communication and computation burden. |
| Approach: | They propose a Federated low-rank adaptation system for large language models that leverages pipelined error-mitigated model aggregation and adaptive matrix-wise parameter freezing to mitigate aggregations. |
| Outcome: | The proposed system improves time-to-target by 2.17-8.48 on real-world datasets. |
EcoLoRA: Communication-Efficient Federated Fine-Tuning of Large Language Models (2025.emnlp-main)
Copied to clipboard
Han Liu, Ruoyao Wen, Srijith Nair, Jia Liu, Wenjing Lou, Chongjie Zhang, William Yeoh, Yevgeniy Vorobeychik, Ning Zhang
| Challenge: | Recurrent exchange of model updates in FL can result in prohibitively high communication costs, hindering the distributed learning process. |
| Approach: | They propose a federated fine-tuning framework that uses a round-robin segment sharing scheme to reduce network bandwidth and adaptive sparsification methods tailored to LoRA’s training dynamics. |
| Outcome: | The proposed framework reduces communication overhead without compromising performance on question-answering and value-alignment tasks. |
Communication-Efficient and Tensorized Federated Fine-Tuning of Large Language Models (2025.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) excel in translation and summarization due to the capabilities of transformer architectures. |
| Approach: | They propose to integrate tensorized adapters into model encoder/decoder blocks to improve model adaptability against data heterogeneity. |
| Outcome: | Experiments on large-scale cross-device FL and large-silo FL show that the proposed methods perform on par or even better than existing federated PEFT approaches while reducing communication cost. |
FedProxy: Federated Fine-Tuning of LLMs via Proxy SLMs and Heterogeneity-Aware Fusion (2026.acl-long)
Copied to clipboard
| Challenge: | Existing methods for fine-tuning Large Language Models (LLMs) suffer from a performance bottleneck . Existing approaches like Offsite-Tuning (OT) secure the LLMs IP . |
| Approach: | They propose a framework that replaces weak adapters with a unified, powerful Proxy Small Language Model (SLM) they propose 'resource-friendly' compression and 'robust optimization' to handle data heterogeneity. |
| Outcome: | Experiments show that FedProxy outperforms OT and centralized fine-tuning methods. |
G-LoRA: Global-Local Decoupled Low-Rank Adaptation (2026.findings-acl)
Copied to clipboard
| Challenge: | Low-Rank Adaptation (LoRA) improves the fine-tuning efficiency and performance of large language models. |
| Approach: | They propose a low-rank adaptive approach that decomposes update matrix into global and local adapters and assigns them to local and global adapters. |
| Outcome: | The proposed method achieves up to 2.7% accuracy improvement over LoRA and its variants on commonsense reasoning, mathematical reasoning, and code generation. |
Towards Federated Low-Rank Adaptation of Language Models with Rank Heterogeneity (2025.naacl-short)
Copied to clipboard
| Challenge: | Low-rank adaptation (LoRA) is an efficient alternative to full-weight adaptation in federated fine-tuning of language models, significantly reducing computational costs. |
| Approach: | They propose a low-rank adaptation method that freezes original weights and trains only the update parametrized as a product of two low-ranked matrices. |
| Outcome: | The proposed method accelerates convergence and enhances the global model’s predictive performance. |
pFedGPT: Hierarchically Optimizing LoRA Aggregation Weights for Personalized Federated GPT Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for fine-tuning Large Language Models (LLMs) struggle with data heterogeneity and adapt shared global knowledge to individual client needs. |
| Approach: | They propose a framework that leverages Hierarchical Bayesian Optimization (HBO) for fine-grained, personalized LoRA aggregation. |
| Outcome: | The proposed framework achieves state-of-the-art (SOTA) performance on personalized FL benchmarks while introducing only minimal (approx. 4%) additional optimization overhead. |
FedReFT: Federated Representation Fine-Tuning with All-But-Me Aggregation (2026.findings-eacl)
Copied to clipboard
| Challenge: | Representation Fine-Tuning (ReFT) adapts large pre-trained models by updating only a small subset of parameters. |
| Approach: | They propose a method that uses sparse intervention layers to steer hidden representations directly to capture rich semantic information. |
| Outcome: | The proposed approach outperforms PEFTs on commonsense reasoning, arithmetic reasoning, and GLUE benchmarks while maintaining a high parameter efficiency. |
CoLA: Collaborative Low-Rank Adaptation (2025.findings-acl)
Copied to clipboard
| Challenge: | The scaling law of Large Language Models (LLMs) reveals diminishing return on performance as model scale increases. |
| Approach: | They propose a more flexible LoRA architecture with an efficient initialization scheme . they propose combining three collaborative strategies to enhance performance . |
| Outcome: | The proposed model outperforms existing methods in low-sample scenarios. |