FedReFT: Federated Representation Fine-Tuning with All-But-Me Aggregation (2026.findings-eacl)
Copied to clipboard
| Challenge: | Representation Fine-Tuning (ReFT) adapts large pre-trained models by updating only a small subset of parameters. |
| Approach: | They propose a method that uses sparse intervention layers to steer hidden representations directly to capture rich semantic information. |
| Outcome: | The proposed approach outperforms PEFTs on commonsense reasoning, arithmetic reasoning, and GLUE benchmarks while maintaining a high parameter efficiency. |
Similar Papers
Communication-Efficient and Tensorized Federated Fine-Tuning of Large Language Models (2025.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) excel in translation and summarization due to the capabilities of transformer architectures. |
| Approach: | They propose to integrate tensorized adapters into model encoder/decoder blocks to improve model adaptability against data heterogeneity. |
| Outcome: | Experiments on large-scale cross-device FL and large-silo FL show that the proposed methods perform on par or even better than existing federated PEFT approaches while reducing communication cost. |
FedProxy: Federated Fine-Tuning of LLMs via Proxy SLMs and Heterogeneity-Aware Fusion (2026.acl-long)
Copied to clipboard
| Challenge: | Existing methods for fine-tuning Large Language Models (LLMs) suffer from a performance bottleneck . Existing approaches like Offsite-Tuning (OT) secure the LLMs IP . |
| Approach: | They propose a framework that replaces weak adapters with a unified, powerful Proxy Small Language Model (SLM) they propose 'resource-friendly' compression and 'robust optimization' to handle data heterogeneity. |
| Outcome: | Experiments show that FedProxy outperforms OT and centralized fine-tuning methods. |
FedPETuning: When Federated Learning Meets the Parameter-Efficient Tuning Methods of Pre-trained Language Models (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing research on federated learning (FL) for pre-trained language models (PLMs) with increasing concerns about data privacy, enterprises or institutions are not allowed to collect data from end devices or local clients to a centralized server for fine-tuning PLMs. |
| Approach: | They investigate the parameter-efficient tuning of pre-trained language models (PLMs) and develop a federated benchmark for four representative PETuning methods . |
| Outcome: | The proposed method can defend against privacy attacks and maintain acceptable performance with reducing heavy resource consumption. |
FedEx-LoRA: Exact Aggregation for Federated and Efficient Fine-Tuning of Large Language Models (2025.acl-long)
Copied to clipboard
| Challenge: | Existing methods for low-rank averaging of LoRA adapters result in inexact updates. |
| Approach: | They propose a method which adds a residual error term to the pre-trained frozen weight matrix to achieve exact updates with minimal computational and communication overhead. |
| Outcome: | The proposed method achieves exact updates with minimal computational and communication overhead, preserving LoRA’s efficiency. |
Punctuation-Steered Representation Fine-Tuning (2026.acl-short)
Copied to clipboard
| Challenge: | Existing methods for parameter-efficient fine-tuning (PeFT) are limited due to their prohibitive size and computational demands. |
| Approach: | They propose a method that fine-tunes punctuation representations to achieve performance improvements. |
| Outcome: | The proposed method improves performance by altering the representation space alone . but it results in suboptimal performance due to the effects of the method on the output . |
RB-LoRA: Rank-Balanced Aggregation for Low-Rank Adaptation with Federated Fine-Tuning (2026.findings-eacl)
Copied to clipboard
| Challenge: | Low-rank adaptation (LoRA) improves fine-tuning of foundation models by updating only compact adapter matrices . varying client device capabilities lead to different adapter ranks, causing rank heterogeneity that undermines aggregation. |
| Approach: | They propose a rank-balanced aggregation framework that decomposes each update into rank-wise components and aligns them using analytically derived weights. |
| Outcome: | Experiments on language and vision models show that RB-LoRA improves under one and three rounds of communication in federated learning environments. |
PEFT-Bench: A Parameter-Efficient Fine-Tuning Methods Benchmark (2026.eacl-long)
Copied to clipboard
| Challenge: | Parameter-Efficient Fine-Tuning (PEFT) methods reduce the number of trainable parameters while maintaining strong downstream performance. |
| Approach: | They propose a unified benchmark for evaluating diverse PEFT methods on autoregressive LLMs. |
| Outcome: | The proposed methods reduce trainable parameters while maintaining strong downstream performance. |
Parameter-Efficient Fine-Tuning without Introducing New Latency (2023.acl-long)
Copied to clipboard
| Challenge: | Parameter-efficient fine-tuning of pre-trained language models has been demonstrated to be effective, but its inherent characteristics limit its performance. |
| Approach: | They propose to generate a sparse mask in a task-agnostic manner by modifying only a small subset of existing parameters and adding new parameters. |
| Outcome: | The proposed method surpasses existing methods on the GLUE benchmark by a significant margin. |
GMFL: Efficient Global Masking for Federated LLM Fine-tuning (2026.acl-long)
Copied to clipboard
| Challenge: | Low-Rank Adaptation (LoRA) has emerged as a prominent solution to mitigate the communication and computation costs in federated fine-tuning of Large Language Models (LLMs). |
| Approach: | They propose a plug-and-play layer freezing mechanism to integrate with existing federated fine-tuning frameworks. |
| Outcome: | The proposed solution reduces communication overhead and lowers computational costs while preserving the performance of the underlying federated fine-tuning methods. |
PEFT-Factory: Unified Parameter-Efficient Fine-Tuning of Autoregressive Large Language Models (2026.eacl-demo)
Copied to clipboard
| Challenge: | Parameter-Efficient Fine-Tuning (PEFT) methods address the increasing size of Large Language Models (LLMs). |
| Approach: | They propose a framework for efficient fine-tuning Large Language Models (LLMs) they aim to train only a small percentage of the full model's parameters . |
| Outcome: | Xu et al., 2023; Ding e t al, 2024; Lialin e al. 2023) show that using PEFT methods can improve performance. |