DLoRA: Distributed Parameter-Efficient Fine-Tuning Solution for Large Language Model (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models (LLMs) can handle tasks such as text generation, translation and summarization with remarkable proficiency. |
| Approach: | They propose a framework that enables scalable PEFT operations to be performed collaboratively between the cloud and user devices. |
| Outcome: | The proposed framework reduces computation and communication workload over user devices while achieving superior accuracy and privacy protection. |
Similar Papers
UORA: Uniform Orthogonal Reinitialization Adaptation in Parameter Efficient Fine-Tuning of Large Models (2025.acl-long)
Copied to clipboard
| Challenge: | Existing methods such as LoRA and VeRA use a low-rank approximation method that reduces the number of trainable parameters without compromising performance. |
| Approach: | They propose a parameter-efficient fine-tuning approach that leverages a low-rank approximation method that reduces the number of trainable parameters without compromising performance. |
| Outcome: | The proposed approach outperforms existing methods on GLUE and E2E benchmarks and is effective in instruction-tuning large language models and image classification models. |
PEFT-Factory: Unified Parameter-Efficient Fine-Tuning of Autoregressive Large Language Models (2026.eacl-demo)
Copied to clipboard
| Challenge: | Parameter-Efficient Fine-Tuning (PEFT) methods address the increasing size of Large Language Models (LLMs). |
| Approach: | They propose a framework for efficient fine-tuning Large Language Models (LLMs) they aim to train only a small percentage of the full model's parameters . |
| Outcome: | Xu et al., 2023; Ding e t al, 2024; Lialin e al. 2023) show that using PEFT methods can improve performance. |
PEFT-Bench: A Parameter-Efficient Fine-Tuning Methods Benchmark (2026.eacl-long)
Copied to clipboard
| Challenge: | Parameter-Efficient Fine-Tuning (PEFT) methods reduce the number of trainable parameters while maintaining strong downstream performance. |
| Approach: | They propose a unified benchmark for evaluating diverse PEFT methods on autoregressive LLMs. |
| Outcome: | The proposed methods reduce trainable parameters while maintaining strong downstream performance. |
Communication-Efficient and Tensorized Federated Fine-Tuning of Large Language Models (2025.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) excel in translation and summarization due to the capabilities of transformer architectures. |
| Approach: | They propose to integrate tensorized adapters into model encoder/decoder blocks to improve model adaptability against data heterogeneity. |
| Outcome: | Experiments on large-scale cross-device FL and large-silo FL show that the proposed methods perform on par or even better than existing federated PEFT approaches while reducing communication cost. |
VEEF-Multi-LLM: Effective Vocabulary Expansion and Parameter Efficient Finetuning Towards Multilingual Large Language Models (2025.coling-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have a significant disadvantage for low-resource languages . VEEF-Multi-LLM-8B excels in multilingual instruction-following tasks . |
| Approach: | They propose a low-resource multilingual large language model that expands the vocabulary for multilingual support. |
| Outcome: | The proposed model outperforms existing models in multilingual instruction-following tasks, but lags behind English-centric models in some tasks. |
From Bottom to Top: Extending the Potential of Parameter Efficient Fine-Tuning (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to fine-tune large language models primarily focus on the interaction between different layers, ignoring the fact that different layers store different information. |
| Approach: | They propose a Parameter Efficient Fine-Tuning method which freeze pre-trained parameters and fine-tunes only a few task-specific parameters. |
| Outcome: | The proposed methods reduce parameter count to nearly half by omitting fine-tuning in the middle layers. |
An Empirical Study on Parameter-Efficient Fine-Tuning for MultiModal Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Multimodal Large Language Models fine-tuned with multimodal instruction-following data have demonstrated formidable capabilities in multimodal tasks. |
| Approach: | They propose to employ four PEFT methods to fine-tune the LLM component of open-source MLLMs. |
| Outcome: | The proposed method is the best performing on seven datasets, while fine-tuning the connector layers leads to improved performance in most MLLMs. |
Flexora: Flexible Low-Rank Adaptation for Large Language Models (2025.acl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) have revolutionized artificial intelligence, but performance on specific tasks is limited by knowledge boundaries. |
| Approach: | They propose a method that automatically selects the most critical layers for fine-tuning to optimize performance across diverse downstream tasks. |
| Outcome: | The proposed method outperforms baseline models and natural language tasks. |
MAPLE: Multilingual Evaluation of Parameter Efficient Finetuning of Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Prior work on multilingual evaluation has shown that there is a large gap between the performance of Large Language Models on English and other languages. |
| Approach: | They propose to finetune Llama-2 and Mistral models on two datasets to determine their effect on model performance on six downstream tasks covering forty one languages. |
| Outcome: | The proposed model can improve on six multilingual tasks while degrading on high-resource languages. |
DaMoC: Efficiently Selecting the Optimal Large Language Model for Fine-tuning Domain Tasks Based on Data and Model Compression (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models excel in general tasks but struggle with domain-specific ones, requiring fine-tuning with specific data. |
| Approach: | They propose a Data and Model Compression Framework that categorizes data filtering methodologies into three distinct paradigms: (1) distribution-aware methods, (2) quality-a aware methods, and (3) hybrid approaches considering both dimensions. |
| Outcome: | The proposed framework can select the optimal LLM while saving approximately 20-fold in training time. |