AdaZeta: Adaptive Zeroth-Order Tensor-Train Adaption for Memory-Efficient Large Language Models Fine-Tuning (2024.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in memory-efficient zeroth-order methods have limited their widespread adoption due to performance drops and a high risk of divergence. |
| Approach: | They propose a memory-efficient zeroth-order framework to improve performance and convergence of the MeZO methods by using only forward passes. |
| Outcome: | The proposed framework improves performance and convergence of the proposed methods on Roberta-Large and Llama-2-7B models. |
Similar Papers
MUZO: Leveraging Multiple Queries and Momentum for Zeroth-Order Fine-Tuning of Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for fine-tuning large language models incur memory overhead due to the need for activation storage for back-propagation (BP). |
| Approach: | They propose a method that estimates gradients through finite differences without activation storage for back-propagation. |
| Outcome: | The proposed method demonstrates superior performance in fine-tuning various LLMs. |
RoZO: Geometry-Aware Zeroth-Order Fine-Tuning on Low-Rank Adapters for Black-Box Large Language Models (2026.eacl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) have demonstrated exceptional performance across a wide range of tasks, yet fine-tuning them efficiently under black-box or memory-constrained settings remains challenging. |
| Approach: | They propose a Riemannian zeroth-order optimization framework that constrains updates to the tangent space of the LoRA manifold. |
| Outcome: | The proposed framework achieves more stable convergence, tighter variance bounds, and superior performance compared to existing ZO methods. |
Global Adaptive Momentum Meets Local Personalized Perturbation: Efficient Federated LLM Fine-Tuning with Zeroth-Order Gradients (2026.acl-long)
Copied to clipboard
| Challenge: | federated fine-tuning of large language models provides privacy-preserving approach to deploying pervasive generative AI services. |
| Approach: | They propose a federated framework for fine-tuning large language models . they propose unified optimization and local personalized perturbation for ZO gradients . |
| Outcome: | The proposed framework outperforms existing methods for integrating ZO gradients in federated learning over diverse heterogeneous data settings. |
Universally Empowering Zeroth-Order Optimization via Adaptive Layer-wise Sampling (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for fine-tuning Large Language Models are slow and lack of performance. |
| Approach: | They propose a Zeroth-Order optimization framework that uses forward passes to fine-tune Large Language Models. |
| Outcome: | The proposed framework achieves 1.7 to 3.0 wall-clock acceleration on LLaMA and OPT models. |
Hi-ZFO: Hierarchical Zeroth- and First-Order LLM Fine-Tuning via Importance-Guided Tensor Selection (2026.findings-acl)
Copied to clipboard
| Challenge: | generative tasks require a high degree of exploratory capacity, but zeroth-order methods suffer from slow convergence . generative task-specific methods tend to converge toward local minima, causing noise and inefficient estimation . |
| Approach: | They propose a framework that synergizes FO precision with exploratory capability of ZO estimation. |
| Outcome: | The proposed framework synergizes precision of FO gradients with exploratory capability of ZO estimation. |
HELENE: Hessian Layer-wise Clipping and Gradient Annealing for Accelerating Fine-tuning LLM with Zeroth-order Optimization (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) face memory challenges due to the high cost of backpropagation. |
| Approach: | They propose a zeroth-order (ZO) optimization that matches memory usage to inference . they propose scalable and memory-efficient zeroth order (ZE) optimizer that integrates annealed A-GNB gradients with diagonal Hessian estimation and layer-wise clipping as a second-order pre-conditioner. |
| Outcome: | The proposed algorithm outperforms state-of-the-art methods with an average speedup of 20 over MeZO on RoBERTa-large and OPT-1.3B. |
QuZO: Quantized Zeroth-Order Fine-Tuning for Large Language Models (2025.emnlp-main)
Copied to clipboard
Jiajun Zhou, Yifan Yang, Kai Zhen, Ziyue Liu, Yequan Zhao, Ershad Banijamali, Athanasios Mouchtaris, Ngai Wong, Zheng Zhang
| Challenge: | Large Language Models (LLMs) are quantized to lower precision to reduce memory cost and latency in inference. |
| Approach: | They propose a quantized zeroth-order framework for fine-tuning Large Language Models (LLMs) using low-precision forward passes. |
| Outcome: | The proposed method achieves comparable results to first-order methods in FP8 and superior accuracy in INT8 and INT4 training. |
MobiZO: Enabling Efficient LLM Fine-Tuning at the Edge via Inference Engines (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are currently pre-trained and fine-tuned on large cloud servers . fine-timing on resource-constrained edge devices presents significant memory and computational demands . |
| Approach: | They propose a resource-efficient fine-tuning framework for LLMs specifically designed for edge devices. |
| Outcome: | Experiments show that MobiZO achieves substantial runtime speedups and memory savings while improving fine-tuning accuracy. |
LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Models (2023.emnlp-main)
Copied to clipboard
Zhiqiang Hu, Lei Wang, Yihuai Lan, Wanyu Xu, Ee-Peng Lim, Lidong Bing, Xing Xu, Soujanya Poria, Roy Lee
| Challenge: | Large language models (LLMs) have shown unprecedented performance across various tasks. |
| Approach: | They propose an easy-to-use framework that integrates adapters into LLMs . they evaluate adapters on 14 datasets from two different reasoning tasks . |
| Outcome: | The proposed framework can be used to fine-tune open-access language models with task-specific data and instruction data. |
LoRETTA: Low-Rank Economic Tensor-Train Adaptation for Ultra-Low-Parameter Fine-Tuning of Large Language Models (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing methods for parameter-efficient fine-tuning are limited by the growing number of trainable parameters with the rapid deployment of Large Language Models (LLMs). |
| Approach: | They propose a parameter-efficient framework that reduces trainable parameters through tensor-train decomposition. |
| Outcome: | The proposed methods achieve comparable or better performance than most widely used methods with up to 100 fewer parameters on the LLaMA-2-7B models. |