Papers by Huanyi Xie
High-Throughput and Memory-Efficient Zeroth-Order Fine-tuning LLMs with Distributed Parallel Computing (2026.findings-acl)
Copied to clipboard
| Challenge: | Zeroth-order (ZO) optimization is a memory-efficient alternative to fine-tuning large language models (LLMs). |
| Approach: | They propose a zeroth-order (ZO) optimization framework that offloads model parameters to CPU memory and overlapping transformer block transfer with dual forward computation on a single GPU. |
| Outcome: | The proposed framework achieves a 3x speedup over ZO2 on an OPT-175B model while maintaining memory efficiency and improving training throughput. |