Enables high-performance Zeroth-Order (ZO) fine-tuning of LLMs by leveraging online curvature signals.
March 24, 2026
Original Paper
CurvZO: Adaptive Curvature-Guided Sparse Zeroth-Order Optimization for Efficient LLM Fine-Tuning
arXiv · 2603.21725
The Takeaway
ZO optimization is memory-efficient but usually slow; this paper introduces a way to use scalar feedback to estimate curvature and prune parameter updates. It achieves up to 2x speedups and significant accuracy gains, making LLM fine-tuning much more viable on consumer hardware.
From the abstract
Fine-tuning large language models (LLMs) with backpropagation achieves high performance but incurs substantial memory overhead, limiting scalability on resource-constrained hardware. Zeroth-order (ZO) optimization provides a memory-efficient alternative by relying solely on forward passes, yet it typically suffers from slow or unstable convergence due to high-variance gradient estimates. Sparse ZO updates partially address this issue by perturbing only a subset of parameters, but their effective