Four Sentences Summary:

• The growing scale of LLMs imposes memory constraints, especially for on-device training applications.
• ZO optimization techniques are proposed as a solution to reduce memory consumption without the need for back-propagation.
• A comprehensive benchmarking study is conducted across multiple LLM families and fine-tuning schemes.
• Results highlight the effectiveness of ZO techniques, including block-wise descent and hybrid training strategies.

Bullet Points:

• Emphasizes the shift toward BP-free optimization methods.
• Introduces enhancements like gradient sparsity and hybrid training.
• Unveils task alignment’s role in optimizing performance.
• Explores a wide array of ZO techniques beyond traditional ZO-SGD.
• Provides open-source codes for reproducible experiments.

Personal Insight:

• This paper is significant as it addresses the critical barrier of memory efficiency in deploying LLMs, particularly on resource-constrained devices. The insights could guide further research into algorithmically simpler yet effective LLM training strategies.