In this pioneering research, the authors re-examine the efficacy of zeroth-order optimization in the context of fine-tuning Large Language Models with an eye towards minimal memory consumption. By testing a variety of ZO algorithms across LLM families like Roberta, OPT, and others, they surface key principles for optimizing task performance with reduced memory overhead:

• The research assesses different ZO strategies, contrasting them with first-order methods that are more traditional but memory-intensive.
• A focus on task-specific alignment and advanced optimization techniques serves as a harbinger for more refined and efficient LLM usage.
• New approaches such as block-wise descent and hybrid training, and a focus on gradient sparsity, suggest pathways to finetuning without the heavy memory needs typically required.
• The findings point to the potential for more sophisticated on-device applications of LLMs.

The importance of this research lies in its potential to democratize advanced AI capabilities, making them accessible for devices with limited memory resources. The availability of the research code heightens its impact, fostering further innovation in the field.