PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Models

As LLMs expand, their extensive parameters render traditional fine-tuning prohibitively costly. PiSSA, a groundbreaking PEFT method, optimizes a significantly smaller parameter space to maintain or even surpass full-parameter fine-tuning outcomes. Employing SVD to factorize a matrix within the model, PiSSA utilizes principal singular values and vectors to initialize the trainable matrices A and B, while maintaining the residual matrix frozen during fine-tuning.

PiSSA’s key benefits are:

• Focused Optimizations: Adapting the most crucial components of the matrix while keeping noisy parts unchanged.
• Rapid Convergence: Accelerating the fine-tuning process with better performance end results.

While PiSSA shares architecture similarities with LoRA, its unique initialization method provides a distinct advantage. Learn more about PiSSA’s effect on LLMs. PiSSA is a testament to the ongoing innovation in efficient fine-tuning methods, offering a path forward for training increasingly larger LLMs with both parameter and computation efficiency.