In the realm of Large Language Models (LLMs), Knowledge Distillation (KD) plays a pivotal role. This article presents a comprehensive survey of the application of KD, focusing on algorithm, skill improvements, and their practical implications, including model compression and self-improvement:

• Highlights the critical function of KD in imparting advanced knowledge to smaller, more efficient models.
• KD helps in the compression of LLMs and enhancing their self-learning capabilities by using themselves as teachers.
• The article discusses the synergy between data augmentation and KD, enhancing the performance of LLMs.
• Authors propose future research directions, focusing on generating training data that helps models approximate human-like understanding.

This paper is essential for understanding how KD can be utilized to make LLMs more efficient and capable. It opens up possibilities for future research in generating skill-specific training that enhances models’ contextual capabilities.