Knowledge Labs
Logo
Introduction to Machine LearningDeep Learning IntroductionProbability for Machine LearningStatistics for Machine LearningNatural Language Processing (NLP)Data Science IntroductionAI for EveryoneRetrieval Augmented Generation (RAG)AI for ArtistsComprehensive Guide to Model Context ProtocolComprehensive Guide to Economy & Financial SystemsComprehensive Guide to Babylon.js Game Engine
Deep Learning Introduction
/Advanced Learning Paradigms
/Knowledge Distillation

Knowledge Distillation

.

Knowledge Distillation

Knowledge distillation is a model compression technique where a smaller "student" model is trained to mimic the behavior of a larger "teacher" model. The teacher’s soft targets (probability distributions) provide richer information than hard labels, enabling the student to achieve comparable performance with fewer parameters and less computation.

Imagine a master chef teaching an apprentice. Rather than having the apprentice go through all experiments, the master shares refined techniques and shortcuts so that the apprentice achieves similar results without all the background knowledge.