Kullback-Leibler Divergence (KL Divergence)

Kullback-Leibler Divergence (or relative entropy) measures the information gained when updating beliefs from distribution Q to distribution P:

D_KL(P||Q) = ∑ P(x) log(P(x)/Q(x))

KL divergence is always non-negative and equals zero only when P=Q. Importantly, it is asymmetric: D_KL(P||Q) ≠ D_KL(Q||P), making it not a true distance metric but rather a directed measure of dissimilarity.

Applications in Machine Learning:

• Variational Inference: Objective function measuring how closely approximate posterior matches true posterior
• Generative Models: Regularization term in VAEs ensuring learned latent space follows desired distribution
• Reinforcement Learning: Constraining policy updates in algorithms like PPO and TRPO
• Distribution Shift Detection: Identifying when test data diverges from training distribution