Knowledge Labs
Logo
Introduction to Machine LearningDeep Learning IntroductionProbability for Machine LearningStatistics for Machine LearningNatural Language Processing (NLP)Data Science IntroductionAI for EveryoneRetrieval Augmented Generation (RAG)AI for ArtistsComprehensive Guide to Model Context ProtocolComprehensive Guide to Economy & Financial SystemsComprehensive Guide to Babylon.js Game Engine
Probability for Machine Learning
/Information & Computational Methods
/Information Theory
/Entropy

Entropy

.

Entropy

Entropy represents the average unpredictability or uncertainty in a random variable. Intuitively, it measures how 'surprising' outcomes are on average—a high-entropy system is highly unpredictable, while a low-entropy system is more ordered and predictable.

For a discrete random variable X with possible values {x₁, x₂, ..., xₙ} and probability mass function P(X), the entropy H(X) is defined as:

H(X) = -∑ P(xᵢ) log₂ P(xᵢ)

The logarithm base determines the units—base 2 gives entropy in bits, while natural logarithm (base e) gives entropy in nats. This formula captures several intuitive properties:

  • Events with probability 1 (certainty) contribute zero entropy
  • Maximum entropy occurs with uniform distributions (maximum uncertainty)
  • Entropy is always non-negative

Entropy provides the foundation for information theory, connecting directly to information content by quantifying the average number of bits needed to encode messages from a given source. This relationship makes entropy essential for data compression, communication systems, and machine learning algorithms that must identify patterns amid noise.