Bayesian Optimization

Bayesian Optimization represents a sophisticated approach to black-box optimization particularly suited for expensive-to-evaluate objective functions. Unlike methods that require many function evaluations, Bayesian optimization uses a probabilistic model (typically a Gaussian process) to approximate the objective function and guide the selection of the most promising points to evaluate next.

The method operates through a sequential strategy that balances exploration and exploitation. First, it builds a surrogate model of the objective function based on previous evaluations. This model captures both the estimated function value at any point and the uncertainty in that estimate. Then, it uses an acquisition function that combines information about predicted values and uncertainties to determine the next most informative point to evaluate.

Common acquisition functions include Expected Improvement (which balances the value of exploring uncertain regions against exploiting regions with high predicted performance) and Upper Confidence Bound (which explicitly manages the exploration-exploitation tradeoff through a tunable parameter).

This approach has become the method of choice for hyperparameter tuning in machine learning, where each evaluation might require training a neural network for hours or days. It's also valuable in experimental design, drug discovery, material science, and other domains where each function evaluation is time-consuming or expensive. By making intelligent decisions about which points to evaluate, Bayesian optimization can find high-quality solutions with remarkably few function evaluations—often 10-100 times fewer than required by other global optimization methods.