Nelder-Mead Method

The Nelder-Mead method (also known as the simplex method) represents one of the most widely used direct search techniques for multidimensional unconstrained optimization without derivatives. Unlike population-based methods, it maintains just a single geometric figure—a simplex with n+1 vertices in n-dimensional space—and evolves this shape to explore the objective function landscape.

The algorithm iteratively transforms the simplex through a series of geometric operations—reflection, expansion, contraction, and shrinking—based on function evaluations at the vertices. These operations adaptively reshape and move the simplex to follow the landscape's contours, generally flowing toward better solutions while adjusting its shape to match the local geometry of the function being optimized.

This elegant approach makes remarkably efficient use of function evaluations, typically requiring far fewer calls to the objective function than many other gradient-free methods. Its adaptive behavior allows it to handle varying scales and correlations between different dimensions, naturally stretching along promising directions and contracting in others.

Despite its age (developed in the 1960s), the Nelder-Mead method remains a workhorse optimization technique, particularly well-suited for problems with up to 10-20 variables where function evaluations are expensive. It excels at finding local optima of non-differentiable functions and is widely implemented in scientific computing environments due to its reliability and relative simplicity.