|
Line Search
In optimization, line search is a basic iterative approach to find a local minimum \mathbf^* of an objective function f:\mathbb R^n\to\mathbb R. It first finds a descent direction along which the objective function f will be reduced, and then computes a step size that determines how far \mathbf should move along that direction. The descent direction can be computed by various methods, such as gradient descent or quasi-Newton method. The step size can be determined either exactly or inexactly. One-dimensional line search Suppose ''f'' is a one-dimensional function, f:\mathbb R\to\mathbb R, and assume that it is unimodal, that is, contains exactly one local minimum ''x''* in a given interval 'a'',''z'' This means that ''f'' is strictly decreasing in ,x*and strictly increasing in *,''z'' There are several ways to find an (approximate) minimum point in this case. Zero-order methods Zero-order methods use only function evaluations (i.e., a value oracle) - not derivatives: * ... [...More Info...] [...Related Items...] OR: [Wikipedia] [Google] [Baidu] |
Optimization (mathematics)
Mathematical optimization (alternatively spelled ''optimisation'') or mathematical programming is the selection of a best element, with regard to some criteria, from some set of available alternatives. It is generally divided into two subfields: discrete optimization and continuous optimization. Optimization problems arise in all quantitative disciplines from computer science and engineering to operations research and economics, and the development of solution methods has been of interest in mathematics for centuries. In the more general approach, an optimization problem consists of maxima and minima, maximizing or minimizing a Function of a real variable, real function by systematically choosing Argument of a function, input values from within an allowed set and computing the Value (mathematics), value of the function. The generalization of optimization theory and techniques to other formulations constitutes a large area of applied mathematics. Optimization problems Opti ... [...More Info...] [...Related Items...] OR: [Wikipedia] [Google] [Baidu] |
Superlinear Convergence
In mathematical analysis, particularly numerical analysis, the rate of convergence and order of convergence of a sequence that converges to a limit are any of several characterizations of how quickly that sequence approaches its limit. These are broadly divided into rates and orders of convergence that describe how quickly a sequence further approaches its limit once it is already close to it, called asymptotic rates and orders of convergence, and those that describe how quickly sequences approach their limits from starting points that are not necessarily close to their limits, called non-asymptotic rates and orders of convergence. Asymptotic behavior is particularly useful for deciding when to stop a sequence of numerical computations, for instance once a target precision has been reached with an iterative root-finding algorithm, but pre-asymptotic behavior is often crucial for determining whether to begin a sequence of computations at all, since it may be impossible or imprac ... [...More Info...] [...Related Items...] OR: [Wikipedia] [Google] [Baidu] |
Pattern Search (optimization)
Pattern search (also known as direct search, derivative-free search, or black-box search) is a family of numerical optimization methods that does not require a gradient. As a result, it can be used on functions that are not continuous or differentiable. One such pattern search method is "convergence" (see below), which is based on the theory of positive bases. Optimization attempts to find the best match (the solution that has the lowest error value) in a multidimensional analysis space of possibilities. History The name "pattern search" was coined by Hooke and Jeeves. An early and simple variant is attributed to Fermi and Metropolis when they worked at the Los Alamos National Laboratory. It is described by Davidon, as follows: Convergence Convergence is a pattern search method proposed by Yu, who proved that it converges using the theory of positive bases.*Yu, Wen Ci. 1979. �Positive basis and a class of direct search techniques��. ''Scientia Sinica'' 'Zhongguo Kexue ... [...More Info...] [...Related Items...] OR: [Wikipedia] [Google] [Baidu] |
Learning Rate
In machine learning and statistics, the learning rate is a tuning parameter in an optimization algorithm that determines the step size at each iteration while moving toward a minimum of a loss function. Since it influences to what extent newly acquired information overrides old information, it metaphorically represents the speed at which a machine learning model "learns". In the adaptive control literature, the learning rate is commonly referred to as gain. In setting a learning rate, there is a trade-off between the rate of convergence and overshooting. While the descent direction is usually determined from the gradient of the loss function, the learning rate determines how big a step is taken in that direction. A too high learning rate will make the learning jump over minima but a too low learning rate will either take too long to converge or get stuck in an undesirable local minimum. In order to achieve faster convergence, prevent oscillations and getting stuck in undesi ... [...More Info...] [...Related Items...] OR: [Wikipedia] [Google] [Baidu] |
Grid Search
Grid, The Grid, or GRID may refer to: Space partitioning * Regular grid, a tessellation of space with translational symmetry, typically formed from parallelograms or higher-dimensional analogs ** Grid graph, a graph structure with nodes connected in a regular grid ** Square grid, a grid of squares ** Triangular grid, a grid of triangles ** Hexagonal grid, a grid of hexagons ** Unstructured grid, a tessellation of a space by simple shapes such as triangles or tetrahedra in an irregular pattern * Grid reference system, a coordinate system relative to a particular map projection * Grid (spatial index), a discretization of a geometric domain into a set of contiguous cells, used to organize information ** Discrete global grid (DGG), a grid that covers the entire Earth's surface * Grid (graphic design) (or typographic grid), organized lines for guiding graphic design * Grid plan, a city design with streets running at right angles * Grid paper, paper with a regular grid printe ... [...More Info...] [...Related Items...] OR: [Wikipedia] [Google] [Baidu] |
Trust Region
In mathematical optimization, a trust region is the subset of the region of the objective function that is approximated using a model function (often a quadratic). If an adequate model of the objective function is found within the trust region, then the region is expanded; conversely, if the approximation is poor, then the region is contracted. The fit is evaluated by comparing the ratio of expected improvement from the model approximation with the actual improvement observed in the objective function. Simple thresholding of the ratio is used as the criterion for expansion and contraction—a model function is "trusted" only in the region where it provides a reasonable approximation. Trust-region methods are in some sense dual to line-search methods: trust-region methods first choose a step size (the size of the trust region) and then a step direction, while line-search methods first choose a step direction and then a step size. The general idea behind trust region methods is kn ... [...More Info...] [...Related Items...] OR: [Wikipedia] [Google] [Baidu] |
Local Minimum
In mathematical analysis, the maximum and minimum of a function are, respectively, the greatest and least value taken by the function. Known generically as extremum, they may be defined either within a given range (the ''local'' or ''relative'' extrema) or on the entire domain (the ''global'' or ''absolute'' extrema) of a function. Pierre de Fermat was one of the first mathematicians to propose a general technique, adequality, for finding the maxima and minima of functions. As defined in set theory, the maximum and minimum of a set are the greatest and least elements in the set, respectively. Unbounded infinite sets, such as the set of real numbers, have no minimum or maximum. In statistics, the corresponding concept is the sample maximum and minimum. Definition A real-valued function ''f'' defined on a domain ''X'' has a global (or absolute) maximum point at ''x''∗, if for all ''x'' in ''X''. Similarly, the function has a global (or absolute) minimum point at ''x''∗ ... [...More Info...] [...Related Items...] OR: [Wikipedia] [Google] [Baidu] |
Simulated Annealing
Simulated annealing (SA) is a probabilistic technique for approximating the global optimum of a given function. Specifically, it is a metaheuristic to approximate global optimization in a large search space for an optimization problem. For large numbers of local optima, SA can find the global optimum. It is often used when the search space is discrete (for example the traveling salesman problem, the boolean satisfiability problem, protein structure prediction, and job-shop scheduling). For problems where finding an approximate global optimum is more important than finding a precise local optimum in a fixed amount of time, simulated annealing may be preferable to exact algorithms such as gradient descent or branch and bound. The name of the algorithm comes from annealing in metallurgy, a technique involving heating and controlled cooling of a material to alter its physical properties. Both are attributes of the material that depend on their thermodynamic free energy ... [...More Info...] [...Related Items...] OR: [Wikipedia] [Google] [Baidu] |
Wolfe Conditions
In the unconstrained minimization problem, the Wolfe conditions are a set of inequalities for performing inexact line search, especially in quasi-Newton methods, first published by Philip Wolfe in 1969. In these methods the idea is to find \min_x f(\mathbf) for some smooth f\colon\mathbb R^n\to\mathbb R. Each step often involves approximately solving the subproblem \min_ f(\mathbf_k + \alpha \mathbf_k) where \mathbf_k is the current best guess, \mathbf_k \in \mathbb R^n is a search direction, and \alpha \in \mathbb R is the step length. The inexact line searches provide an efficient way of computing an acceptable step length \alpha that reduces the objective function In mathematical optimization and decision theory, a loss function or cost function (sometimes also called an error function) is a function that maps an event or values of one or more variables onto a real number intuitively representing some "cost ... 'sufficiently', rather than minimizing the objective function ... [...More Info...] [...Related Items...] OR: [Wikipedia] [Google] [Baidu] |
Backtracking Line Search
In (unconstrained) mathematical optimization, a backtracking line search is a line search method to determine the amount to move along a given search direction. Its use requires that the objective function is differentiable and that its gradient is known. The method involves starting with a relatively large estimate of the step size for movement along the line search direction, and iteratively shrinking the step size (i.e., "backtracking") until a decrease of the objective function is observed that adequately corresponds to the amount of decrease that is expected, based on the step size and the local gradient of the objective function. A common stopping criterion is the Armijo–Goldstein condition. Backtracking line search is typically used for gradient descent (GD), but it can also be used in other contexts. For example, it can be used with Newton's method if the Hessian matrix is positive definite. Motivation Given a starting position \mathbf and a search direction \mathbf ... [...More Info...] [...Related Items...] OR: [Wikipedia] [Google] [Baidu] |
Conjugate Gradient Method
In mathematics, the conjugate gradient method is an algorithm for the numerical solution of particular systems of linear equations, namely those whose matrix is positive-semidefinite. The conjugate gradient method is often implemented as an iterative algorithm, applicable to sparse systems that are too large to be handled by a direct implementation or other direct methods such as the Cholesky decomposition. Large sparse systems often arise when numerically solving partial differential equations or optimization problems. The conjugate gradient method can also be used to solve unconstrained optimization problems such as energy minimization. It is commonly attributed to Magnus Hestenes and Eduard Stiefel, who programmed it on the Z4, and extensively researched it. The biconjugate gradient method provides a generalization to non-symmetric matrices. Various nonlinear conjugate gradient methods seek minima of nonlinear optimization problems. Description of the problem addres ... [...More Info...] [...Related Items...] OR: [Wikipedia] [Google] [Baidu] |
Regula Falsi
In mathematics, the ''regula falsi'', method of false position, or false position method is a very old method for solving an equation with one unknown; this method, in modified form, is still in use. In simple terms, the method is the trial and error technique of using test ("false") values for the variable and then adjusting the test value according to the outcome. This is sometimes also referred to as "guess and check". Versions of the method predate the advent of algebra and the use of equations. As an example, consider problem 26 in the Rhind papyrus, which asks for a solution of (written in modern notation) the equation . This is solved by false position. First, guess that to obtain, on the left, . This guess is a good choice since it produces an integer value. However, 4 is not the solution of the original equation, as it gives a value which is three times too small. To compensate, multiply (currently set to 4) by 3 and substitute again to get , verifying that the solution ... [...More Info...] [...Related Items...] OR: [Wikipedia] [Google] [Baidu] |
