Gradient Descent

Gradient descent updates a parameter vector opposite the local gradient to reduce a differentiable objective function. The notation x_k+1=x_k−η∇f(x_k) shows which quantities are connected. Here, x_k: Parameter vector at iteration k (per parameter); η: Positive step size (x² per f); ∇f(x_k): Gradient of the objective (f per x). η has units x²/f so that η∇f has the same units as x; parameters are often scaled to be dimensionless. An application depends on whether the supplied quantities match the definitions and reference directions. Common uses include Machine learning, parameter estimation, and numerical optimization of large objectives.. Read the variables and their dimensions before substituting numbers; this often reveals whether the result should be a length, rate, probability, or dimensionless statistic.

Start with a diagram or a list of known quantities, assign each one a symbol, and convert units to a consistent form. Then choose the appropriate equation and substitute values only afterward. Example: For f(x)=½(x−3)², x₀=0, η=0.1: f′(0)=−3 and x₁=0−0.1·(−3)=0.3. Check: the step moves toward minimizer 3 and lowers f from 4.5 to 3.645. Verification is part of the calculation: For a strongly convex, L-smooth function, a suitably chosen constant η converges; steps that are too large can oscillate or diverge, while very small steps converge slowly Also compare the magnitude and unit with the context. Round only at the end, since early rounding can affect multi-step calculations, especially when small differences or sums are involved.

For a strongly convex, L-smooth function, a suitably chosen constant η converges; steps that are too large can oscillate or diverge, while very small steps converge slowly. Local minima, saddle points, and scaling affect practical optimization. Stopping criteria may use the gradient norm or change in objective value. These are not minor footnotes: when assumptions fail, the formula can produce a precise but inappropriate result. Before calculating, identify which values are observed, estimated, or modeled, and whether special cases change the equation. Machine learning, parameter estimation, and numerical optimization of large objectives. For numerical methods, also check step size and stability; for statistics, sampling design and distribution; for calculus, differentiability, boundary, and orientation. Use another method when these conditions do not hold.

Both relations belong to the same topic, but answer different questions. “Gradient Descent”: Gradient descent updates a parameter vector opposite the local gradient to reduce a differentiable objective function. The companion relation “Newton's Method for Optimization” Newton's optimization method corrects parameters using the inverse Hessian and gradient to find stationary points of a sufficiently smooth function. Choose based on the quantity sought and the model: Machine learning, parameter estimation, and numerical optimization of large objectives. The worked example For f(x)=½(x−3)², x₀=0, η=0.1: f′(0)=−3 and x₁=0−0.1·(−3)=0.3. Check: the step moves toward minimizer 3 and lowers f from 4.5 to 3.645. shows which quantities actually enter this relation. Compare both formulas using the same data and identify the companion’s extra assumption. This clarifies which relation fits.

A stationary point is not automatically a global minimum. Check curvature, step size and convergence; one successful step does not establish a global result. The symbols must also match their definitions exactly: x_k: Parameter vector at iteration k (per parameter); η: Positive step size (x² per f); ∇f(x_k): Gradient of the objective (f per x). Do not calculate mechanically; write the assumption beside the setup, then check dimensions and sign. Also note: For a strongly convex, L-smooth function, a suitably chosen constant η converges; steps that are too large can oscillate or diverge, while very small steps converge slowly The example For f(x)=½(x−3)², x₀=0, η=0.1: f′(0)=−3 and x₁=0−0.1·(−3)=0.3. Check: the step moves toward minimizer 3 and lowers f from 4.5 to 3.645. lets you follow these checks directly. If the assumptions do not fit, choose another relation from the chapter instead of forcing values into an unsuitable formula.