Optimization method AMSGrad in multilayer neural networks

Abstract

The most common method for optimizing neural networks is the gradient descent method. Gradient descent is an optimization algorithm that tracks the negative gradient of the objective function to find the minimum of the error function.

The limitation of gradient descent is that it applies a single learning rate to all input variables. Extensions of gradient descent, such as the Adaptive Moment Estimation (Adam) algorithm, use different learning rates for each input variable, but this can result in the learning rate quickly decreasing to very small values.

The AMSGrad method is an enhanced version of the Adam method, which aims to improve the convergence properties of the algorithm by avoiding large abrupt changes in the learning rate for each input variable. Technically, gradient descent is called a first-order optimization algorithm because it explicitly uses the first-order derivative of the objective function.