Stochastic multilevel methods for deep learning

Beschreibung

This thesis introduces a multilevel stochastic gradient descent algorithm (MLSGD) to accelerate neural network training through multilevel techniques. The core contribution of this thesis is the development and analysis of MLSGD. As in traditional multilevel methods, prolongation and restriction operators enable transitions between levels. To ensure first-order coherence, a gradient correction is added to the objective function as well as additional conditions including step size regularization and an angle condition. We analyze the convergence properties of the method under the assumption of fixed step sizes. Additionally, we investigate the influence of stochastic directions in the gradient correction as a replacement for full gradients as well as the effect of variance reduction in both cases. Finally, we evaluate the practical performance of the method as well as the effect of stochastic gradient correction and variance reduction. To this end, MLSGD is applied to the image classification dataset CIFAR-10 and compared to stochastic gradient descent. We explore two different approaches to constructing a hierarchy, based either on network depth or image resolution. For both approaches, we construct suitable prolongation and restriction operators.

Autor*in

Isabel Jacob

Themen in »Stochastic multilevel methods for deep learning«

stochastic gradient descent multilevel optimization deep learning

Stimmen zu »Stochastic multilevel methods for deep learning«

Details

Link teilen