SPASS - Probability, stochastic analysis and statistics in Pisa

Mercoledi 25 Maggio

Dipartimento di Matematica, Sala Seminari

• 14:00 Enrico Malatesta (Bocconi): Phase transitions in the landscape of solutions of overparametrized neural networks.

Current deep neural networks are nonlinear devices composed of a number of parameters that far exceed the number of data points. Understanding how these systems can fit the data almost perfectly through variants of gradient descent algorithms and achieve exceptional levels of prediction accuracy without overfitting are key conceptual challenges. In this talk I will show how common techniques used in machine learning (e.g. the choice of the activation function or the loss) deeply affect the loss landscape, tending to mild its roughness. Then we shed light on the role of overparameterization in non-convex neural networks. By analytically studying a non-convex model of random features, we identify a novel (non-equilibrium) phase transition, that we call “Local Entropy” transition, controlled by the degree of overparameterization. In non-convex models this transition is strictly different to the SAT/UNSAT threshold and it coincides with the appearance of highly entropic minima of the error loss function. Those minima, in turn, are found to be highly attractive to the learning algorithms currently used in deep learning.