(Deep) Learning Theory, PhD course in mathematics @ UNIPI (2023-2024)

Class times: Aula Riunioni, Mathematics Department, UNIPI, (and streamed online through the teams platform on the class teams channel)
Monday 10:30 - 12:30,
Thursday 14:00 - 16:00 (the THU lecture will approximately take place once every two weeks).

The class will start on MON, DEC 4th 2023 and end before the start of the Spring semester.
If these lecture times have conflicts with your classes, please contact the course instructor.

Class format: Lectures will be in hybrid format: they will be held live and also streamed through the class team channel

Please enroll in the teams channel of the class for announcements and up-to-date information.

Instructor: Andrea Agazzi,

E-mail: andrea.agazzi at unipi.it (please include "DLT" in email title).

Corse summary:
Deep learning has emerged as a powerful approach to solving complex problems in artificial intelligence, and understanding the underlying theory is crucial for practitioners and researchers alike. This postgraduate course offers an introduction to the theory behind deep learning, focusing specifically on mathematically rigorous results on the subject.
The course will start with an overview of the rudiments of statistical learning theory, such as loss functions, empirical risk minimization, kernel methods, generalization, and regularization. We will then thoroughly discuss the fundamentals of neural networks theory, covering topics such as architecture, activation functions, expressivity, approximation theorems, and training through (stochastic) gradient descent. The third part of the course will be devoted to some aspects of the optimization theory of neural networks. In particular, we will discuss the training dynamics of neural networks in the infinitely wide limit in two contrastive regimes: the neural tangent kernel regime and the mean-field regime. Depending on time and interest, other topics that might be covered are: (deep) reinforcement learning, generalization bounds for stochastic gradient descent and time-series learning with recurrent neural networks.

Class requirements: The class is open to advanced master students. While no previous knowledge of machine learning theory is expected, a solid background in analysis and probability will be necessary to reach an in-depth understanding of the topics treated in the course.

Textbooks and references (to be updated during the course):

• Learning Theory from First Principles F. Bach (available online),
• Deep learning theory lecture notes M. Telgarsky (available online),
• Six Lectures on Linearized Neural Networks A. Montanari (available online here),

Lecture notes:
04.12.2023 Lecture 1
07.12.2023 no class
11.12.2023 Lecture 2
14.12.2023 Lecture 3 and sandbox
18.12.2023 Lecture 4
WINTER BREAK
08.01.2024 no class
11.01.2024 Lecture 5
15.01.2024 Lecture 6
19.01.2024 at 10:30 Lecture 7
22.01.2024 no class
25.01.2024 no class
29.01.2024 Lecture 8
01.02.2024 Lecture 9
05.02.2024 no class
08.02.2024 Lecture 10
12.02.2024 Lecture 11
15.02.2024 Lecture 12
19.02.2024 Lecture 13
22.02.2024 (lecture starts at 15:00) Lecture 14


Papers for presentations:

• Neural Ordinary Differential Equations,
  
• Sampling Can Be Faster Than Optimization,
  
• The promises and pitfalls of Stochastic Gradient Langevin Dynamics,
  
• Limitations of Lazy Training of Two-layers Neural Networks,
  
• Implicit Bias of Gradient Descent for Wide Two-layer Neural Networks Trained with the Logistic Loss on Non-Convex Losses in High-Dimenstional Inference,
  
• A Mean-field Analysis of Deep ResNet and Beyond: Towards Provable Optimization Via Overparameterization From Depth,
  
• Kernel and Rich Regimes in Overparametrized Models,
  
• Global optimality of softmax policy gradient with single hidden layer neural networks in the mean-field regime ,
  
• Tight Bounds on the Smallest Eigenvalue of the Neural Tangent Kernel for Deep ReLU Networks,
  
• On Energy-Based Models with Overparametrized Shallow Neural Networks,
  
• STOCHASTIC GRADIENT DESCENT WITH NOISE OF MACHINE LEARNING TYPE PART II: CONTINUOUS TIME ANALYSIS,
  
• Quantitative Gaussian Approximation of Randomly Initialized Deep Neural Networks,
  
• Gradient flow dynamics of shallow ReLU networks for square loss and orthogonal inputs,
  
• Label noise (stochastic) gradient descent implicitly solves the Lasso for quadratic parametrisation,
  
• Learning sparse features can lead to overfitting in neural networks,
  
• On the relationship between multivariate splines and infinitely-wide neural networks,
  
• Convergence Rates for Momentum Stochastic Gradient Descent with Noise of Machine Learning Type,
  
• Stochastic Modified Flows, Mean-Field Limits and Dynamics of Stochastic Gradient Descent,
  
• Learning time-scales in two-layers neural networks,
  
• Neural (Tangent Kernel) Collapse,
  
• High-Dimensional SGD Aligns with Emerging Outlier Eigenspaces,
  
• A Mathematical Theory on Transformers,