Course: Mathematical Foundation for Deep Learning

Syllabus: Syllabus_7970.pdf

Instructors: Yanzhao Cao, Yimin Zhong

Where: Parker Hall 246

When: TTh 3:30 - 4:45pm

About: The course focuses solely on the mathematical aspects of deep learning, including classical approximation theory, convergence analysis, and some related topics.

Prerequisites: solid vector calculus, linear algebra, ODE. Preferred: real analysis, probability, PDE.

Homework: There will be some homework, including some coding.

Presentation: TBA

Exam: No exams.

Reference books:

$LaTeX: \oplus$ Mathematical Foundations of Deep Learning Models and Algorithms, by Konstantinos Spiliopoulos, Richard B. Sowers, Justin Sirignano, 2025 (https://mathdl.github.io/)

$LaTeX: \oplus$ Learning Theory from First Principles, by Francis Bach, 2024 (https://www.di.ens.fr/~fbach/ltfp/)

Some Reference Papers/Reports (Covered in this course):

List of Papers
	Paper/Report	URL	About
1 (covered)	DeVore, Ronald A., Ralph Howard, and Charles Micchelli. "Optimal nonlinear approximation." Manuscripta mathematica 63.4 (1989): 469-478.	Optimal-Nonlinear-Approximation(DeVore,Howard,Micchelli,1989).pdf	general theory for stable approximation
2(suggest reading)	Yarotsky, Dmitry. "Optimal approximation of continuous functions by very deep ReLU networks." Conference on learning theory. PMLR, 2018.	Optimal-Approximation-Deep-ReLU.pdf	approximation of deep ReLU network
3 (suggest reading)	Morris, Sidney. "Hilbert 13: Are there any genuine continuous multivariate real-valued functions?." Bulletin of the American Mathematical Society 58.1 (2021): 107-118.	Are_There_Any_Genuine_Cont_Mult_Real_Val_Func.pdf	Kolmogorov-Arnold Superposition Theorem
4	Bertsekas, Dimitri P., and John N. Tsitsiklis. "Gradient convergence in gradient methods with errors." SIAM Journal on Optimization 10.3 (2000): 627-642.	Grad_Conv_In_Gradient_Met_W_Err.pdf	Convergence of gradient descent to saddle points
5 (covered)	Barron, Andrew R. "Universal Approximation Bounds for Superpositions of a Sigmoidal Function." IEEE TRANSACTIONS ON INFORMATION THEORY 39.3 (1993).	Universal_Approximation_Bounds_For_Superpositions_Of_A_Sigmoidal_Function.pdf	Barron Space approximation
6 (covered)	Cybenko, George. "Approximation by superpositions of a sigmoidal function." Mathematics of control, signals and systems 2.4 (1989): 303-314.	Approx_Sigmoidal.pdf	Uniform approximation by sigmoidal function
7	Polyak, Boris, and Pavel Shcherbakov. "Lyapunov functions: An optimization theory perspective." IFAC-PapersOnLine 50.1 (2017): 7456-7461.	Lyapunov_Func_Opt_Theory_Persp.pdf	Tutorial on Lyapunov Functions with application to heavy ball method.
8 (HW2)	Jones, Lee K. "A simple lemma on greedy approximation in Hilbert space and convergence rates for projection pursuit regression and neural network training." The annals of Statistics (1992): 608-613.	Jones-SimpleLemmaGreedy-1992.pdf	A greedy algorithm for approximation in Barron-like space.
9 (covered)	Maiorov, Vitaly E. "On best approximation by ridge functions." Journal of Approximation Theory 99.1 (1999): 68-94.	Best_Approx_Ridge.pdf	Approximation by Ridge functions.