The mathematical core of deep learning is function approximation by neural networks trained on data using stochastic gradient descent.

I will present a collection of results on training dynamics for the deep linear network (DLN). The DLN is a phenomenological model of deep learning for linear functions that was introduced by computer scientists. It allows a comprehensive analysis that reveals interesting ties with several areas of mathematics and several lessons for 'true' deep learning.

This is joint work with several co-authors: Nadav Cohen (Tel Aviv), Kathryn Lindsey (Boston College), Alan Chen, Zsolt Veraszto and Tianmin Yu (Brown).