Fundamental concepts

Machine learning from a theoretical standpoint is still a relatively new domain, and there’s a lot of theory missing, which would explain why certain fundamental concepts work in some cases and not in others. As of now, machine learning and data science are more practical and experimental sciences, similar to experimental physics, rather than state-and-prove pure mathematics.

Nevertheless, I firmly believe that we’re going to see a more mathematical approach to neural networks, which would help clarify fundamental concepts. Such a theory would likely be within probability theory with elements of dynamical systems (training dynamics), representation theory (feature engineering, representation characteristics), and probably much more. As such, it would find a considerable appeal among mathematicians and theoretical computer scientists.

Currently, we don’t even know how to answer the basic questions, like:

• how to select models...