Overview of Neural Networks

Greetings to you, fellow sentient being; welcome to our exciting journey. The journey itself is to understand the concepts and inner workings behind an elusively powerful computing paradigm: the artificial neural network (ANN). While this notion has been around for almost half a century, the ideas accredited to its birth (such as what an agent is, or how an agent may learn from its surroundings), date back to Aristotelian times, and perhaps even to the dawn of civilization itself. Unfortunately, people in the time of Aristotle were not blessed with the ubiquity of big data, or the speeds of Graphical Processing Unit (GPU)-accelerated and massively parallelized computing, which today open up some very promising avenues for us. We now live in an era where the majority of our species has access to the building blocks and tools required to assemble artificially...

Defining our goal

Essentially, our task here is to conceive a mechanism that is capable of dealing with any data that it is introduced to. In doing so, we want this mechanism to detect any underlying patterns present in our data, in order to leverage it for our own benefit. Succeeding at this task means that we will be able to translate any form of raw data into knowledge, in the form of actionable business insights, burden-alleviating services, or life-saving medicines. Hence, what we actually want is to construct a mechanism that is capable of universally approximating any possible function that could represent our data; the elixir of knowledge, if you will. Do step back and imagine such a world for a moment; a world where the deadliest diseases may be cured in minutes. A world where all are fed, and all may choose to pursue the pinnacle of human achievement in any discipline...

Knowing our tools

We will mainly be working with the two most popular deep learning frameworks that exist, and are freely available to the public at large. This does not mean that we will completely limit our implementations and exercises to these two platforms. It may well occur that we experiment with other prominent deep learning frameworks and backends. We will, however, try to use either TensorFlow or Keras, due to their widespread popularity, large support community, and flexibility in interfacing with other prominent backend and frontend frameworks (such as Theano, Caffe, or Node.js, respectively). We will now provide a little background information on Keras and TensorFlow:

Keras

Many have named Keras the lingua franca...

The fundamentals of neural learning

We begin our journey with an attempt to gain a fundamental understanding of the concept of learning. Moreover, what we are really interested in is how such a rich and complex phenomenon as learning has been implemented on what many call the most advanced computer known to humankind. As we will observe, scientists seem to continuously find inspiration from the inner workings of our own biological neural networks. If nature has indeed figured out a way to leverage loosely connected signals from the outside world and patch them together as a continuous flow of responsive and adaptive awareness (something most humans will concur with), we would indeed like to know exactly what tricks and treats it may have used to do so. Yet, before we can move on to such topics, we must establish a baseline to understand why the notion of neural networks are far...

The fundamentals of data science

Let's get acquainted with some basic terminologies and concepts of data science. We will get into some theory and then move on to understand some complex terms such as entropy and dimensionality.

Information theory

Before a deeper dive into various network architectures and some hands-on examples, it would be a pity if we did not elaborate a little on the pivotal notion of gaining information through processing real-world signals. We speak of the science of quantifying the amount of information present in a signal, also referred to as information theory. While we don't wish to provide a deep mathematical overview on this notion, it is useful to know some background on learning from...

Summary

In this chapter, we gained a functional overview of biological neural networks, with a small and brief preview covering concepts such as neural learning and distributed representations. We also refreshed our memory on some classic data science dilemmas that are equally relevant for neural networks as they are for other ML techniques. In the following chapter, we will finally dive into the much-anticipated learning mechanism loosely inspired by our biological neural networks, as we explore the basic architecture of an ANN. We amicably describe ANNs in such a manner because, despite aiming to work as effectively as their biological counterparts, they are not quite there yet. In the next chapter, you will go over the main implementation considerations involved in designing ANNs and progressively discover the complexity that such an endeavour entails.

...

Further reading

• Symbolic versus connectionist learning: http://www.cogsci.rpi.edu/~rsun/sun.encyc01.pdf
• History of artificial intelligence: http://sitn.hms.harvard.edu/flash/2017/history-artificial-intelligence/
• History of the human brain: http://www.mybrain.co.uk/public/learn_history4.php