As in the previous chapter, bear in mind that a real-life project goes through a three-dimensional method in some form or other. First, it's important to think and talk about the problem in need of solving without jumping onto a laptop. Once that is done, bear in mind that the foundation of machine learning and deep learning relies on mathematics. Finally, once the problem has been discussed and mathematically represented, it is time to develop the solution.

Designing datasets

The reinforcement learning program described in the first chapter can solve a variety of problems involving unlabeled classification in an unsupervised decision-making process. The Q function can be applied to drone, truck, or car deliveries. It can also be applied to decision making in games or real life.

However, in a real-life case study problem (such as defining the reward matrix in a warehouse for the AGV, for example), the difficulty will be to produce an efficient matrix using the proper features.

For example, an AGV requires information coming from different sources: daily forecasts and real-time warehouse flows.

The warehouse manages thousands of locations and hundreds of thousands of inputs and outputs. Trying to fit too many features into the model would be counterproductive. Removing both features and worthless data requires careful consideration.

A simple neuron can provide an efficient way to attain the standardization of the input data.

Using the McCulloch-Pitts neuron

To create the reward matrix, R, a robust model for processing the inputs of the huge volumes in a warehouse must be reduced to a limited number of features.

In one model, for example, the thousands to hundreds of thousands of inputs can be described as follows:

• Forecast product arrivals with a low priority weight: w₁ = 10
• Confirmed arrivals with a high priority weight: w₂ = 70
• Unplanned arrivals decided by the sales department: w₃ = 75
• Forecasts with a high priority weight: w₄ = 60
• Confirmed arrivals that have a low turnover and so have a low weight: w₅ = 20

The weights have been provided as constants. A McCulloch-Pitts neuron does not modify weights. A perceptron neuron does as we will see beginning with Chapter 8, Solving the XOR Problem with a Feedforward Neural Network. Experience shows that modifying weights is not always necessary.

These weights form a vector, as shown here:

Each element of the vector represents the weight of a feature of a product stored in optimal locations. The ultimate phase of this process will produce a reward matrix, R, for an MDP to optimize itineraries between warehouse locations.

Let's focus on our neuron. These weights, used through a system such as this one, can attain up to more than 50 weights and parameters per neuron. In this example, 5 weights are implemented. However, in real-life case, many other parameters come into consideration, such as unconfirmed arrivals, unconfirmed arrivals with a high priority, confirmed arrivals with a very low priority, arrivals from locations that probably do not meet security standards, arrivals with products that are potentially dangerous and require special care, and more. At that point, humans and even classical software cannot face such a variety of parameters.

The reward matrix will be size 6×6. It contains six locations, A to F. In this example, the six locations, l1 to l6, are warehouse storage and retrieval locations.

A 6×6 reward matrix represents the target of the McCulloch-Pitts layer implemented for the six locations.

When experimenting, the reward matrix, R, can be invented for testing purposes. In real-life implementations, you will have to find a way to build datasets from scratch. The reward matrix becomes the output of the preprocessing phase. The following source code shows the input of the reinforcement learning program used in the first chapter. The goal of this chapter describes how to produce the following reward matrix that we will be building in the next sections.

# R is The Reward Matrix for each location in a warehouse (or any other problem)
R = ql.matrix([ [0,0,0,0,1,0],
                [0,0,0,1,0,1],
                [0,0,100,1,0,0],
                [0,1,1,0,1,0],
                [1,0,0,1,0,0],
                [0,1,0,0,0,0] ])

For the warehouse example that we are using as for any other domain, the McCulloch-Pitts neuron sums up the weights of the input vector described previously to fill in the reward matrix.

Each location will require its neuron, with its weights.

INPUTS -> WEIGHTS -> BIAS -> VALUES

• Inputs are the flows in a warehouse or any form of data.
• Weights will be defined in this model.
• Bias is for stabilizing the weights. Bias does exactly what it means. It will tilt weights. It is very useful as a referee that will keep the weights on the right track.
• Values will be the output.

The McCulloch-Pitts neuron

The McCulloch-Pitts neuron dates back to 1943. It contains inputs, weights, and an activation function. Part of the preprocessing phase consists of selecting the right model. The McCulloch-Pitts neuron can represent a given location efficiently.

The following diagram shows the McCulloch-Pitts neuron model:

Figure 2.1: The McCulloch-Pitts neuron model

This model contains several input x weights that are summed to either reach a threshold that will lead, once transformed, to the output, y = 0, or 1. In this model, y will be calculated in a more complex way.

MCP.py written with TensorFlow 2 will be used to illustrate the neuron.

In the following source code, the TensorFlow variables will contain the input values (x), the weights (W), and the bias (b). Variables represent the structure of your graph:

# The variables
x = tf.Variable([[0.0,0.0,0.0,0.0,0.0]], dtype = tf.float32)
W = tf.Variable([[0.0],[0.0],[0.0],[0.0],[0.0]], dtype =
    tf.float32)
b = tf.Variable([[0.0]])

In the original McCulloch-Pitts artificial neuron, the inputs (x) were multiplied by the following weights:

The mathematical function becomes a function with the neuron code triggering a logistic activation function (sigmoid), which will be explained in the second part of the chapter. Bias (b) has been added, which makes this neuron format useful even today, shown as follows:

# The Neuron
def neuron(x, W, b):
    y1=np.multiply(x,W)+b
    y1=np.sum(y1)
    y = 1 / (1 + np.exp(-y1))
    return y

Before starting a session, the McCulloch-Pitts neuron (1943) needs an operator to set its weights. That is the main difference between the McCulloch-Pitts neuron and the perceptron (1957), which is the model for modern deep learning neurons. The perceptron optimizes its weights through optimizing processes. Chapter 8, Solving the XOR Problem with a Feedforward Neural Network, describes why a modern perceptron was required.

The weights are now provided, and so are the quantities for each input value, which are stored in the x vector at l₁, one of the six locations of this warehouse example:

The weight values will be divided by 100, to represent percentages in terms of 0 to 1 values of warehouse flows in a given location. The following code deals with the choice of one location, l₁ only, its values, and parameters:

# The data
x_1 = [[10, 2, 1., 6., 2.]]
w_t = [[.1, .7, .75, .60, .20]]
b_1 = [1.0]

The neuron function is called, and the weights (w_t) and the quantities (x_1) of the warehouse flow are entered. Bias is set to 1 in this model. No session needs to be initialized; the neuron function is called:

# Computing the value of the neuron
value=neuron(x_1,w_t,b_1)

The neuron function neuron will calculate the value of the neuron. The program returns the following value:

value for threshold calculation:0.99999

This value represents the activity of location l₁ at a given date and a given time. This example represents only one of the six locations to compute. For this location, the higher the value, the closer to 1, the higher the probable saturation rate of this area. This means there is little space left to store products at that location. That is why the reinforcement learning program for a warehouse is looking for the least loaded area for a given product in this model.

Each location has a probable availability:

A = Availability = 1 – load

The probability of a load of a given storage point lies between 0 and 1.

High values of availability will be close to 1, and low probabilities will be close to 0, as shown in the following example:

>>> print("Availability of location x:{0:.5f}".format(
...       round(availability,5)))
Availability of location x:0.00001

For example, the load of l₁ has a probable rounded load of 0.99, and its probable availability is 0.002 maximum. The goal of the AGV is to search and find the closest and most available location to optimize its trajectories. l₁ is not a good candidate at that day and time. Load is a keyword in production or service activities. The less available resources have the highest load rate.

When all six locations' availabilities have been calculated by the McCulloch-Pitts neuron—each with its respective quantity inputs, weights, and bias—a location vector of the results of this system will be produced. Then, the program needs to be implemented to run all six locations and not just one location through a recursive use of the one neuron model:

A(L) = {a(l₁),a(l₂),a(l₃),a(l₄),a(l₅),a(l₆)}

The availability, 1 – output value of the neuron, constitutes a six-line vector. The following vector, l_v, will be obtained by running the previous sample code on all six locations.

As shown in the preceding formula, l_v is the vector containing the value of each location for a given AGV to choose from. The values in the vector represent availability. 0.0002 means little availability; 0.9 means high availability. With this choice, the MDP reinforcement learning program will optimize the AGV's trajectory to get to this specific warehouse location.

The l_v is the result of the weighing function of six potential locations for the AGV. It is also a vector of transformed inputs.

The Python-TensorFlow architecture

Implementation of the McCulloch-Pitts neuron can best be viewed as shown in the following graph:

Figure 2.2: Implementation of the McCulloch-Pitts neuron

A data flow graph will also help optimize a program when things go wrong as in classical computing.

Now that the value of each location of L = {l₁, l₂, l₃, l₄, l₅, l₆} contains its availability in a vector, the locations can be sorted from the most available to the least available location. From there, the reward matrix, R, for the MDP process described in Chapter 1, Getting Started with Next-Generation Artifcial Intelligence through Reinforcement Learning, can be built.

Overall architecture

At this point, the overall architecture contains two main components:

1. Chapter 1: A reinforcement learning program based on the value-action Q function using a reward matrix that will be finalized in this chapter. The reward matrix was provided in the first chapter as an experiment, but in the implementation phase, you'll often have to build it from scratch. It sometimes takes weeks to produce a good reward matrix.
2. Chapter 2: Designing a set of 6×1 neurons that represents the flow of products at a given time at six locations. The output is the availability probability from 0 to 1. The highest value indicates the highest availability. The lowest value indicates the lowest availability.

At this point, there is some real-life information we can draw from these two main functions through an example:

• An AGV is automatically moving in a warehouse and is waiting to receive its next location to use an MDP, to calculate the optimal trajectory of its mission.
• An AGV is using a reward matrix, R, that was given during the experimental phase but needed to be designed during the implementation process.
• A system of six neurons, one per location, weighing the real quantities and probable quantities to give an availability vector, l_v, has been calculated. It is almost ready to provide the necessary reward matrix for the AGV.

To calculate the input values of the reward matrix in this reinforcement learning warehouse model, a bridge function between l_v and the reward matrix, R, is missing.

That bridge function is a logistic classifier based on the outputs of the n neurons that all perform the same tasks independently or recursively with one neuron.

At this point, the system:

• Took corporate data
• Used n neurons calculated with weights
• Applied an activation function

The activation function in this model requires a logistic classifier, a commonly used one.

Logistic classifier

The logistic classifier will be applied to l_v (the six location values) to find the best location for the AGV. This method can be applied to any other domain. It is based on the output of the six neurons as follows:

input × weight + bias

What are logistic functions? The goal of a logistic classifier is to produce a probability distribution from 0 to 1 for each value of the output vector. As you have seen so far, artificial intelligence applications use applied mathematics with probable values, not raw outputs.

The main reason is that machine learning/deep learning works best with standardization and normalization for workable homogeneous data distributions. Otherwise, the algorithms will often produce underfitted or overfitted results.

In the warehouse model, for example, the AGV needs to choose the best, most probable location, l_i. Even in a well-organized corporate warehouse, many uncertainties (late arrivals, product defects, or some unplanned problems) reduce the probability of a choice. A probability represents a value between 0 (low probability) and 1 (high probability). Logistic functions provide the tools to convert all numbers into probabilities between 0 and 1 to normalize data.

Logistic function

The logistic sigmoid provides one of the best ways to normalize the weight of a given output. The activation function of the neuron will be the logistic sigmoid. The threshold is usually a value above which the neuron has a y = 1 value; or else it has a y = 0 value. In this model, the minimum value will be 0.

The logistic function is represented as follows:

• e represents Euler's number, or 2.71828, the natural logarithm.
• x is the value to be calculated. In this case, s is the result of the logistic sigmoid function.

The code has been rearranged in the following example to show the reasoning process that produces the output, y, of the neuron:

    y1=np.multiply(x,W)+b
    y1=np.sum(y1)
    y = 1 / (1 + np.exp(-y1)) #logistic Sigmoid

Thanks to the logistic sigmoid function, the value for the first location in the model comes out squashed between 0 and 1 as 0.99, indicating a high probability that this location will be full.

To calculate the availability of the location once the 0.99 value has been taken into account, we subtract the load from the total availability, which is 1, as follows:

Availability = 1 – probability of being full (value)

Or

availability = 1 – value

As seen previously, once all locations are calculated in this manner, a final availability vector, l_v, is obtained.

When analyzing l_v, a problem has stopped the process. Individually, each line appears to be fine. By applying the logistic sigmoid to each output weight and subtracting it from 1, each location displays a probable availability between 0 and 1. However, the sum of the lines in l_v exceeds 1. That is not possible. A probability cannot exceed 1. The program needs to fix that.

Each line produces a [0, 1] solution, which fits the prerequisite of being a valid probability.

In this case, the vector l_v contains more than one value and becomes a probability distribution. The sum of l_v cannot exceed 1 and needs to be normalized.

The softmax function provides an excellent method to normalize l_v. Softmax is widely used in machine learning and deep learning.

Bear in mind that mathematical tools are not rules. You can adapt them to your problem as much as you wish as long as your solution works.

Softmax

The softmax function appears in many artificial intelligence models to normalize data. Softmax can be used for classification purposes and regression. In our example, we will use it to find an optimized goal for an MDP.

In the case of the warehouse example, an AGV needs to make a probable choice between six locations in the l_v vector. However, the total of the l_v values exceeds 1. l_v requires normalization of the softmax function, S. In the source code, the l_v vector will be named y.

The following code used is SOFTMAX.py.

1. y represents the l_v vector:

   # y is the vector of the scores of the lv vector in the warehouse example:
   y = [0.0002, 0.2, 0.9,0.0001,0.4,0.6]
   

2. is the exp(i) result of each value in y (l_v in the warehouse example), as follows:

   y_exp = [math.exp(i) for i in y]
   

3. is the sum of as shown in the following code:

   sum_exp_yi = sum(y_exp)
   

Now, each value of the vector is normalized by applying the following function:

softmax = [round(i / sum_exp_yi, 3) for i in y_exp]

softmax(l_v) provides a normalized vector with a sum equal to 1, as shown in this compressed version of the code. The vector obtained is often described as containing logits.

The following code shows one version of a softmax function:

def softmax(x):
    return np.exp(x) / np.sum(np.exp(x), axis=0)

l_v is now normalized by softmax(l_v) as follows.

The last part of the softmax function requires softmax(l_v) to be rounded to 0 or 1. The higher the value in softmax(l_v), the more probable it will be. In clear-cut transformations, the highest value will be close to 1, and the others will be closer to 0. In a decision-making process, the highest value needs to be established as follows:

print("7C.
Finding the highest value in the normalized y vector : ",ohot)

The output value is 0.273 and has been chosen as the most probable location. It is then set to 1, and the other, lower values are set to 0. This is called a one-hot function. This one-hot function is extremely helpful for encoding the data provided. The vector obtained can now be applied to the reward matrix. The value 1 probability will become 100 in the R reward matrix, as follows:

The softmax function is now complete. Location l₃ or C is the best solution for the AGV. The probability value is multiplied by 100, and the reward matrix, R, can now receive the input.

We now have the data for the reward matrix. The best way to understand the mathematical aspect of the project is to draw the result on a piece of paper using the actual warehouse layout from locations A to F.

Locations={l1-A, l2-B, l3-C, l4-D, l5-E, l6-F}

Line C of the reward matrix ={0, 0, 100, 0, 0, 0}, where C (the third value) is now the target for the self-driving vehicle, in this case, an AGV in a warehouse.

Figure 2.3: Illustration of a warehouse transport problem

We obtain the following reward matrix, R, described in Chapter 1, Getting Started with Next-Generation Artificial Intelligence through Reinforcement Learning:

This reward matrix is exactly the one used in the Python reinforcement learning program using the Q function from Chapter 1. The output of this chapter is thus the input of the R matrix. The 0 values are there for the agent to avoid those values. The 1 values indicate the reachable cells. The 100 in the C×C cell is the result of the softmax output. This program is designed to stay close to probability standards with positive values, as shown in the following R matrix taken from the mdp01.py of Chapter 1:

R = ql.matrix([ [0,0,0,0,1,0],
                [0,0,0,1,0,1],
                [0,0,100,1,0,0],
                [0,1,1,0,1,0],
                [1,0,0,1,0,0],
                [0,1,0,0,0,0] ])

At this point:

• The output of the functions in this chapter generated a reward matrix, R, which is the input of the MDP described in Chapter 1, Getting Started with Next-Generation Artificial Intelligence through Reinforcement Learning.
• The MDP process was set to run for 50,000 episodes in Chapter 1.
• The output of the MDP has multiple uses, as we saw in this chapter and Chapter 1.

The building blocks are in place to begin evaluating the execution and performances of the reinforcement learning program, as we will see in Chapter 3, Machine Intelligence – Evaluation Functions and Numerical Convergence.

Using a McCulloch-Pitts neuron with a logistic activation function in a one-layer network to build a reward matrix for reinforcement learning shows one way to preprocess a dataset.

Processing real-life data often requires a generalization of a logistic sigmoid function through a softmax function, and a one-hot function applied to logits to encode the data.

Machine learning functions are tools that must be understood to be able to use all or parts of them to solve a problem. With this practical approach to artificial intelligence, a whole world of projects awaits you.

This neuronal approach is the parent of the multilayer perceptron that will be introduced starting in Chapter 8, Solving the XOR Problem with a Feedforward Neural Network.

This chapter went from an experimental black box machine learning and deep learning to white box implementation. Implementation requires a full understanding of machine learning algorithms that often require fine-tuning.

However, artificial intelligence goes beyond understanding machine learning algorithms. Machine learning or deep learning require evaluation functions. Performance or results cannot be validated without evaluation functions, as explained in Chapter 3, Machine Intelligence – Evaluation Functions and Numerical Convergence.

In the next chapter, the evaluation process of machine intelligence will be illustrated by examples that show the limits of human intelligence and the rise of machine power.

1. Raw data can be the input to a neuron and transformed with weights. (Yes | No)
2. Does a neuron require a threshold? (Yes | No)
3. A logistic sigmoid activation function makes the sum of the weights larger. (Yes | No)
4. A McCulloch-Pitts neuron sums the weights of its inputs. (Yes | No)
5. A logistic sigmoid function is a log10 operation. (Yes | No)
6. A logistic softmax is not necessary if a logistic sigmoid function is applied to a vector. (Yes | No)
7. A probability is a value between –1 and 1. (Yes | No)

• The original McCulloch-Pitts neuron 1943 paper: http://www.cse.chalmers.se/~coquand/AUTOMATA/mcp.pdf
• TensorFlow variables: https://www.tensorflow.org/beta/guide/variables