Function Optimizations: How and Why

bAnd now it's time to have some fun. We are going to develop a very powerful, three-dimensional application that you won't find anywhere else. This application will allow you to visualize how individual functions optimize over time, with a two and three-dimensional graphic of each. The source code for this application is located in the instructions given for access of the book's source code. This application will be very unique in that we will use a mixture of open source and third-party controls to create an unbelievably powerful application. Open source doesn't handle everything all the time, and for those of you serious about graphics I wanted to expose you to some controls aside from the open source standards such as ZedGraph, Microsoft Charting Controls, and others. As you will see in a moment, the difference is astounding...

Technical requirements

You will be required to have a basic knowledge of .NET development using Microsoft Visual Studio and C#. You will need to download the code for this chapter from the book's website.

Check out the following video to see Code in Action: http://bit.ly/2ppBmvI.

Getting started

Before we get started, let me show you the product we are going to create. When we are done you will have an application that allows you to view graphically how a function minimizes or maximizes over iterations. This is contrary to the typical text-based representations of such systems, as follows:

3D Plot

As you can see, this is a very visual application. Let's go ahead and break this down into sections that we will refer to as we progress.

The first section is the three-dimensional plot and is located on our main page. Three dimensions can provide much more insight into the path that each particle in the swarm takes, and the swarm itself is taking. It is also much easier to see when a particle or the swarm converges on the global minimum. For this plot we will use the incredible Nevron chart control. You can find more information about this Chart control...

Function minimization and maximization

Function minimization and maximization are the process of finding the smallest and largest value of a given function. Let's talk briefly about that value.

If the value is within a given range, then it is called the local extrema; if it is within the entire domain of a function then it is called the global extrema. Let's say we have a function f, and it's defined against a domain X. The maximum, or global, point at x* is f(x*) is greater than or equal to f(x) for all x in the domain X. Conversely, the function's global minimum point at x* is f(x*) is less than or equal to f(x) for all x in the domain X.

In a simpler fashion, the maximum point is also called the maximum value, and the minimum point is called the minimum value, of the function. The global maximum or minimum is either the highest or lowest function value in...

Hyperparameters and tuning

Hyperparameters in general are parameters used for tuning various machine learning functions. This is no different in our application.

On the front screen of our application, the following is what our hyperparameter panel looks like. We will discuss each hyperparameter in detail:

Parameters

Function

This is a list of all available functions for optimization. Simply select the function you would like to use, set the strategy and other parameters, and click the Run button. Please see the function optimization reference for more detailed information about each function. There are over 50 different functions currently available at the time of writing, and we will cover in a later chapter how you can...

Understanding visualizations

In this section we will go over some of the many that you will see in our program. This includes both two and three dimensional plots.

Understanding two-dimensional visualizations

For our application, we have several two-dimensional visualizations that we need to explain. The first is the two-dimensional plot of the function optimization, be it maximized or minimized. This visualization is shown as follows. Remember, for this we are using Microsoft Chart control, which is available from: https://www.microsoft.com/en-us/download/details.aspx?id=14422:

2D Visualization

For each particle that we are plotting (up to a maximum of 10), we will have a different marker style. A marker is the diamonds...

Plotting results

The following section details how our information is displayed once results have been obtained.

Playing back results

Once the swarm optimization is complete, the job of playing back the results comes to the forefront. Our main function in replaying plots is called PlaybackPlot. Let's discuss the function in detail:

private void PlaybackPlot()
{

Get our current iteration, as follows:

int iterN = GetCurrentIteration();
_Closing = false;

If we have played back all our points, then leave, as follows:

if (iterN >= maxIter)
return;
PlotStatusN = PlotStatus.Play;

Update the progress bar, as follows:

progressBarX1.Visible = true;
progressBarX1.Minimum = 0;
progressBarX1.Maximum = maxIter;

Go through all iterations...

Adding new optimization functions

One of the beautiful things about our visual test workbench is the ease of which we can add new optimization functions for testing.

The purpose of functions

Some problems are evaluated in terms of quality versus correct or incorrect. Such problems are known as optimization problems because the goal is the identification of the optimal value. Functions (sometimes called cost functions, objective functions, error functions, and so on.) achieve that goal by mapping n-dimensional real-valued items into one-dimensional real-valued items (some folks will prefer the termspaces over items' as it more closely aligns with the total search space we have talked about).

There are two types of function...

Summary

In this chapter, we discussed functions, what they are, and why we use them. We developed a very powerful and flexible application with which to test function optimization. We also showed you the complete process of adding a new function and how to run it once added. You are now free to add as many new functions as you like; just follow the process outlined herein and you should have no problems.

In the next chapter, we are going to learn how to replace back propagation with particle swarm-optimized algorithms, so hold on to your hats!