Benchmarking and Neural Network Optimization

Benchmarking is a standard against which we compare solutions to find out whether they are good or not. In the context of deep learning, we might set benchmarks for an existing model that is performing pretty well. We might test our model against factors such as accuracy, the amount of data handled, memory consumption, and JVM garbage collection tuning. In this chapter, we briefly talk about the benchmarking possibilities with your DL4J applications. We will start with general guidelines and then move on to more DL4J-specific benchmarking settings. At the end of the chapter, we will look at a hyperparameter tuning example that shows how to find the best neural network parameters in order to yield the best results.

In this chapter, we will cover the following recipes:

• DL4J/ND4J specific configuration
• Setting up heap spaces and garbage...

Technical requirements

The code for this chapter is located at https://github.com/PacktPublishing/Java-Deep-Learning-Cookbook/tree/master/12_Benchmarking_and_Neural_Network_Optimization/sourceCode/cookbookapp/src/main/java.

After cloning our GitHub repository, navigate to the Java-Deep-Learning-Cookbook/12_Benchmarking_and_Neural_Network_Optimization/sourceCode directory. Then import the cookbookapp project as a Maven project by importing pom.xml.

The following are links to two examples:

• Hyperparameter tuning example: https://github.com/PacktPublishing/Java-Deep-Learning-Cookbook/blob/master/12_Benchmarking_and_Neural_Network_Optimization/sourceCode/cookbookapp/src/main/java/HyperParameterTuning.java
• Arbiter UI example: https://github.com/PacktPublishing/Java-Deep-Learning-Cookbook/blob/master/12_Benchmarking_and_Neural_Network_Optimization/sourceCode/cookbookapp/src/main/java...

DL4J/ND4J-specific configuration

Apart from general benchmarking guidelines, we need to follow additional benchmarking configurations that are DL4J/ND4J-specific. These are important benchmarking configurations that target the hardware and mathematical computations.

Because ND4J is the JVM computation library for DL4J, benchmarks mostly target mathematical computations. Any benchmarks discussed with regard to ND4J can then also be applied to DL4J. Let's discuss DL4J/ND4J-specific benchmarks.

Getting ready

Make sure you have downloaded cudNN from the following link: https://developer.nvidia.com/cudnn. Install it before attempting to configure it with DL4J. Note that cuDNN doesn't come as a bundle with CUDA. So...

Setting up heap spaces and garbage collection

Memory heap spaces and garbage collection are frequently discussed yet are often the most frequently ignored benchmarks. With DL4J/ND4J, you can configure two types of memory limit: on-heap memory and off-heap memory. Whenever an INDArray is collected by the JVM garbage collector, the off-heap memory will be de-allocated, assuming that it is not being used anywhere else. In this recipe, we will set up heap spaces and garbage collection for benchmarking.

How to do it...

1. Add the required VM arguments to the Eclipse/IntelliJ IDE, as shown in the following example:

-Xms1G -Xmx6G -Dorg.bytedeco.javacpp.maxbytes=16G -Dorg.bytedeco.javacpp.maxphysicalbytes=20G

For example, in IntelliJ...

Using asynchronous ETL

We use synchronous ETL for demonstration purposes. But for production, asynchronous ETL is preferable. In production, the existence of a single low-performance ETA component can cause a performance bottleneck. In DL4J, we load data to the disk using DataSetIterator. It can load the data from disk or, memory, or simply load data asynchronously. Asynchronous ETL uses an asynchronous loader in the background. Using multithreading, it loads data into the GPU/CPU and other threads take care of compute tasks. In the following recipe, we will perform asynchronous ETL operations in DL4J.

How to do it...

1. Create asynchronous iterators with asynchronous prefetch:

DatasetIterator asyncIterator = new AsyncMultiDataSetIterator...

Using arbiter to monitor neural network behavior

Hyperparameter optimization/tuning is the process of finding the optimal values for hyperparameters in the learning process. Hyperparameter optimization partially automates the process of finding optimal hyperparameters using certain search strategies. Arbiter is part of the DL4J deep learning library and is used for hyperparameter optimization. Arbiter can be used to find high-performing models by tuning the hyperparameters of the neural network. Arbiter has a UI that visualizes the results of the hyperparameter tuning process.

In this recipe, we will set up arbiter and visualize the training instance to take a look at neural network behavior.

How to do it...

1. Add the arbiter...

Performing hyperparameter tuning

Once search spaces are defined using ParameterSpace or OptimizationConfiguration, with a possible range of values, the next step is to complete network configuration using MultiLayerSpace or ComputationGraphSpace. After that, we start the training process. We perform multiple training sessions during the hyperparameter tuning process.

In this recipe, we will perform and visualize the hyperparameter tuning process. We will be using MultiLayerSpace for the demonstration.

How to do it...

1. Add a search space for the layer size using IntegerParameterSpace:

ParameterSpace<Integer> layerSizeParam = new IntegerParameterSpace(startLimit,endLimit);

2. Add a search space for the learning rate using...