Scala for Machine Learning

Scientists define observations as sets or vectors of features. Classification problems rely on the estimation of the similarity between vectors of observations. One technique consists of comparing two vectors by computing the normalized inner product. A co-vector is defined as a linear map α of a vector to the inner product (field).

Tip

Inner product

M1: The definition of a <.> inner product and a α co-vector is as follows:

Let's define a vector as a constructor from any _ => Vector[_] field (or Function1[_, Vector]). A co-vector is then defined as the mapping function of a vector to its Vector[_] => _ field (or Function1[Vector, _]).

Let's define a two-dimensional (two types or fields) higher kind structure, Hom, that can be defined as either a vector or co-vector by fixing one of the two types:

type Hom[T] = {
  type Right[X] = Function1[X,T] // Co-vector
  type Left[X] = Function1[T,X]   // Vector
 }

The projections of the higher kind, Hom, to the Right or Left single parameter types are known as functors, which are as follows:

A covariant functor of a variable is a map F: C => C such that:

The definition of the F[U => V] := F[U] => F[V]covariant functor in Scala is as follows:

trait Functor[M[_]] {
  def map[U,V](m: M[U])(f: U =>V): M[V]
}

For example, let's consider an observation defined as a n dimension vector of a T type, Obs[T]. The constructor for the observation can be represented as Function1[T,Obs]. Its ObsFunctor functor is implemented as follows:

trait ObsFunctor[T] extends Functor[(Hom[T])#Left] { self =>
  override def map[U,V](vu: Function1[T,U])(f: U =>V): 
    Function1[T,V] = f.compose(vu)
}

The functor is qualified as a covariant functor because the morphism is applied to the return type of the element of Obs as Function1[T, Obs]. The Hom projection of the two parameters types to a vector is implemented as (Hom[T])#Left.

A contravariant functor of one variable is a map F: C => C such that:

The definition of the F[U => V] := F[V] => F[U] contravariant functor in Scala is as follows:

trait CoFunctor[M[_]] {
  def map[U,V](m: M[U])(f: V =>U): M[V]
}

Note that the input and output types in the f morphism are reversed from the definition of a covariant functor. The constructor for the co-vector can be represented as Function1[Obs,T]. Its CoObsFunctor functor is implemented as follows:

trait CoObsFunctor[T] extends CoFunctor[(Hom[T])#Right] {
  self =>
    override def map[U,V](vu: Function1[U,T])(f: V =>U): 
       Function1[V,T] = f.andThen(vu)
}

Monads are structures in algebraic topology that are related to the category theory. Monads extend the concept of a functor to allow a composition known as the monadic composition of morphisms on a single type. They enable the chaining or weaving of computation into a sequence of steps or pipeline. The collections bundled with the Scala standard library (List, Map, and so on) are constructed as monads [1:1].

Monads provide the ability for those collections to perform the following functions:

An example is as follows:

trait Monad[M[_]] {
  def unit[T](a: T): M[T]
  def map[U,V](m: M[U])(f U =>V): M[V]
  def flatMap[U,V](m: M[U])(f: U =>M[V]): M[V]
}

Monads are therefore critical in machine learning as they enable you to compose multiple data transformation functions into a sequence or workflow. This property is applicable to any type of complex scientific computation [1:2].

The purpose of data clustering is to partition a collection of data into a number of clusters or data segments. Practically, a clustering algorithm is used to organize observations into clusters by minimizing the distance between observations within a cluster and maximizing the distance between observations across clusters. A clustering algorithm consists of the following steps:

Data clustering is also known as data segmentation or data partitioning.

Dimension reduction techniques aim at finding the smallest but most relevant set of features needed to build a reliable model. There are many reasons for reducing the number of features or parameters in a model, from avoiding overfitting to reducing computation costs.

There are many ways to classify the different techniques used to extract knowledge from data using unsupervised learning. The following taxonomy breaks down these techniques according to their purpose, although the list is far from being exhaustive, as shown in the following diagram:

Taxonomy of unsupervised learning algorithms

In order to simplify the description of a statistics formula, we adopt the following simplification: the probability of an X event is the same as the probability of the discrete X random variable to have a value x: p(X) = p(X=x).

The notation for the joint probability is p(X,Y) = p(X=x,Y=y).

The notation for the conditional probability is p(X|Y) = p(X=x|Y=y).

Generative models attempt to fit a joint probability distribution, p(X,Y), of two X and Y events (or random variables), representing two sets of observed and hidden x and y variables. Discriminative models compute the conditional probability, p(Y|X), of an event or random variable Y of hidden variables y, given an event or random variable X of observed variables x. Generative models are commonly introduced through the Bayes' rule. The conditional probability of a Y event, given an X event, is computed as the product of the conditional probability of the X event, given the Y event, and the probability of the X event normalized by the probability of the Y event [1:5].

Note

Bayes' rule

Joint probability for independent random variables, X=x and Y=y, is given by:

Conditional probability of a random variable, Y = y, given X = x, is given by:

Bayes' formula is given by:

The Bayes' rule is the foundation of the Naïve Bayes classifier, as described in the Introducing the multinomial Naïve Bayes section in Chapter 5, Naïve Bayes Classifiers.

Contrary to generative models, discriminative models compute the conditional probability p(Y|X) directly, using the same algorithm for training and classification.

Generative and discriminative models have their respective advantages and disadvantages. Novice data scientists learn to match the appropriate algorithm to each problem through experimentation. Here is a brief guideline describing which type of models make sense according to the objective or criteria of the project:

We can further refine the taxonomy of supervised learning algorithms by segregating arbitrarily between sequential and random variables for generative models and breaking down discriminative methods as applied to continuous processes (regression) and discrete processes (classification):

Taxonomy of supervised learning algorithms

This is a lightweight library that provides developers with a foundation of small, ready-to-use Java classes that can be easily weaved into a machine learning problem. The examples used throughout the book require Version 3.5 or higher.

The math library supports the following:

For more information, visit http://commons.apache.org/proper/commons-math.

We need Apache Public License 2.0; the terms are available at http://www.apache.org/licenses/LICENSE-2.0.

The installation and deployment of the Apache Commons Math library are quite simple. The steps are as follows:

You can also download commons-math3-3.5-src.zip from the Source section.

The library supports a variety of configurable plots and charts (scatter, dial, pie, area, bar, box and whisker, stacked, and 3D). We use JFreeChart to display the output of data processing and algorithms throughout the book, but you are encouraged to explore this great library on your own, as time permits.

It is distributed under the terms of the GNU Lesser General Public License (LGPL), which permits its use in proprietary applications.

To install and deploy JFreeChart, perform the following steps:

For the sake of readability of the code, the following primitive types will be used:

type DblPair = (Double, Double)
type DblArray = Array[Double]
type DblMatrix = Array[DblArray]
type DblVector = Vector[Double]
type XSeries[T] = Vector[T]         // One dimensional vector
type XVSeries[T] = Vector[Array[T]] // multi-dimensional vector

The times series introduced in the Time series in Scala section in Chapter 3, Data Preprocessing, is implemented as XSeries[T] or XVSeries[T] of a parameterized T type.

Implicit conversion is an important feature of the Scala programming language. It allows developers to specify a type conversion for an entire library in a single place. Here are a few of the implicit type conversions that are used throughout the book:

object Types {
  Object ScalaMl {  
   implicit def double2Array(x: Double): DblArray = 
      Array[Double](x)
   implicit def dblPair2Vector(x: DblPair): Vector[DblPair] = 
      Vector[DblPair](x._1,x._2)
   ...
  }
}

The objective is to create a model for stock price using its daily trading volume and volatility. Throughout the book, we will rely on financial data to evaluate and discuss the merits of different data processing and machine learning methods. In this example, the data is extracted from Yahoo Finances using the CSV format with the following fields:

The YahooFinancials enumerator extracts the historical daily trading information from the Yahoo finance site:

type Fields = Array[String]
object YahooFinancials extends Enumeration {
   type YahooFinancials = Value
   val DATE, OPEN, HIGH, LOW, CLOSE, VOLUME, ADJ_CLOSE = Value

   def toDouble(v: Value): Fields => Double =   //1
   (s: Fields) => s(v.id).toDouble
   def toDblArray(vs: Array[Value]): Fields => DblArray = //2
       (s: Fields) => vs.map(v => s(v.id).toDouble)
  …
}

The toDouble method converts an array of string into a single value (line 1) and toDblArray converts an array of string into an array of values (line 2). The YahooFinancials enumerator is described in the Data sources section in Appendix A, Basic Concepts in detail.

Let's create a simple program that loads the content of the file, executes some simple preprocessing functions, and creates a simple model. We selected the CSCO stock price between January 1, 2012 and December 1, 2013 as our data input.

Let's consider the two variables, price and volume, as shown in the following screenshot. The top graph displays the variation of the price of Cisco stock over time and the bottom bar chart represents the daily trading volume on Cisco stock over time:

Price-Volume action for Cisco stock 2012-2013

The second step is loading the dataset from a local or remote data storage. Typically, large datasets are loaded from a database or distributed filesystems such as Hadoop Distributed File System (HDFS). The load method takes an absolute pathname, extract, and transforms the input data from a file into a time series of a Vector[DblPair] type:

def load(fileName: String): Try[Vector[DblPair]] = Try {
   val src =  Source.fromFile(fileName)  //3
   val data = extract(src.getLines.map(_.split(",")).drop(1)) //4
   src.close //5
   data
 }

The data file is extracted through an invocation of the Source.fromFile static method (line 3), and then the fields are extracted through a map before the header (first row in the file) is removed using drop (line 4). The file has to be closed to avoid leaking of the file handle (line 5).

The purpose of the extract method is to generate a time series of two variables (relative stock volatility and relative stock daily trading volume):

def extract(cols: Iterator[Array[String]]): XVSeries[Double]= {
  val features = Array[YahooFinancials](LOW, HIGH, VOLUME) //6
  val conversion = YahooFinancials.toDblArray(features)  //7
  cols.map(c => conversion(c)).toVector   
      .map(x => Array[Double](1.0 - x(0)/x(1), x(2)))  //8
}

The only purpose of the extract method is to convert the raw textual data into a two-dimensional time series. The first step consists of selecting the three features to extract LOW (the lowest stock price in the session), HIGH (the highest price in the session), and VOLUME (trading volume for the session) (line 6). This feature set is used to convert each line of fields into a corresponding set of three values (line 7). Finally, the feature set is reduced to the following two variables (line 8):

val cols = Source.fromFile.getLines.map(_.split(",")).toArray.drop(1)

val lines = Source.fromFile.getLines
val fields = lines.map(_.split(",")).toArray
val cols = fields.drop(1)

The next step is to normalize the data in the range [0.0, 1.0] to be trained by the binomial logistic regression. It is time to introduce an immutable and flexible normalization class.

Immutable normalization

The logistic regression relies on the sigmoid curve or logistic function is described in the Logistic function section in Chapter 6, Regression and Regularization. The logistic functions are used to segregate training data into classes. The output value of the logistic function ranges from 0 for x = - INFINITY to 1 for x = + INFINITY. Therefore, it makes sense to normalize the input data or observation over [0, 1].

Note

Normalize or not normalize?

The purpose of normalizing data is to impose a single range of values for all the features, so the model does not favor any particular feature. Normalization techniques include linear normalization and Z-score. Normalization is an expensive operation that is not always needed.

The normalization is a linear transformation of the raw data that can be generalized to any range [l, h].

Note

Linear normalization

M2: [0, 1] Normalization of features {x_i} with minimum x_min and maximum x_max values:

M3: [l, h] Normalization of features {xi}:

The normalization of input data in supervised learning has a specific requirement: the classification and prediction of new observations have to use the normalization parameters (min and max) extracted from the training set, so all the observations share the same scaling factor.

Let's define the MinMax normalization class. The class is immutable: the minimum, min, and maximum, max, values are computed within the constructor. The class takes a time series of a parameterized T type and values as arguments (line 8). The steps of the normalization process are defined as follows:

1. Initialize the minimum values for a given time series during instantiation (line 9).

2. Compute the normalization parameters (line 10) and normalize the input data (line 11).

3. Normalize any new data points reusing the normalization parameters (line 14):

   class MinMax[T <: AnyVal](val values: XSeries[T]) (f : T => Double) { //8
     val zero = (Double.MaxValue, -Double.MaxValue)
     val minMax = values./:(zero)((mM, x) => { //9
       val min = mM._1
       val max = mM._2
      (if(x < min) x else min, if(x > max) x else max)
     })
     case class ScaleFactors(low:Double ,high:Double, ratio: Double)
     var scaleFactors: Option[ScaleFactors] = None //10
   
     def min = minMax._1
     def max = minMax._2
     def normalize(low: Double, high: Double): DblVector //11
     def normalize(value: Double): Double
   }

The class constructor computes the tuple of minimum and maximum values, minMax, using a fold (line 9). The scaleFactors scaling parameters are computed during the normalization of the time series (line 11), which are described as follows. The normalize method initializes the scaling factor parameters (line 12) before normalizing the input data (line 13):

def normalize(low: Double, high: Double): DblVector = 
  setScaleFactors(low, high).map( scale => { //12
    values.map(x =>(x - min)*scale.ratio + scale.low) //13
  }).getOrElse(/* … */)

def setScaleFactors(l: Double, h: Double): Option[ScaleFactors]={
    // .. error handling code
   Some(ScaleFactors(l, h, (h - l)/(max - min))
}

Subsequent observations use the same scaling factors extracted from the input time series in normalize (line 14):

def normalize(value: Double):Double = setScaleFactors.map(scale => 
   if(value < min) scale.low
   else if (value > max) scale.high
   else (value - min)* scale.high + scale.low
).getOrElse( /* … */)

The MinMax class normalizes single variable observations.

Note

The statistics class

The class that extracts the basic statistics from a Stats dataset, which is introduced in the Profiling data section in Chapter 2, Hello World!, inherits the MinMax class.

The test case with the binomial logistic regression uses a multiple variable normalization, implemented by the MinMaxVector class, which takes observations of the XVSeries[Double] type as inputs:

class MinMaxVector(series: XVSeries[Double]) {
  val minMaxVector: Vector[MinMax[Double]] = //15
      series.transpose.map(new MinMax[Double](_))
  def normalize(low: Double, high: Double): XVSeries[Double]
}

The constructor of the MinMaxVector class transposes the vector of array of observations in order to compute the minimum and maximum value for each dimension (line 15).

The price action chart has a very interesting characteristic.

Analyzing data

At a closer look, a sudden change in price and increase in volume occurs about every three months or so. Experienced investors will undoubtedly recognize that these price-volume patterns are related to the release of quarterly earnings of Cisco. Such a regular but unpredictable pattern can be a source of concern or opportunity if risk can be properly managed. The strong reaction of the stock price to the release of corporate earnings may scare some long-term investors while enticing day traders.

The following graph visualizes the potential correlation between sudden price change (volatility) and heavy trading volume:

Price-volume correlation for the Cisco stock 2012-2013

The next section is not required for the understanding of the test case. It illustrates the capabilities of JFreeChart as a simple visualization and plotting library.

Plotting data

Although charting is not the primary goal of this book, we thought that you will benefit from a brief introduction to JFreeChart.

Note

Plotting classes

This section illustrates a simple Scala interface to JFreeChart Java classes. Reading this is not required for the understanding of machine learning. The visualization of the results of a computation is beyond the scope of this book.

Some of the classes used in visualization are described in the Appendix A, Basic Concepts.

The dataset (volatility and volume) is converted into internal JFreeChart data structures. The ScatterPlot class implements a simple configurable scatter plot with the following arguments:

• config: This includes information, labels, fonts, and so on, of the plot

• theme: This is the predefined theme for the plot (black, white background, and so on)

The code will be as follows:

class ScatterPlot(config: PlotInfo, theme: PlotTheme) { //16
  def display(xy: Vector[DblPair], width: Int, height) //17
  def display(xt: XVSeries[Double], width: Int, height)
  // ….
}

The PlotTheme class defines a specific theme or preconfiguration of the chart (line 16). The class offers a set of display methods to accommodate a wide range of data structures and configuration (line 17).

Note

Visualization

The JFreeChart library is introduced as a robust charting tool. The code related to plots and charts is omitted from the book in order to keep the code snippets concise and dedicated to machine learning. On a few occasions, output data is formatted as a CSV file to be imported into a spreadsheet.

The ScatterPlot.display method is used to display the normalized input data used in the binomial logistic regression as follows:

val plot = new ScatterPlot(("CSCO 2012-2013", 
   "Session High - Low", "Session Volume"), new BlackPlotTheme)
plot.display(volatility_vol, 250, 340)

A scatter plot of volatility and volume for the Cisco stock 2012-2013

The scatter plot shows a level of correlation between session volume and session volatility and confirms the initial finding in the stock price and volume chart. We can leverage this information to classify trading sessions by their volatility and volume. The next step is to create a two class model by loading a training set, observations, and expected values, into our logistic regression algorithm. The classes are delimited by a decision boundary (also known as a hyperplane) drawn on the scatter plot.

Visualizing labels—the normalized variation of the stock price between the opening and closing of the trading session is selected as the label for this classifier.

The objective of this training is to build a model that can discriminate between volatile and nonvolatile trading sessions. For the sake of the exercise, session volatility is defined as the relative difference between the session highest price and lower price. The total trading volume within a session constitutes the second parameter of the model. The relative price movement within a trading session (that is, closing price/open price - 1) is our expected values or labels.

Logistic regression is commonly used in statistics inference.

Tip

M4: Logistic regression model

The first weight w₀ is known as the intercept. The binomial logistic regression is described in the Logistic regression section in Chapter 6, Regression and Regularization, in detail.

The following implementation of the binomial logistic regression classifier exposes a single classify method to comply with our desire to reduce the complexity and life cycle of objects. The model weights parameters are computed during training when the LogBinRegression class/model is instantiated. As mentioned earlier, the sections of the code nonessential to the understanding of the algorithm are omitted.

The LogBinRegression constructor has five arguments (line 18):

The code is as follows:

class LogBinRegression(
     obsSet: Vector[DblArray], 
     expected: Vector[Int],
     maxIters: Int, 
     eta: Double, 
     eps: Double) {  //18

   val model: LogBinRegressionModel = train  //19
   def classify(obs: DblArray): Try[(Int, Double)]   //20
   def train: LogBinRegressionModel
   def intercept(weights: DblArray): Double
   …
}

The LogBinRegressionModel model is generated through training during the instantiation of the LogBinRegression logistic regression class (line 19):

case class LogBinRegressionModel(val weights: DblArray)

The model is fully defined by its weights, as described in the mathematical formula M3. The weights(0) intercept represents the mean value of the prediction for observations for which variables are zero. The intercept does not have any specific meaning for most of the cases and it is not always computable.

The code will be as follows:

def intercept(weights: DblArray): Double = {
  val zeroObs = obsSet.filter(!_.exists( _ > 0.01))
  if( zeroObs.size > 0)
    zeroObs.aggregate(0.0)((s,z) => s + dot(z, weights), 
       _ + _ )/zeroObs.size
  else 0.0
}

The classify methods takes new observations as inputs and compute the index of the classes (0 or 1) the observations belong to and the actual likelihood (line 20).

Selecting an optimizer

The goal of the training of a model using expected values is to compute the optimal weights that minimizes the error or cost function. We select the batch gradient descent algorithm to minimize the cumulative error between the predicted and expected values for all the observations. Although there are quite a few alternative optimizers, the gradient descent is quite robust and simple enough for this first chapter. The algorithm consists of updating the weights w_i of the regression model by minimizing the cost.

Note

Cost function

M5: Cost (or compound error = predicted – expected):

M6: The batch gradient descent method to update model weights w_i is as follows:

For those interested in learning about of optimization techniques, the Summary of optimization techniques section in the Appendix A, Basic Concepts presents an overview of the most commonly used optimizers. The batch descent gradient method is also used for the training of the multilayer perceptron (refer to The training epoch section under The multilayer perceptron section in Chapter 9, Artificial Neural Networks).

The execution of the batch gradient descent algorithm follows these steps:

1. Initialize the weights of the regression model.

2. Shuffle the order of observations and expected values.

3. Aggregate the cost or error for the entire observation set.

4. Update the model weights using the cost as the objective function.

5. Repeat from step 2 until either the maximum number of iterations is reached or the incremental update of the cost is close to zero.

The purpose of shuffling the order of the observations between iterations is to avoid the minimization of the cost reaching a local minimum.

Tip

Batch and stochastic gradient descent

The stochastic gradient descent is a variant of the gradient descent that updates the model weights after computing the error on each observation. Although the stochastic gradient descent requires a higher computation effort to process each observation, it converges toward the optimal value of weights fairly quickly after a small number of iterations. However, the stochastic gradient descent is sensitive to the initial value of the weights and the selection of the learning rate, which is usually defined by an adaptive formula.

def train: LogBinRegressionModel = {
  val nWeights = obsSet.head.length + 1  //21
  val init = Array.fill(nWeights)(Random.nextDouble )  //22
  val weights = gradientDescent(obsSet.zip(expected),0.0,0,init)
  new LogBinRegressionModel(weights)   //23
}

type LabelObs = Vector[(DblArray, Int)]

@tailrec
def gradientDescent(
      obsAndLbl: LabelObs, 
      cost: Double, 
      nIters: Int, 
      weights: DblArray): DblArray = {  //24
  
  if(nIters >= maxIters) 
       throw new IllegalStateException("..")//25
  val shuffled = shuffle(obsAndLbl)   //26
  val errorGrad = shuffled.map{ case(x, y) => {  //27
      val error = sigmoid(dot(x, weights)) - y
      (error, x.map( _ * error))  //28
   }}.unzip

   val scale = 0.5/obsAndLbl.size
   val newCost = errorGrad._1   //29
.aggregate(0.0)((s,c) =>s + c*c, _ + _ )*scale
   val relativeError = cost/newCost - 1.0
   
   if( Math.abs(relativeError) < eps)  weights  //30
   else {
     val derivatives = Vector[Double](1.0) ++ 
                 errorGrad._2.transpose.map(_.sum) //31
     val newWeights = weights.zip(derivatives)
                       .map{ case (w, df) => w - eta*df)  //32
     newWeights.copyToArray(weights)
     gradientDescent(shuffled, newCost, nIters+1, newWeights)//33
   }
}

val SPAN = 5
def shuffle(labelObs: LabelObs): LabelObs = { 
  shuffle(new ArrayBuffer[Int],0,0).map(labelObs( _ )) //34
}

val maxChunkSize = Random.nextInt(SPAN)+2  //35

@tailrec
def shuffle(indices: ArrayBuffer[Int], count: Int, start: Int): 
      Array[Int] = {
  val end = start + Random.nextInt(maxChunkSize) //36
  val isOdd = ((count & 0x01) != 0x01)
  if(end >= sz) 
    indices.toArray ++ slice(isOdd, start, sz) //37
  else 
    shuffle(indices ++ slice(isOdd, start, end), count+1, end)
}

def slice(isOdd: Boolean, start: Int, end: Int): Array[Int] = {
  val r = Range(start, end).toArray
  (if(isOdd) r else r.reverse)
}

def classify(obs: DblArray): Try[(Int, Double)] = 
  val linear = dot(obs, model.weights)  //37
  val prediction = sigmoid(linear)
  (if(linear > 0.0) 1 else 0, prediction) //38
})

def dot(obs: DblArray, weights: DblArray): Double =
   weights.zip(Array[Double](1.0) ++ obs)
          .aggregate(0.0){case (s, (w,x)) => s + w*x, _ + _ }

def dot(x: DblArray, w: DblArray): Double = 
  x.zip(w.drop(1)).map {case (_x,_w) => _x*_w}.sum + w.head

The first step is to define the configuration parameters for the test: the maximum number of NITERS iterations, the EPS convergence criteria, the ETA learning rate, the decision boundary used to label the BOUNDARY training observations, and the path to the training and test sets:

val NITERS = 800; val EPS = 0.02; val ETA = 0.0001
val path_training = "resources/data/chap1/CSCO.csv"
val path_test = "resources/data/chap1/CSCO2.csv"

The various activities of creating and testing the model, loading, normalizing data, training the model, loading, and classifying test data is organized as a workflow using the monadic composition of the Try class:

for {
  volatilityVol <- load(path_training)    //39
  minMaxVec <- Try(new MinMaxVector(volatilityVol))    //40
  normVolatilityVol <- Try(minMaxVec.normalize(0.0,1.0))//41
  classifier <- logRegr(normVolatilityVol)    //42
  testValues <- load(path_test)    //43
  normTestValue0 <- minMaxVec.normalize(testValues(0))  //44
  class0 <- classifier.classify(normTestValue0)   //45
  normTestValue1 <- minMaxVec.normalize(testValues(1))    
  class1 <- classifier.classify(normTestValues1)
} yield {
   val modelStr = model.toString
   …
}

First, the daily trading volatility and volume for the volatilityVol stock price is loaded from file (line 39). The workflow initializes the multi-dimensional MinMaxVec normalizer (line 40) and uses it to normalize the training set (line 41). The logRegr method instantiates the binomial classifier logistic regression (line 42). The testValues test data is loaded from file (line 43), normalized using MinMaxVec already applied to the training data (line 44), and classified (line 45).

The load method extracts data (observations) of a XVSeries[Double] type from the file. The heavy lifting is done by the extract method (line 46), and then the file handle is closed (line 47) before returning the vector of raw observations:

def load(fileName: String): Try[XVSeries[Double], XSeries[Double]] =  {
  val src =  Source.fromFile(fileName)
  val data = extract(src.getLines.map( _.split(",")).drop(1)) //46
  src.close; data //47
}

The private logRegr method has the following two purposes:

The code is as follows:

def logRegr(obs: XVSeries[Double]): Try[LogBinRegression] = Try {
    val expected = normalize(labels._2).get  //48
    new LogBinRegression(obs, expected, NITERS, ETA, EPS)  //49
}

The method labels observations by evaluating if they belong to any one of the two classes delimited by the BOUNDARY condition, as illustrated in the scatter plot in a previous section.

The training run is performed with three different values of the learning rate. The following chart illustrates the convergence of the batch gradient descent in the minimization of the cost, given different values of learning rates:

Impact of the learning rate on the batch gradient descent on the convergence of the cost (error)

As expected, the execution of the optimizer with a higher learning rate produces a steepest descent in the cost function.

The execution of the test produces the following model:

iters = 495

weights: 0.859-3.6177923,-64.927832

input (0.0088, 4.10E7) normalized (0.063,0.061) class 1 prediction 0.515

input (0.0694, 3.68E8) normalized (0.517,0.641) class 0 prediction 0.001