Scala for Machine Learning, Second Edition - Second Edition

Functors and monads are important concepts in functional programming.

Functors and monads are derived from category and group theory; they allow developers to create a high-level abstraction, as illustrated in the following Scala libraries:

In mathematics, a category M is a structure that is defined by the following:

Covariant, contravariant functors, and bi-functors are well-understood concepts in algebraic topology that are related to manifold and vector bundles. They are commonly used in differential geometry for the generation of non-linear models.

Higher kinded types (HKTs) are abstractions of types. They generate a new type from existing types. Let's consider the following parameterized trait:

trait M[T] {  .  } 

A higher kinded type H over a trait M is defined as follows:

       trait H[M[_]]; class H[M[_]] 

Functors and monads are higher kinded types.

How are higher kinded types relevant to data analysis and machine learning?

Scientists define observations as sets or vectors of features.

Classification problems rely on the estimation of the similarity between vectors of observations. One technique consists of comparing two vectors by computing the normalized inner (or dot) product. A co-vector is defined as a linear map α of vector to the inner product (field).

Note

Inner product

M1: Definition of inner product <.> and co-vector α:

Let's define a vector as a constructor from any field, _ => Vector[_]. A co-vector is then defined as the mapping function of a vector to its field: Vector[_].

Let's then define a two-dimension (two types or fields) higher kinded structure, Hom, that can be defined as either a vector or a co-vector by fixing one of the two types:

type Hom[T] = {
  type Right[X] = Function1[X,T] // Co-vector
  type Left[X] = Function1[T,X]   // Vector
 }

The projections of the higher-kind Hom to Right or Left single parameter types are known as functors:

A covariant functor is a mapping function, such as F: C => C, with the following properties:

The definition of the covariant functor is F[U => V] := F[U] => F[V]. Its implementation in Scala is:

trait Functor[M[_]]{ def map[U,V](m: M[U])(f: U=>V): M[V] }

For example, let's consider an observation defined as an n dimension vector of type T, Obs[T]. The constructor for the observation can be represented as Function1[T,Obs]. Its functor, ObsFunctor, is implemented as:

trait ObsFunctor[T] extends Functor[(Hom[T])#Left] { self =>
  override def map[U,V](vu: Function1[T,U])(f: U =>V): 
    Function1[T,V] = f.compose(vu)
}

The functor is qualified as a covariant functor because the morphism is applied to the return type of the element of Obs, Function1[T, Obs]. The projection of the two parameters types Hom to a vector is implemented as (Hom[T])#Left.

A contravariant functor is a mapping function, F: C => C, with the following properties:

The definition of the contravariant functor is F[U => V] := F[V] => F[U], as follows:

trait CoFunctor[M[_]]{ def map[U,V](m: M[U])(f: V=>U): M[V]}

Note that the input and output types in the morphism f are reversed from the definition of a covariant functor. The constructor for the co-vector can be represented as Function1[Obs,T]. Its functor, CoObsFunctor, is implemented as:

trait CoObsFunctor[T] extends CoFunctor[(Hom[T])#Right] {
  self =>
      override def map[U,V](vu: Function1[U,T])(f: V =>U): 
       Function1[V,T] = f.andThen(vu)
}

Monads are structures in algebraic topology related to category theory. Monads extend the concept of functors to allow composition known as the monadic composition of morphisms on a single type. They enable the chaining or weaving of computation into a sequence of steps sometimes known as a data pipeline. The collections bundled with the Scala standard library (List, Map…) are constructed as monads [1:1].

Monads provide the ability for those collections to do the following:

The following Scala definition of a monad as a trait illustrates the concept of a higher kinded Monad trait for type M:

trait Monad[M[_]] {
  def unit[T](a: T): M[T]   
  def map[U,V](m: M[U])(f U =>V): M[V]              
  def flatMap[U,V](m: M[U])(f: U =>M[V]): M[V] 
}

Monads are therefore critical in machine learning as they enable the composition of multiple data transformation functions into a sequence or workflow. This property is applicable to any type of complex scientific computation [1:2].

The purpose of data clustering is to partition a collection of data into a number of clusters or data segments. Practically, a clustering algorithm is used to organize observations into clusters by minimizing the distance between observations within a cluster and maximizing the distance between observations across clusters. A clustering algorithm consists of the following steps:

Data clustering is also known as data segmentation or data partitioning.

Dimension reduction techniques aim to find the smallest, yet most relevant, set of features needed to build a reliable model. There are many reasons for reducing the number of features or parameters in a model, from avoiding overfitting to reducing computation costs.

There are many ways to classify the different techniques used to extract knowledge from data using unsupervised learning. The taxonomy breaks down these techniques according to their purpose, although the list is far from being exhaustive, as shown in the following diagram:

Taxonomy of unsupervised learning algorithms

In order to simplify the description of a statistics formula, we adopt the following simplification: the probability of an event X is the same as the probability of the discrete random variable X having a value x: p(X) = p(X=x):

Generative models attempt to fit a joint probability distribution p(X,Y) of two events (or random variables), X and Y, representing two set of observed and hidden variables, x, y. Discriminative models compute the conditional probability p(Y| X) of an event or random variable Y of hidden variables y, given an event or random variable X of observed variables x. Generative models are commonly introduced through Bayes' rule. The conditional probability of an event Y given an event X is computed as the product of the conditional probability of the event X given the event Y and the probability of the event X, normalized by the probability of event Y [1:6].

Note

Bayes' rule

Joint probability for independent random variables X=x and Y=y:

Conditional probability of a random variable Y = y, given X = x:

Bayes' formula

Bayes' rule is the foundation of the Naïve Bayes classifier, which is described in the Introducing the multinomial Naïve Bayes section in Chapter 6, Naïve Bayes Classifiers.

The description and installation instructions for the Eclipse Scala IDE version 4.0 and higher is available at http://scala-ide.org/docs/user/gettingstarted.html.

You can also download the IntelliJ IDEA Scala plugin version 13 or higher from the JetBrains website at http://confluence.jetbrains.com/display/SCA/.

This is a lightweight library that provides developers with a foundation of small, ready-to-use Java classes that can be easily weaved into a machine learning problem. The examples used throughout the book require version 3.5 or higher.

The math library supports the following:

For more information, visit http://commons.apache.org/proper/commons-math.

We need Apache Public License 2.0; the terms are available at https://www.apache.org/licenses/LICENSE-2.0.

The installation and deployment of the Apache Commons Math library are quite simple. The steps are as follows:

the source commons-math3-3.6-src.zip from the source section.

The library supports a variety of configurable plots and charts (scatter, dial, pie, area, bar, box and whisker, stacked, and 3D). We use JFreeChart to display the output of data processing and algorithm throughout the book, but you are encouraged to explore this great library on your own, as time permits.

It is distributed under the terms of the GNU Lesser General Public License (LGPL), which permits its use in proprietary applications.

To install and deploy JFreeChart, perform the following steps:

Most of the Scala classes discussed in the book are parameterized with a type associated to the discrete/categorical value (Int) or continuous value (Double) [1:10]. For this book, context bounds are used instead of view bounds, as follows:

class A[T: ToInt](param: Param//implicit conversion to Int
class C[T: ToDouble](param: Param)//implicit conversion to Double

For the sake of readability of the implementation of algorithms, code non-essential to the understanding of a concept or algorithm, such as error checking, comments, exception, or import, is omitted. The following code elements are shown in the code snippets presented in the book:

The complete list of Scala code elements omitted in the code snippets in the book can be found in the Code snippets format section in the Appendix.

The algorithms presented in this book share the same primitive types, generic operators, and implicit conversions. For the sake of the readability of the code, the following primitive types will be used:

type DblPair = (Double, Double)
type DblArray = Array[Double]
type DblMatrix = Array[DblArray]
type DblVec = Vector[Double]
type XSeries[T] = Vector[T]         // One dimensional vector
type XVSeries[T] = Vector[Array[T]] // multi-dimensional vector

Time series, introduced in the Time series section in Chapter 3, Data Preprocessing, are implemented as XSeries[T] or XVSeries[T] of the parameterized type T. Make a note of these six types; they are used across the entire book.

The conversion between the primitive types listed above and types introduced in the particular library (that is, the Apache Commons Math library) is described in the relevant chapters.

It is usually a good idea to reduce the number of states of an object. A method invocation transitions an object from one state to another. The larger the number of methods or states, the more cumbersome the testing process becomes.

For example, there is no point in creating a model that is not defined (trained). Therefore, making the training of a model as part of the constructor of the class it implements makes a lot of sense. Therefore, the only public methods of a machine learning algorithm are the following:

The objective here is to create a model for stock price using its daily trading volume and volatility. Throughout the book, we will rely on financial data to evaluate and discuss the merits of different data processing and machine learning methods. In this example, the data is extracted from Yahoo Finances using the CSV format with the following fields:

The enumerator YahooFinancials extracts historical daily trading information from the Yahoo finance site:

type Features = Array[Double]
type Weights = Array[Double]
type ObsSet = Vector[Features]
type Fields = Array[String]

object YahooFinancials extends Enumeration {
  type YahooFinancials = Value
  val DATE, OPEN, HIGH, LOW, CLOSE, VOLUME, ADJ_CLOSE=Value
  def toDouble(v: Value): Fields => Double =   //1
     (s: Fields) => s(v.id).toDouble
   def toArray(vs: Array[Value]): Fields => Features = //2 
       (s: Fields) => vs.map(v => s(v.id).toDouble)   
  
}

The method toDouble converts an array of a string into a single value (line 1) and toArray converts an array of a string into an array of values (line 2). The enumerator YahooFinancials is described in detail in the Data sources section in the Appendix.

Let's create a simple program that loads the content of the file, executes some simple preprocessing functions, and creates a simple model. We selected the CSCO stock price between January 1, 2012 and December 1, 2013 as our data input.

Let's consider two variables, price and volume, as illustrated by the following screenshot. The top graph displays the variation of the price of Cisco stock over time and the bottom bar chart represents the daily trading volume on Cisco stock over time:

Price-volume action for Cisco stock 2012-2013

The second step is loading the dataset from local or remote data storage. Typically, a large dataset is loaded from a database or distributed filesystem such as Hadoop Distributed File System (HDFS). The load method takes an absolute path name, extract, and transforms the input data from a file into a time series of type Vector[DblPair]:

def load(fileName: String): Try[Vector[DblPair]] = Try {
  val src =  Source.fromFile(fileName)  //3
  val data = extract(src.getLines.map(_.split(",")).drop(1))//4
  src.close //5
  data
 }

The data file is extracted through a invocation of the static method Source.fromFile (line 3), then the fields are extracted through a map before the header (the first row in the file) is removed using drop (line 4). The file has to be closed to avoid leaking the file handle (line 5).

The purpose of the extract method is to generate a time series of two variables (relative stock volatility and relative stock daily trading volume):

def extract(cols: Iterator[Fields]): ObsSet = {
  val features = Array[YahooFinancials](LOW, HIGH, VOLUME) //6
  val conversion = toArray(features)  //7
  cols.map(conversion(_)).toVector   
      .map(x => Array[Double](1.0 - x(0)/x(1), x(2)))  //8
}

The only purpose of the extract method is to convert the raw textual data into a two-dimension time series. The first step consists of selecting the three features to extract: LOW (lowest stock price in the session), HIGH (highest price in the session), and VOLUME (trading volume for the session) (line 6). This feature set is used to convert each line of the fields into a corresponding set of three values (line 7). Finally, the feature set is reduced to two variables (line 8):

The next step is to normalize the data in the range [0.0, 1.0] to be trained by the binomial logistic regression. It is time to introduce an immutable and flexible normalization class.

Immutable normalization

Logistic regression relies on the sigmoid curve or logistic function described in the Logistic function section in Chapter 9, Regression and Regularization. The logistic function is used to segregate training data into classes. The output value of the logistic function ranges from 0 for x = - INFINTY to 1 for x = + INFINITY. Therefore, it makes sense to normalize the input data or observation over [0, 1].

Note

To normalize or not normalize?

The purpose of normalizing data is to impose a single range of values for all the features, so the model does not favor any particular feature. Normalization techniques include linear normalization and Z-score. Normalization is an expensive operation that is not always needed.

Normalization is a linear transformation on the raw data that can be generalized to any range [l, h].

Note

Linear normalization

M2: [0, 1] Normalization features {x_i} with minimum xmin, maximum xmax values:

M3: [l, h] Normalization of features {x_i}:

The normalization of input data in supervised learning has a specific requirement: the classification and prediction of new observations have to use the normalization parameters (min, max) extracted from the training set, so all observations share the same scaling factor.

Let's define the normalization class MinMax. The class is immutable: the minimum, min, and maximum, max, values are computed within the constructor. The class takes a time series of the parameterized type T values as an argument (line 8). The steps of the normalization process are defined as follows:

1. Initialize the minimum values for a given time series during instantiation (line 9).

2. Compute the normalization parameters (line 10) and normalize the input data (line 11).

3. Normalize any new data point reusing the normalization parameters (line 14):

   class MinMax[T : ToDouble](val values: Vector[T])
     { //8
     val zero = (Double.MaxValue, -Double.MaxValue)
     val (min, max) = values./:(zero){ case ((mn, mx),x) => {
       val z = implicitly[ToDouble[T]].apply(x)
       (if(z < mn) z else mn, if(z > mx) z else mx)  //9
     }}

     case class ScaleFactors(
       low:Double, high:Double, ratio: Double
     )
     
     var scaleFactors: Option[ScaleFactors] = None //10
     def normalize(low: Double, high: Double): Vector[Double]//11
     def normalize(value: Double): Double
   }

The class constructor computes the tuple of minimum and maximum values minMax using a fold (line 9). The scaling parameters scaleFactors are computed during the normalization of the time series (line 11), described as follows. The method normalize initializes the scaling factors parameters (line 12) before normalizing the input data (line 13):

def normalize(low: Double, high: Double): Vector[Double] = 
  setScaleFactors(low, high).map( scale => { //12
    values.map(x =>{
      val z = implicitly[ToDouble[T]].apply(x)
      (z - min)*scale.ratio + scale.low //13
    }) 
  }).getOrElse(/* … */)

def setScaleFactors(l: Double, h: Double): Option[ScaleFactors]={
    // .. error handling code
   Some(ScaleFactors(l, h, (h - l)/(max - min))
}

Subsequent observations use the same scaling factors extracted from the input time series in normalize (line 14):

def normalize(value: Double): Double = setScaleFactors.map(
scale => 
   if(value < min) scale.low
   else if (value > max) scale.high
   else (value - min)* scale.high + scale.low
).getOrElse( /* … */)

The class MinMax normalizes single variable observations.

Note

Statistics class

The class that extracts the basic statistics from a dataset, Stats, introduced in the Profiling data section in Chapter 2, Data Pipelines, inherits the class MinMax.

The test case with the binomial logistic regression uses a multiple variable normalization, implemented by the class MinMaxVector which takes observations of type Vector[Array[Double]] as input:

class MinMaxVector(series: Vector[Double]) {
  val minMaxVector: Vector[MinMax[Double]] = //15
      series.transpose.map(new MinMax[Double](_))
  def normalize(low: Double, high: Double): Vector[Double]
}

The constructor of the class MinMaxVector transposes the vector of an array of observations in order to compute the minimum and maximum values for each dimension (line 15).

The price action chart has a very interesting characteristic.

Analyzing data

At a closer look, a sudden change in price and increase in volume occurs about every 3 months or so. Experienced investors will undoubtedly recognize that these price-volume patterns are related to the release of quarterly earnings of Cisco. Such a regular but unpredictable pattern can be a source of concern or opportunity if risk can be properly managed. The strong reaction of the stock price to the release of corporate earnings may scare some long-term investors while enticing day traders.

The following graph visualizes the potential correlation between sudden price change (volatility) and heavy trading volume:

Price-volume correlation for Cisco stock 2012-2013

The next section is not required for the understanding of the test case. It illustrates the capabilities of JFreeChart as a simple visualization and plotting library.

Plotting data

Although charting is not the primary goal of this book, we thought that you would benefit from a brief introduction to JFreeChart.

Note

Plotting classes

This section illustrates a simple Scala interface to JFreeChart java classes. Its reading is not required for the understanding of machine learning. The visualization of the results of a computation is beyond the scope of this book.

Some of the classes used in visualization are described in the Appendix.

The dataset (volatility, volume) is converted into internal JFreeChart data structures.

The following code snippet defines the key components of a simple scatter plot:

class ScatterPlot(config: PlotInfo, theme: PlotTheme) {//16
  def display(xy: Vector[DblPair], width: Int, height) //17

  // ….
}

The class ScatterPlot implements a simple configurable scatter plot with the following arguments:

• config: Information, labels, and fonts of the plot

• theme: Predefined theme for the plot (black, white background, and so on)

The class PlotTheme defines a specific theme or preconfiguration of the chart (line 16). The class offers a set of methods with the name display to accommodate for a wide range of data structures and configuration (line 17).

Note

Visualization

The JFreeChart library is introduced as a robust charting tool. The code related to plots and charts is omitted throughout the book in order to keep the code snippets concise. On a few occasions, output data is formatted in an CSV file to be imported into a spreadsheet.

Visualizing model features

The ScatterPlot.display method is used to display the normalized input data used in the binomial logistic regression, as follows:

val plot = new ScatterPlot(("CSCO 2012-13 Model features", 
   "Normalized session volatility", "Normalized session Volume"), 
    new BlackPlotTheme)
plot.display(volatilityVolume, 250, 340)

The invocation of the method display generates the following output:

Scatter plot of volatility and volume for Cisco stock 2012-2013

The scatter plot shows some level of correlation between session volume and session volatility and confirms the initial finding in the stock price and volume chart. We can leverage this information to classify trading sessions by their volatility and volume. The next step is to create a two-class model by loading a training set, observations, and expected values into our logistic regression algorithm. The classes are delimited by a decision boundary (also known as a hyperplane) drawn onto the scatter plot.

Visualizing label

The normalized variation of the stock price between the opening and closing of the trading session is selected as the label for this classifier:

Classifier training label: normalized variation of stock price within a trading session

The objective of this training is to build a model that can discriminate between volatile and non-volatile trading sessions. For the sake of the exercise, session volatility is defined as the relative difference between a session's highest price and lowest price. The total trading volume within a session constitutes the second parameter of the model. The relative price movement within a trading session (that is, closing price/open price -1) is our expected value or label.

Logistic regression is commonly used in statistics inference.

Note

Logistic regression model (M4)

Given a model with weight w_i, the margin f and the logistic function l are defined as:

The first weight w₀ is known as the intercept. The binomial logistic regression is described in detail in the Logisticregression section in Chapter 9, Regularization and Regression.

The following implementation of the binomial logistic regression classifier exposes a single method, classify, to comply with our desire to reduce the complexity and life cycle of objects. The model parameters, weights, are computed during training when the class/model LogBinRegression is instantiated. As mentioned earlier, the sections of the code non-essential to the understanding of the algorithm are omitted.

The constructor LogBinRegression has five arguments (line 18):

The model LogBinRegressionModel is generated through training during the instantiation of the logistic regression class, LogBinRegression (line 19):

case class LogBinRegressionModel(
   weights: Weights, 
   losses: List[Double]
)  

The model is fully defined by its weights as described in the mathematical formula M4. The intercept weights(0) represents the mean value of the prediction for observations whose variables are zero. The list losses contain the logistic loss collected at each iteration. It is used for debugging purposes. The intercept does not have a specific meaning in most cases and it is not always computable.

The following code snippet implements the computation of the intercept given a model, Weights:

def intercept(weights: Weights): Double = {
  val zeroObs = obsSet.filter(_.exists(_ > 0.01))
  if( zeroObs.size > 0)
    -zeroObs.aggregate(0.0)(
      (s,z) => s + dot(z, weights), _ + _ 
     )/zeroObs.size
  else 0.0
}

The classify method takes new observations as input and computes the index of the classes (0 or 1) that the observations belong to, along with the actual likelihood (line 20).

Selecting an optimizer

The goal of the training of a model using expected values is to compute the optimal weights that minimize the error or loss function.

Note

Least squares or logistic loss

The sum of least squares loss is more often used for regression problems while the logistic loss is more commonly applied to classification.

We select the Stochastic Gradient Descent (SGD) algorithm to minimize the cumulative error between the predicted and expected values for all the observations. Although there are quite a few alternative optimizers, the SGD is quite robust and simple enough for this first chapter. The algorithm consists of updating the weights wi of the regression model by minimizing the cost.

Note

Cost functions

M5: Logistic loss

M6: SGD method to update model weights at iteration t, wt:

For those interested in learning about about optimization techniques, the Summary of optimization techniques section in the Appendix presents an overview of the most commonly used optimizers. The stochastic descent gradient is used for the training of the multilayer perceptron (refer to the The training epoch subsection in the The multilayer perceptron (MLP) section of Chapter 10, Multilayer Perceptron for more detail).

The execution of the SGD algorithm follows these steps:

1. Initialize the weights of the regression model.

2. Shuffle the order of observations and expected pair of values.

3. Select the first pair of observations and expected value.

4. Compute the loss for this pair.

5. Update the model weights using the derivatives of the loss over each weight.

6. Repeat from step 3 until either the maximum number of iterations is reached or the incremental update of the loss is close to zero.

The purpose of shuffling the order of the observations between iterations is to avoid the minimization of the cost reaching a local minimum.

Note

Batch and SGD

The SGD is a variant of the gradient descent which updates the model weights after computing the error on each observation. Although the SGD requires a higher computation effort to process each observation, it converges toward the optimal value of weights fairly quickly after a small number of iterations. However, the SGD is sensitive to the initial value of the weights and the selection of the learning rate, which is usually defined by an adaptive formula.

def train: LogBinRegressionModel = {
   val init = Array.fill(nWeights)(Random.nextDouble) //22
   val (weights, losses) = sgd(
      0,init, List[Double]()
   )
   new LogBinRegressionModel(weights, losses.reverse)  //23
}

val shuffled = shuffle(observations.zip(expected)) //24
@tailrec
def sgd(   nIters: Int, 
   weights: Weights,//25
   losses: List[Double]): (Weights, List[Double]
 ) = {  //26 
  if(nIters >= maxIters) 
     (weights, losses)  //27
  else {
     val (x, y) = shuffled(nIters % observations.size)
     val (newLoss, grad) = { 
      val yDot = y * margin(x, weights)
       val gradient = derivativeLoss(y, yDot)
         (logisticLoss(yDot),  // 28
           Array[Double](gradient) ++ x.map(_ *gradient) )//29
    }

     if(newLoss < eps)  //30
       (weights, newLoss :: losses)  //31
     else {
       val newWeights = weights.zip(grad).map{ 
          case (w, df) => w - eta*df //33
       } 
       sgd(
         nIters+1, //34
         newWeights,
         newLoss :: losses)
     }
}

def logisticLoss(z: Double): Double = 
  log(1.0 + exp(-z)) / observations.size //30
def derivedLoss(y: Double, yDot: Double):Double = 
 -y / (1.0 + exp(yDot))

def margin(observation: Features, weights: Weights):Double =
  weights.drop(1).zip(observation.view)
             .aggregate(weights.head)(dot, _ + _)

def classify(obs: Features): Try[(Int, Double)] = 
  val linear = margin(obs, model.weights) 
           + model.weights(0)  //37
  val prediction = sigmoid(linear)
  (if(linear > 0.0) 1 else 0, prediction) //38
})

def margin(obs: Features, weights: Weights): Double =
   weights.drop(1).zip(obs.view)
          .aggregate(0.0){case (s, (w,x)) => s + w*x, _ + _ }

The first step is to define the configuration parameters for the test: the maximum number of iterations, NITERS, convergence criterion EPS, learning rate ETA, and decision boundary used to label training observations, BOUNDARY, and the path to the training and test sets:

val NITERS = 4096; val EPS = 0.001; val ETA = 0.0001
val path_training = "supervised/regression/CSCO.csv"
val path_test = "supervised/regression/CSCO2.csv"

The various activities of creating and testing the model, loading, normalizing data, training the model, loading, and classifying test data is organized as a workflow using the monadic composition of the Try class:

for {
     path <- getPath(path_training)
      (volatility, vol) <- load(path)
      minMaxVec <- Try(new MinMaxVector(volatility))
      normVolatilityVol <- Try(minMaxVec.normalize(0.0, 1.0)) 
      classifier <- logRegr(normVolatilityVol, vol)

      testValues <- load(path_test) 
      normTestValue0 <- minMaxVec.normalize(testValues._1(0))
      class0 <- classifier.classify(normTestValue0)
      normTestValue1 <- minMaxVec.normalize(testValues._1(1)) 
      class1 <- classifier.classify(normTestValue1) 
} yield {
   val modelStr = model.toString
}

At first, the daily trading volatility and volume for the stock price (volatility, Vol) pairs are loaded from file (line 39). The workflow initializes the multi-dimensional normalizer, MinMaxVec (line 40), and uses it to normalize the training set (line 41). The logRegr method instantiates the binomial logistic regression, classifier (line 42). The test data, testValues, is loaded from file (line 43), normalized using the MinMaxVec, which has been already applied to training data (line 44) and classified (line 45).

The method load extracts the data (observations) of type XVSeries[Double] from the file. The heavy lifting is done by the extract method (line 46), and then the file handle is closed (line 47) before returning the vector of raw observations:

type Labels = (Vector[Features], Vector[Double])

def load(fileName: String): Try[Labels] =  {
  val src = Source.fromFile(fileName)
  val data = extract(src.getLines.map( _.split(",")).drop(1)) //46
  src.close; data //47
}

The method logRegr, implemented in the following code snippet, has two purposes:

The training run is performed with three different values of the learning rate. The following chart illustrates the convergence of the batch gradient descent in the minimization of the cost given different values of learning rates:

Impact of learning rate on the SGD on the convergence of the loss

As expected, the execution of the optimizer with a higher learning rate produces the steepest descent in the cost function.

The execution of the test produces the following model:

iters = 495
weights: 0.859,-3.617,-6.927
input (0.0088, 4.10E7) normalized (0.063,0.061) class 1 prediction 0.5156
input (0.0694, 3.68E8) normalized (0.517,0.641) class 0 prediction 0.0012

These values may differ between experiments as the initial weights of the model are initialized randomly.