In this chapter, we will cover:
Unsupervised machine is a type of learning technique in which we try to draw inferences either directly or indirectly (through latent factors) from a set of unlabeled observations. In simple terms, we are trying to find the hidden knowledge or structures in a set of data without initially labeling the training data.
While most machine learning library implementation break down when applied to large datasets (iterative, multi-pass, a lot of intermediate writes), the Apache Spark Machine Library succeeds by providing machine library algorithms designed for parallelism and extremely large datasets using memory for intermediate writes out of the box.
At the most abstract level, we can think of unsupervised learning as:
The...
In this recipe, we will load a set of features (for example, x, y, z coordinates) using a LIBSVM file and then proceed to use KMeans() to instantiate an object. We will then set the of desired clusters to three and then use kmeans.fit() to action the algorithm. Finally, we will the centers for the three clusters that we found.
It is really important to note that Spark does not implement KMeans++, contrary to popular literature, instead it implements KMeans || (pronounced as KMeans Parallel). See the following recipe and the sections following the code for a complete explanation of the algorithm as it is implemented in Spark.
package spark.ml.cookbook.chapter8
Log4j.Logger to reduce...In this recipe, we will download the glass and try to identify and label each glass using a bisecting KMeans algorithm. The Bisecting KMeans is a hierarchical version of the K-Mean algorithm implemented in Spark using the BisectingKMeans() API. While this algorithm is conceptually like KMeans, it can considerable speed for some use cases where the hierarchical path is present.
The dataset we used for this recipe is the Glass Identification Database. The study of the classification of types of glass was motivated by criminological research. Glass could be considered as evidence if it is correctly identified. The data can be found at NTU (Taiwan), already in LIBSVM format.
The dataset contains 11 features and 214 rows.
In this recipe, we will Spark's implementation of expectation maximization (EM) GaussianMixture(), which calculates the maximum likelihood given a set of as input. It assumes a Gaussian in which each point can be sampled from K of sub-distributions (cluster memberships).
Start a new project in IntelliJ or in an IDE of your choice. Make sure the necessary JAR files are included.
Set up the package location where the program will reside:
package spark.ml.cookbook.chapter8.
import org.apache.log4j.{Level, Logger} import org.apache.spark.mllib.clustering.GaussianMixture import org.apache.spark.mllib.linalg.Vectors import org.apache.spark.sql.SparkSession
Create Spark's session object:
val spark = SparkSession .builder .master("local[*]") .appName("myGaussianMixture") .config("spark.sql.warehouse.dir", ".") .getOrCreate()
Let us take...
This is a classification method for the vertices of a given their similarities as defined by their edges. It uses the GraphX library which is ships out of the box with Spark to implement the algorithm. Power Iteration Clustering is similar to other Eigen Vector/Eigen Value algorithms, but without the overhead of matrix decomposition. It is suitable when you have a large sparse matrix (for example, graphs depicted as a sparse matrix).
GraphFrames will be the replacement/interface proper for the GraphX library going forward (https://databricks.com/blog/2016/03/03/introducing-graphframes.html).
Start a new project in IntelliJ or in an IDE of your choice. Make sure the necessary JAR files are included.
Set up the package location where the program will reside:
package spark.ml.cookbook.chapter8
Log4j.Logger to reduce...In this recipe, we will explore the Latent Dirichlet Allocation (LDA) algorithm in Spark 2.0. The LDA we use in this recipe is completely different from linear analysis. Both Latent Dirichlet Allocation and linear discrimination analysis are referred to as LDA, but they are extremely different techniques. In this recipe, when we use the LDA, we refer to Latent Dirichlet Allocation. The chapter on text analytics is also relevant to understanding the LDA.
LDA is often used in natural language processing which tries to classify a large body of document (for example, emails from the Enron fraud case) into a discrete number of topics or themes so it can be understood. LDA is also a good candidate for selecting articles based on one's interest (for example, as you turn a page and spend time on a specific topic) in a given magazine article or page.
Spark streaming is a powerful facility lets you combine near real time and batch in the same paradigm. The streaming KMeans interface lives at the intersection of ML clustering and Spark streaming, and takes full advantage of the core facilities provided by Spark streaming itself (for example, fault tolerance, exactly once delivery semantics, and so on).
package spark.ml.cookbook.chapter8.
import org.apache.log4j.{Level, Logger} import org.apache.spark.mllib.clustering.StreamingKMeans import org.apache.spark.mllib.linalg.Vectors import org.apache.spark.mllib.regression.LabeledPoint import org.apache.spark.sql.SparkSession import org.apache.spark.streaming.{Seconds, StreamingContext}
We set up the following parameters...
© 2017 Packt Publishing Limited All Rights Reserved
