Mastering Apache Spark 2.x - Second Edition

Advanced analytics on your Big Data with latest Apache Spark 2.x

Mastering Apache Spark 2.x - Second Edition

This ebook is included in a Mapt subscription
Romeo Kienzler

Advanced analytics on your Big Data with latest Apache Spark 2.x
$0.00
$34.00
$49.99
$29.99p/m after trial
RRP $39.99
RRP $49.99
Subscription
eBook
Print + eBook
Start 30 Day Trial
Subscribe and access every Packt eBook & Video.
 
  • 4,000+ eBooks & Videos
  • 40+ New titles a month
  • 1 Free eBook/Video to keep every month
Start Free Trial
 
Preview in Mapt

Book Details

ISBN 139781786462749
Paperback332 pages

Book Description

Apache Spark is an in-memory cluster-based parallel processing system that provides a wide range of functionalities such as graph processing, machine learning, stream processing, and SQL. This book aims to take your knowledge of Spark to the next level by teaching you how to expand Spark’s functionality and implement your data flows and machine/deep learning programs on top of the platform.

The book commences with an overview of the Spark ecosystem. It will introduce you to Project Tungsten and Catalyst, two of the major advancements of Apache Spark 2.x.

You will understand how memory management and binary processing, cache-aware computation, and code generation are used to speed things up dramatically. The book extends to show how to incorporate H20, SystemML, and Deeplearning4j for machine learning, and Jupyter Notebooks and Kubernetes/Docker for cloud-based Spark. During the course of the book, you will learn about the latest enhancements to Apache Spark 2.x, such as interactive querying of live data and unifying DataFrames and Datasets.

You will also learn about the updates on the APIs and how DataFrames and Datasets affect SQL, machine learning, graph processing, and streaming. You will learn to use Spark as a big data operating system, understand how to implement advanced analytics on the new APIs, and explore how easy it is to use Spark in day-to-day tasks.

Table of Contents

Chapter 1: A First Taste and What’s New in Apache Spark V2
Spark machine learning
Spark Streaming
Spark SQL
Spark graph processing
Extended ecosystem
What's new in Apache Spark V2?
Cluster design
Cluster management
Cloud-based deployments
Performance
Cloud
Summary
Chapter 2: Apache Spark SQL
The SparkSession--your gateway to structured data processing
Importing and saving data
Understanding the DataSource API
DataFrames
Using SQL
Using Datasets
User-defined functions
RDDs versus DataFrames versus Datasets
Summary
Chapter 3: The Catalyst Optimizer
Understanding the workings of the Catalyst Optimizer
Managing temporary views with the catalog API
The SQL abstract syntax tree
How to go from Unresolved Logical Execution Plan to Resolved Logical Execution Plan
Code generation
Summary
Chapter 4: Project Tungsten
Memory management beyond the Java Virtual Machine Garbage Collector
Cache-friendly layout of data in memory
Code generation
Summary
Chapter 5: Apache Spark Streaming
Overview
Errors and recovery
Streaming sources
Summary
Chapter 6: Structured Streaming
The concept of continuous applications
Windowing
Increased performance with good old friends
How transparent fault tolerance and exactly-once delivery guarantee is achieved
Example - connection to a MQTT message broker
Summary
Chapter 7: Apache Spark MLlib
Architecture
Classification with Naive Bayes
Clustering with K-Means
Artificial neural networks
Summary
Chapter 8: Apache SparkML
What does the new API look like?
The concept of pipelines
Model evaluation
CrossValidation and hyperparameter tuning
Winning a Kaggle competition with Apache SparkML
Summary
Chapter 9: Apache SystemML
Why do we need just another library?
A cost-based optimizer for machine learning algorithms
Performance measurements
Apache SystemML in action
Summary
Chapter 10: Deep Learning on Apache Spark with DeepLearning4j and H2O
H2O
Deeplearning4j
Summary
Chapter 11: Apache Spark GraphX
Overview
Graph analytics/processing with GraphX
Summary
Chapter 12: Apache Spark GraphFrames
Architecture
Examples
Summary
Chapter 13: Apache Spark with Jupyter Notebooks on IBM DataScience Experience
Why notebooks are the new standard
Learning by example
Summary
Chapter 14: Apache Spark on Kubernetes
Bare metal, virtual machines, and containers
Understanding the core concepts of Docker
Understanding Kubernetes
Using Kubernetes for provisioning containerized Spark applications
Example--Apache Spark on Kubernetes
Summary

What You Will Learn

  • Examine Advanced Machine Learning and DeepLearning with MLlib, SparkML, SystemML, H2O and DeepLearning4J
  • Study highly optimised unified batch and real-time data processing using SparkSQL and Structured Streaming
  • Evaluate large-scale Graph Processing and Analysis using GraphX and GraphFrames
  • Apply Apache Spark in Elastic deployments using Jupyter and Zeppelin Notebooks, Docker, Kubernetes and the IBM Cloud
  • Understand internal details of cost based optimizers used in Catalyst, SystemML and GraphFrames
  • Learn how specific parameter settings affect overall performance of an Apache Spark cluster
  • Leverage Scala, R and python for your data science projects

Authors

Table of Contents

Chapter 1: A First Taste and What’s New in Apache Spark V2
Spark machine learning
Spark Streaming
Spark SQL
Spark graph processing
Extended ecosystem
What's new in Apache Spark V2?
Cluster design
Cluster management
Cloud-based deployments
Performance
Cloud
Summary
Chapter 2: Apache Spark SQL
The SparkSession--your gateway to structured data processing
Importing and saving data
Understanding the DataSource API
DataFrames
Using SQL
Using Datasets
User-defined functions
RDDs versus DataFrames versus Datasets
Summary
Chapter 3: The Catalyst Optimizer
Understanding the workings of the Catalyst Optimizer
Managing temporary views with the catalog API
The SQL abstract syntax tree
How to go from Unresolved Logical Execution Plan to Resolved Logical Execution Plan
Code generation
Summary
Chapter 4: Project Tungsten
Memory management beyond the Java Virtual Machine Garbage Collector
Cache-friendly layout of data in memory
Code generation
Summary
Chapter 5: Apache Spark Streaming
Overview
Errors and recovery
Streaming sources
Summary
Chapter 6: Structured Streaming
The concept of continuous applications
Windowing
Increased performance with good old friends
How transparent fault tolerance and exactly-once delivery guarantee is achieved
Example - connection to a MQTT message broker
Summary
Chapter 7: Apache Spark MLlib
Architecture
Classification with Naive Bayes
Clustering with K-Means
Artificial neural networks
Summary
Chapter 8: Apache SparkML
What does the new API look like?
The concept of pipelines
Model evaluation
CrossValidation and hyperparameter tuning
Winning a Kaggle competition with Apache SparkML
Summary
Chapter 9: Apache SystemML
Why do we need just another library?
A cost-based optimizer for machine learning algorithms
Performance measurements
Apache SystemML in action
Summary
Chapter 10: Deep Learning on Apache Spark with DeepLearning4j and H2O
H2O
Deeplearning4j
Summary
Chapter 11: Apache Spark GraphX
Overview
Graph analytics/processing with GraphX
Summary
Chapter 12: Apache Spark GraphFrames
Architecture
Examples
Summary
Chapter 13: Apache Spark with Jupyter Notebooks on IBM DataScience Experience
Why notebooks are the new standard
Learning by example
Summary
Chapter 14: Apache Spark on Kubernetes
Bare metal, virtual machines, and containers
Understanding the core concepts of Docker
Understanding Kubernetes
Using Kubernetes for provisioning containerized Spark applications
Example--Apache Spark on Kubernetes
Summary

Book Details

ISBN 139781786462749
Paperback332 pages
Read More

Read More Reviews