Learning PySpark

Build data-intensive applications locally and deploy at scale using the combined powers of Python and Spark 2.0
Preview in Mapt

Learning PySpark

Tomasz Drabas, Denny Lee

1 customer reviews
Build data-intensive applications locally and deploy at scale using the combined powers of Python and Spark 2.0
Mapt Subscription
FREE
$29.99/m after trial
eBook
$10.00
RRP $35.99
Save 72%
Print + eBook
$44.99
RRP $44.99
What do I get with a Mapt Pro subscription?
  • Unlimited access to all Packt’s 5,000+ eBooks and Videos
  • Early Access content, Progress Tracking, and Assessments
  • 1 Free eBook or Video to download and keep every month after trial
What do I get with an eBook?
  • Download this book in EPUB, PDF, MOBI formats
  • DRM FREE - read and interact with your content when you want, where you want, and how you want
  • Access this title in the Mapt reader
What do I get with Print & eBook?
  • Get a paperback copy of the book delivered to you
  • Download this book in EPUB, PDF, MOBI formats
  • DRM FREE - read and interact with your content when you want, where you want, and how you want
  • Access this title in the Mapt reader
What do I get with a Video?
  • Download this Video course in MP4 format
  • DRM FREE - read and interact with your content when you want, where you want, and how you want
  • Access this title in the Mapt reader
$0.00
$10.00
$44.99
$29.99 p/m after trial
RRP $35.99
RRP $44.99
Subscription
eBook
Print + eBook
Start 30 Day Trial

Frequently bought together


Learning PySpark Book Cover
Learning PySpark
$ 35.99
$ 10.00
Mastering Machine Learning Algorithms Book Cover
Mastering Machine Learning Algorithms
$ 35.99
$ 10.00
Buy 2 for $20.00
Save $51.98
Add to Cart

Book Details

ISBN 139781786463708
Paperback274 pages

Book Description

Apache Spark is an open source framework for efficient cluster computing with a strong interface for data parallelism and fault tolerance. This book will show you how to leverage the power of Python and put it to use in the Spark ecosystem. You will start by getting a firm understanding of the Spark 2.0 architecture and how to set up a Python environment for Spark.

You will get familiar with the modules available in PySpark. You will learn how to abstract data with RDDs and DataFrames and understand the streaming capabilities of PySpark. Also, you will get a thorough overview of machine learning capabilities of PySpark using ML and MLlib, graph processing using GraphFrames, and polyglot persistence using Blaze. Finally, you will learn how to deploy your applications to the cloud using the spark-submit command.

By the end of this book, you will have established a firm understanding of the Spark Python API and how it can be used to build data-intensive applications.

Table of Contents

Chapter 1: Understanding Spark
What is Apache Spark?
Spark Jobs and APIs
Spark 2.0 architecture
Summary
Chapter 2: Resilient Distributed Datasets
Internal workings of an RDD
Creating RDDs
Global versus local scope
Transformations
Actions
Summary
Chapter 3: DataFrames
Python to RDD communications
Catalyst Optimizer refresh
Speeding up PySpark with DataFrames
Creating DataFrames
Simple DataFrame queries
Interoperating with RDDs
Querying with the DataFrame API
Querying with SQL
DataFrame scenario – on-time flight performance
Spark Dataset API
Summary
Chapter 4: Prepare Data for Modeling
Checking for duplicates, missing observations, and outliers
Getting familiar with your data
Visualization
Summary
Chapter 5: Introducing MLlib
Overview of the package
Loading and transforming the data
Getting to know your data
Creating the final dataset
Predicting infant survival
Summary
Chapter 6: Introducing the ML Package
Overview of the package
Predicting the chances of infant survival with ML
Parameter hyper-tuning
Other features of PySpark ML in action
Summary
Chapter 7: GraphFrames
Introducing GraphFrames
Installing GraphFrames
Preparing your flights dataset
Building the graph
Executing simple queries
Understanding vertex degrees
Determining the top transfer airports
Understanding motifs
Determining airport ranking using PageRank
Determining the most popular non-stop flights
Using Breadth-First Search
Visualizing flights using D3
Summary
Chapter 8: TensorFrames
What is Deep Learning?
What is TensorFlow?
Introducing TensorFrames
TensorFrames – quick start
Summary
Chapter 9: Polyglot Persistence with Blaze
Installing Blaze
Polyglot persistence
Abstracting data
Data operations
Summary
Chapter 10: Structured Streaming
What is Spark Streaming?
Why do we need Spark Streaming?
What is the Spark Streaming application data flow?
Simple streaming application using DStreams
A quick primer on global aggregations
Introducing Structured Streaming
Summary
Chapter 11: Packaging Spark Applications
The spark-submit command
Deploying the app programmatically
Databricks Jobs
Summary

What You Will Learn

  • Learn about Apache Spark and the Spark 2.0 architecture
  • Build and interact with Spark DataFrames using Spark SQL
  • Learn how to solve graph and deep learning problems using GraphFrames and TensorFrames respectively
  • Read, transform, and understand data and use it to train machine learning models
  • Build machine learning models with MLlib and ML
  • Learn how to submit your applications programmatically using spark-submit
  • Deploy locally built applications to a cluster

Authors

Table of Contents

Chapter 1: Understanding Spark
What is Apache Spark?
Spark Jobs and APIs
Spark 2.0 architecture
Summary
Chapter 2: Resilient Distributed Datasets
Internal workings of an RDD
Creating RDDs
Global versus local scope
Transformations
Actions
Summary
Chapter 3: DataFrames
Python to RDD communications
Catalyst Optimizer refresh
Speeding up PySpark with DataFrames
Creating DataFrames
Simple DataFrame queries
Interoperating with RDDs
Querying with the DataFrame API
Querying with SQL
DataFrame scenario – on-time flight performance
Spark Dataset API
Summary
Chapter 4: Prepare Data for Modeling
Checking for duplicates, missing observations, and outliers
Getting familiar with your data
Visualization
Summary
Chapter 5: Introducing MLlib
Overview of the package
Loading and transforming the data
Getting to know your data
Creating the final dataset
Predicting infant survival
Summary
Chapter 6: Introducing the ML Package
Overview of the package
Predicting the chances of infant survival with ML
Parameter hyper-tuning
Other features of PySpark ML in action
Summary
Chapter 7: GraphFrames
Introducing GraphFrames
Installing GraphFrames
Preparing your flights dataset
Building the graph
Executing simple queries
Understanding vertex degrees
Determining the top transfer airports
Understanding motifs
Determining airport ranking using PageRank
Determining the most popular non-stop flights
Using Breadth-First Search
Visualizing flights using D3
Summary
Chapter 8: TensorFrames
What is Deep Learning?
What is TensorFlow?
Introducing TensorFrames
TensorFrames – quick start
Summary
Chapter 9: Polyglot Persistence with Blaze
Installing Blaze
Polyglot persistence
Abstracting data
Data operations
Summary
Chapter 10: Structured Streaming
What is Spark Streaming?
Why do we need Spark Streaming?
What is the Spark Streaming application data flow?
Simple streaming application using DStreams
A quick primer on global aggregations
Introducing Structured Streaming
Summary
Chapter 11: Packaging Spark Applications
The spark-submit command
Deploying the app programmatically
Databricks Jobs
Summary

Book Details

ISBN 139781786463708
Paperback274 pages
Read More
From 1 reviews

Read More Reviews

Recommended for You

Modern Python Cookbook Book Cover
Modern Python Cookbook
$ 39.99
$ 10.00
Frank Kane's Taming Big Data with Apache Spark and Python Book Cover
Frank Kane's Taming Big Data with Apache Spark and Python
$ 31.99
$ 10.00
Python Data Analysis Cookbook Book Cover
Python Data Analysis Cookbook
$ 39.99
$ 10.00
Apache Spark 2.x Cookbook Book Cover
Apache Spark 2.x Cookbook
$ 39.99
$ 10.00
Python: End-to-end Data Analysis Book Cover
Python: End-to-end Data Analysis
$ 71.99
$ 10.00
Python Data Analysis - Second Edition Book Cover
Python Data Analysis - Second Edition
$ 39.99
$ 10.00