Big Data Processing with Apache Spark

Efficiently tackle large datasets and big data analysis with Spark and Python

Manuel Ignacio Franco Galeano

Data Processing with Apache Spark

Copyright © 2018 Packt Publishing

All rights reserved. No part of this course may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews.

Every effort has been made in the preparation of this course to ensure the accuracy of the information presented. However, the information contained in this course is sold without warranty, either express or implied. Neither the author, nor Packt Publishing, and its dealers and distributors will be held liable for any damages caused or alleged to be caused directly or indirectly by this course.

Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this course by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information.

Author: Manuel Franco

Reviewer: Amit Nandi

Managing Editor: Edwin Moses

Acquisitions Editor: Aditya Date

Production Editor: Nitesh Thakur

Editorial Board: David Barnes, Ewan Buckingham, Simon Cox, Manasa Kumar, Alex Mazonowicz, Douglas Paterson, Dominic Pereira, Shiny Poojary, Saman Siddiqui, Erol Staveley, Ankita Thakur, and Mohita Vyas

First Published: October 2018

Production Reference: 1311018

ISBN: 978-1-78980-881-0

Table of Contents

Preface   i

Introduction to Spark Distributed Processing   1

Introduction   2

Introduction to Spark and Resilient Distributed Datasets   4

Spark Components   5

Spark Deployment Modes   5

Spark Standalone   6

Apache Mesos   6

Other Deployment Options   7

Resilient Distributed Datasets   7

Python Shell and SparkContext   8

Parallelized Collections   8

RDD Creation from External Data Sources   9

Exercise 1: Basic Interactive Analysis with Python   10

Operations Supported by the RDD API   12

Map Transformations   12

Reduce Action   13

Working with Key-Value Pairs   13

Join Transformations   13

Set Operations   14

Exercise 2: Map Reduce Operations   14

Activity 1: Statistical Operations on Books   19

Self-Contained Python Spark Programs   20

Introduction to Functional Programming   21

Exercise 3: Standalone Python Programs   23

Introduction to SQL, Datasets, and DataFrames   27

Exercise 4: Downloading the Reduced Version of the movielens Dataset   28

Exercise 5: RDD Operations in DataFrame Objects   31

Summary   32

Introduction to Spark Streaming   35

Introduction   36

Introduction to Streaming Architectures   36

Back-Pressure, Write-Ahead Logging, and Checkpointing   38

Introduction to Discretized Streams   38

Consuming Streams from a TCP Socket   39

TCP Input DStream   39

Map-Reduce Operations over DStreams   40

Exercise 6: Building an Event TCP Server   41

Activity 2: Building a Simple TCP Spark Stream Consumer   43

Parallel Recovery of State with Checkpointing   44

Keeping the State in Streaming Applications   45

Join Operations   46

Exercise 7: TCP Stream Consumer from Multiple Sources   47

Activity 3: Consuming Event Data from Three TCP Servers   50

Windowing Operations   51

Exercise 8: Distributed Log Server   51

Introduction to Structured Streaming   59

Result Table and Output Modes in Structured Streaming   60

Exercise 9: Writing Random Ratings   61

Exercise 10: Structured Streaming   64

Summary   65

Spark Streaming Integration with AWS   67

Introduction   68

Spark Integration with AWS Services   68

Previous Requirements   68

AWS Kinesis Data Streams Basic Functionality   69

Integrating AWS Kinesis and Python   70

Exercise 11: Listing Existing Streams   70

Exercise 12: Creating a New Stream   72

Exercise 13: Deleting an Existing Stream   73

Exercise 14: Pushing Data to a Stream   75

AWS S3 Basic Functionality   77

Creating, Listing, and Deleting AWS S3 Buckets   78

Exercise 15: Listing Existing Buckets   78

Exercise 16: Creating a Bucket   80

Exercise 17: Deleting a Bucket   81

Kinesis Streams and Spark Streams   83

Activity 4: AWS and Spark Pipeline   84

Summary   86

Spark Streaming, ML, and Windowing Operations   89

Introduction   90

Spark Integration with Machine Learning   90

The MovieLens Dataset   90

Introduction to Recommendation Systems and Collaborative Filtering   91

Exercise 18: Collaborative Filtering and Spark   92

Exercise 19: Creating a TCP Server that Publishes User Ratings   96

Exercise 20: Spark Streams Integration with Machine Learning   99

Activity 5: Experimenting with Windowing Operations   101

Summary   102

Appendix A   105