Hands-On PySpark for Big Data Analysis [Video]

More Information
  • Work on real-life messy datasets with PySpark to get practical Big Data experience
  • Design for both offline and online use cases with Spark Notebooks to increase productivity
  • Analyse and discover patterns with Spark SQL to improve your business intelligence
  • Get rapid-fire feedback with PySpark’s interactive shell to speed up development time
  • Quickly iterate through your solution by setting up PySpark for your own computer
  • Using Spark Notebooks to quickly iterate through your new ideas

Data is an incredible asset, especially when there are lots of it. Exploratory data analysis, business intelligence, and machine learning all depend on processing and analyzing Big Data at scale.

How do you go from working on prototypes on your local machine, to handling messy data in production and at scale?

This is a practical, hands-on course that shows you how to use Spark and it's Python API to create performant analytics with large-scale data. Don't reinvent the wheel, and wow your clients by building robust and responsible applications on Big Data.

All the code and supporting files for this course are available on Github at - https://github.com/PacktPublishing/Hands-On-Pyspark-for-Big-Data-Analysis

Style and Approach

This hands-on course is divided into clear bite-size chunks so you can learn at your own pace and focus on the areas of most interest to you. It’s practical and packed with step-by-step instructions, working examples, and helpful advice from our expert author. You will learn how PySpark provides an easy to use, performant way to do data analysis with Big Data.

  • Work with large amounts of data with agility using distributed datasets and in-memory caching
  • Source data from all popular data hosting platforms, including HDFS, Hive, JSON, and S3
  • Deploy Big Data analytics to production using PySpark’s easy to use API
Course Length 1 hours 52 minutes
ISBN 9781789530056
Date Of Publication 31 Dec 2018


Rudy Lai

Colibri Digital is a technology consultancy company founded in 2015 by James Cross and Ingrid Funie. The company works to help its clients navigate the rapidly changing and complex world of emerging technologies, with deep expertise in areas such as big data, data science, machine learning, and Cloud computing.

Over the past few years, they have worked with some of the World's largest and most prestigious companies, including a tier 1 investment bank, a leading management consultancy group, and one of the World's most popular soft drinks companies, helping each of them to better make sense of its data, and process it in more intelligent ways.
The company lives by its motto: Data -> Intelligence -> Action.

Rudy Lai is the founder of QuantCopy, a sales acceleration startup using AI to write sales emails for prospects. By taking in leads from your pipelines, QuantCopy researches them online and generates sales emails from that data. It also has a suite of email automation tools to schedule, send, and track email performance—key analytics that all feed back into how our AI generates content.

Prior to founding QuantCopy, Rudy ran HighDimension.IO, a machine learning consultancy, where he experienced first-hand the frustrations of outbound sales and prospecting. As a founding partner, he helped startups and enterprises with HighDimension.IO's Machine-Learning-as-a-Service, allowing them to scale up data expertise in the blink of an eye.

In the first part of his career, Rudy spent 5+ years in quantitative trading at leading investment banks such as Morgan Stanley. This valuable experience allowed him to witness the power of data, but also the pitfalls of automation using data science and machine learning. Quantitative trading was also a great platform from which you can learn about reinforcement learning and supervised learning topics in depth and in a commercial setting.

Rudy holds a Computer Science degree from Imperial College London, where he was part of the Dean's List, and received awards such as the Deutsche Bank Artificial Intelligence prize.