In the first part of the book, which sets the foundations, you can follow along without participating in hands-on examples (although it’s recommended). The latter part of the book will be more practice-driven. If you want to get hands-on from the beginning, the code for this chapter can be found in the GitHub repository of this book at:

https://github.com/PacktPublishing/Time-Series-Analysis-with-Spark/tree/main/ch1

Note

Refer to this GitHub repository for the latest revisions of the code, which will be commented on if updated post-publication. The updated code (if any) might differ from what is presented in the book's code sections.

The following hands-on sections will give you further details to get started with time series analysis.

In this section, we will develop an understanding of what time series are and some related terms. This will be illustrated by hands-on examples to visualize time series. We will look at different types of time series and what characterizes them. This knowledge of the nature of time series is necessary for us to choose the appropriate time series analysis approach in the upcoming chapters.

Let’s start with an example of a time series with the average temperature in Mauritius every year since 1950. A short sample of the data is shown in Table 1.1.

Let’s go through the hands-on exercise to load a time series dataset and visualize it. We will try to create the visual representation we’ve already seen in Figure 1.1.

Development environment

In order to run the code, you will need a Python development environment where you can install Apache Spark and other required libraries. Specific libraries will be detailed, together with installation instructions, in the corresponding chapters when required.

PaaS

An easy way to get going with these requirements is by using Databricks Community Edition, which is free. This comes with a notebook-based development interface, as well as compute with pre-installed Spark and some other libraries.

The instructions to sign up for Databricks Community Edition can be found here:

https://docs.databricks.com/en/getting-started/community-edition.html

Community Edition’s compute size is limited as it is a free cloud-based...

This section aims to further your understanding of a time series by analyzing its components and detailing several terms introduced so far. This will set you on track for the rest of the book, to use the right methods based on the nature of the time series you are analyzing.

Time series models can be broken down into three main components: trend, seasonality, and residuals:

Note

The mathematical representations in this book will follow a simplified English notation, in favour of a broad audience. Refer to the following great resource on time series for mathematical formulations: Forecasting: Principles and Practice: https://otexts.com/fpp3/.

As you will see in the next hands-on section, this breakdown into components is derived from the model fitted to the time series data. For most real-life datasets, the breakdown is only an approximation of reality by the model. As such, each model will come up with its own identification...

We will be going through the code to create the data visualization in Figure 1.13. The code for this section is in the notebook file named ts-spark_ch1_3.dbc.

The location URL is as follows: https://github.com/PacktPublishing/Time-Series-Analysis-with-Spark/raw/main/ch1/ts-spark_ch1_3.dbc

The dataset is synthetic and generated as three different sine curves representing three overlapping seasonalities.

The following code is an extract from the notebook. Let’s look at it at a high level:

1. The import statements add libraries for numerical calculations and for drawing graphs:

   import numpy as np
   from plotly.subplots import make_subplots

   NumPy is an open source Python library for scientific computing significantly more efficient in terms of computation and memory use than standard Python. We will use it here for its mathematical functions.

2. We then generate a number of sine curves, using np.sin, to represent different seasonalities...

This section is probably the most important in this early part of the book. In the introductory section, we mentioned some key considerations for time series, such as the preservation of chronological order, regularity, and stationarity. Here, we map out the key challenges and additional considerations when analyzing time series in real-life projects. In doing so, it allows you to plan your learning and practice accordingly, with guidance in the relevant sections of this book as well as further reading.

According to Hidden Technical Debt in Machine Learning Systems a well-known paper published in 2015, only a fraction of the effort is with the code in advanced analytics projects. The rest of the time is mostly spent on other considerations such as data preparation and infrastructure.

The solutions to these challenges are very specific to your context. The aim in this chapter is to bring these considerations, as summarized in...

Time series are everywhere, and this chapter gave us an introduction to what they are, their components, and the challenges in working with them. We started with some simple code to explore time series, setting the foundation for further practice in upcoming chapters. The concepts discussed in this first chapter will be built upon to get us to the point of analyzing time series at scale by the end of this book.

Now that you understand the “what” for time series, in the next chapter, we will be looking at the “why,” which will pave the way to applications in various domains.

This section serves as a repository of sources that can help you build on your understanding of the topic:

• Climate Chaos Helped Spark the French Revolution: https://time.com/6107671/french-revolution-history-climate/
• Databricks Community Edition: https://docs.databricks.com/en/getting-started/community-edition.html
• Climate Change Knowledge Portal: https://climateknowledgeportal.worldbank.org/country/mauritius
• Forecasting: Principles and Practice by Rob J Hyndman and George Athanasopoulos: https://otexts.com/fpp3/
• Hidden Technical Debt in Machine Learning Systems (Sculley et al., 2015): https://papers.neurips.cc/paper/5656-hidden-technical-debt-in-machine-learning-systems.pdf