Handling and Manipulating Date, Time, and Time Series Data

Time series data is ubiquitous when it comes to algorithmic trading. So, handling, managing, and manipulating time series data is essential to performing algorithmic trading successfully. This chapter has various recipes that demonstrate how algorithmic trading can be done using the Python standard library and pandas, which is a Python data analysis library.

For our context, time series data is a series of data consisting of equally spaced timestamps and multiple data points describing trading data in that particular time frame.

When handling time series data, the first thing you should know is how to read, modify, and create Python objects that understand date and time. The Python standard library includes the datetime module, which provides the datetime and timedelta objects, which can handle everything about the date...

Technical requirements

You will need the following to successfully execute the recipes in this chapter:

• Python 3.7+
• Python package:
• pandas ($ pip install pandas)

For all the recipes in this chapter, you will need the Jupyter notebook for this chapter, found at https://github.com/PacktPublishing/Python-Algorithmic-Trading-Cookbook/tree/master/Chapter01.

You can also open a new Jupyter notebook and try the hands-on exercises directly as they are shown in the recipes. Note that the output for some of these recipes might differ for you as they depend on the date, time, and time zone information provided at the time.

Creating datetime objects

The datetime module provides a datetime class, which can be used to accurately capture information relating to timestamps, dates, times, and time zones. In this recipe, you will create datetime objects in multiple ways and introspect their attributes.

How to do it…

Follow these steps to execute this recipe:

1. Import the necessary module from the Python standard library:

>>> from datetime import datetime

2. Create a datetime object holding the current timestamp using the now() method and print it:

>>> dt1 = datetime.now()
>>> print(f'Approach #1: {dt1}')

We get the following output. Your output will differ:

Approach #1: 2020-08-12 20:55:39.680195

3. Print the attributes of dt1 related to date and time:

>>> print(f'Year: {dt1.year}')
>>> print(f'Month: {dt1.month}')
>>> print(f'Day: {dt1.day}')
>>> print(f'Hours: {dt1.hour}')
>>> print(f&apos...

Creating timedelta objects

The datetime module provides a timedelta class, which can be used to represent information related to date and time differences. In this recipe, you will create timedelta objects and perform operations on them.

How to do it…

Follow along with these steps to execute this recipe:

1. Import the necessary module from the Python standard library:

>>> from datetime import timedelta

2. Create a timedelta object with a duration of 5 days. Assign it to td1 and print it:

>>> td1 = timedelta(days=5)
>>> print(f'Time difference: {td1}')

We get the following output:

Time difference: 5 days, 0:00:00

3. Create a timedelta object with a duration of 4 days. Assign it to td2 and print it:

>>> td2 = timedelta(days=4)
>>> print(f'Time difference: {td2}')

We get the following output:

Time difference: 4 days, 0:00:00

4. Add td1 and td2 and print the output:

>>> print(f'Addition: {td1} + {td2} = {td1 ...

Operations on datetime objects

The datetime and timedelta classes support various mathematical operations to get dates in the future or the past. Using these operations returns another datetime object. . In this recipe, you would create datetime, date, time, and timedelta objects and perform mathematical operations on them.

How to do it…

Follow along with these steps to execute this recipe:

1. Import the necessary modules from the Python standard library:

>>> from datetime import datetime, timedelta

2. Fetch today's date. Assign it to date_today and print it:

>>> date_today = date.today()              
>>> print(f"Today's Date: {date_today}")

We get the following output. Your output may differ:

Today's Date: 2020-08-12

3. Add 5 days to today's date using a timedelta object. Assign it to date_5days_later and print it:

>>> date_5days_later = date_today + timedelta(days=5)
>>> print(f"Date 5 days later: {date_5days_later...

Modifying datetime objects

Often, you may want to modify existing datetime objects to represent a different date and time. This recipe includes code to demonstrate this.

How to do it…

Follow these steps to execute this recipe:

1. Import the necessary modules from the Python standard library:

>>> from datetime import datetime

2. Fetch the current timestamp. Assign it to dt1 and print it:

>>> dt1 = datetime.now()
>>> print(dt1)

We get the following output. Your output would differ:

2020-08-12 20:55:46.753899

3. Create a new datetime object by replacing the year, month, and day attributes of dt1. Assign it to dt2 and print it :

>>> dt2 = dt1.replace(year=2021, month=1, day=1)
>>> print(f'A timestamp from 1st January 2021: {dt2}')

We get the following output. Your output would differ:

A timestamp from 1st January 2021: 2021-01-01 20:55:46.753899

4. Create a new datetime object by specifying all the attributes directly. Assign it to dt3...

Converting a datetime object to a string

This recipe demonstrates the conversion of the datetime objects into strings which finds application in printing and logging. Also, this is helpful while sending timestamps as JSON data over web APIs.

How to do it…

Execute the following steps for this recipe:

1. Import the necessary modules from the Python standard library:

>>> from datetime import datetime

2. Fetch the current timestamp along with time zone information. Assign it to now and print it:

>>> now = datetime.now().astimezone()

3. Cast now to a string and print it::

>>> print(str(now))

We get the following output. Your output may differ:

2020-08-12 20:55:48.366130+05:30

4. Convert now to a string with a specific date-time format using strftime() and print it:

>>> print(now.strftime("%d-%m-%Y %H:%M:%S %Z"))

We get the following output. Your output may differ:

12-08-2020 20:55:48 +0530

How it works...

In step 1, you import the datetime class...

Creating a datetime object from a string

This recipe demonstrates the conversion of well-formatted strings into datetime objects. This finds application in reading timestamps from a file. Also, this is helpful while receiving timestamps as JSON data over web APIs.

How to do it…

Execute the following steps for this recipe:

1. Import the necessary modules from the Python standard library:

>>> from datetime import datetime

2. Create a string representation of timestamp with date, time, and time zone. Assign it to now_str:

>>> now_str = '13-1-2021 15:53:39 +05:30'

3. Convert now_str to now, a datetime.datetime object. Print it:

>>> now = datetime.strptime(now_str, "%d-%m-%Y %H:%M:%S %z")
>>> print(now)

We get the following output:

2021-01-13 15:53:39+05:30

4. Confirm that now is of the datetime type:

>>> print(type(now))

We get the following output:

<class 'datetime.datetime'>

How it works...

In step 1, you...

The datetime object and time zones

There are two types of datetime objects—time zone-naive and time zone-aware. Time zone-naive objects do not hold time zone information and timezone-aware objects hold time zone information. This recipe demonstrates multiple time zone related operations on datetime objects: creating time zone-naive and time zone-aware objects, adding time zone information to time zone-aware objects, removing time zone information from time zone-naive objects, and comparing time zone-aware and time zone-naive objects.

How to do it…

Execute the following steps for this recipe:

1. Import the necessary modules from the Python standard library:

>>> from datetime import datetime

2. Create a time zone-naive datetime object. Assign it to now_tz_naive and print it:

>>> now_tz_unaware = datetime.now()
>>> print(now_tz_unaware)

We get the following output. Your output may differ:

2020-08-12 20:55:50.598800

3. Print the time zone information of...

Creating a pandas.DataFrame object

Now that we are done with handling date and time, let's move on to handling time series data. The pandas library has a pandas.DataFrame class, which is useful for handling and manipulating such data. This recipe starts by creating these objects.

How to do it...

Execute the following steps for this recipe:

1. Import the necessary modules from the Python standard library:

>>> from datetime import datetime
>>> import pandas

2. Create a sample time-series data as a list of dictionary objects. Assign it to time_series data:

>>> time_series_data = \
[{'date': datetime.datetime(2019, 11, 13, 9, 0),   
  'open': 71.8075, 'high': 71.845,  'low': 71.7775, 
  'close': 71.7925, 'volume': 219512},
{'date': datetime.datetime(2019, 11, 13, 9, 15),  
 'open': 71.7925, 'high': 71.8,    'low': 71.78,   
 'close': 71.7925, 'volume&apos...

DataFrame manipulation—renaming, rearranging, reversing, and slicing

After creating a DataFrame object, you can perform various operations on it. This recipe covers the following operations on DataFrame objects. Renaming a column, rearranging columns, reversing the DataFrame, and slicing the DataFrame to extract a row, column, and a subset of data.

Getting ready

Make sure the df object is available in your Python namespace. Refer to Creating a pandas.DataFrame object recipe of this chapter to set up this object.

How to do it…

Execute the following steps for this recipe:

1. Rename the date column to timestamp for df. Print it:

>>> df.rename(columns={'date':'timestamp'}, inplace=True)
>>> df

We get the following output:

            timestamp    open    high     low   close volume
0 2019-11-13 09:00:00 71.8075 71.8450 71.7775 71.7925 219512
1 2019-11-13 09:15:00 71.7925 71.8000 71.7800 71.7925  59252
2 2019-11-13 09:30:00 71.7925 71.8125 71...

DataFrame manipulation—applying, sorting, iterating, and concatenating

Adding to the previous recipe, this recipe demonstrates more operations that can be performed on DataFrame objects: applying a function to all elements in a column, sorting based on a column, iterating over the rows, and concatenating multiple DataFrame objects vertically and horizontally.

Getting ready

Make sure you have followed the previous recipe before trying out this recipe. Ensure you have df in your Python namespace from the previous recipe.

How to do it…

Execute the following steps for this recipe:

1. Import the necessary modules

>>> import random
>>> import pandas

2. Modify the values in the timestamp column of df with a different date and time format DD-MM-YYYY HH:MM:SS:

>>> df['timestamp'] = df['timestamp'].apply(
                        lambda x: x.strftime("%d-%m-%Y %H:%M:%S"))
>>> df

We get the following output:

            timestamp...

Converting a DataFrame into other formats

This recipe demonstrates the conversion of DataFrame objects into other formats, such as .csv files, json objects, and pickle objects. Conversion into a .csv file makes it easier to further work on the data using a spreadsheet application. The json format is useful for transmitting DataFrame objects over web APIs. The pickle format is useful for transmitting DataFrame objects created in one Python session to another Python session over sockets without having to recreate them.

Getting ready

Make sure the object df is available in your Python namespace. Refer to Creating a pandas.DataFrame object recipe of this chapter to set up this object.

How to do it…

Execute the following steps for this recipe:

1. Convert and save df as a CSV file:

>>> df.to_csv('dataframe.csv', index=False)

2. Convert df to a JSON string:

>>> df.to_json()

We get the following output:

'{
    "timestamp":{
        "0":...

Creating a DataFrame from other formats

In this recipe, you will create DataFrame objects from other formats, such as .csv files, .json strings, and pickle files. A .csv file created using a spreadsheet application, valid JSON data received over web APIs, or valid pickle objects received over sockets can all be processed further using Python by converting them to DataFrame objects.

Loading pickled data received from untrusted sources can be unsafe. Please use read_pickle() with caution. You can find more details here: https://docs.python.org/3/library/pickle.html. If you are using this function on the pickle file created in the previous recipe, it is perfectly safe to use read_pickle().

Getting ready

Make sure you have followed the previous recipe before starting this recipe.

How to do it…

Execute the following steps for this recipe:

1. Create a DataFrame object by reading a CSV file:

>>> pandas.read_csv('dataframe.csv')

We get the following output:

         ...