Retrieving and Handling Market Data with Python

If you look at the general logical diagram of a trading application’s architecture, which we devised in Chapter 1, Developing Trading Strategies – Why They Are Different, you can see that now we’re moving from the very first module named Receive data to the second one, Cleanup and filter, which features the Store and Retrieve functionality.

In previous chapters, we mentioned many times that any algo trading application is based on market data, and success in algo trading (that is, the ability to make money and not lose it) depends on the quality of the data and its consistency. So, let’s proceed to determine which data we really need to collect and how to make sure the collected data is consistent, then decide about the internal format for the trading app, and the way to store, update, retrieve, and delete data (if necessary).

Upon finishing this chapter, you will have a clear understanding of the most...

Navigating through data

The problem with data collection starts at the very beginning: every data provider offers its own data, quite often in its own format. Some data providers offer only compressed data or snapshots (see later in this chapter) while others broadcast tick-by-tick data; very few data providers also offer order book (depth of market or DOM; see Chapter 3, FX Market Overview from a Developer’s Standpoint) data.

So, first and above all, you should decide on the data granularity: whether your trading algorithm requires tick-by-tick or compressed data, and if you need DOM data or not. At this stage, you may feel lost with these questions, but don’t worry—you will get a very clear understanding of your data needs when you’ve accomplished further chapters of this book.

Let’s consider all three cases and see how we can actually handle ticks, snapshots, and DOM market data.

Tick data and snapshots

I would like to reiterate that...

Data compression – keeping the amounts to a reasonable minimum

In the previous section, we already considered one of the most popular data compression techniques used by data providers: snapshots. The difference is that a tick represents a single event (such as a new trade or a change in bid or ask) and a single price value, but a snapshot instead discards information about individual ticks and replaces it with the following prices per period:

• Price of the first tick of the period (or open)
• Maximum price for the period (or high)
• Minimum price for the period (or low)
• Price of the last tick of the period (or close)

For example, if the period is 1 minute and during this minute 100 trades were placed, then the snapshot will replace 100 ticks (or 100 prices) with just 4 prices.

The resulting snapshots are called bars when plotted on charts. Very frequently, traders and developers use bars instead of snapshots. Graphically, a bar is typically presented...

Working with saved and live data – keeping your app universal

In the previous chapter, we quickly observed various methods to get market data from LMAX, one of the most important ECNs in the FX market. Without going into too much technical detail, we can assume that most other brokers, trading venues, and data vendors use more or less the same protocols and APIs based on socket connections. So, it should not be a problem to re-adapt your code to start retrieving data from a new source.

However, from the previous chapter, I hope you also remember that despite similarities in the transport layer of the connection, almost every data source has its own features that can only be found in its documentation (and sometimes, unfortunately, only in direct talks with its technical support).

This means that even if you implemented—for example—Financial Information eXchange (FIX) protocol version 4.4 with one broker, it is quite possible that you will have to modify something...

Summary

In this chapter, we learned how to process both live and historical market data and, most importantly, how to do it in an efficient way. We are now familiar with time-based bars, which are most heavily used in trading strategies. We also learned about the solution that helps keep our trading applications flexible and quickly switch from one data source to another, which will help when switching from testing to production in the future.

Now, we are ready to move on to using the processed market data in our trading strategies. Let’s see how we can do this in the next chapter.