For many years, I traded in various markets, educated developers in trading, and taught some essential programming and algorithm (algo) development to traders. Eventually, I concluded that the main problem for both groups (the developers and traders) was not a lack of knowledge in programming, but an incorrect or insufficient understanding of the subject domain, that is, financial markets and their mechanics.

This was quite a surprise for me. I thought that coding, debugging, and handling errors should be the main obstacle, but in reality, the problem is in finding useful, correct, and adequate information about markets, their structure, and operations, especially in regard to algo trading. I don’t mean that it’s somehow undisclosed or otherwise hidden from the general audience – not at all. The problem is that it’s really hard to find a source that not only covers everything – from the market structure to money and risk management – but also shows clear connections between very specific features of a market and less-specific features that a trading algorithm should have in order to be successful in this market.

For example, many authors consider entries and exits in a trading strategy as something that can be executed immediately and with equal success at any moment, any time of the day. However, in the real market, this is not the case. I saw lots of strategies that only worked on paper because their developers didn’t take into consideration the fact that they were dealing with a real, complex structure with lots of limitations and shortcomings, and not with an abstract idealistic model.

This example is really primitive, but I hope it explains the most important point that is missed by many researchers and developers: we do not deal with abstract datasets and time series where each data point has an equal value and where any action can be taken at any moment with equal possibility; we work with a very complex structure that has lots of features that make this structure a dynamic non-stationary system. And to be successful in algo trading, these features should be accounted for.

Without knowing how markets are organized and how they operate, our trading algorithms will inevitably fail sooner or later.

If put in a more positive manner, understanding the market will help us in making trading algorithms more robust.

This book is my humble attempt to bridge the gap between the world of quantitative trading (driven by numbers) and the real market (driven by humans). I am convinced that today, any long-term success in trading can be achieved only with good knowledge of both.

Let’s begin with the basics. We will learn about the very basic market terminology and the essential concept of a trading strategy, get familiar with the risk in various forms and aspects with a special focus on ordering and transactional risk, and also mention market data processing in a very general sense.

In this chapter, we will discuss the following topics:

• Trading strategies – it’s all about how you manage risk
• Automated trading – operational risk and regulatory limitations
• Making actual trading decisions – trading logic and credit risk
• Ordering – transactional risk
• The general architecture of a trading application

Let’s start from the beginning by giving definitions. I know it sounds like an academic textbook, but I promise that it’s going to get more and more interesting very quickly.

According to the definition provided by Cambridge Dictionary, trade is "the activity of buying and selling, or exchanging, goods and/or services between people or countries.” This means that if I exchange my money for meat or vegetables, then I am at a grocery market. If I exchange my money for obligations to buy or sell live cattle or wheat in the future, then I am in a derivatives market. And if I exchange my money for another type of money, I am in the foreign exchange, forex or FX market.

What I buy and sell in a particular market is called an asset. If I buy or sell something that can be delivered (wheat, gold, stocks, or even money) it’s called the underlying. If I buy or sell an obligation or the right to buy or sell the underlying at a certain price in the future, then it’s called a derivative.

Underlying and derivatives are traded at trading venues – previously, physical locations such as exchanges, but today, more electronic networks where traders, liquidity providers, market makers, and other market participants match their orders.

If I went to a grocery market to sell potatoes, then I am on the sell side. If I came to buy potatoes, I am on the buy side. In financial markets, buy-side market participants are mostly referred to as price takers (because they can only take prices offered by the sell side), and sell-side market participants are referred to as price givers.

Not all trading venues welcome everyone as a direct participant. Most of them work via a network of brokers – entities that accept orders from their clients and route them to one or more trading venues, possibly internally netting client positions and sometimes even acting as the counterparty for their clients.

In Chapter 3, FX Market Overview from a Developer’s Standpoint, we will consider all these entities in more detail to see why we should pay particular attention to their behavior and how to actually use them to make our trading strategies more robust.

Let’s proceed with a very quick look into the essence of a trading strategy and see the main risks that are natural to the systematic or algo trading business.

Trading strategies – how we make money in financial markets

A trading strategy is a set of rules that tells me when to buy and when to sell the asset if I am on the buy side, or when and how much to offer to the market if I am on the sell side. But in any case, the ultimate goal of a trading strategy is to make money.

If I am on the buy side, I can make money only in the following two ways:

• I buy low and I sell high, or vice versa (earning on the price difference)
• I buy and receive dividends

The former is called active trading, and the latter is normally called investing, or getting passive income. In this book, we will consider only active trading.

There are two major classes of active trading strategies on the buy side. The first one is called directional trading and in essence, it’s when we buy, wait, and sell. If we manage to sell at a price greater than the price at which we bought, we make money.

The second class of buy-side trading strategies is arbitrage. Such a strategy identifies moments when the same asset is priced differently at different trading venues (so-called real or pure arbitrage), or when there is a tradable difference between the price of the underlying and a derivative, different derivatives, or even more complex setups that consist of multiple instruments (so-called statistical arbitrage or stat arb).

Sounds a bit overwhelming? Don’t worry, we will now consider each case in more detail.

Trading app – what can be simpler?

At this point, if you have sufficient experience in application development and a reasonable knowledge of markets, you might exclaim the following:

“Why do we need all this? It is so easy to build a trading application: you only need to get market data, calculate entry and exit orders, and send them to the broker!”

So, the suggested general architecture of a trading application from that standpoint may look as simple as this:

Figure 1.1 – Initial architecture of a trading application

However, as we will see very soon, this is an overly simplified view and it lacks at least one key feature that makes developing trading strategies different from developing other applications: it doesn’t include risk management.

So, what about that risk?

Before discussing ways to make money in trading, let’s first pay closer attention to avoiding losses – at least because according to all statistics, more than 70% of active traders, unfortunately, lose money.

Before we proceed, let me note that the following classification of risks does not fully correspond to official legal and academic classifications. This informal classification is used here for the sake of simplicity, to sort out rather complex matters quickly and with an acceptable degree of comfort.

All risks in active trading can be generally put into three major categories, as follows:

• Operational risks are those associated with how you place trades and depend on your own business procedures as well as third parties, such as brokers, trading venues, and regulators.
• Systemic risks are those pertaining to the very market itself and the logic of a trading strategy.
• Transactional risks are those that make the execution of orders different from expectations; this is the main reason why many strategies that work on paper are unable to make money in reality.

Now that we are a bit more familiar with the idea of a trading strategy and know that the main problem any systematic trader should solve is risk mitigation, let’s dive a little deeper into risks specific to the algo trading business.

Operational risk is the risk of direct or indirect loss resulting from inadequate or failed internal procedures, people, and systems, or from external events (Bank for International Settlements, Basel Committee on Banking Supervision, Operational Risk Supporting Documentation to the New Basel Capital Accord (Basel: BIS, 2002), p. 2, https://www.bis.org/publ/bcbsca07.pdf).

Since, in this book, we will talk mostly about developing trading algos with Python and not about running a trading business, the main operational risk in this context could be that you don’t follow your own strategy or intervene in the algorithmic trading process discretionarily.

Another risk that may be considered operational (although it is normally considered as money management) is the improper use of leverage. In essence, leverage is a credit line provided by the broker that allows you to buy more than you have in your account. If the leverage is too high, you are at risk of being unable to enter the market, or in certain cases, even worse – liquidating your positions that are rapidly losing money.

Broker risk can also be attributed to operational risks as the broker is the very entity that provides you with access to the market, gives you a credit line to open positions, and does the clearing and settlement. Some brokers also act as market makers for their clients, netting their positions internally and acting as the counterparty for their own clients, which may lead to a conflict of interest, and even worse – loss of money if the broker didn’t have sufficient capital to perform these operations.

Last, but not least, we should note that algorithmic and/or automated trading may be fully or partially prohibited in certain jurisdictions. So always check with the respective market regulators to make sure you can run your algo trading at all.

Key takeaways

Always perform a background check of all counterparties, especially your broker. Be careful with leverage and check local regulatory documents on algo trading.

Enough on operational risk – at least for a quick start – and let’s move on to another kind of risk that is common for any trading activity but becomes particularly problematic for algo trading: the risk of basing trading decisions on incorrect market data.

Market data is often considered something that cannot contribute to the overall risk in systematic trading. However, this is a massive mistake. There are two key risks associated with market data:

• Issues with receiving data
• Issues with received data

In the next two subsections, we will dive deeper into the preceding risks.

Receiving data – when size does matter

There are two forms in which we get market data: real-time or historical. In both cases, we obtain it from a data vendor, a broker, or directly from an exchange. The difference is that real-time data is used for actual trading (as it reflects what is going on in the market right now) and historical data is used only for research and development, to rebuild hypothetical trades and estimate the theoretical performance of a trading algorithm.

The issues with receiving data are mostly related to real-time data.

Let’s now add some more definitions as we need to acquire some common terminology to move on with market data and ordering.

A request to buy an asset at a certain price is called a bid. It’s like you go to a market and shout, “I want to buy this asset at this price. Is anyone willing to sell it to me?”

A request to sell an asset at a certain price is called an ask or offer. It means that you are ready to sell it to anyone willing to accept your price.

In financial markets, both requests are realized by buy side traders with a limit order (see Chapter 10, Types of Orders and Their Simulation in Python, for a detailed discussion on types of orders).

When another counterparty agrees to place a trade at the order price, a new trade is registered and its information is included in the data stream and distributed across data vendors, brokers, and other recipients. Such a record is called a tick. In other words, a tick is a minimal piece of information in the market data and normally consists of the following fields:

• date
• time
• price
• traded volume
• counterparty1
• counterparty2

The last two fields contain information about actual counterparties and are normally not disclosed or distributed to protect the market participants. Traded volume means the amount of the asset that was traded (number of contracts, or just the amount of money if we are talking about forex).

The main problem with receiving market data in its raw form is that it’s simply overwhelmingly huge. There are so many market participants and so many trading venues that streaming all transactions for just one asset (which is also called a “financial instrument”) may easily reach megabytes per second – receiving it is already a challenge by itself (don’t worry, we are not going to work with data feeds of this sort in this book). Next, even if we are able to receive a data stream with such a throughput, we need to store and handle this data somehow, and thus a very fast database is required. And finally, we need to be able to process this amount of data at an adequate speed, so we need blazingly fast computers.

But there is good news. Despite some strategies (mostly arbitrage and high-frequency trading) do require raw market data in the format just described (also frequently referred to as time and sales data) to identify trading opportunities, most directional trading algorithms are far less sensitive to lack of information about each and every trade. So, data vendors provide data in a compressed format. This becomes possible because most of the raw market data contains sequences of ticks with identical prices, and removing them won’t distort the price movements. This happens because there may be many market participants placing trades at the same price at almost the same time, so by excluding these sequences, we lose information about each transaction but retain information about any change in price. Such a market data stream is often referred to as filtered or cleaned. Besides that, some trades are made at bids, others at asks, and while both bids and asks remain the same, these trades form sequences of trades where prices seem to be different. However, in reality, they are always at the distance of the difference between bids and asks. This difference doesn’t mean that the market price changes. Such a phenomenon is called a bounce and is normally also excluded from cleaned data.

Some vendors and brokers go even further and send snapshots of the market data instead of a filtered data stream. A snapshot is sent at regular time intervals, for example, 100 ms or 1 s, and contains only the following information:

• Date
• Time
• Price at the beginning of the interval (also known as open, or just O)
• Maximum price during the interval (also known as high, or H)
• Minimum price during the interval (also known as low, or L)
• Price at the end of the interval (also known as close, or C)
• Traded volume

Therefore, instead of thousands of ticks, we receive only one tick with seven data fields. This approach dramatically reduces the throughput but is obviously somewhat destructive to the data, and snapshot data may not be suitable for some strategies.

Key takeaway

Be careful with choosing the source of data, especially for live trading, and always make sure it contains sufficient information for your strategy.

Received data – looking at it from a critical angle

After we have successfully received the data, we should make sure it makes sense. Often, data, especially tick data, contains erroneous prices. These prices may be received due to a number of reasons, which we will discuss in detail in Chapter 5, Retrieving and Handling Market Data with Python.

Erroneous, otherwise known as non-market, prices may cause trouble for systematic traders because a single wrong quote may trigger an algorithm to buy or sell something, and such a trade should not have happened according to the strategy logic.

Sometimes, these incorrect quotes can be seen if plotted on a chart. The human eye intuitively expects data points to be within a certain reasonable range and easily catches the outliers, as can be seen in the following chart:

Figure 1.2 – Non-market prices seen on a tick chart

In case we receive snapshots or other compressed data, there could be missing intervals when we receive no quotes. It can happen because of the following:

• The market is closed (scheduled or due to an emergency)
• The data server is down
• The connection is broken

Key takeaway

A robust trading app should have a module capable of checking data consistency and connection persistence.

Alright, we are now aware of the operational risks and know how harmful incorrectly handling market data could be. Anything else? Of course, here comes the main risk: systemic.

In directional trading, systemic risks are mostly realized when you or your trading strategy supposed the price would go in one direction, but in reality, it went in the opposite direction. Don’t worry, this situation is absolutely normal for systematic trading, and no one can guarantee that any strategy will generate a win rate of 100%.

There are two major types of trading strategies: data-driven and event-driven. Data-driven strategies analyze the price time series (which we discussed in the Retrieving market data – quality and consistency as keys to success section) in order to find a certain pattern or sequence that then triggers an order. Event-driven strategies wait for a certain event to happen – for example, a new tick at a certain price and with a certain volume, or a release of political news or economic indicators. In both cases, a trading app should have rules to not only open positions but also to liquidate them – again, based on price data or events (or both).

In general, if a strategy generates some wins and some losses, there are only two ways for it to make money:

• Generate more wins than losses
• Have the average win greater than the average loss

If your trading algorithm does not have a routine that handles situations when the market goes against the position, then most likely, the average loss across a statistically significant number of trades will be greater than the average win, and it will become really problematic to make money with such a strategy.

Don’t forget that forex is traded using leverage, which means the ability to trade much greater amounts of money than you actually have in your account. In the case that your trading algorithm has poor risk management logic and incorrect position sizing, an open position may quickly drain your account to zero if the price goes the opposite way, and even worse – some brokers will even let you go negative and you will be left with debt instead of profit.

Key takeaway

Systemic risk management and position sizing algorithms are crucial parts of an algo trading application.

Systemic risk is very important, but there’s good news for systematic traders: it is possible to mitigate this risk by carefully testing the strategy before going to production and adjusting it so the systemic risk is minimized. But there is one more risk that is hard to mitigate during the research and development phase: transactional risk.

Transactional risks are the real problem in the first place for arbitrage, but they also affect directional strategies. In simple terms, this is a risk of the following:

• Entering or exiting the market at a wrong price
• Entering or exiting the market at a wrong time
• Entering or exiting the market with a wrong trading size
• Not entering or exiting the market at all

All four situations are more than possible in all markets and are even quite frequent during periods of insufficient liquidity (see Chapter 3, FX Market Overview from a Developer’s Standpoint, for a more detailed discussion of liquidity issues).

Key takeaway

Transactional risks are managed by a set of algorithms that are also an essential part of any trading application.

Well, it’s been quite a trip across the various risks, and we now understand that the initial idea of a trading application with a simple and straightforward linear logic definitely won’t work in real life. Now, we can suggest something (unfortunately) more complex, but (fortunately) more realistic.

Now, we can improve our initial diagram (see Figure 1.1) representing the architecture of a trading application. Although it is still very general and high level, it now radically differs from what we suggested in the beginning:

Figure 1.3 – More complete general architecture of a trading app

Here, we can see that the real trading application features a number of blocks or modules, and some of them receive feedback from others. After we have received market data, we should clean it up and add storage and retrieval facilities, as pieces of data may be reused in the following code. Then, we generate trading signals according to the strategy logic. But before sending these signals to the market in the form of orders, we should do some checks to make sure that the order size is comfortable for both the strategy and the market. After that, we actually trade – send orders to the market controlling their execution. And finally, we track the open positions and manage exposure according to the running profit or loss.

In this chapter, we familiarized ourselves with the core terminology and key concepts of FX markets, learned about counterparties and transactions, understood the intrinsic issues with market data, reviewed various risks, and drafted the first mockup of a trading application’s architecture. We now know that a robust trading app is more about risk management, checking various situations that may happen in real life before and after the trade, and making corrections on the fly. This is what makes the development of trading apps different.

In the next chapter, let us see how using Python in algo trading helps improve the research and development of trading applications.