Best Practices and Python Performance

After going through the preceding chapters and learning various things about Python, we have come to the last chapter. Here, we want to discuss some general strategies that you can implement and how to write code that works faster, is cleaner, and is easier to maintain. These approaches can be used for data-oriented code—or any other type of code, for that matter.

This chapter is split into three parts. The first section will discuss how you can analyze and speed up your code, the second section will cover best practices for maintaining your code so that you'll code faster and cleaner, and in the third and final section, we'll go through a brief overview of the non-Python technologies that you might find useful for your projects.

The following topics will be covered in this chapter:

• Ways to monitor performance and identify...

Technical requirements

The code for this chapter can be found in this book's GitHub repository (https://github.com/PacktPublishing/Learn-Python-by-Building-Data-Science-Applications), which is stored in the Chapter20 folder. The code requires an array of packages to be installed, including the following:

• numpy
• scipy
• numba
• dask
• black
• wily
• hypothesis
• line_profiler
• python-graphviz and graphviz

Speeding up your Python code

In the previous chapter, we talked about different best practices, approaches, and ways to boost code performance. As a toy example for performance, we'll build our own KNN model, which we used in Chapter 13, Training a Machine Learning Model. As a reminder, KNN is a simple ML model that predicts the target variable by identifying K closest records in the training set, then taking a mode (for classification) or weighted average (for regression) of the target variable. Obviously, there are quite a few implementations of KNN already, and so we will use one as an example.

For starters, let's write a naive implementation; it has already been fairly optimized through the use of NumPy commands. First, let's import all the Euclidean distance measuring functions and define a function to get the N-closest records. Take a look at the following...

Using best practices for coding in your project

In this section, we'll switch to another, although adjacent, topic—best practices for maintaining good quality code. Here, we will define "good" in a broad way—as dry, concise, expressive, easy to read, change, and build upon. To illustrate this topic, we will review the wikiwwii package we built in Chapter 15, Packaging and Testing with Poetry and PyTest.

All the changes we make to the package throughout this chapter are stored on the best-practices branch in this book's GitHub repository.

Code formatting with black

First of all, let's talk about formatting. It may sound like a minor issue—and it generally is—but formatting...

Beyond this book – packages and technologies to look out for

Throughout this book, we've shared a wide range of Python frameworks and libraries for data-driven development. However, there are some tools we couldn't fit in, but that you need to be aware of. We'll discuss some of them here. In particular, we want to cover three somewhat connected topics—Python flavors, Docker containers, and Kubernetes.

Different Python flavors

In the Numba section, we showed you how to use Numba to speed Python code up. To do so, Numba uses a modern compilation engine. It does so by exploiting the C nature of Python. Another project, Cython, does the same—it compiles Python code into C using a somewhat different...

Summary

In this final chapter, we covered multiple topics on code performance and quality and discussed a few important technologies beyond Python. In particular, we discussed how the combination of efficient code, a better understanding of requirements, and smart usage of appropriate data structures can significantly speed up the performance of code—in our case, a hundred times more performant! Then, we discussed how we can deal with big data by computing in parallel on multiple CPUs—or multiple machines in the cluster.

In the second part of this chapter, we discussed a few ways to keep code quality under control—by running sophisticated non-deterministic test suits, automating code formatting, and tracking code maintainability.

Both code performance and quality are important. Knowing ways to measure and improve both are necessary skills for a professional...

Questions

1. How can we measure which line in the code took the most time to complete?
2. Does NumPy run faster than Pandas?
3. When should we use Numba? What are the challenges and benefits of using Numba?
4. When should we use Dask?
5. Does code formatting matter? Why is Black better than linters?
6. How does Hypothesis help you test your code?

Further reading

Architectural Patterns and Best Practices with Python, by Anand Balachandran Pillai: https://www.packtpub.com/application-development/architectural-patterns-and-best-practices-python-video