Julia High Performance - Second Edition

4.7 (3 reviews total)
By Avik Sengupta , Alan Edelman
  • Instant online access to over 8,000+ books and videos
  • Constantly updated with 100+ new titles each month
  • Breadth and depth in over 1,000+ technologies
  1. Julia is Fast

About this book

Julia is a high-level, high-performance dynamic programming language for numerical computing. If you want to understand how to avoid bottlenecks and design your programs for the highest possible performance, then this book is for you.

The book starts with how Julia uses type information to achieve its performance goals, and how to use multiple dispatches to help the compiler emit high-performance machine code. After that, you will learn how to analyze Julia programs and identify issues with time and memory consumption. We teach you how to use Julia's typing facilities accurately to write high-performance code and describe how the Julia compiler uses type information to create fast machine code. Moving ahead, you'll master design constraints and learn how to use the power of the GPU in your Julia code and compile Julia code directly to the GPU. Then, you'll learn how tasks and asynchronous IO help you create responsive programs and how to use shared memory multithreading in Julia. Toward the end, you will get a flavor of Julia's distributed computing capabilities and how to run Julia programs on a large distributed cluster.

By the end of this book, you will have the ability to build large-scale, high-performance Julia applications, design systems with a focus on speed, and improve the performance of existing programs.

Publication date:
June 2019
Publisher
Packt
Pages
218
ISBN
9781788298117

 

Julia is Fast

In many ways, the history of programming languages has been driven by, and certainly intertwined with, the needs of numerical and scientific computing. The first high-level programming language, Fortran, was created to solve scientific computing problems, and continues to be important in the field even to this day. In recent years, the rise of data science as a specialty has brought additional focus to numerical computing, particularly for statistical uses. In this area, somewhat counter-intuitively, both specialized languages such as R and general-purpose languages such as Python are in widespread use. The rise of Hadoop and Spark has spread the use of Java and Scala respectively among this community. In the midst of all this, Matlab has had a strong niche within engineering communities, while Mathematica remains unparalleled for symbolic operations.

A new language for scientific computing therefore has a very high barrier to overcome, and it's been only a few short years since the Julia language was introduced to the world. In that time, however, its innovative features, combining the ease of use of a dynamic language and the performance of a statically compiled language, have created a growing niche within the numerical computing world. Based on multiple dispatch as its defining paradigm, Julia is a very pleasant language to program in, making mathematical abstractions very easy to express. However, it was the claim of high performance that drew the earliest adopters.

This, then, is a book that celebrates writing high-performance programs. With Julia, this is not only possible, but also reasonably straightforward, in a low-overhead, dynamic language.

As a reader of this book, you have likely already written your first few Julia programs. We will assume that you have successfully installed Julia, and have a working programming environment available. We expect you are familiar with very basic Julia syntax, but we will discuss and review many of those concepts throughout the book as we introduce them.

In this chapter, we will describe some of the underlying design elements of Julia that contribute to its well-deserved reputation as a fast language:

  • Julia – fast and dynamic
  • Designed for speed
  • How fast can Julia be?
 

Julia – fast and dynamic

It is a widely believed myth in programming language communities that high-performance languages and dynamic languages are completely disjointed sets. The perceived wisdom is that, if you want programmer productivity, you should use a dynamic language, such as Ruby, Python, or R. On the other hand, if you want fast code execution, you should use a statically typed language, such as C or Java.

There are always exceptions to this rule. However, for most mainstream programmers, this is a strongly held belief. This usually manifests itself in what is known as the two-language problem. This is something that is especially prominent in scientific computing. This is the situation where the performance-critical inner kernel is written in C, but is then wrapped and used from a dynamic, higher-level language. Code written in traditional, scientific computing environments such as R, Matlab, or NumPy follows this paradigm.

Code written in this fashion is not without its drawbacks, however. Even though it looks like it gets you the best of both worlds—fast computation, while allowing the programmer to use a high-level languagethis is a path full of hidden dangers. For one, someone will have to write the low-level kernel. So, you need two different skill sets. If you are lucky enough to find the low-level code in C for your project, you are fine. However, if you are doing anything new or original, or even slightly different from the norm, you will find yourself writing both C and a high-level language. This will severely limit the number of contributors that your projects or research will get: to be really productive, those contributors really have to be familiar with two languages.

Secondly, when running code routinely written in two languages, there can be severe and unforeseen performance pitfalls. When you can drop down to C code quickly, everything is fine. However, if, for time reasons, effort, skill or changing requirements, you cannot write a performance-intensive part of your algorithm in C, you'll find your program taking hundreds or even thousands of times longer than you expected.

Julia is the first modern language to make a reasonable effort to solve the two-language problem. It is a high-level, dynamic language with powerful features that make for very productive programming. At the same time, code written in Julia usually runs very quickly, almost as quickly as code written in statically typed languages.

The rest of this chapter describes some of the underlying design decisions that make Julia such a fast language. We'll also look at some evidence of the performance claims about Julia. The rest of the book shows you how to write your Julia programs to be as fast and lean as possible. We will discuss how to measure and reason about performance in Julia, and how to avoid some potential performance roadblocks.

For all the content in this book, we will usually illustrate our points with small, self-contained programs. We hope that this will enable you grasp the crux of the issue, without getting distracted by unnecessary elements of a larger program. We expect that this methodology will therefore provide you with instinctive intuition about Julia's performance profile.

Julia has a refreshingly simple performance modelthus, writing fast Julia code is a matter of understanding a few key elements of computer architecture, and how the Julia compiler interacts with it. We hope that, by the end of this book, your instincts are developed well enough to design and write your own Julia code with the fastest possible performance.

Finally, Julia will work for you at both ends of the compute spectrum. On one hand, its performance and expressiveness allows it to run embedded use cases on low-powered processors and it is fully supported on ARM processors, and works well on the Raspberry Pi, which makes it a perfect environment for teaching programming. At the other end of the spectrum, Julia has been used to run large-scale machine learning applications on some of the world's largest super-computers. The Celeste project used Julia Build and Atlas of the Sky, where the computation ran at an amazing 1.5 petaflops (1 petaflop is 10^15 floating point operations per second, or a thousand million million), using 1.3 million threads. This was the first time any dynamic language had broken the petaflop barrier. So, Julia can run on machines large and small, scaling massively in both directions.

Versions of Julia:
The code and examples in this book are targeted at version 1.2 of the language, which is the most recently released version at the time of publication. Since there will be no breaking changes in the 1.x series of Julia, most of the code in this book should work on version 1.0 onward, which was released in August of 2018.

 

Designed for speed

When the creators of Julia launched the language into the world, they said the following in a blog post entitled Why We Created Julia, which was published in early 2012:

"We want a language that's open source, with a liberal license. We want the speed of C with the dynamism of Ruby. We want a language that's homoiconic, with true macros like Lisp, but with obvious, familiar mathematical notation like Matlab. We want something as usable for general programming as Python, as easy for statistics as R, as natural for string processing as Perl, as powerful for linear algebra as Matlab, as good at gluing programs together as the shell. Something that is dirt simple to learn, yet keeps the most serious hackers happy. We want it interactive and we want it compiled. (Did we mention it should be as fast as C?)"

High-performance, indeed nearly C-level performance, has therefore been one of the founding principles of the language. It's built from the ground up to enable the fast execution of code.

In addition to being a core design principle, it has also been a necessity from the early stages of its development. A very large part of Julia's standard library, including very basic low-level operations, is written in Julia itself. For example, the + operation to add two integers is defined in Julia itself. (Refer to: https://github.com/JuliaLang/julia/blob/e1def102429941705bc16009e35a74abcdb6f88e/base/int.jl#L38.) Similarly, the basic for loop uses the standard iteration mechanism available to all user-defined types. Broadcasting, which is a fundamental low-level operation in the compiler, can be completely overridden by custom array types (this is used heavily in CUDA arrays, for example). All of this means that the compiler had to be very fast from the very beginning to create a usable language. The creators of Julia did not have the luxury of escaping to C for even the core elements of the library.

We will note throughout the book the many design decisions that have been made with an eye to high performance, but there are three main elements that create the basis for Julia's speed: a high performance Just in Time compiler, LLVM to generate machine code, and a type system that allows expressive code.

JIT and LLVM

Julia is a Just In Time (JIT) compiled language, rather than an interpreted one. This allows Julia to be dynamic, without having the overhead of interpretation. This compilation infrastructure is built on top of LLVM—more information about it is available on its website: http://llvm.org.

The LLVM compiler infrastructure project originated at the University of Illinois. It now has contributions from a very large number of corporate as well as independent developers. As a result of all this work, it is now a very high-quality, yet modular, system for many different compilation and code generation activities.

Julia uses LLVM for its JIT compilation needs. The Julia runtime code generator produces LLVM Intermediate Representation (IR) and hands it over to LLVM's JIT compiler, which in turn generates machine code that is executed on the CPU. As a result, sophisticated compilation techniques that are built into LLVM are ready and available to Julia, from simple ones (such as Loop Unrolling or Loop Deletion) to state-of-the-art ones (such as SIMD Vectorization). These compiler optimizations form a very large body of work and, in this sense, the existence of LLVM is very much a pre-requisite to the existence of Julia. It would have been an almost impossible task for a small team of developers to build this compiler and code generation infrastructure from scratch.

Just-In-Time compilation:
A technique in which the code in a high-level language is converted to machine code for execution on the CPU at runtime. This is in contrast to interpreted languages, whose runtime executes the source language directly.

This usually has a significantly higher overhead. On the other hand, Ahead of Time (AOT) compilation refers to the technique of converting a source language into machine code as a separate step prior to running the code. In this case, the converted machine code can usually be saved to disk as an executable file.

Types, type inference, and code specialization

While LLVM provides the basic infrastructure that allows fast machine code to be produced, it must be noted that adding an LLVM compiler to any language will not necessarily make it execute faster. Julia's syntax and semantics have been carefully designed to allow high-performance execution, and a large part of this is due to how Julia uses types in the language. We will, of course, have much more to say about types in Julia throughout this book. At this stage, suffice it to say that Julia's concept of types is a key ingredient of its performance.

The Julia compiler attempts to infer the type of all data used in a program, and compiles different versions of functions specialized to particular types of its arguments. To take a simple example, consider the ^ (power) function. This function can be called with integer or floating point (i.e, fractional, or decimal) arguments. The mathematical definitions and, thus, the implementation of this function are very different for integers and floats. So, Julia will compile, on demand, two versions of the code, one for integer arguments, and one for floating point arguments, and insert the appropriate call in the code when it compiles the program. This means that, at runtime, fast, straight-line code without any type checks will be executed on the CPU.

Julia allows us to introspect the native code that runs on the CPU. Using this facility, we can see that very different code is generated for integer and floating point arguments. So, let's look at the following machine code, generated for squaring an integer:

julia> @code_native 3^2
pushl %eax
decl %eax
movl $202927424, %eax ## imm = 0xC186D40
addl %eax, (%eax)
addb %al, (%eax)
calll *%eax
popl %ecx
retl
We omitted some boilerplate output when showing the result of the @code macros, in order to focus on the relevant parts. Run this code yourself to see the full output.

Let's now look at the following code, generated for squaring a floating point value:

julia> @code_native 3.5^2
vcvtsi2sdl %edi, %xmm1, %xmm1
decl %eax
movl $1993314664, %eax ## imm = 0x76CF9168
.byte 0xff .byte 0x7f .byte 0x00
addb %bh, %bh
loopne 0x68
nopw %cs:(%eax, %eax)

You will notice that the code looks very different (although the actual meaning of the code is not relevant for now). You will notice that there are no runtime type checks in the code. This gets to the heart of Julia's design and its performance claims. 

The ability of the compiler to reason about types is due to the combination of a sophisticated dataflow-based algorithm, and careful language design that allows this information to be inferred from most programs before execution begins. Put in another way, the language is designed to make it easy to statically analyze its data types.

If there is a single reason for Julia being such a high-performance language, this is it. This is why Julia is able to run at C-like speeds while still being a dynamic language. Type inference and code specialization are as close to a secret sauce as Julia gets. It is notable that, outside this type inference mechanism, the Julia compiler is quite simple. It does not include many of the advanced Just in Time optimizations that Java and JavaScript compilers are known to use. When the compiler has enough information about the types within the code, it can generate optimized, straight-line code without many of these advanced techniques.

Detailed information about the implementation of type inference and code specialization in Julia can be found in the paper Julia: A Fresh Approach to Numerical Computing. Jeff Bezanson, Alan Edelman, Stefan Karpinski, and Viral B. Shah (2017) SIAM Review, 59: 65–98. doi: 10.1137/141000671. URL: https://julialang.org/research/julia-fresh-approach-BEKS.pdf

It is useful to note here that, unlike some optionally typed dynamic languages, simply adding type annotations to your code does not make Julia go any faster. Type inference means that the compiler is usually able to figure out the types of variables when necessary. Hence, you can usually write high-level code without fighting with the compiler about types, and still achieve superior performance. 

 

How fast can Julia be?

The best evidence of Julia's performance claims is when you write your own code. We encourage you to run and measure all the code snippets in the book. To start, we will provide an indication of how fast Julia can be by comparing a similar algorithm on multiple languages.

As an example, consider the algorithm to compute a Mandelbrot set. Given a complex number, z, the function computes whether, after a certain number of iterations, the  function converges or not. Plotting the imaginary numbers where that function diverges on a 2D plane produces the following iconic fractal image that is associated with this set:

The following code computes the divergence point based on this logic. Calling this function over all points on a 2D plane will produce the Mandelbrot set:

function mandel(c)
z = c
maxiter = 80
for n in 1:maxiter
if abs(z) > 2
return n - 1
end
z = z^2 + c
end
return maxiter
end

You will notice that this code contains no type annotations, or any special markup, even though it operates on complex numbers. It looks remarkably clean, and the idea that the same mathematical operations can apply to many different kinds of mathematical objects is key to Julia's expressiveness.

The same algorithm implemented in modern C would look as follows:

int mandel(double complex z) {
int maxiter = 80;
double complex c = z;
for (int n = 0; n < maxiter; ++n) {
if (cabs(z) > 2.0) {
return n;
}
z = z*z+c;
}
return maxiter;
}
Downloading the example code:
You can download the example code files for this book from your account at http://www.packtpub.com. If you purchased this book elsewhere, you can visit http://www.packtpub.com/support and register to have the files emailed directly to you.

You can download the code files by taking the following steps:
  1. Log in or register on our website using your email address and password
  2. Let the mouse pointer hover over the SUPPORT tab at the top
  3. Click on Code Downloads & Errata
  4. Enter the name of the book in the Search box
  5. Select the book for which you're looking to download the code files
  6. Choose from the drop-down menu where you purchased this book
  7. Click on Code Download
Once the file is downloaded, please make sure that you unzip or extract the folder using the latest version of the following:
  • WinRAR/7-Zip for Windows
  • Zipeg/iZip/UnRarX for Mac
  • 7-Zip/PeaZip for Linux

By timing this code in Julia and C, as well as re-implementing it in many other languages (all of which are available within the Microbencmarks project at https://github.com/JuliaLang/Microbenchmarks), we can note that Julia's performance claims are certainly borne out for this small program. Plotting these timing results in the following chart, we see that Julia can perform at a level similar to C and other statically typed and compiled languages:

This is, of course, a micro benchmark, and therefore cannot be extrapolated too much. However, I hope you will agree that it is certainly possible to achieve exceptional performance in Julia, without having to fall back to low-level languages for performance-critical code. The rest of the book will attempt to show how we can achieve performance close to this standard, for many different kinds of code bases. We will learn how to squeeze the maximum possible performance from the CPU, without any of the overhead that typically plagues dynamic languages.

 

Summary

In this chapter, we noted that Julia is a language that is built from the ground up for high performance. Its design and implementation have always been focused on providing the highest possible performance on a modern CPU.

The rest of the book will show you how to use the power of Julia fully, to write the fastest possible code in this language. In the next chapter, we will discuss how to measure the speed of Julia code, and identify performance bottlenecks. You will also learn about some of the tools that are built into Julia for this purpose.

About the Authors

  • Avik Sengupta

    Avik Sengupta is the Vice President of engineering at Julia Computing, contributor to open source Julia and maintainer of several Julia packages. Avik is the co-founder of two startups in the financial services and AI sectors and creator of large complex trading systems for the world's leading investment banks. Prior to Julia Computing, Avik was co-founder and CTO at AlgoCircle and at Itellix, director at Lab49 and head of algorithmic solutions at Decimal Point Analytics. Avik earned his MS in Computational Finance at Carnegie Mellon and MBA Finance at the Indian Institute of Management in Bangalore.

    Browse publications by this author
  • Alan Edelman

    Those who are new to Julia often ask me what makes it so special, or how it achieves such high performance. Having to think on my feet in order to answer this question has proven challenging. This was the case, at least, until Avik Sengupta came along with the Julia High Performance book. Now all I have to do is tell the enquirers to read the book as it consists of just the right combination of details to answer their questions. In an easy-to-read concise set of chapters - most of which contain words like "performance" and "fast" - Avik takes you through examples that you can run yourself in order to see how fast and easy Julia is to use. There are computer science words that have become Julia words. It is a pleasure to learn these in a manner that is easy to follow. Just a few examples of this are "JIT", "Multiple Dispatch", "type system", "generated functions", "CUDA", and "SIMD". After learning about Julia's design, you will learn to measure performance. From there, you will appreciate Julia's type system. You will then master using arrays along with making fast function calls and fast numbers. Finally, you will learn to write parallel Julia programs. With Julia High Performance, you'll pick up the key essentials of Julia in no time. You can then join the friendly, fast growing, online community of Julia programmers. Welcome to the world of Julia! Read this book and you will soon join us in loving the Julia language. Alan Edelman Professor of Applied Mathematics Computer Science & AI Labs Member MIT Co-creator, The Julia Language

    Browse publications by this author

Latest Reviews

(3 reviews total)
Great book. Learn from some of the people involved in creating the language.
Good book, hope will improve and explain the content in detail in future. Also I will suggest to add content about real life big project using the concept of high performance at the end of chapter 7, 8, 9 and 10.
Great book providing an in-depth understanding of Julia

Recommended For You

Book Title
Access this book, plus 8,000 other titles for FREE
Access now