What is GPGPU?

A graphics processing unit (GPU) is a specialized processor originally designed to render images on a computer screen. This requires calculating the color of millions of pixels on the screen and updating it multiple times a second. Calculating the value of each pixel is independent and can therefore be computed in parallel. GPUs were designed to perform millions of these computations in parallel to enable fast rendering of images for applications such as computer games.

A few decades ago, researchers realized they could employ the massive parallelization power of GPUs to dramatically speed up calculations unrelated to graphics. However, writing non-rendering GPU programs used to be extremely difficult and out of reach for most programmers. That was until CUDA was released.

What is CUDA?

CUDA is a platform first released in 2007 by the company NVIDIA for making GPGPU more accessible. NVIDIA is the market leader in designing chips for GPUs. CUDA includes a high-level API, a programming model, and several libraries for running non-rendering algorithms on NVIDIA GPUs. CUDA unleashed the power of GPGPU in numerous domains by enabling programmers to easily write massively parallel algorithms.

CUDA is the main enabler of GPGPU and, by extension, the AI boom. It is what makes NVIDIA hardware the de facto standard for GPGPU. This unique market position is reflected in NVIDIA's remarkable stock price performance over the past few years:

Figure 1.1 – NVIDIA stock price evolution over time (source data: Kaggle)

Originally, CUDA was designed to be used primarily through CUDA C with NVIDIA's NVCC compiler. As a result, CUDA programming has traditionally been taught using C or C++. Python libraries seeking to leverage GPU acceleration often had to wrap CUDA C code, which adds complexity.

In recent years, NVIDIA has invested significantly in making CUDA more accessible directly from Python, broadening its appeal to a wider audience of programmers. These efforts are centralized in the CUDA Python project (see https://nvidia.github.io/cuda-python/latest), a collection of tools and libraries that enable Python developers to write and interact with CUDA code more seamlessly.

In the first half of this book, we will focus on numba.cuda, a key component of the CUDA Python ecosystem. Numba is a just-in-time (JIT) compiler for Python, and numba.cuda is an extension that allows us to write CUDA-like code using Python syntax. This approach eliminates the need to mix multiple programming languages, enabling us to write all our code in familiar Python. Still, numba.cuda allows us to write low-level code, which will help you understand what goes on under the hood.

In the second half of the book, we will introduce high-level libraries that abstract these primitives and simplify common functionality such as array programming, data manipulation, and machine learning. In the final chapters, we will integrate these concepts and explore real-world applications, including image processing and atomistic simulations. CUDA is the foundation for nearly all the tools discussed throughout this book.

Why GPGPU is attractive

Like a CPU, a GPU contains many cores: physical processing units capable of performing arithmetic operations at extremely high speeds. Think of each core as a tiny calculator, executing simple operations rapidly. However, the key difference lies in scale and design: while a modern CPU typically features a few complex cores optimized for fast, sequential execution, a GPU may contain thousands of simpler cores specifically designed for massively parallel execution. When a problem can be divided into many smaller, independent tasks, and those tasks can be distributed across all these GPU cores, dramatic speedups can be achieved compared to traditional CPU execution.

To quantify this difference, we can calculate the theoretical maximum compute speed of a device in floating-point operations per second (FLOPS) using the following equation:

The clock frequency is the internal metronome of the device that indicates how fast it can perform computational work, a bit like a heartbeat. It is expressed in cycles per second, or Hz. The number of clock cycles required for a single operation depends on the type of operation we want to perform (e.g., 32-bit versus 64-bit floating-point numbers) and the architecture of the device; for some operations on some devices, it is possible to perform multiple in a single cycle.

Let's use this equation to compare the theoretical compute capacity for 32-bit floating-point numbers of an Intel Core i9 processor CPU, which has 24 cores, to that of an NVIDIA GeForce RTX 4080 GPU with 9,728 cores; both are consumer-grade hardware. The CPU has a clock frequency of 3.5 GHz, and with the AVX instruction set, each core can perform 16 floating-point operations per clock cycle. By plugging these values into the equation, we can estimate that the CPU has a theoretical compute capacity of a little over 1 TerraFlop (TFLOP). The GPU has a slightly lower clock frequency of 2,210 MHz, and the cores can only perform 2 operations per cycle. Still, because the GPU has so many more cores than the CPU, the theoretical compute capacity is a whopping 43 TFLOPS!

Despite the GPU's reputation as being an expensive and power-hungry device, it is much more economical and energy efficient than the CPU when measured per TFLOP. At a retail price of around 1,100 USD and a power draw of up to 300 W, the RTX 4080 GPU costs 26 USD and consumes 7 W per TFLOP. The Intel processor retails around 500 USD and draws up to 125 W; the GPU is therefore roughly 20 times cheaper and energy efficient when measured per TFLOP!

When is GPGPU useful?

The calculation of theoretical compute capacity illustrates that the biggest factor in GPU compute power is the massive number of cores. CPUs generally outperform GPUs in terms of clock frequency and number of operations per clock cycle. This means that if we only use a single GPU core, the calculation will be slower than on the CPU. Only code that can make simultaneous use of a large number of cores will benefit from GPGPU.

GPU cores are not interchangeable with CPU cores. Whereas CPU cores operate more or less independently from each other, GPU cores are grouped into larger units called streaming multiprocessors (SMs). Cores in the same SM do not operate independently and execute instructions in a coordinated manner; we will explore the details in Chapter 5. The consequence is that a GPU is best suited for one specific type of parallelism: data parallelism.

Work can be parallelized in two fundamental ways: task parallelism and data parallelism. Task parallelism involves executing distinct and independent tasks concurrently across different workers (e.g., CPU cores). Data parallelism, on the other hand, divides a single task into identical operations executed simultaneously across multiple data elements, such as applying the same operation to every element in an array.

GPUs are optimized for data parallelism because their architecture allows thousands of threads (sequences of instructions) to execute the same instruction in lockstep, maximizing throughput for uniform workloads such as matrix operations or image processing. We will see in Chapter 3 that the data parallelism model is almost baked into the CUDA programming model.

GPUs struggle with task parallelism and divergent execution paths. When threads within a warp (a group of 32 GPU threads) follow different control flows (such as taking different branches in an if-else statement), performance degrades significantly due to a phenomenon called warp divergence. We will explain warps and warp divergence in more detail in Chapter 5.

 

Task parallelism on the GPU

There are strategies for employing task parallelism on the GPU using techniques such as warp specialization: assigning different tasks to different warps. However, this can result in very complex code and is reserved for advanced use cases.

Application areas of GPGPU

In the previous section, we mentioned that the GPU excels at accelerating computations by leveraging data parallelism. It turns out that many problems from diverse fields benefit from this approach. In recent years, deep learning, AI, and blockchain have received significant attention, but other application areas include the following:

• Scientific computation and simulations in fields such as quantum mechanics, fluid dynamics, and astrophysics
• Bioinformatics, analysis of genetic data, and predictions of protein structures (e.g., AlphaFold)
• Signal processing: image, video, and audio
• Computer vision and object detection
• Medical imaging and CT reconstruction
• Computational finance
• Operations research

We aim to illuminate some of these application areas through example cases in this book.