Where to Go from Here

This book has been a journey, much like a daring mountain hike… but now, at last, we have arrived at the end of our trek. We now stand upon the summit of mount introductory-GPU-programming, and we stand proud as we gaze back upon our native village of serial-programming-ville and smile as we think about the quaint naivity of our old one-dimensional programming traditions, where we considered forking a process in Unix to be our entire understanding of the notion of parallel programming. We have braved many pitfalls and dangers to arrive at this point, and we may have even made such mishaps as installing a broken NVIDIA driver module in Linux, or maybe downloading the wrong Visual Studio version over a slow 100k connection while visiting our parents for vacation. But these setbacks were only temporary, leaving wounds that developed into calluses that...

Furthering your knowledge of CUDA and GPGPU programming

The first option you have is, of course, to learn more about CUDA and General-Purpose GPU (GPGPU) programming in particular. In this case, you have probably already found a good application of this and want to write even more advanced or optimized CUDA code. You may find it interesting for its own sake, or perhaps you want to get a job as a CUDA/GPU programmer. With a strong GPU programming foundation in place (which was provided by this book), we will now look at some of the advanced topics in this field that we are now prepared to learn about.

Multi-GPU systems

The first major topic that comes to mind would be to learn how to program systems with more than one GPU installed...

Graphics

Obviously, the G in GPU stands for graphics, which we really haven't seen much of in this book. Even though machine learning applications are now NVIDIA's bread and butter, it all started with rendering nice-looking graphics. We will provide some resources to get you started here, whether you want to develop video game engines, render CGI movies, or develop CAD software. CUDA can actually be used hand in hand with graphics applications, and is actually used in professional software such as Adobe's Photoshop and After Effects, as well as in many recent video games such as the Mafia and Just Cause series. We will briefly cover some of the major APIs you might consider starting with here.

OpenGL

The Open...

Machine learning and computer vision

Of course, the elephant in the room of this chapter is machine learning and its fraternal twin computer vision. It goes without saying that machine learning (particularly the subfields of deep neural networks and convolutional neural networks) is what is keeping a roof over NVIDIA CEO Jensen Huang's head these days. (Okay, we admit that was the understatement of the decade...) If you need a reminder as to why GPUs are so applicable and useful in this field, please take another look at Chapter 9, Implementation of a Deep Neural Network. A large number of parallel computations and mathematical operations, as well as the user-friendly mathematical libraries, have made NVIDIA GPUs the hardware backbone of the machine learning industry.

The basics...

Blockchain technology

Last, but certainly not least, is blockchain technology. This is the underlying cryptographic technology that powers cryptocurrencies such as Bitcoin and Ethereum. This is, of course, a very new field, which was first described by Bitcoin's mysterious creator, Satoshi Nakamoto, in a white paper published in 2008. GPUs were applied to this field almost immediately after its invention—generating a unit of currency comes down to brute-force cracking a cryptographic puzzle, and a GPU can attempt to brute-force crack more combinations in parallel than any other piece of hardware available to the general public today. This process is known as mining.

Those who are interested in blockchain technology are suggested to read Satoshi Nakamoto's original white paper on Bitcoin, which is available here: https://bitcoin.org/bitcoin.pdf.

GUIMiner, an open...

Summary

In this chapter, we went over some of the options and paths for those that are interested in furthering their background in GPU programming, which is beyond the scope of this book. The first path we covered was expanding your background in pure CUDA and GPGPU programming—some of the things you can learn about that weren't covered in this book include programming systems with multiple GPUs and networked clusters. We also looked at some of the parallel programming languages/APIs besides CUDA, such as MPI and OpenCL. Next, we discussed some of the well-known APIs available to those who are interested in applying GPUs to rendering graphics, such as Vulkan and DirectX 12. We then looked at machine learning and went into some of the basic backgrounds that you should have as well as some of the major frameworks available for developing deep neural networks. Finally...

Questions

Use Google or some other search engine to find at least one application of GPU programming that is not featured in this chapter.
Try to find at least one programming language or API that can be used to program a GPU that is not featured in this chapter.
Look up Google's new Tensor Processing Unit (TPU) chips. How do these differ from GPUs?
Do you think it is a better idea to connect computers together into a cluster using Wi-Fi or wired Ethernet cables?