In order to learn how to work with LLVM, it is best to begin by compiling LLVM from source. LLVM is an umbrella project and the GitHub repository contains the source of all projects belonging to LLVM. Each LLVM project is in a top-level directory of the repository. Besides cloning the repository, your system must also have all tools installed that are required by the build system. In this chapter, you will learn about the following topics:
- Getting the prerequisites ready, which will show you how to set up your build system
- Cloning the repository and building from source, which will cover how to get the LLVM source code, and how to compile and install the LLVM core libraries and clang with CMake and Ninja
- Customizing the build process, which will talk about the various possibilities for influencing the build process
Compiling LLVM versus installing binaries
You can install LLVM binaries from various sources. If you are using Linux, then your distribution contains the LLVM libraries. Why bother compiling LLVM yourself?
First, not all install packages contain all the files required for developing with LLVM. Compiling and installing LLVM yourself prevents this problem. Another reason stems from the fact that LLVM is highly customizable. With building LLVM, you learn how you can customize LLVM, and this will enable you to diagnose problems that may arise if you bring your LLVM application to another platform. And last, in the third part of this book, you will extend LLVM itself, and for this, you need the skill of building LLVM yourself.
However, it is perfectly fine to avoid compiling LLVM for the first steps. If you want to go on this route, then you only need to install the prerequisites as described in the next section.
Many Linux distributions split LLVM into several packages. Please make sure that you install the development package. For example, on Ubuntu, you need to install the
llvm-dev package. Please also make sure that you install LLVM 17. For other versions, the examples in this book may require changes.
Getting the prerequisites ready
To work with LLVM, your development system should run a common operating system such as Linux, FreeBSD, macOS, or Windows. You can build LLVM and clang in different modes. A build with debug symbols enabled can take up to 30 GB of space. The required disk space depends heavily on the chosen build options. For example, building only the LLVM core libraries in release mode, targeting only one platform, requires about 2 GB of free disk space, which is the bare minimum needed.
To reduce compile times, a fast CPU (such as a quad-core CPU with a 2.5 GHz clock speed) and a fast SSD are also helpful. It is even possible to build LLVM on a small device such as a Raspberry Pi – it only takes a lot of time. The examples within this book were developed on a laptop with an Intel quad-core CPU running at a 2.7 GHz clock speed, with 40 GB RAM and 2.5 TB SSD disk space. This system is well suited for the development task.
Your development system must have some prerequisite software installed. Let’s review the minimal required version of these software packages.
To check out the source from GitHub, you need Git (https://git-scm.com/). There is no requirement for a specific version. The GitHub help pages recommend using at least version 1.17.10. Due to known security issues found in the past, it is recommended to use the latest available version, which is 2.39.1 at the time of writing.
The LLVM project uses CMake (https://cmake.org/) as the build file generator. At least the 3.20.0 version is required. CMake can generate build files for various build systems. In this book, Ninja (https://ninja-build.org/) is used because it is fast and available on all platforms. The latest version, 1.11.1, is recommended.
Obviously, you also need a C/C++ compiler. The LLVM projects are written in modern C++, based on the C++17 standard. A conforming compiler and standard library are required. The following compilers are known to work with LLVM 17:
- gcc 7.1.0 or later
- clang 5.0 or later
- Apple clang 10.0 or later
- Visual Studio 2019 16.7 or later
Please be aware that with further development of the LLVM project, the requirements for the compiler are most likely to change. In general, you should use the latest compiler version available for your system.
Python (https://python.org/) is used during the generation of the build files and for running the test suite. It should be at least the 3.8 version.
Although not covered in this book, there can be reasons that you need to use Make instead of Ninja. In this case, you need to use GNU Make (https://www.gnu.org/software/make/) version 3.79 or later. The usage of both build tools is very similar. It is sufficient to replace
ninja in each command with
make for the scenarios described below.
LLVM also depends on the
zlib library (https://www.zlib.net/). You should have at least version 18.104.22.168 installed. As usual, we recommend using the latest version, 1.2.13.
To install the prerequisite software, the easiest way is to use the package manager from your operating system. In the following sections, the commands required to install the software are shown for the most popular operating systems.
$ sudo apt -y install gcc g++ git cmake ninja-build zlib1g-dev
Fedora and RedHat
$ sudo dnf –y install gcc gcc-c++ git cmake ninja-build \ zlib-devel
On FreeBSD 13 or later, you have to use the
pkg package manager. FreeBSD differs from Linux-based systems in that the clang compiler is already installed. To install all other packages at once, you type the following:
$ sudo pkg install –y git cmake ninja zlib-ng
For development on OS X, it is best to install Xcode from the Apple store. While the Xcode IDE is not used in this book, it comes with the required C/C++ compilers and supporting utilities. For installation of the other tools, the package manager Homebrew (https://brew.sh/) can be used. To install all packages at once, you type the following:
$ brew install git cmake ninja zlib
Like OS X, Windows does not come with a package manager. For the C/C++ compiler, you need to download Visual Studio Community 2022 (https://visualstudio.microsoft.com/vs/community/), which is free for personal use. Please make sure that you install the workload named Desktop Development with C++. You can use the package manager Scoop (https://scoop.sh/) to install the other packages. After installing Scoop as described on the website, you open x64 Native Tools Command Prompt for VS 2022 from your Windows menu. To install the required packages, you type the following:
$ scoop install git cmake ninja python gzip bzip2 coreutils $ scoop bucket add extras $ scoop install zlib
Please watch the output from Scoop closely. For the Python and
zlib packages, it advises adding some registry keys. These entries are needed so that other software can find these packages. To add the registry keys, you’d best copy and paste the output from Scoop, which looks like the following:
$ %HOMEPATH%\scoop\apps\python\current\install-pep-514.reg $ %HOMEPATH%\scoop\apps\zlib\current\register.reg
After each command, a message window from the registry editor will pop up asking whether you really want to import those registry keys. You need to click on Yes to finish the import. Now all prerequisites are installed.
The LLVM code base is very large. To comfortably navigate the source, we recommend using an IDE that allows you to jump to the definition of classes and search through the source. We find Visual Studio Code (https://code.visualstudio.com/download), which is an extensible cross-platform IDE, very comfortable to use. However, this is no requirement for following the examples in this book.
Cloning the repository and building from source
- Configure Git.
- Clone the repository.
- Create the build directory.
- Generate the build system files.
- Finally, build and install LLVM.
Let’s begin with configuring Git.
The LLVM project uses Git for version control. If you have not used Git before, then you should do some basic configuration of Git first before continuing: to set the username and email address. Both pieces of information are used if you commit changes.
One can check whether they previously had an email and username already configured in Git with the following commands:
$ git config user.email $ git config user.name
The preceding commands will output the respective email and username that you already have set when using Git. However, in the event that you are setting the username and email for the first time, the following commands can be entered for first-time configuration. In the following commands, you can simply replace
Jane with your name and
firstname.lastname@example.org with your email:
$ git config --global user.email "email@example.com" $ git config --global user.name "Jane"
By default, Git uses the vi editor for commit messages. If you prefer another editor, then you can change the configuration in a similar way. To use the nano editor, you type the following:
$ git config --global core.editor nano
For more information about Git, please see the Git Version Control Cookbook (https://www.packtpub.com/product/git-version-control-cookbook-second-edition/9781789137545).
Now you are ready to clone the LLVM repository from GitHub.
Cloning the repository
On all non-Windows platforms, you type the following command to clone the repository:
$ git clone https://github.com/llvm/llvm-project.git
Only on Windows, add the option to disable auto-translation of line endings. Here, you type the following:
$ git clone --config core.autocrlf=false \ https://github.com/llvm/llvm-project.git
This Git command clones the latest source code from GitHub into a local directory named
llvm-project. Now change the current directory into the new
llvm-project directory with the following command:
$ cd llvm-project
Inside the directory are all LLVM projects, each one in its own directory. Most notably, the LLVM core libraries are in the
llvm subdirectory. The LLVM project uses branches for subsequent release development (“release/17.x”) and tags (“llvmorg-17.0.1”) to mark a certain release. With the preceding clone command, you get the current development state. This book uses LLVM 17. To check out the first release of LLVM 17 into a branch called
llvm-17, you type the following:
$ git checkout -b llvm-17 llvmorg-17.0.1
With the previous steps, you cloned the whole repository and created a branch from a tag. This is the most flexible approach.
Git also allows you to clone only a branch or a tag (including history). With
git clone --branch release/17.x https://github.com/llvm/llvm-project, you only clone the
release/17.x branch and its history. You then have the latest state of the LLVM 17 release branch, so you only need to create a branch from the release tag like before if you need the exact release version. With the additional
–-depth=1 option, which is known as a shallow clone with Git, you prevent the cloning of the history, too. This saves time and space but obviously limits what you can do locally, including checking out a branch based on the release tags.
Creating a build directory
Unlike many other projects, LLVM does not support inline builds and requires a separate build directory. Most easily, this is created inside the
llvm-project directory, which is your current directory. Let us name the build directory,
build, for simplicity. Here, the commands for Unix and Windows systems differ. On a Unix-like system, you use the following:
$ mkdir build
And on Windows, use the following:
$ md build
Now you are ready to create the build system files with the CMake tool inside this directory.
Generating the build system files
$ cmake -G Ninja -DCMAKE_BUILD_TYPE=Release \ -DLLVM_ENABLE_PROJECTS=clang -B build -S llvm
-G option tells CMake for which system to generate build files. Often-used values for that option are as follows:
Ninja– for the Ninja build system
Unix Makefiles– for GNU Make
Visual Studio 17 VS2022– for Visual Studio and MS Build
Xcode– for Xcode projects
–B option, you tell CMake the path of the build directory. Similarly, you specify the source directory with the
–S option. The generation process can be influenced by setting various variables with the
–D option. Usually, they are prefixed with
CMAKE_ (if defined by CMake) or
LLVM_ (if defined by LLVM).
As mentioned previously, we are also interested in compiling clang alongside LLVM. With the
LLVM_ENABLE_PROJECTS=clang variable setting, this allows CMake to generate the build files for clang in addition to LLVM. Furthermore, the
CMAKE_BUILD_TYPE=Release variable tells CMake that it should generate build files for a release build.
The default value for the
–G option depends on your platform, and the default value for the build type depends on the toolchain. However, you can define your own preference with environment variables. The
CMAKE_GENERATOR variable controls the generator, and the
CMAKE_BUILD_TYPE variable specifies the build type. If you use bash or a similar shell, then you can set the variables with the following:
$ export CMAKE_GENERATOR=Ninja $ export CMAKE_BUILD_TYPE=Release
If you are using the Windows command prompt instead, then you set the variables with the following:
$ set CMAKE_GENERATOR=Ninja $ set CMAKE_BUILD_TYPE=Release
$ cmake -DLLVM_ENABLE_PROJECTS=clang -B build -S llvm
You will find more about CMake variables in the Customizing the build process section.
Compiling and installing LLVM
After the build files are generated, LLVM and clang can be compiled with the following:
$ cmake –-build build
This command runs Ninja under the hood because we told CMake to generate Ninja files in the configuration step. However, if you generate build files for a system such as Visual Studio, which supports multiple build configurations, then you need to specify the configuration to use for the build with the
--config option. Depending on the hardware resources, this command runs for between 15 minutes (server with lots of CPU cores, memory, and fast storage) and several hours (dual-core Windows notebook with limited memory).
By default, Ninja utilizes all available CPU cores. This is good for the speed of compilation but may prevent other tasks from running; for example, on a Windows-based notebook, it is almost impossible to surf the internet while Ninja is running. Fortunately, you can limit the resource usage with the
Let’s assume you have four CPU cores available and Ninja should only use two (because you have parallel tasks to run); you then use this command for compilation:
$ cmake --build build –j2
After compilation is finished, a best practice is to run the test suite to check whether everything works as expected:
$ cmake --build build --target check-all
Again, the runtime of this command varies widely with the available hardware resources. The
check-all Ninja target runs all test cases. Targets are generated for each directory containing test cases. Using
check-llvm instead of
check-all runs the LLVM tests but not the clang tests;
check-llvm-codegen runs only the tests in the
CodeGen directory from LLVM (that is, the
You can also do a quick manual check. One of the LLVM applications is llc, the LLVM compiler. If you run it with the
-version option, it shows the LLVM version, the host CPU, and all supported architectures:
$ build/bin/llc --version
If you have trouble getting LLVM compiled, then you should consult the Common Problems section of the Getting Started with the LLVM System documentation https://releases.llvm.org/17.0.1/docs/GettingStarted.html#common-problems) for solutions to typical problems.
As the last step, you can install the binaries:
$ cmake --install build
On a Unix-like system, the install directory is
/usr/local. On Windows,
C:\Program Files\LLVM is used. This can be changed, of course. The next section explains how.
Customizing the build process
The CMake system uses a project description in the
CMakeLists.txt file. The top-level file is in the
llvm/CMakeLists.txt. Other directories also have
CMakeLists.txt files, which are recursively included during the generation process.
Based on the information provided in the project description, CMake checks which compilers are installed, detects libraries and symbols, and creates the build system files, for example,
Makefile (depending on the chosen generator). It is also possible to define reusable modules, for example, a function to detect whether LLVM is installed. These scripts are placed in the special
cmake directory (
llvm/cmake), which is searched automatically during the generation process.
The build process can be customized with the definition of CMake variables. The command-line option
–D is used to set a variable to a value. The variables are used in the CMake scripts. Variables defined by CMake itself are almost always prefixed with
CMAKE_ and these variables can be used in all projects. Variables defined by LLVM are prefixed with
LLVM_ but they can only be used if the project definition includes the use of LLVM.
Variables defined by CMake
Some variables are initialized with the value of environment variables. Most notable are
CXX, which define the C and C++ compilers to be used for building. CMake tries to locate a C and a C++ compiler automatically, using the current shell search path. It picks the first compiler found. If you have several compilers installed, for example, gcc and clang or different versions of clang, then this might not be the compiler you want for building LLVM.
Suppose you like to use clang17 as a C compiler and clang++17 as a C++ compiler. Then, you can invoke CMake in a Unix shell in the following way:
$ CC=clang17 CXX=clang++17 cmake –B build –S llvm
This sets the value of the environment variables only for the invocation of
cmake. If necessary, you can specify an absolute path for the compiler executables.
CC is the default value of the
CMAKE_C_COMPILER CMake variable, and
CXX is the default value of the
CMAKE_CXX_COMPILER CMake variable. Instead of using the environment variables, you can set the CMake variables directly. This is equivalent to the preceding call:
$ cmake –DCMAKE_C_COMPILER=clang17 \ -DCMAKE_CXX_COMPILER=clang++17 –B build –S llvm
Other useful variables defined by CMake are as follows:
This is a path prefix that is prepended to every path during installation. The default is
Different types of build require different settings. For example, a debug build needs to specify options to generate debug symbols and usually link against debug versions of system libraries. In contrast, a release build uses optimization flags and links against production versions of libraries. This variable is only used for build systems that can only handle one build type, for example, Ninja or Make. For IDE build systems, all variants are generated and you have to use the mechanism of the IDE to switch between build types. Possible values are as follows:
The default build type is taken from the
These are extra flags used when compiling C and C++ source files. The initial values are taken from the
This specifies additional directories that are searched for CMake modules. The specified directories are searched before the default ones. The value is a semicolon-separated list of directories.
If the Python interpreter is not found or if the wrong one is picked in case you have installed multiple versions, you can set this variable to the path of the Python binary. This variable only has an effect if the Python module of CMake is included (which is the case for LLVM).
Table 1.1 - Additional useful variables provided by CMake
$ cmake --help-variable CMAKE_BUILD_TYPE
You can also list all variables with the following command:
$ cmake --help-variable-list
This list is very long. You may want to pipe the output to
more or a similar program.
Using LLVM-defined build configuration variables
The build configuration variables defined by LLVM work in the same way as those defined by CMake except that there is no built-in help. The most useful variables are found in the following tables, where they are divided into variables that are useful for first-time users installing LLVM, and also variables for more advanced LLVM users.
Variables useful for first-time users installing LLVM
LLVM supports code generation for different CPU architectures. By default, all these targets are built. Use this variable to specify the list of targets to build, separated by semicolons. The current targets are
In addition to the official targets, the LLVM source tree also contains experimental targets. These targets are under development and often do not yet support the full functionality of a backend. The current list of experimental targets is
This is a list of the projects you want to build, separated by semicolons. The source for the projects must be on the same level as the
To build clang and bolt together with LLVM, you specify
Table 1.2 - Useful variables for first-time LLVM users
Variables for advanced users of LLVM
If set to
This enables some expensive checks that can really slow down compilation speed or consume large amounts of memory. The default value is
LLVM tools such as
LLVM automatically includes thread support if a threading library is detected (usually the
The LLVM projects do not use C++ exception handling and therefore turn exception support off by default. This setting can be incompatible with other libraries your project is linking with. If needed, you can enable exception support by specifying
LLVM uses a lightweight, self-build system for runtime type information. The generation of C++ RTTI is turned off by default. Like the exception handling support, this may be incompatible with other libraries. To turn generation of C++ RTTI on, you specify
Compiling LLVM should generate no warning messages if possible. The option to print warning messages is therefore turned on by default. To turn it off, you specify
The LLVM source should be C/C++ language standard-conforming; hence, pedantic checking of the source is enabled by default. If possible, compiler-specific extensions are also disabled. To reverse this setting, you specify
If set to
Usually, the tablegen tool is built with the same options as all other parts of LLVM. At the same time, tablegen is used to generate large parts of the code generator. As a result, tablegen is much slower in a debug build, increasing the compile time noticeably. If this option is set to
If the build compiler is gcc or clang, then turning on this option will instruct the compiler to generate the DWARF debug information in a separate file. The reduced size of the object files reduces the link time of debug builds significantly. The default is
Table 1.3 - Useful variables for advanced LLVM users
LLVM defines many more CMake variables. You can find the complete list in the LLVM documentation about CMake https://releases.llvm.org/17.0.1/docs/CMake.html#llvm-specific-variables. The preceding list contains only the ones you are most likely to need.
In this chapter, you prepared your development machine to compile LLVM. You cloned the GitHub repository and compiled your own version of LLVM and clang. The build process can be customized with CMake variables. You learned about useful variables and how to change them. Equipped with this knowledge, you can tweak LLVM to your needs.
In the next section, we will be taking a closer look at the structure of a compiler. We will be exploring the different components found inside the compiler, as well as different types of analyses that occur in it – specifically, the lexical, syntactical, and semantic analyses. Finally, we will also briefly touch on interfacing with an LLVM backend for code generation.