Building a BeagleBone Black Super Cluster: Build and configure your own parallel computing Beowulf cluster using BeagleBone Black ARM systems

In general, the power of a computer is defined by the speed of its components. Most critical are the main memory, its bus interconnection, and the central processing unit (CPU). The slowest interconnecting path between faster components is called the bottleneck and defines the overall system's performance. On modern CPUs, the memory interface is on-die, which means that it is integrated into the silicon chip. This offers maximum performance and avoids any bottlenecks between the CPU and the main memory. This is also the case with BBB.

BBB's heart is the Sitara ™ ARM ® Cortex-A8 32-bit processor from Texas Instruments with a programmable clock driven at 1 GHz by factory settings. Interestingly, on some of my boards, the XAM3359AZCZ100 version of the CPU is used, which has a maximum frequency of 800 MHz according to TI. Despite this, it is driven at 1000 MHz on BBB without any problems. According to the manual, it was changed to XAM3358 in the board revision C. The architecture is RISC, which means that the CPU only needs one clock cycle to execute a command. This makes the Cortex-A8 faster than a typical Intel P3 with the same clock frequency if one focuses on standard instructions. Also, the CPU supports the NEON instruction set, which gives BBB a great advantage over Raspberry Pi.

Under full processor utilization, the board gets quite warm. This is not a problem if it is used as a standalone board. However, if there is more than one board in a small area or even if the boards are stacked on top of each other, they will need some cooling in order to avoid overheating. An example of a cheap cooling system is given in Chapter 2, Building a Beowulf Cluster.

The figure in the next section shows us the internal CPU architecture that also provides very sophisticated features such as a touchscreen controller and a 3D graphics acceleration on-die.

The actual purpose of BBB was to provide a development platform that is easy to program and can be used to control or regulate other hardware. In other words, it must be able to measure and output signals that are easy to interface in the hardware and software. To provide this capability, two 46-pin headers are available for general-purpose I/O (GPIO). These pins use a 3.3 volt logic for I/O. A higher voltage level might result in damaging the CPU. The board also provides two programmable real-time units (PRUs) for time-critical applications. Please refer to the hardware manual for a detailed description. In this book, no I/O operation is described, as they are not used in the BBB cluster.

Besides general-purpose I/O, the board also provides the following:

The system can be powered by either the 5V power jack or the USB client port using the preinstalled operating system. However, using the operating system as described in this book, we can see that only the 5V power jack will work. This means that the boards in the final cluster configuration cannot be powered up using a USB power supply. Please refer to Chapter 2, Building a Beowulf Cluster, for information on how to build a cheap and efficient power supply. This book only describes the usage of the RJ45 network interface for installation, configuration, and user control via SSH. Thus, it is not necessary to make use of the HDMI port in cluster applications. The onboard network controller works with 10/100 megabit/s.

The internal architecture of BBB's CPU

The following image and list gives you an overview of the important onboard ports, plugs, and control buttons; a indicates a 5V power jack, b indicates an RJ45 Ethernet port, c indicates a general-purpose I/O with serial header on its right, d indicates a USB host port, e indicates status LEDs, f indicates the reset button, g indicates the power button, h indicates the CPU, i indicates the boot-selection button, j indicates a USB client port, k indicates the microSD slot, and l indicates the mini HDMI port:

While the power and reset buttons are used in the same manner as they are used on PCs, the boot-selection button is used to select the boot sector located in either the internal ROM or on the external microSD card. This function is very useful in order to recover a misconfigured or non-bootable system.

BBB is equipped with 512 MB of DDR3 RAM, which possesses a higher bandwidth compared to the competitor, which is Raspberry Pi, which uses DDR2 RAM. It is not possible to extend the physical RAM on a single BBB; however, this is not important when performing cloud computing because this makes use of distributed memory.

For nonvolatile storage, 2 or 4 GB of 8-bit eMMC flash memory—depending on your board revision—is available onboard. This is enough to install all the required software along with the operating system in order to perform the highly sophisticated computations described in this book. Any dynamic libraries will be installed in a common network shared folder. The storage capacity can also be extended using the free microSD card slot that provides more virtual memory or additional storage capacity. You should refer to the documentation of your specific board version to see supported microSD card sizes. In this book, an additional card with 16 GB size will be used for the master-node board, which will be explained in Chapter 3, Operating System Setup and Configuration.

Whenever you encounter a boot problem and cannot start up your BBB with the operating system on the internal eMMC flash, you can use a failsafe microSD card that you set up earlier in order to boot up the system. For this purpose, you just have to insert the microSD card into the slot and push the boot-selection button before you power on the system and hold it until it boots. It is always a good idea to keep a working system image on a microSD card for such purposes.

Linux operating systems are available for a vast variety of different computer platforms such as PowerPC, x86, IA-64, ARM, and many more. An interesting feature of the CPU used on BBBs is the implementation of floating point instructions. This means that mathematical operations based on non-integer values can be executed by hardware rather than software, and thus they are carried out much faster. ARMhf stands for ARM hard float architecture. To make use of this advantage, a special operating system image is used throughout this book, namely the Ubuntu-12.04-armhf image from John Clark.

The Ubuntu 12.04 Linux distribution, compiled for the ARMhf platform, can be obtained from the home page of John Clark's website at http://www.armhf.com/download/. Please keep in mind that Linux systems and software are updated a lot. If you are not able to get exactly the same software as that used in this book, you can try a newer one or download the version from the download section accompanying this book. You will find more details on how to download and install the operating system in Chapter 2, Building a Beowulf Cluster.

Although there are a lot of already existing helpful software packages, it is very important to understand that Linux is an open source operating system written for an open source community. Usually, Windows users are frustrated when they gain first contact with open source software, because they are used to having already working and easy-to-install software. A huge disadvantage is that these software packets are compiled for a standard platform and might not be optimized to a specific computer that they are installed on. Another problem is that if software components of the operating system are updated but older versions are required by the user software, instabilities might arise or completely different interfaces might disrupt the software functionality completely.

Linux is a highly modular operating system. The whole system is built on the philosophy of open software, which means that every part of the operating system can be compiled from available open source code. This source code is then compiled by standard programming languages such as C, C++, FORTRAN, Assembler, and others in order to build binary code specifically optimized for certain hardware. The technique by which software is built does not differ much from Windows or Linux operating systems. For the beginner, it might be hard to produce a working compiled program starting from source code because usually, there are a lot of software dependencies such as missing software libraries or other programs that code is based upon. In this case, it might be hard to find all the required libraries, especially when newer versions that have changed in interfaces such as function definitions are available. On the other hand, a dependency can soon lead to several others so that the search for all the required libraries grows exponentially and takes a lot of time.

Also, on certain hardware, some well-established compiling parameters do not work and have to be modified or bug fixes have to be found. This can make the simple task of "just compiling software" an unsolvable problem for beginners. Thanks to the rising community of hobby programmers and Linux enthusiasts, there are a lot of forums online that can be searched for such problems. Often, solutions are present, and if not, there can be hints that point us in the right direction, at least.

The following sections will explain the basics of creating software on Linux operating systems with standard programming environments. It is written for hobby enthusiasts who might or might not have already tried and programmed their own software. Although existing knowledge is very helpful, it is not required in order to understand the following explanations.

Each computer program consists of binary code, which means a sequence of two states usually described as zero and one. A specific state is called a bit. Four of these bits make up a so-called nibble and eight make up a byte. Several bytes can be described as a word, a double word, or a quad word. The following table gives you a small summary of the most important data sizes:

A central processor does nothing else except interpreting bit sequences as commands. These commands are called instructions and tell the CPU what to do. This is the lowest level of programming, which is the so-called machine language. Machine language is, except to certain freaky people, not human-readable. For example, it is not obvious that the binary code 1011 0100 0100 1100 1100 1101 0010 0001 is the end of an MS-DOS program.

Low-level programming means the direct programming of machine language. Of course, this has to happen in a human-readable way. One possibility to simplify 1011 0100 0100 1100 is given by using another number system, such as the hexadecimal system, resulting in 0xB4 0x4C. This is better, but it's still not readable by humans. The final simplification is the invention of so-called mnemonics. For example, on Intel x86-platforms, 0xB4 0x4C would mean mov ah, 0x4C in this mnemonics language. Now, one can understand that this code sets the CPU register named ah to the value of 0x4C. This language as well as the software that translates this back into bits is called Assembler.

Assembler has its advantages and disadvantages. One big advantage is that the resulting software does exactly what you programmed. This means that there is no optimization that modifies your code and you can program very effectively in size and speed. One big disadvantage, however, is the problem that each CPU has its own instruction set. This means that our Sitara CPU will not understand the preceding example, because it has no ah register. For any real problems we want to solve using computers, we are primarily interested in the nature of the problem and not the nature of the CPU used in the computer. To make programming independent of the used computer platform, there exist so-called high-level programming languages.

High-level programming languages consist of keywords, syntax, and grammar, as with every spoken language. The keywords define the vocabulary that can be used, whereas syntax and grammar define the exact utilization and order of these keywords. To understand this, we should have a look at how a simple loop will look in a low-level language compared to a high-level language.

The following example shows us that a simple loop already needs four different instructions in Assembler, while it can be realized by a relatively simple for keyword in the high-level language C++:

mov cx, 0
@mark1:
    other code..
inc cx
cmp cx, 345
jne @mark1

for (int j = 0; j < 345; j++) {
    other code...
}

While C++ is a general-purpose high-level programming language, there are also languages that are more specifically optimized. One example is FORTRAN, which is mainly used for mathematical problems due to its ability to define matrices and other mathematical structures very easily.

Once the required code has been written and is ready to be translated from its human-readable form into machine language, there is a certain sequence of tools that have to be used. This toolset is called the compiler toolchain:

The whole process is depicted in the following diagram:

Software compilation and simple toolchain

Another important feature of the linker is its capability to embed the required libraries into the executable file. This is called static linking. The program can then be used on other computers that do not have that specific library installed. The opposite of static linking is dynamic linking. In this case, only a stub of the library is linked into the program that tells the OS which library to provide. Dynamically-linked programs are smaller in size but always need their libraries.

In all examples, the main focus will be on C++. Some of the modules are only available as FORTRAN code; however, once compiled, their functions can also be accessed from C++ programs.