Hands-On System Programming with Go

3 (2 reviews total)
By Alex Guerrieri
  • Instant online access to over 7,500+ books and videos
  • Constantly updated with 100+ new titles each month
  • Breadth and depth in over 1,000+ technologies
  1. An Introduction to System Programming

About this book

System software and applications were largely created using low-level languages such as C or C++. Go is a modern language that combines simplicity, concurrency, and performance, making it a good alternative for building system applications for Linux and macOS.

This Go book introduces Unix and systems programming to help you understand the components the OS has to offer, ranging from the kernel API to the filesystem. You'll then familiarize yourself with Go and its specifications. You'll also learn how to optimize input and output operations with files and streams of data, which are useful tools in building pseudo-terminal applications. You'll gain insights into how processes communicate with each other, and learn about processes and daemon control using signals, pipes, and exit codes. This book will also enable you to understand how to use network communication using various protocols, including TCP and HTTP. As you advance, you'll focus on Go's best feature - concurrency, which will help you handle communication with channels and goroutines, other concurrency tools to synchronize shared resources, and the context package to write elegant applications.

By the end of this book, you will have learned how to build concurrent system applications using Go

Publication date:
July 2019
Publisher
Packt
Pages
458
ISBN
9781789804072

 

An Introduction to System Programming

This chapter is an introduction to system programming, exploring a range of topics from its original definition to how it has shifted in time with system evolution. This chapter provides some basic concepts and an overview of Unix and its resources, including the kernel and the application programming interfaces (API). Many of these concepts are defined here and are used in the rest of the book.

The following topics will be covered in this chapter:

  • What is system programming?
  • Application programming interfaces
  • Understanding how the protection ring works
  • An overview of system calls
  • The POSIX standard
 

Technical requirements

This chapter does not require you to install any special software if you're on Linux.

If you are a Windows user, you can install the Windows Subsystem for Linux (WSL). Follow these steps in order to install WSL:

  1. Open PowerShell as administrator and run the following: 
Enable-WindowsOptionalFeature -Online -FeatureName Microsoft-Windows-Subsystem-Linux
  1. Restart your computer when prompted.
  2. Install your favorite Linux distribution from the Microsoft Store.

 

Beginning with system programming

Over the years, the IT landscape has shifted dramatically. Multicore CPUs that challenge the Von Neumann machine, the internet, and distributed systems are just some of the changes that occurred in the last 30 years. So, where does system programming stand in this landscape?

Software for software

Let's start with the standard textbook definition first.

System programming (or systems programming) is the activity of programming computer system software. The primary distinguishing characteristic of system programming when compared to application programming is that application programming aims to produce software that provides services directly to the user (for example, a word processor), whereas system programming aims to produce software and software platforms that provide services to other software, and are designed to work in performance-constrained environments, for example, operating systems, computational science applications, game engines and AAA video games, industrial automation, and software as a service applications.

The definition highlights two main concepts of what system applications are as follows:

  • Software that is used by other software, not directly by the final user.
  • The software is hardware aware (it knows how the hardware works), and is oriented toward performance.

This makes it possible to easily recognize as system software operating system kernels, hardware drivers, compilers, and debuggers, and not as system software, a chat client, or a word processor.

Historically, system programs were created using Assembly and C. Then came the shells and the scripting languages that were used to tie together the functionality offered by system programs. Another characteristic of system languages was the control of the memory allocation.

Languages and system evolution

In the last decade, scripting languages gained popularity to the point at which some had significant performance improvement and entire systems were built with them. For example, let's just think about the V8 Engine for JavaScript and the PyPy implementation of Python, which dramatically shifted the performance of these languages.

Other languages, such as Go, proved that garbage collection and performance are not mutually exclusive. In particular, Go managed to replace its own memory allocator written in C with a native version written in Go in release 1.5, improving it to the point where the performance was comparable.

In the meantime, systems started to get distributed and the applications started to get shipped in containers, orchestrated by other system software, such as Kubernetes. These systems are meant to sustain huge throughput and achieve it in two main ways:

  • By scaling—augmenting the number or the resources of the machines that are hosting the system
  • By optimizing the software in order to be more resource effective

System programming and software engineering

Some of the practices of system programming—such as having an application that is tied to the hardware, performance oriented, and working on an environment that is resource constrained—are an approach that can also be valid when building distributed systems, where constraining resource usage allows the reduction of the number of instances needed. It looks like system programming is a good way of addressing generic software engineering problems.

This means that learning the concept of system programming with regards to using the resource of the machine efficiently—from memory usage to filesystem access—will be useful when building any type of application.

 

Application programming interfaces

APIs are series subroutine definitions, communication protocols, and tools for building software. The most important aspects of an API are the functionalities that it offers, combined with its documentation, which facilitates the user in the usage and implementation of the software itself in another software. An API can be the interface that allows an application software to use a system software.

An API usually has a specific release policy that is meant to be used by a specific group of recipients. This can be the following:

  • Private and for internal use only
  • Partner and usable by determined groups only—this may include companies that want to integrate the service with theirs
  • Public and available for every user

Types of APIs

We'll see that there are several types of APIs, from the ones used to make different application software work together, to the inner ones exposed by the operating system to other software.

 

Understanding the protection ring

The protection ring, also referred to as hierarchical protection domainsis the mechanism used to protect a system against failure. Its name is derived from the hierarchical structure of its levels of permission, represented by concentric rings, with privilege decreasing when moving to the outside rings. Between each ring there are special gates that allow the outer ring to access the inner ring resources in a restricted manner.

Architectural differences

The number and order of rings depend on the CPU architecture. They are usually numbered with decreasing privilege, making ring 0 the most privileged one. This is true for i386 and x64 architecture that use four rings (from ring 0 to ring 3) but it's not true for ARM, which uses reverse order (from EL3 to EL0). Most operating systems are not using all four levels; they end up using a two level hierarchy—user/application (ring 3) and kernel (ring 0).

Kernel space and user space

A software that runs under an operating system will be executed at user (ring 3) level. In order to access the machine resources, it will have to interact with the operating system kernel (that runs at ring 0). Here's a list of some of the operations a ring 3 application cannot do:

  • Modify the current segment descriptor, which determines the current ring
  • Modify the page tables, preventing one process from seeing the memory of other processes
  • Use the LGDT and LIDT instructions, preventing them from registering interrupt handlers
  • Use I/O instructions such as in and out that would ignore file permissions and read directly from disk

The access to the content of the disk, for instance, will be mediated by the kernel that will verify that the application has permission to access the data. This kind of negotiation improves security and avoids failures, but comes with an important overhead that impacts the application performance.

Some applications can be designed to run directly on the hardware without the framework provided by an operating system. This is true for real-time systems, where there is no compromise on response times and performance.

 

Diving into system calls

System calls are the way operating systems provide access to the resources for the applications. It is an API implemented by the kernel for accessing the hardware safely.

Services provided

There are some categories that we can use to split the numerous functions offered by the operating system. These include the control of the running applications and their flow, the filesystem access, and the network.

Process control

This type of services includes load, which adds a program to memory and prepares for its execution before passing control to the program itself, or execute, which runs an executable file in the context of a pre-existing process. Other operations that belong to this category are as follows:

  • end and abort—the first requires the application to exit while the second forces it.
  • CreateProcess, also known as fork on Unix systems or NtCreateProcess in Windows.
  • Terminate process.
  • Get/set process attributes.
  • Wait for time, wait event, or signal event.
  • Allocate and free memory.

File management

The handling of files and filesystems belongs to file management system calls. There are create and delete files that make it possible to add or remove an entry from the filesystem, and open and close operations that make it possible to gain control of a file in order to execute read and write operations. It is also possible to read and change file attributes.

Device management

Device management handles all other devices but the filesystem, such as frame buffers or display. It includes all operations from the request of a device, including the communication to and from it (read, write, seek), and its release. It also includes all the operations of changing device attributes and logically attaching and detaching them.

Information maintenance

Reading and writing the system date and time belongs to the information maintenance category. This category also takes care of other system data, such as the environment. Another important set of operations that belongs here is the request and the manipulation of processes, files, and device attributes.

Communication

All the network operations from handling sockets to accepting connections fall into the communication category. This includes the creation, deletion, and naming of connections, and sending and receiving messages.

The difference between operating systems

Windows has a series of different system calls that cover all the kernel operations. Many of these correspond exactly with the Unix equivalent. Here's a list of some of the overlapping system calls:

Windows

Unix

Process control

CreateProcess()
ExitProcess()
WaitForSingleObject()

fork()
exit()
wait()

File manipulation CreateFile()
ReadFile()
WriteFile()
CloseHandle()

open()
read()
write()
close()

File protection SetFileSecurity()
InitializeSecurityDescriptor()
SetSecurityDescriptorGroup()

chmod()
umask()
chown()

Device management SetConsoleMode()
ReadConsole()
WriteConsole()
ioctl()
read()
write()
Information maintenance GetCurrentProcessID()
SetTimer()
Sleep()
getpid()
alarm()
sleep()
Communication CreatePipe()
CreateFileMapping()
MapViewOfFile()
pipe()
shmget()
mmap()
 

Understanding the POSIX standard

In order to ensure consistency between operating systems, IEEE formalized some standards for operating systems. These are described in the following sections.

POSIX standards and features

Portable Operating System Interface (POSIX) for Unix represents a series of standards for operating system interfaces. The first version dates back to 1988 and covers a series of topics like filenames, shells, and regular expressions.

There are many features defined by POSIX, and they are organized in four different standards, each one focusing on a different aspect of the Unix compliance. They are all named POSIX followed by a number.

POSIX.1 – core services

POSIX.1 is the 1988 original standard, which was initially named POSIX but was renamed to make it possible to add more standards to the family without giving up the name. It defines the following features:

  • Process creation and control
  • Signals:
  • Floating point exceptions
  • Segmentation/memory violations
  • Illegal instructions
  • Bus errors
  • Timers
  • File and directory operations
  • Pipes
  • C library (standard C)
  • I/O port interface and control
  • Process triggers

POSIX.1b and POSIX.1c – real-time and thread extensions

POSIX.1b focuses on real-time applications and on applications that need high performance. It focus on these aspects:

  • Priority scheduling
  • Real-time signals
  • Clocks and timers
  • Semaphores
  • Message passing
  • Shared memory
  • Asynchronous and synchronous I/O
  • Memory locking interface

POSIX.1c introduces the multithread paradigm and defines the following:

  • Thread creation, control, and cleanup
  • Thread scheduling
  • Thread synchronization
  • Signal handling

POSIX.2 – shell and utilities 

POSIX.2 specifies standards for both a command-line interpreter and utility programs as cd, echo, or ls.

OS adherence

Not all operating systems are POSIX compliant. Windows was born after the standard and it is not compliant, for instance. From a certification point of view, macOS is more compliant than Linux, because the latter uses another standard built on top of POSIX.

Linux and macOS

Most Linux distributions follow the Linux Standard Base (LSB), which is another standard that includes POSIX and much more, focusing on maintaining the inter-compatibility between different Linux distributions. It is not considered officially compliant because the developers didn't go into the process of certification.

However, macOS became fully compatible in 2007 with the Snow Leopard distribution, and it has been POSIX-certified since then.

Windows

Windows is not POSIX compliant, but there are many attempts to make it so. There are open source initiatives such as Cygwin and MinGW that provide a less POSIX-compliant development environment, and support C applications using the Microsoft Visual C runtime library. Microsoft itself has made some attempts at POSIX compatibility, such as the Microsoft POSIX subsystem. The latest compatibility layer made by Microsoft is the Windows Linux Subsystem, which is an optional feature that can be activated in Windows 10 and has been well received by developers (including myself).

 

Summary

In this chapter, we saw what system programming means—writing system software that has some strict requirements, such as being tied to the hardware, using a low-level language, and working in a resource-constrained environment. Its practices can be really useful when building distributed systems that normally require optimizing resource usage. We discussed APIs, definitions that allows software to be used by other software, and listed the different types—the ones in the operating system, libraries and frameworks, and remote and web APIs.

We analyzed how, in operating systems, the access to resources is arranged in hierarchical levels called protection rings that prevent uncontrolled usage in order to improve security and avoid failures from the applications. The Linux model simplifies this hierarchy to just two levels called user and kernel space. All the applications are running in the user space, and in order to access the machine's resources they need the kernel to intercede.

Then we saw one specific type of API called system calls that allows the applications to request resources to the kernel, and mediates process control, access and management of files, and devices and network communications.

We gave an overview of the POSIX standard, which defines Unix system interoperability. Among the features defined, there are also the C API, CLI utilities, shell language, environment variables, program exit status, regular expressions, directory structures, filenames, and command-line utility API conventions.

In the next chapter, we will explore the Unix operating system resources such as the filesystem and the Unix permission model. We will look at what processes are, how they communicate with each other, and how they handle errors.

 

Questions

  1. What is the difference between application and system programming?
  2. What is an API? Why are APIs so important?
  3. Could you explain how protection rings work?
  4. Can you make some examples of what cannot be done in user space?
  5. What's a system call?
  6. Which calls are used in Unix to manage a process?
  7. Why is POSIX useful?
  8. Is Windows POSIX compliant?

About the Author

  • Alex Guerrieri

    Alex Guerrieri is a software developer who specializes in backend and distributed systems. Go has been his favorite tool for the job since first using it in 2013. He holds a degree in computer science engineering and has been working in the field for more than 6 years. Originally from Italy, where he completed his studies and started his career as a full-stack developer, he now lives in Spain, where he previously worked in two start-ups—source{d} and Cabify. He is now working for three companies—as a software crafter for BBVA, one of the biggest Spanish banks; as a software architect for Security First, London, a company focusing on digital and physical security; and as a cofounder of DauMau, the company behind Vidsey, software that generates videos in a procedural manner.

    Browse publications by this author

Latest Reviews

(2 reviews total)
Great to understand, easy to learn
System advertised June 2019 availability and upon purchase customer support told me July end availability.

Recommended For You

Book Title
Unlock this full book FREE 10 day trial
Start Free Trial