Interacting with the Operating System

This section is an add-on to the introduction to Python. It puts Python into the context of the operating system on your computer and shows how Python is used in a command window, also called a console. We demonstrate how system commands and a Python command interact with each other and how you make a new application.

Among many other operating systems and various dialects, there are three main operating systems in use on desktop computers and notebooks: Windows 10, macOS, and Linux. In this chapter, we discuss how to use Python within the Linux world. All examples are tested in the widely distributed Ubuntu environment. The principles presented here apply also to the other big operating systems. The presentation is nevertheless restricted to Linux as a completely freely accessible environment.

First, we assume we have written and tested...

17.1 Running a Python program in a Linux shell

When you open a terminal window with the terminal app, you obtain a window with a command prompt; see Figure 17.1:

Figure 17.1: A terminal window in Ubuntu 20.04

The terminal windows come with a command prompt, often prefixed by the username and the computer's name followed by the directory name. This depends on the individual settings.

To execute the Python commands written in a file named myprogram.py, you have two choices:

• Executing the command python myprogram.py
• Executing the command myprogram.py directly

The second variant needs some preparation. First, you have to give permission to execute that file, and secondly, you have to tell the system which command is needed to execute that file. The permission to execute that file is given by the command:

chmod myprogram.py o+x

chmod stands for changing the file mode. The command is followed by the filename and finally the desired new modes, here o+x.

The modes that are given...

17.2 The module sys

The module sys provides tools to communicate from a Python script with system commands. We can provide the Python script as a command directly from the command line with arguments and we can output the results to the console.

17.2.1 Command-line arguments

To illustrate the use of command-line arguments, we consider the following piece of code, which we save in a file called demo_cli.py:

#! /usr/bin/env  python 
import sys
text=f"""
You called the program{sys.argv[0]}
with the {len(sys.argv)-1} arguments, namely {sys.argv[1:]}"""
print(text)

After giving execution permissions to the file by chmod o+x demo_cli.py, we can execute it in the shell with arguments; see Figure 17.3:

Figure 17.3: Executing a Python script with three arguments on a terminal command line

The three arguments given in the console are accessible in the Python script via the list sys.argv. The first element in this list—the element with index 0—is the name of the script. The other elements are the given arguments as strings.

Arguments are given to the call of the Python script. They should not be confounded with user input during the execution of a script. 

17.2.2 Input and output streams

In the preceding example, we used the command print to display the generated message in the terminal (or even in the Python shell). A priory input to the script is obtained via arguments and the variable sys.argv. The counterpart to print is the command input, which prompts for data from the terminal (or from the Python shell).

In Section 14.1: File handling, we saw how to provide a script with data and how to output data from a script by the use of file objects and related methods. The module sys makes it possible to treat the keyboard as a file object for input (for example, readline, readlines) and the console as a file object for output (for example, write, writelines).

The information flow is organized in UNIX by three streams:

• The standard input stream: STDIN  
• The standard output stream: STDOUT
• The standard error stream: STDERR

These streams correspond to file objects that can be accessed in Python...

Redirecting streams

The standard input is expecting a data stream from the keyboard. But the input can be redirected from a file instead. This is done by using the redirection symbol < in Linux. We demonstrate this by using the same script as previously, but now a data file, intest.txt, provides the script with data; see Figure 17.5:

Figure 17.5: Screenshot to demonstrate the redirection of input (sys.stdin)

No modification in the script itself is required. It can be used in either way.

The same holds for outputs. By default the output is displayed in the terminal, but also here there is the option to redirect the output to a file. In that case, the redirect symbol is >; see Figure 17.6:

Figure 17.6: Screenshot of a redirected input and redirected output

Now, a file with the name result.txt is created and the output of the script is written to it. If there was already a file with this name, its content is overwritten. If instead the output should be appended to an already-existing...

Building a pipe between a Linux command and a Python script

In the last section, we saw how to redirect the input and output of Python programs to files. The data flow between different Python programs or between a Python program and a Linux command goes, in that case, via a file. If the data is not used elsewhere or should be saved for later use, this is a tedious process: creating, naming, and deleting a file just for directly passing information from one piece of code to another. The alternative is to use a Linux pipe that lets the data flow in a direct stream from one command to another.

Let's start with a pure Linux example and then apply the pipe construction to Python.

The Linux command ifconfig displays a lot of information about the actual network configuration of a Linux computer. Among this information, you find the IP number(s), which are the current network addresses in use. To automatically find out whether a computer, for example, a notebook, is connected via a certain...

17.3 How to execute Linux commands from Python

We saw in the last section how to execute a Python command from the Linux terminal. In this section, we consider the important case of how to execute Linux commands within a Python program.

17.3.1 The modules subprocess and shlex

To execute system commands within Python, we need to import the module subprocess first. The high-level tool provided by this module is run. With this tool, you quickly access Linux commands within Python and can process their output.

The more sophisticated tool is Popen, which we will shortly present when explaining how to mimic Linux pipes in Python.

A complete process: subprocess.run

We demonstrate this tool with the most standard and simple UNIX command, ls—the command for listing the content of a directory. It comes with various optional arguments; for example, ls -l displays the list with extended information.

To execute this command within a Python script, we use subprocess.run. The simplest usage is using only one argument, a list with the Linux command split into several text strings:

import subprocess as sp
res = sp.run(['ls','-l'])

The module shlex provides a special tool for performing this split: 

_import shlex
command_list = shlex.split('ls -l') # returns ['ls', '-l']

It also respects empty spaces in filenames and does not use those as separators.

The command run displays the result of the Linux command and the subprocess.CompletedProcess object res.

To execute UNIX commands in this way is quite useless. Mostly, you want to process the output. Therefore...

Creating processes: subprocess.Popen

What happens when you apply subprocess.run on a Linux command that starts a process that requires user input to terminate?

A simple example of such a program is xclock. It opens a new window displaying a clock until the window is closed by the user.

As the command subprocess.run creates a CompletedProcess object, the following Python script:

import subprocess as sp
res=sp.run(['xclock'])

starts a process and waits until it ends, that is, until somebody closes the window with the clock; see Figure 17.9:

Figure 17.9: The xclock window

This makes a difference to subprocess.Popen. It creates a _Popen object. The process itself becomes a Python object. It need not be completed to become an accessible Python object:

import subprocess as sp
p=sp.Popen(['xclock'])

The process is completed by either a user action on the clock window or by explicitly terminating the process with:

p.terminate()

With Popen, we can construct Linux pipes in Python...

17.4 Summary

In this chapter, we demonstrated the interaction of a Python script with system commands. Either a Python script can be called in a way as if it would be a system command or a Python script can itself create system processes. The chapter is based on Linux systems such as Ubuntu and serves only as a demonstration of concepts and possibilities. It allows putting scientific computing tasks in an application context, where often different software have to be combined. Even hardware components might come into play.