Automating Your System Administration and Deployment Tasks Over SSH

Exclusive offer: get 50% off this eBook here
Python Network Programming Cookbook

Python Network Programming Cookbook — Save 50%

Over 70 detailed recipes to develop practical solutions for a wide range of real-world network programming tasks with this book and ebook

$26.99    $13.50
by Dr. M. O. Faruque Sarker | March 2014 | Cookbooks Networking & Telephony

In this article, by Dr. M. O. Faruque Sarker, author of Python Network Programming Cookbook, we will cover the following recipes:

  • Executing a remote shell command using telnet
  • Copying a file to a remote machine by SFTP
  • Printing a remote machine's CPU information
  • Installing a Python package remotely
  • Running a MySQL command remotely

This article promotes some interesting Python libraries. The recipes are presented aiming at the system administrators and advanced Python programmers who like to write code that connects to remote systems and executes commands. The article begins with lightweight recipes with a built-in Python library, telnetlib. It then brings Paramiko, a well-known remote access library. Finally, the powerful remote system administration library, fabric, is presented. The fabric library is loved by developers who regularly script for automatic deployments, for example, deploying web applications or building custom application binaries.

(For more resources related to this topic, see here.)

Executing a remote shell command using telnet

If you need to connect an old network switch or router via telnet, you can do so from a Python script instead of using a bash script or an interactive shell. This recipe will create a simple telnet session. It will show you how to execute shell commands to the remote host.

Getting ready

You need to install the telnet server on your machine and ensure that it's up and running. You can use a package manager that is specific to your operating system to install the telnet server package. For example, on Debian/Ubuntu, you can use apt-get or aptitude to install the telnetd package, as shown in the following command:

$ sudo apt-get install telnetd $ telnet localhost

How to do it...

Let us define a function that will take a user's login credentials from the command prompt and connect to a telnet server.

Upon successful connection, it will send the Unix 'ls' command. Then, it will display the output of the command, for example, listing the contents of a directory.

Listing 7.1 shows the code for a telnet session that executes a Unix command remotely as follows:

#!/usr/bin/env python # Python Network Programming Cookbook -- Chapter - 7 # This program is optimized for Python 2.7. # It may run on any other version with/without modifications. import getpass import sys import telnetlib     def run_telnet_session():   host = raw_input("Enter remote hostname e.g. localhost:")   user = raw_input("Enter your remote account: ")   password = getpass.getpass()     session = telnetlib.Telnet(host)     session.read_until("login: ")   session.write(user + "\n")   if password:     session.read_until("Password: ")     session.write(password + "\n")       session.write("ls\n")   session.write("exit\n")     print session.read_all()   if __name__ == '__main__':   run_telnet_session()

If you run a telnet server on your local machine and run this code, it will ask you for your remote user account and password. The following output shows a telnet session executed on a Debian machine:

$ python 7_1_execute_remote_telnet_cmd.py Enter remote hostname e.g. localhost: localhost Enter your remote account: faruq Password:   ls exit Last login: Mon Aug 12 10:37:10 BST 2013 from localhost on pts/9 Linux debian6 2.6.32-5-686 #1 SMP Mon Feb 25 01:04:36 UTC 2013 i686   The programs included with the Debian GNU/Linux system are free software; the exact distribution terms for each program are described in the individual files in /usr/share/doc/*/copyright.   Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent permitted by applicable law. You have new mail. faruq@debian6:~$ ls       down              Pictures               Videos Downloads         projects               yEd Dropbox           Public env               readme.txt faruq@debian6:~$ exit logout

How it works...

This recipe relies on Python's built-in telnetlib networking library to create a telnet session. The run_telnet_session() function takes the username and password from the command prompt. The getpass module's getpass() function is used to get the password as this function won't let you see what is typed on the screen.

In order to create a telnet session, you need to instantiate a Telnet() class, which takes a hostname parameter to initialize. In this case, localhost is used as the hostname. You can use the argparse module to pass a hostname to this script.

The telnet session's remote output can be captured with the read_until() method. In the first case, the login prompt is detected using this method. Then, the username with a new line feed is sent to the remote machine by the write() method (in this case, the same machine accessed as if it's remote). Similarly, the password was supplied to the remote host.

Then, the ls command is sent to be executed. Finally, to disconnect from the remote host, the exit command is sent, and all session data received from the remote host is printed on screen using the read_all() method.

Copying a file to a remote machine by SFTP

If you want to upload or copy a file from your local machine to a remote machine securely, you can do so via Secure File Transfer Protocol (SFTP).

Getting ready

This recipe uses a powerful third-party networking library, Paramiko, to show you an example of file copying by SFTP, as shown in the following command. You can grab the latest code of Paramiko from GitHub (https://github.com/paramiko/paramiko) or PyPI:

$ pip install paramiko

How to do it...

This recipe takes a few command-line inputs: the remote hostname, server port, source filename, and destination filename. For the sake of simplicity, we can use default or hard-coded values for these input parameters.

In order to connect to the remote host, we need the username and password, which can be obtained from the user from the command line.

Listing 7.2 explains how to copy a file remotely by SFTP, as shown in the following code:

#!/usr/bin/env python # Python Network Programming Cookbook -- Chapter - 7 # This program is optimized for Python 2.7. # It may run on any other version with/without modifications.   import argparse import paramiko import getpass     SOURCE = '7_2_copy_remote_file_over_sftp.py' DESTINATION ='/tmp/7_2_copy_remote_file_over_sftp.py '     def copy_file(hostname, port, username, password, src, dst):   client = paramiko.SSHClient()   client.load_system_host_keys()   print " Connecting to %s \n with username=%s... \n" %(hostname,username)   t = paramiko.Transport((hostname, port))   t.connect(username=username,password=password)   sftp = paramiko.SFTPClient.from_transport(t)   print "Copying file: %s to path: %s" %(SOURCE, DESTINATION)   sftp.put(src, dst)   sftp.close()   t.close()     if __name__ == '__main__':   parser = argparse.ArgumentParser(description='Remote file copy')   parser.add_argument('--host', action="store", dest="host", default='localhost')   parser.add_argument('--port', action="store", dest="port", default=22, type=int)   parser.add_argument('--src', action="store", dest="src", default=SOURCE)   parser.add_argument('--dst', action="store", dest="dst", default=DESTINATION)     given_args = parser.parse_args()   hostname, port =  given_args.host, given_args.port   src, dst = given_args.src, given_args.dst     username = raw_input("Enter the username:")   password = getpass.getpass("Enter password for %s: " %username)     copy_file(hostname, port, username, password, src, dst)

If you run this script, you will see an output similar to the following:

$ python 7_2_copy_remote_file_over_sftp.py Enter the username:faruq Enter password for faruq:  Connecting to localhost  with username=faruq... Copying file: 7_2_copy_remote_file_over_sftp.py to path: /tmp/7_2_copy_remote_file_over_sftp.py

How it works...

This recipe can take the various inputs for connecting to a remote machine and copying a file over SFTP.

This recipe passes the command-line input to the copy_file() function. It then creates a SSH client calling the SSHClient class of paramiko. The client needs to load the system host keys. It then connects to the remote system, thus creating an instance of the transport class. The actual SFTP connection object, sftp, is created by calling the SFTPClient.from_transport() function of paramiko. This takes the transport instance as an input.

After the SFTP connection is ready, the local file is copied over this connection to the remote host using the put() method.

Finally, it's a good idea to clean up the SFTP connection and underlying objects by calling the close() method separately on each object.

Python Network Programming Cookbook Over 70 detailed recipes to develop practical solutions for a wide range of real-world network programming tasks with this book and ebook
Published: March 2014
eBook Price: $26.99
Book Price: $44.99
See more
Select your format and quantity:

Printing a remote machine's CPU information

Sometimes, we need to run a simple command on a remote machine over SSH. For example, we need to query the remote machine's CPU or RAM information. This can be done from a Python script as shown in this recipe.

Getting ready

You need to install the third-party package, Paramiko, as shown in the following command, from the source available from GitHub's repository at https://github.com/paramiko/paramiko:

$ pip install paramiko

How to do it...

We can use the paramiko module to create a remote session to a Unix machine.

Then, from this session, we can read the remote machine's /proc/cpuinfo file to extract the CPU information.

Listing 7.3 gives the code for printing a remote machine's CPU information, as follows:

#!/usr/bin/env python # Python Network Programming Cookbook -- Chapter - 7 # This program is optimized for Python 2.7. # It may run on any other version with/without modifications.   import argparse import getpass import paramiko   RECV_BYTES = 4096 COMMAND = 'cat /proc/cpuinfo'   def print_remote_cpu_info(hostname, port, username, password):   client = paramiko.Transport((hostname, port))   client.connect(username=username, password=password)     stdout_data = []   stderr_data = []   session = client.open_channel(kind='session')   session.exec_command(COMMAND)   while True:     if session.recv_ready():       stdout_data.append(session.recv(RECV_BYTES))       if session.recv_stderr_ready():         stderr_data.append(session.recv_stderr(RECV_BYTES))       if session.exit_status_ready():         break        print 'exit status: ', session.recv_exit_status()   print ''.join(stdout_data)   print ''.join(stderr_data)     session.close()   client.close()   if __name__ == '__main__':   parser = argparse.ArgumentParser(description='Remote file copy')   parser.add_argument('--host', action="store", dest="host", default='localhost')   parser.add_argument('--port', action="store", dest="port", default=22, type=int)      given_args = parser.parse_args()   hostname, port =  given_args.host, given_args.port     username = raw_input("Enter the username:")   password = getpass.getpass("Enter password for %s: " %username)   print_remote_cpu_info(hostname, port, username, password)

Running this script will show the CPU information of a given host, in this case, the local machine, as follows:

$ python 7_3_print_remote_cpu_info.py Enter the username:faruq Enter password for faruq: exit status:  0 processor    : 0 vendor_id    : GenuineIntel cpu family    : 6 model        : 42 model name    : Intel(R) Core(TM) i5-2400S CPU @ 2.50GHz stepping    : 7 cpu MHz        : 2469.677 cache size    : 6144 KB fdiv_bug    : no hlt_bug        : no f00f_bug    : no coma_bug    : no fpu        : yes fpu_exception    : yes cpuid level    : 5 wp        : yes flags        : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca
cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx rdtscp lm
constant_tsc up pni monitor ssse3 lahf_lm
bogomips    : 4939.35 clflush size    : 64 cache_alignment    : 64 address sizes    : 36 bits physical, 48 bits virtual power management:

How it works...

First, we collect the connection parameters such as hostname, port, username, and password. These parameters are then passed to the print_remote_cpu_info() function.

This function creates an SSH client session by calling the transport class of paramiko. The connection is made thereafter using the supplied username and password. We can create a raw communication session using open_channel() on the SSH client. In order to execute a command on the remote host, exec_command() can be used.

After sending the command to the remote host, the response from the remote host can be caught by blocking the recv_ready() event of the session object. We can create two lists, stdout_data and stderr_data, and use them to store the remote output and error messages.

When the command exits in the remote machine, it can be detected using the exit_status_ready() method, and the remote session data can be concatenated using the join() string method.

Finally, the session and client connection can be closed using the close() method on each object.

Installing a Python package remotely

While dealing with the remote host in the previous recipes, you may have noticed that we need to do a lot of stuff related to the connection setup. For efficient execution, it is desirable that they become abstract and only the relevant high-level part is exposed to the programmers. It is cumbersome and slow to always explicitly set up connections to execute commands remotely.

Fabric (http://fabfile.org/), a third-party Python module, solves this problem. It only exposes as many APIs as can be used to efficiently interact with remote machines.

In this recipe, a simple example of using Fabric will be shown.

Getting ready

We need Fabric to be installed first. You can install Fabric using the Python packing tools, pip or easy_install, as shown in the following command. Fabric relies on the paramiko module, which will be installed automatically.

$ pip install fabric

Here, we will connect the remote host using the SSH protocol. So, it's necessary to run the SSH server on the remote end. If you like to test with your local machine (pretending to access as a remote machine), you may install the openssh server package locally. On a Debian/Ubuntu machine, this can be done with the package manager, apt-get, as shown in the following command:

$ sudo apt-get install openssh-server

How to do it...

Here's the code for installing a Python package using Fabric.

Listing 7.4 gives the code for installing a Python package remotely as follows:

#!/usr/bin/env python # Python Network Programming Cookbook -- Chapter - 7 # This program is optimized for Python 2.7. # It may run on any other version with/without modifications.   from getpass import getpass from fabric.api import settings, run, env, prompt   def remote_server():   env.hosts = ['127.0.0.1']   env.user = prompt('Enter user name: ')   env.password = getpass('Enter password: ')   def install_package():   run("pip install yolk")

Fabric scripts are run in a different way as compared to the normal Python scripts. All functions using the fabric library must be referred to a Python script called fabfile.py. There's no traditional __main__ directive in this script. Instead, you can define your method using the Fabric APIs and execute these methods using the command-line tool, fab. So, instead of calling python <script>.py, you can run a Fabric script, which is defined in a fabfile.py script and located under the current directory, by calling fab one_function_name another_function_name.

So, let's create a fabfile.py script as shown in the following command. For the sake of simplicity, you can create a file shortcut or link from any file to a fabfile.py script. First, delete any previously created fabfile.py file and create a shortcut to fabfile:

$ rm -rf fabfile.py $ ln -s 7_4_install_python_package_remotely.py fabfile.py

If you call the fabfile now, it will produce the following output after installing the Python package, yolk, remotely as follows:

$ ln -sfn 7_4_install_python_package_remotely.py fabfile.py $ fab remote_server install_package Enter user name: faruq Enter password: [127.0.0.1] Executing task 'install_package' [127.0.0.1] run: pip install yolk [127.0.0.1] out: Downloading/unpacking yolk [127.0.0.1] out:   Downloading yolk-0.4.3.tar.gz (86kB): [127.0.0.1] out:   Downloading yolk-0.4.3.tar.gz (86kB): 100%  86kB [127.0.0.1] out:   Downloading yolk-0.4.3.tar.gz (86kB):           [127.0.0.1] out:   Downloading yolk-0.4.3.tar.gz (86kB): 86kB downloaded [127.0.0.1] out:   Running setup.py egg_info for package yolk [127.0.0.1] out:     Installing yolk script to /home/faruq/env/bin [127.0.0.1] out: Successfully installed yolk [127.0.0.1] out: Cleaning up... [127.0.0.1] out:   Done. Disconnecting from 127.0.0.1... done.

How it works...

This recipe demonstrates how a system administration task can be done remotely using a Python script. There are two functions present in this script. The remote_server() function sets up the Fabric env environment variables, for example, the hostname, user, password, and so on.

The other function, install_package(), calls the run() function. This takes the commands that you usually type in the command line. In this case, the command is pip install yolk. This installs the Python package, yolk, with pip. As compared to the previously described recipes, this method of running a remote command using Fabric is easier and more efficient.

Python Network Programming Cookbook Over 70 detailed recipes to develop practical solutions for a wide range of real-world network programming tasks with this book and ebook
Published: March 2014
eBook Price: $26.99
Book Price: $44.99
See more
Select your format and quantity:

Running a MySQL command remotely

If you ever need to administer a MySQL server remotely, this recipe is for you. It will show you how to send database commands to a remote MySQL server from a Python script. If you need to set up a web application that relies on a backend database, this recipe can be used as a part of your web application setup process.

Getting ready

This recipe also needs Fabric to be installed first. You can install Fabric using the Python packing tools, pip or easy_install, as shown in the following command. Fabric relies on the paramiko module, which will be installed automatically.

$ pip install fabric

Here, we will connect the remote host using the SSH protocol. So, it's necessary to run the SSH server on the remote end. You also need to run a MySQL server on the remote host. On a Debian/Ubuntu machine, this can be done with the package manager, apt-get, as shown in the following command:

$ sudo apt-get install openssh-server mysql-server

How to do it...

We defined the Fabric environment settings and a few functions for administering MySQL remotely. In these functions, instead of calling the mysql executable directly, we send the SQL commands to mysql via echo. This ensures that arguments are passed properly to the mysql executable.

Listing 7.5 gives the code for running MySQL commands remotely, as follows:

#!/usr/bin/env python # Python Network Programming Cookbook -- Chapter - 7 # This program is optimized for Python 2.7. # It may run on any other version with/without modifications. from getpass import getpass from fabric.api import run, env, prompt, cd   def remote_server():   env.hosts = ['127.0.0.1'] # Edit this list to include remote hosts   env.user =prompt('Enter your system username: ')   env.password = getpass('Enter your system user password: ')   env.mysqlhost = 'localhost'   env.mysqluser = 'root'prompt('Enter your db username: ')   env.password = getpass('Enter your db user password: ')   env.db_name = ''   def show_dbs():   """ Wraps mysql show databases cmd"""   q = "show databases"   run("echo '%s' | mysql -u%s -p%s" %(q, env.mysqluser, env.mysqlpassword))     def run_sql(db_name, query):   """ Generic function to run sql"""   with cd('/tmp'):     run("echo '%s' | mysql -u%s -p%s -D %s" %(query, env.mysqluser, env.mysqlpassword, db_name))   def create_db():   """Create a MySQL DB for App version"""   if not env.db_name:     db_name = prompt("Enter the DB name:")   else:     db_name = env.db_name   run('echo "CREATE DATABASE %s default character set utf8 collate utf8_unicode_ci;"|mysql --batch --user=%s --password=%s -- host=%s'\     % (db_name, env.mysqluser, env.mysqlpassword, env.mysqlhost), pty=True)   def ls_db():   """ List a dbs with size in MB """   if not env.db_name:     db_name = prompt("Which DB to ls?")   else:     db_name = env.db_name   query = """SELECT table_schema                                        "DB Name",   Round(Sum(data_length + index_length) / 1024 / 1024, 1) "DB Size in MB"     FROM   information_schema.tables             WHERE table_schema = \"%s\"     GROUP  BY table_schema """ %db_name   run_sql(db_name, query)     def empty_db():   """ Empty all tables of a given DB """   db_name = prompt("Enter DB name to empty:")   cmd = """   (echo 'SET foreign_key_checks = 0;';   (mysqldump -u%s -p%s --add-drop-table --no-data %s |   grep ^DROP);   echo 'SET foreign_key_checks = 1;') | \   mysql -u%s -p%s -b %s   """ %(env.mysqluser, env.mysqlpassword, db_name, env.mysqluser, env.mysqlpassword, db_name)   run(cmd)

In order to run this script, you should create a shortcut, fabfile.py. From the command line, you can do this by typing the following command:

$ ln -sfn 7_5_run_mysql_command_remotely.py fabfile.py

Then, you can call the fab executable in various forms.

The following command will show a list of databases (using the SQL query, show databases):

$ fab remote_server show_dbs

The following command will create a new MySQL database. If you haven't defined the Fabric environment variable, db_name, a prompt will be shown to enter the target database name. This database will be created using the SQL command, CREATE DATABASE <database_name> default character set utf8 collate utf8_unicode_ci;.

$ fab remote_server create_db

This Fabric command will show the size of a database:

$ fab remote_server ls_db()

The following Fabric command will use the mysqldump and mysql executables to empty a database. This behavior of this function is similar to the truncating of a database, except it removes all the tables. The result is as if you created a fresh database without any tables:

$ fab remote_server empty_db()

The following will be the output:

$ $ fab remote_server show_dbs [127.0.0.1] Executing task 'show_dbs' [127.0.0.1] run: echo 'show databases' | mysql -uroot -p<DELETED> [127.0.0.1] out: Database [127.0.0.1] out: information_schema [127.0.0.1] out: mysql [127.0.0.1] out: phpmyadmin [127.0.0.1] out:     Done. Disconnecting from 127.0.0.1... done.   $ fab remote_server create_db [127.0.0.1] Executing task 'create_db' Enter the DB name: test123 [127.0.0.1] run: echo "CREATE DATABASE test123 default character set utf8
collate utf8_unicode_ci;"|mysql --batch --user=root --password=
<DELETED> --host=localhost
  Done. Disconnecting from 127.0.0.1... done. $ fab remote_server show_dbs [127.0.0.1] Executing task 'show_dbs' [127.0.0.1] run: echo 'show databases' | mysql -uroot -p<DELETED> [127.0.0.1] out: Database [127.0.0.1] out: information_schema [127.0.0.1] out: collabtive [127.0.0.1] out: test123 [127.0.0.1] out: testdb [127.0.0.1] out: Done. Disconnecting from 127.0.0.1... done.

How it works...

This script defines a few functions that are used with Fabric. The first function, remote_server(), sets the environment variables. The local loopback IP (127.0.0.1) is put to the list of hosts. The local system user and MySQL login credentials are set and collected via getpass().

The other function utilizes the Fabric run() function to send MySQL commands to the remote MySQL server by echoing the command to the mysql executable.

The run_sql() function is a generic function that can be used as a wrapper in other functions. For example, the empty_db() function calls it to execute the SQL commands. This can keep your code a bit more organized and cleaner.

Summary

In this article we gave you a taste of automating your system administration and deployment tasks over SSH. You can now run commands, install packages, or set up new websites remotely from your laptop.

Resources for Article:


Further resources on this subject:


About the Author :


Dr. M. O. Faruque Sarker

Dr. M. O. Faruque Sarker is a software architect, and DevOps engineer who's currently working at University College London (UCL), United Kingdom. In recent years, he has been leading a number of Python software development projects, including the implementation of an interactive web-based scientific computing framework using the IPython Notebook service at UCL. He is a specialist and an expert in open source technologies, for example, e-learning and web application platforms, agile software development, and IT service management methodologies such as DSDM Atern and ITIL Service management frameworks.

Dr. Sarker received his PhD in multirobot systems from University of South Wales where he adopted various Python open source projects for integrating the complex software infrastructure of one of the biggest multirobot experiment testbeds in UK. To drive his multirobot fleet, he designed and implemented a decoupled software architecture called hybrid event-driven architecture on D-Bus. Since 1999, he has been deploying Linux and open source software in commercial companies, educational institutions, and multinational consultancies. He was invited to work on the Google Summer of Code 2009/2010 programs for contributing to the BlueZ and Tahoe-LAFS open source projects.

Currently, Dr. Sarker has a research interest in self-organized cloud architecture. In his spare time, he likes to play with his little daughter, Ayesha, and is keen to learn about child-centric educational methods that can empower children with self-confidence by engaging with their environment.

Books From Packt


Python 3 Object Oriented Programming
Python 3 Object Oriented Programming

Python Multimedia
Python Multimedia

Expert Python Programming
Expert Python Programming

Python Data Visualization Cookbook
Python Data Visualization Cookbook

Python Geospatial Development - Second Edition
Python Geospatial Development - Second Edition

Python 2.6 Graphics Cookbook
Python 2.6 Graphics Cookbook

Instant Pygame for Python Game Development How-to
Instant Pygame for Python Game Development How-to

Spring Python 1.1
Spring Python 1.1


Code Download and Errata
Packt Anytime, Anywhere
Register Books
Print Upgrades
eBook Downloads
Video Support
Contact Us
Awards Voting Nominations Previous Winners
Judges Open Source CMS Hall Of Fame CMS Most Promising Open Source Project Open Source E-Commerce Applications Open Source JavaScript Library Open Source Graphics Software
Resources
Open Source CMS Hall Of Fame CMS Most Promising Open Source Project Open Source E-Commerce Applications Open Source JavaScript Library Open Source Graphics Software