Mastering Python Networking

Hosts are end nodes on the networks that communicate to other nodes. In today's world, a host can be the traditional computer, or it can be your smart phone, tablet, or TV. With the rise of Internet of Things (IoT), the broad definition of host can be expanded to include IP camera, TV set-top boxes, and the ever-increasing type of sensors that we use in agriculture, farming, automobiles, and more. With the explosion of the number of hosts connected to the internet, all of them need to be addressed, routed, and managed, and the demand for proper networking has never been greater.

Most of the time when we are on the internet, we request for services. The service can be a web page, sending or receiving emails, transferring files, and such. These services are provided by servers. As the name implies, they provide services to multiple nodes and generally have higher levels of hardware specification because of it. In a way, servers are special super nodes on the network that provide additional capabilities to its peers. We will look at servers later on in the client-server section.

If one thinks of servers and hosts as cities and towns, the network components are the roads and highways to connect them together. In fact, the term "information superhighway" comes to mind when describing the network components that transmit the ever increasing bits and bytes across the globe. In the OSI model that we will look at in a bit, these network components are the routers and switches that reside on layer two and three of the model as well as the layer on fiber optics cable, coaxial cable, twisted copper pairs, and some DWDM equipment, to name a few.

Collectively, the hosts, servers, and network components make up the internet as we know today.

In the last section, we looked at different roles that servers, hosts, and network components play in the internetwork. Because of the higher hardware capacity that the servers demand, they are often put together in a central location, so they can be managed more efficiently. We will refer to these locations as datacenters.

Cloud datacenters

With the rise of cloud computing and software or infrastructure as a service, we can say that the datacenters cloud providers build the cloud datacenters. Because of the number of servers they house, they generally demand a much, much higher capacity of power, cooling, network speed, and feed than any enterprise datacenter. In fact, the cloud datacenters are so big they are typically built close to power sources where they can get the cheapest rate for power, without losing too much during transportation of the power. They can also be creative when it comes to cooling down where the datacenter might be build in a generally cold climate, so they can just open the doors and windows to keep the server running at a safe temperature. Any search engine can give you some of the astounding numbers when it comes to the science of building and managing these cloud datacenters for the likes of Amazon, Microsoft, Google, and Facebook:

Utah Data Center (source: https://en.wikipedia.org/wiki/Utah_Data_Center)

The services that the servers at datacenters need to provide are generally not cost efficient to be housed in any single server. They are spread among a fleet of servers, sometimes across many different racks to provide redundancy and flexibility for service owners. The latency and redundancy requirements put a tremendous amount of pressure on the network. The number of interconnection equates to an explosive growth of network equipment; this translates into the number of times these network equipment need to be racked, provisioned, and managed.

CLOS Network (source: https://en.wikipedia.org/wiki/Clos_network)

In a way, the cloud datacenter is where network automation becomes a necessity. If we follow the traditional way of managing the network devices via a Terminal and command-line interface, the number of engineering hours required would not allow the service to be available in a reasonable amount of time. This is not to mention that human repetition is error prone, inefficient, and a terrible waste of engineering talent.

The cloud datacenter is where the author started his path of network automation with Python a number of years ago, and never looked back since.

The Transmission Control Protocol (TCP) is one of the main protocols used on the internet today. If you have opened a web page or have sent an e-mail, you have come across the TCP protocol. The protocol sits at layer 4 of the OSI model, and it is responsible for delivering the data segment between two nodes in a reliable and error-checked manner. The TCP consists of a 128-bit header consists of, among others, source and destination port, sequence number, acknowledgment number, control flags, and checksum:

TCP Header (source:https://en.wikipedia.org/wiki/Transmission_Control_Protocol)

UDP is also a core member of the internet protocol suite. Like TCP, it is on layer 4 of the OSI model that is responsible for delivering data segments between the application and the IP layer. Unlike TCP, their header is only 32-bit that only consists of source and destination port, length, and checksum. The lightweight header makes it ideal for the application to prefer a faster data delivery without setting up the session between two hosts or needing a reliable delivery of data. Perhaps it is hard to imagine in today's fast internet connections, but the extra header made a big difference in the speed of transmission in the early days of X.21 and frame relay links. Although as important as the speed save, not having to maintain various states such as TCP also saves compute resources on the two endpoints.

UDP Header (source: https://en.wikipedia.org/wiki/User_Datagram_Protocol)

You might now wonder why UDP was ever used at all in the modern age; given the lack of reliable transmission, wouldn't we want all the connections to be reliable and error free? If one thinks about some of the multimedia video streaming or Skype call, those applications will benefit from a lighter header when the application just wants to deliver the datagram as fast as possible. You can also consider the DNS lookup process. When the address you type in on the browser is translated into a computer understandable address, the user will benefit from the lightweight process since this has to happen "before" even the first bit of information is delivered to you from your favorite website.

Again, this section does not do justice to the topic of UDP, and the reader is encouraged to explore the topic through various resources if he/she is interested in learning more about UDP.

As network engineers would tell you, network engineers "live" at the IP layer, which is layer 3 on the OSI model. IP has the job of addressing and routing between end nodes, among others. The addressing of IP is probably its most important job. The address space is divided into two parts: the network and the host portion. The subnet mask indicated which portion in the address consist of the network and which is the host. Both IPv4, and later, IPv6 expresses the address in the dotted notation, for example 192.168.0.1. The subnet mask can either be in a dotted notation (255.255.255.0) or use a forward slash to express the number of bits that should be considered in the network bit (/24).

IPv4 Header (source: https://en.wikipedia.org/wiki/IPv4)

The IPv6 header, next generation of IP header of IPv4, has a fixed portion and various extension headers.

IPv6 Fixed Header (source: https://en.wikipedia.org/wiki/IPv6_packet)

The Next Header field in the fixed header section can indicate an extension header to be followed that carries additional information. The extension headers can include routing and fragment information. As much as the protocol designer would like to move from IPv4 to IPv6, the internet today is still pretty much addressed with IPv4.

At the time of writing this book in early 2017, Python is going through a transition period of moving from Python version 2 to Python version 3. Unfortunately Python 3 is not backward compatible with Python 2. When I say transition, keep in mind that Python 3 was released back in 2008, over nine years ago with active development with the most recent release of 3.6. The latest Python 2.x release, 2.7, was released over six years ago in mid 2010. Fortunately, both version can coexist on the same machine. Personally, I use Python 2 as my default interpreter when I type in Python at the Command Prompt, and I use Python 3 when I need to use Python 3. More information is given in the next section about invoking Python interpreter, but here is an example of invoking Python 2 and Python 3 on the Ubuntu Linux machine:

    echou@pythonicNeteng:~$ python
    Python 2.7.12 (default, Nov 19 2016, 06:48:10)
    [GCC 5.4.0 20160609] on linux2
    Type "help", "copyright", "credits" or "license" for 
    more information.
    >>> exit()

    echou@pythonicNeteng:~$ python3
    Python 3.5.2 (default, Nov 17 2016, 17:05:23)
    [GCC 5.4.0 20160609] on linux
    Type "help", "copyright", "credits" or "license" for 
    more information.
    >>> exit()

With the 2.7 release being end of life on extended support only for security updates, most Python frameworks are now supporting Python 3. Python 3 also has lots of good features such as asynchronous I/O that can be taken advantage of when we need to optimize our code. This book will use Python 3 for its code examples.

If the particular library or the framework does not support Python 3, such as Ansible (they are actively working on porting to Python 3), it will be pointed out, so you can use Python 2 instead.

As mentioned, Python is cross platform. Python programs can be run on Windows, Mac, and Linux. In reality, certain care needs to be taken when you need to ensure cross-platform compatibility, such as taking care of the subtle difference backslashes in Windows filenames. Since this book is for DevOps, systems, and network engineers, Linux is the preferred platform for the intended audience, especially in production. The code in this book will be tested on the Linux Ubuntu 16.06 LTS machine. I will also try my best to make sure the code runs the same on Windows and the Mac platform.

If you are interested in the OS details, they are as follows:

    echou@pythonicNeteng:~$ uname -a
    Linux pythonicNeteng 4.4.0-31-generic #50-Ubuntu SMP 
    Wed Jul 13 00:07:12 UTC 2016 x86_64 x86_64 x86_64 
    GNU/Linux
    echou@pythonicNeteng:~$

Python programs are executed by an interpreter, which means the code is fed through this interpreter to be executed by the underlying operating system, and results are displayed . There are several different implementation of the interpreter by the Python development community, such as IronPython and Jython. In this book, we will refer to the most common Python interpreter, CPython, which is in use today.

One way you can use Python is by taking the advantage of the interactive prompt. This is useful when you want to quickly test a piece of Python code or concept without writing a whole program. This is typically done by simply typing in the Python keyword:

    Python 3.5.2 (default, Nov 17 2016, 17:05:23)
    [GCC 5.4.0 20160609] on linux
    Type "help", "copyright", "credits" or "license" for 
    more information.
    >>> print("hello world")
    hello world
    >>>

The interactive mode is one of the Python's most useful features. In the interactive shell, you can type any valid statement or sequence of statements and immediately get a result back. I typically use this to explore around a feature or library that I am not familiar with. Talk about instant gratification!

A more common way to run the Python program, however, is to save your Python file and run via the interpreter after. This will save you from typing the same statements over and over again such as in the interactive shell. Python files are just regular text files that are typically saved with the .py extension. In the *Nix world, you can also add the shebang (#!) line on top to specify the interpreter that is used to run the file. The # character can be used to specify comments that will not be executed by the interpreter. The following file, helloworld.py, has the following statements:

    # This is a comment
    print("hello world") 

This can be executed as follows:

    echou@pythonicNeteng:~/Master_Python_Networking/
    Chapter1$ python helloworld.py
    hello world
    echou@pythonicNeteng:~/Master_Python_Networking/
    Chapter1$

Python has several standard types built into the interpreter:

    >>> a = "networking is fun"
    >>> b = 'DevOps is fun too'
    >>> c = """what about coding?
    ... super fun!"""
    >>>

    >>> vendors = ["Cisco", "Arista", "Juniper"]
    >>> vendors[0]
    'Cisco'
    >>> vendors[1]
    'Arista'
    >>> vendors[2]
    'Juniper'

    >>> datacenters = ("SJC1", "LAX1", "SFO1")
    >>> datacenters[0]
    'SJC1'
    >>> datacenters[1]
    'LAX1'
    >>> datacenters[2]
    'SFO1' 

    >>> a
    'networking is fun'
    >>> a[1]
    'e'
    >>> vendors
    ['Cisco', 'Arista', 'Juniper']
    >>> vendors[1]
    'Arista'
    >>> datacenters
    ('SJC1', 'LAX1', 'SFO1')
    >>> datacenters[1]
    'LAX1'
    >>>
    >>> a[0:2]
    'ne'
    >>> vendors[0:2]
    ['Cisco', 'Arista']
    >>> datacenters[0:2]
    ('SJC1', 'LAX1')
    >>>

    >>> len(a)
    17
    >>> len(vendors)
    3
    >>> len(datacenters)
    3
    >>>
    >>> b = [1, 2, 3, 4, 5]
    >>> min(b)
    1
    >>> max(b)
    5

    >>> a
    'networking is fun'
    >>> a.capitalize()
    'Networking is fun'
    >>> a.upper()
    'NETWORKING IS FUN'
    >>> a
    'networking is fun'
    >>> b = a.upper()
    >>> b
    'NETWORKING IS FUN'
    >>> a.split()
    ['networking', 'is', 'fun']
    >>> a
    'networking is fun'
    >>> b = a.split()
    >>> b
    ['networking', 'is', 'fun']
    >>>

    >>> routers = ['r1', 'r2', 'r3', 'r4', 'r5']
    >>> routers.append('r6')
    >>> routers
    ['r1', 'r2', 'r3', 'r4', 'r5', 'r6']
    >>> routers.insert(2, 'r100')
    >>> routers
    ['r1', 'r2', 'r100', 'r3', 'r4', 'r5', 'r6']
    >>> routers.pop(1)
    'r2'
    >>> routers
    ['r1', 'r100', 'r3', 'r4', 'r5', 'r6']

    >>> datacenter1 = {'spines': ['r1', 'r2', 'r3', 'r4']}
    >>> datacenter1['leafs'] = ['l1', 'l2', 'l3', 'l4']
    >>> datacenter1
    {'leafs': ['l1', 'l2', 'l3', 'l4'], 'spines': ['r1',  
    'r2', 'r3', 'r4']}
    >>> datacenter1['spines']
    ['r1', 'r2', 'r3', 'r4']
    >>> datacenter1['leafs']
    ['l1', 'l2', 'l3', 'l4']

    >>> a = "hello"
    >>> set(a)
    {'h', 'l', 'o', 'e'}
    >>> b = set([1, 1, 2, 2, 3, 3, 4, 4])
    >>> b
    {1, 2, 3, 4}
    >>> b.add(5)
    >>> b
    {1, 2, 3, 4, 5}
    >>> b.update(['a', 'a', 'b', 'b'])
    >>> b
    {1, 2, 3, 4, 5, 'b', 'a'}
    >>> a = set([1, 2, 3, 4, 5])
    >>> b = set([4, 5, 6, 7, 8])
    >>> a.intersection(b)
    {4, 5}
    >>> a.union(b)
    {1, 2, 3, 4, 5, 6, 7, 8}
    >>> 1 *
    {1, 2, 3}
    >>>

Python has the some numeric operators that you would expect; note that the truncating division, (//, also known as floor division) truncates the result to an integer and a floating point and return the integer value. The integer value is returned. The modulo (%) operator returns the remainder value in the division:

    >>> 1 + 2
    3
    >>> 2 - 1
    1
    >>> 1 * 5
    5
    >>> 5 / 1
    5.0
    >>> 5 // 2
    2
    >>> 5 % 2
    1

There are also comparison operators:

    >>> a = 1
    >>> b = 2
    >>> a == b
    False
    >>> a > b
    False
    >>> a < b
    True
    >>> a <= b
    True 

We will also see two of the common membership operators to see whether an object is in a sequence type:

    >>> a = 'hello world'
    >>> 'h' in a
    True
    >>> 'z' in a
    False
    >>> 'h' not in a
    False
    >>> 'z' not in a
    True

The if, else, and elif statements control conditional code execution. As one would expect, the format of the conditional statement is as follows:

    if expression:
      do something
    elif expression:
      do something if the expression meets
    elif expression:
      do something if the expression meets
    ...
    else:
      statement

Here is a simple example:

    >>> a = 10
    >>> if a > 1:
    ...   print("a is larger than 1")
    ... elif a < 1:
    ...   print("a is smaller than 1")
    ... else:
    ...   print("a is equal to 1")
    ...
    a is larger than 1
    >>>

The while loop will continue to execute until the condition is false, so be careful with this one if you don't want to continue to execute:

    while expression:
      do something

    >>> a = 10
    >>> b = 1
    >>> while b < a:
    ...   print(b)
    ...   b += 1
    ...
    1
    2
    3
    4
    5
    6
    7
    8
    9

The for loop works with any object that supports iteration; this means all the built-in sequence types such as lists, tuples, and strings can be used in a for loop. The letter i in the following for loop is an iterating variable, so you can typically pick something that make sense within the context of your code:

    for i in sequence:
      do something

    >>> a = [100, 200, 300, 400]
    >>> for number in a:
    ...   print(number)
    ...
    100
    200
    300
    400

You can also make your own object that supports the iterator protocol and be able to use the for loop for this object:

Most of the time when you find yourself reusing some pieces of code, you should break up the code into a self-contained chunk as functions. This practice allows for better modularity, is easier to maintain, and allows for code reuse. Python functions are defined using the def keyword with the function name, followed by the function parameters. The body of the function consists of Python statements that are to be executed. At the end of the function, you can choose to return a value to the function caller, or by default, it will return the None object if you do not specify a return value:

    def name(parameter1, parameter2):
      statements
      return value

We will see a lot more examples of function in the following chapters, so here is a quick example:

    >>> def subtract(a, b):
    ...   c = a - b
    ...   return c
    ...
    >>> result = subtract(10, 5)
    >>> result
    5
    >>>

Python is an Object-Oriented Programming (OOP) language. The way Python creates objects are with the class keyword. A Python object is most commonly a collection of functions (methods), variables, and attributes (properties). Once a class is defined, you can create instances of such a class. The class serves as the blueprint of the subsequent instances.

The topic of object-oriented programming is outside the scope of this chapter, so here is a simple example of a router object definition:

    >>> class router(object):
    ...   def __init__(self, name, interface_number,
     vendor):
    ...     self.name = name
    ...     self.interface_number = interface_number
    ...     self.vendor = vendor
    ...
    >>>

Once defined, you are able to create as many instances of that class as you'd like:

    >>> r1 = router("SFO1-R1", 64, "Cisco")
    >>> r1.name
    'SFO1-R1'
    >>> r1.interface_number
    64
    >>> r1.vendor
    'Cisco'
    >>>
    >>> r2 = router("LAX-R2", 32, "Juniper")
    >>> r2.name
    'LAX-R2'
    >>> r2.interface_number
    32
    >>> r2.vendor
    'Juniper'
    >>>

Of course, there is a lot more to Python objects and OOP. We will see more examples in the future chapters.

Any Python source file can be used as a module, and any functions and classes you define in that source file can be re-used. To load the code, the file referencing the module needs to use the import keyword. Three things happen when the file is imported:

Remember the subtract() function that you defined using the interactive shell? To reuse the function, we can put it into a file named subtract.py:

    def subtract(a, b):
      c = a - b
      return c

In a file within the same directory of subtract.py, you can start the Python interpreter and import this function:

    Python 2.7.12 (default, Nov 19 2016, 06:48:10)
    [GCC 5.4.0 20160609] on linux2
    Type "help", "copyright", "credits" or "license" for  
    more information.
    >>> import subtract
    >>> result = subtract.subtract(10, 5)
    >>> result
    5

This works because, by default, Python will first search for the current directory for the available modules. If you are in a different directory, you can manually add a search path location using the sys module with sys.path. Remember the standard library that we mentioned a while back? You guessed it, those are just Python files being used as modules.

Packages allow a collection of modules to be grouped together. This further organizes the Python modules into a more namespace protection to further reusability. A package is defined by creating a directory with a name you want to use as the namespace, then yo can place the module source file under that directory. In order for Python to know it as a Python-package, just create a __init__.py file in this directory. In the same example as the subtract.py file, if you were to create a directory called math_stuff and create a __init__.py file:

    echou@pythonicNeteng:~/Master_Python_Networking/
    Chapter1$ mkdir math_stuff
    echou@pythonicNeteng:~/Master_Python_Networking/
    Chapter1$ touch math_stuff/__init__.py
    echou@pythonicNeteng:~/Master_Python_Networking/
    Chapter1$ tree .
    .
    ├── helloworld.py
    └── math_stuff
     ├── __init__.py
     └── subtract.py

    1 directory, 3 files
    echou@pythonicNeteng:~/Master_Python_Networking/
    Chapter1$

The way you will now refer to the module will need to include the package name:

    >>> from math_stuff.subtract import subtract
    >>> result = subtract(10, 5)
    >>> result
    5
    >>>

As you can see, modules and packages are great ways to organize large code files and make sharing Python code a lot easier.

In this chapter, we covered the OSI model and reviewed the network protocol suites, such as TCP, UDP, and IP. We then did a quick review of the Python language, including built-in types, operators, control flows, functions, classes, modules, and packages.

In the next chapter, we will start to look at using Python to programmatically interact with the network equipment.