This book will focus on writing programs for networks that use the Internet protocol suite. Why have we chosen to do this? Well, of the sets of protocols supported by the Python standard library, the TCP/IP protocol is by far the most widely employable. It contains the principle protocols used by the Internet. By learning to program for TCP/IP, you'll be learning how to potentially communicate with just about every device that is connected to this great tangle of network cables and electromagnetic waves.
In this chapter, we will be looking at some concepts and methods around networks and network programming in Python, which we'll be using throughout this book.
This chapter has two sections. The first section, An introduction to TCP/IP networks, offers an introduction to essential networking concepts, with a strong focus on the TCP/IP stack. We'll be looking at what comprises a network, how the Internet Protocol (IP) allows data transfer across and between networks, and how TCP/IP provides us with services that help us to develop network applications. This section is intended to provide a grounding in these essential areas and to act as a point of reference for them. If you're already comfortable with concepts such as IP addresses, routing, TCP and UDP, and protocol stack layers, then you may wish to skip to second part, Network programming with Python.
In the second part, we'll look at the way in which network programming is approached with Python. We'll be introducing the main standard library modules, looking at some examples to see how they relate to the TCP/IP stack, and then we will be discussing a general approach for finding and employing modules that meet our networking needs. We'll also be taking a look at a couple of general issues that we may encounter, when writing applications that communicate over TCP/IP networks.
The Internet protocol suite, often referred to as TCP/IP, is a set of protocols designed to work together to provide end-to-end transmission of messages across interconnected networks.
The following discussion is based on Internet Protocol version 4 (IPv4). Since the Internet has run out of IPv4 addresses, a new version, IPv6, has been developed, which is intended to resolve this situation. However, although IPv6 is being used in a few areas, its deployment is progressing slowly and a majority of the Internet will likely be using IPv4 for a while longer. We'll focus on IPv4 in this section, and then we will discuss the relevant changes in IPv6 in second part of this chapter.
TCP/IP is specified in documents called Requests for Comment (RFCs) which are published by the Internet Engineering Task Force (IETF). RFCs cover a wide range of standards and TCP/IP is just one of these. They are freely available on the IETF's website, which can be found at www.ietf.org/rfc.html. Each RFC has a number, IPv4 is documented by RFC 791, and other relevant RFCs will be mentioned as we progress.
Note that you won't learn how to set up your own network in this chapter because that's a big topic and unfortunately, somewhat beyond the scope of this book. But, it should enable you at least to have a meaningful conversation with your network support people!
So, let's get started with something you're likely to be familiar with, that is, IP addresses. They typically look something like this:
203.0.113.12
They are actually a single 32-bit number, though they are usually written just like the number shown in the preceding example; they are written in the form of four decimal numbers that are separated by dots. The numbers are sometimes called octets or bytes because each one represents 8-bits of the 32-bit number. As such, each octet can only take values from 0 to 255, so valid IP addresses range from 0.0.0.0 to 255.255.255.255. This way of writing IP addresses is called dot-decimal notation.
IP addresses perform two main functions. They are as follows:
They uniquely address each device that is connected to a network
They help the traffic to be routed between networks
You may have noticed that the network-connected devices that you use have IP addresses assigned to them. Each IP address that is assigned to a network device is unique and no two devices can share an IP address.
You can find out what IP addresses have been assigned to your computer by running ip addr (or ipconfig /all on Windows) on a terminal. In Chapter 6, IP and DNS, we'll see how to do this when using Python.
If we run one of these commands, then we can see that the IP addresses are assigned to our device's network interfaces. On Linux, these will have names, such as eth0; on Windows these will have phrases, such as Ethernet adapter Local Area Connection.
You will get the following output when you run the ip addr command on Linux:
$ ip addr 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether b8:27:eb:5d:7f:ae brd ff:ff:ff:ff:ff:ff inet 192.168.0.4/24 brd 192.168.0.255 scope global eth0 valid_lft forever preferred_lft forever
In the preceding example, the IP addresses for the interfaces appear after the word inet.
An interface is a device's physical connection to its network media. It could be a network card that connects to a network cable, or a radio that uses a specific wireless technology. A desktop computer may only have a single interface for a network cable, whereas a Smartphone is likely to have at least two interfaces, one for connecting to Wi-Fi networks and one for connecting to mobile networks that use 4G or other technologies.
An interface is usually assigned only one IP address, and each interface in a device has a different IP address. So, going back to the purposes of IP addresses discussed in the preceding section, we can now more accurately say that their first main function is to uniquely address each device's connection to a network.
Every device has a virtual interface called the
loopback interface, which you can see in the preceding listing as interface 1. This interface doesn't actually connect to anything outside the device, and only the device itself can communicate with it. While this may sound a little redundant, it's actually very useful when it comes to local network application testing, and it can also be used as a means of inter-process communication. The loopback interface is often referred to as
localhost, and it is almost always assigned the IP address 127.0.0.1.
IP addresses can be assigned to a device by a network administrator in one of two ways: statically, where the device's operating system is manually configured with the IP address, or dynamically, where the device's operating system is configured by using the Dynamic Host Configuration Protocol (DHCP).
When using DHCP, as soon as the device first connects to a network, it is automatically allocated an address by a DHCP server from a predefined pool. Some network devices, such as home broadband routers provide a DHCP server service out-of-the-box, otherwise a DHCP server must be set up by a network administrator. DHCP is widely deployed, and it is particularly useful for networks where different devices may frequently connect and disconnect, such as public Wi-Fi hotspots or mobile networks.
The Internet is a huge IP network, and every device that sends data over it is assigned an IP address.
The IP address space is managed by an organization called the Internet Assigned Numbers Authority (IANA). IANA decides the global allocation of the IP address ranges and assigns blocks of addresses to Regional Internet Registries (RIRs) worldwide, who then allocate address blocks to countries and organizations. The receiving organizations have the freedom to allocate the addresses from their assigned blocks as they like within their own networks.
There are some special IP address ranges. IANA has defined ranges of private addresses. These ranges will never be assigned to any organization, and as such these are available for anyone to use for their networks. The private address ranges are as follows:
10.0.0.0 to 10.255.255.255
172.16.0.0 to 172.31.255.255
192.168.0.0 to 192.168.255.255
You may be thinking that if anybody can use them, then would'nt that mean that devices on the Internet will end up using the same addresses, thereby breaking IP's unique addressing property? This is a good question, and this problem has been avoided by forbidding traffic from private addresses from being routed over the public Internet. Wherever a network using private addresses needs to communicate with the public Internet, a technique called Network Address Translation (NAT) is used, which essentially makes the traffic from the private network appear to be coming from a single valid public Internet address, and this effectively hides the private addresses from the Internet. We'll discuss NAT later on.
If you inspect the output of ip addr or ipconfig /all on your home network, then you will find that your devices are using private range addresses, which would have been assigned to them by your broadband router through DHCP.
We'll be talking about network traffic in the following sections, so let's get an idea of what it is.
Many protocols, including the principle protocols in the Internet protocol suite, employ a technique called packetization to help manage data while it's being transmitted across a network.
When a packetizing protocol is given some data to transmit, it breaks it up into small units — sequences of bytes, typically a few thousand bytes long and then it prefixes each unit with some protocol-specific information. The prefix is called a header, and the prefix and data together form a packet. The data within a packet is often called its payload.
What a packet contains is shown in the following figure:
Some protocols use alternative terms for packets, such as frames, but we'll stick with the term packets for now. The header includes all the information that the protocol implementation running on another device needs to be able to interpret what the packet is and how to handle it. For example, the information in an IP packet header includes the source IP address, the destination IP address, the total length of the packet, and the checksum of the data in the header.
Once created, the packets are sent onto the network, where they are independently routed to their destination. Sending the data in packets has several advantages, including multiplexing (where more than one device can send data over the network at once), rapid notification of errors that may occur on the network, congestion control, and dynamic re-routing.
Protocols may call upon other protocols to handle their packets for them; passing their packets to the second protocol for delivery. When both the protocols employ packetization, nested packets result, as shown in the following figure:
This is called encapsulation, and as we'll see shortly, it is a powerful mechanism for structuring network traffic.
A network is a discrete collection of connected network devices. Networks can vary greatly in scale, and they can be made up of smaller networks. Your network-connected devices at home or the network-connected computers in a large office building are examples of networks.
There are quite a few ways of defining a network, some loose, some very specific. Depending on the context, networks can be defined by physical boundaries, administrative boundaries, institutional boundaries, or network technology boundaries.
For this section, we're going to start with a simplified definition of a network, and then work toward a more specific definition, in the form of IP subnets.
So for our simplified definition, our common defining feature of a network will be that all devices on the network share a single point of connection to the rest of the Internet. In some large or specialized networks, you will find that there is more than one point of connection, but for the sake of simplicity we'll stick to a single connection here.
This connection point is called a gateway, and usually it takes the form of a special network device called a router. The job of a router is to direct traffic between networks. It sits between two or more networks and is said to sit at the boundary of these networks. It always has two or more network interfaces; one for each network it is attached to. A router contains a set of rules called a routing table, which tells it how to direct the packets that are passing through it onwards, based on the packets' destination IP addresses.
The gateway forwards the packets to another router, which is said to be upstream, and is usually located at the network's Internet Service Provider (ISP). The ISP's router falls into a second category of routers, that is, it sits outside the networks described earlier, and routes traffic between network gateways. These routers are run by ISPs and other communications entities. They are generally arranged in tiers, and the upper regional tiers route the traffic for some large sections of countries or continents and form the Internet's backbone.
Because these routers can sit between many networks, their routing tables can become very extensive and they need to be updated continuously. A simplified illustration is shown in the following diagram:
The preceding diagram gives us an idea of the arrangement. Each ISP gateway connects an ISP network to the regional routers, and each home broadband router has a home network connected to it. In the real world, this arrangement gets more complicated as one goes toward the top. ISPs will often have more than one gateway connecting them to the regional routers, and some of these will also themselves be acting as regional routers. Regional routers also have more tiers than shown here, and they have many connections between one another, which are in arrangements that are much more complicated than this simple hierarchy. A rendering of a section of the Internet from data gathered in 2005 provides a beautiful illustration of just how complex this becomes, it can be found at http://en.wikipedia.org/wiki/Internet_backbone#/media/File:Internet_map_1024.jpg.
We mentioned that routers are able to route traffic toward a destination network, and implied that this is somehow done by using IP addresses and routing tables. But what's really going on here?
One perhaps obvious method for routers to determine the correct router to forward traffic to would be to program every router's routing table with a route for every IP address. However, in practice, with 4 billion plus IP addresses and constantly changing network routes, this turns out to be a completely infeasible method.
So, how is routing done? The answer lies in another property of IP addresses. An IP address can be interpreted as being made up of two logical parts: a network prefix and a host identifier. The network prefix uniquely identifies the network a device is on, and the device can use this to determine how to handle traffic that it generates, or receives for forwarding. The network prefix is the first n bits of the IP address when it's written out in binary (remember an IP address is really just a 32-bit number). The n bits are supplied by the network administrator as a part of a device's network configuration at the same time that it is given its IP address.
You'll see that n is written in one of two ways. It can simply be appended to the IP address, separated by a slash, as follows:
192.168.0.186/24
This is called CIDR notation. Alternatively, it can be written as a subnet mask, which is sometimes just called a netmask. This is the way in which you will usually see n being specified in a device's network configuration. A subnet mask is a 32-bit number written in dot-decimal notation, just like an IP address.
255.255.255.0
This subnet mask is equivalent to /24. We get n from it by looking at it in binary. A few examples are as follows:
255.0.0.0 = 11111111 00000000 00000000 00000000 = /8 255.192.0.0 = 11111111 11000000 00000000 00000000 = /10 255.255.255.0 = 11111111 11111111 11111111 00000000 = /24 255.255.255.240 = 11111111 11111111 11111111 11110000 = /28
n is simply the number of 1 bits in the subnet mask. (It's always the leftmost bits that are set to 1 because this allows us to quickly get the Network prefix in binary by doing a bitwise AND operation on the IP address and the subnet mask).
So, how does this help in routing? When a network device generates network traffic that needs to be sent across a network, it first compares the destination's IP address with its own network prefix. If the destination IP address has the same network prefix as that of the sending device, then the sending device will recognise that the destination device is on the same network and, therefore, it can then send the traffic directly to it. If the network prefixes differ, then it will send the message to its default gateway, which will forward it on towards the receiving device.
When a router receives traffic that has to be forwarded, it first checks whether the destination IP address matches the network prefix of any of the networks that it's connected to. If that is the case, then it will send the message directly to the destination device on that network. If not, it will consult its routing table. If it finds a matching rule, then it sends the message to the router that it found listed, and if there are no explicit rules defined, then it will send the traffic to its own default gateway.
When we create a network with a given network prefix, in the 32-bits of the IP address, the digits to the right of the network prefix are available for assignment to the network devices. We can calculate the number of the available addresses by raising 2 to the power of the number of available bits. For example, in a /28 network prefix, we have 4 bits left, which means that 16 addresses are available. In reality, we are able to assign fewer addresses, since two of the addresses in the calculated range are always reserved. These are: the first address in the range, which is called the
network address and the last address in the range, which is called the
broadcast address.
This range of addresses, which is identified by its network prefix, is called a subnet. Subnets are the basic unit of assignment when IANA, an RIR or an ISP allocates IP address blocks to organizations. Organizations assign subnets to their various networks.
Organizations can further partition their addresses into subnets simply by employing a longer network prefix than the one they had been assigned. They might do this either to make more efficient use of their addresses or to create a hierarchy of networks, which can be delegated across the organization.
We've discussed connecting to network devices by using IP addresses. However, unless you work with networks or in systems administration, it is unlikely that you will get to see an IP address very often, even though many of us use the Internet every day. When we browse the web or send an e-mail, we usually connect to servers using host names or domain names. These must somehow map to the servers' IP addresses. But how is this done?
Documented as RFC 1035, the Domain Name System (DNS) is a globally distributed database of mappings between hostnames and IP addresses. It is an open and hierarchical system with many organizations choosing to run their own DNS servers. DNS is also a protocol, which devices use to query DNS servers for resolving hostnames to IP addresses (and vice-versa).
The nslookup tool comes with most Linux and Windows systems and it lets us query DNS on the command line, as follows:
$ nslookup python.org Server: 192.168.0.4 Address: 192.168.0.4#53 Non-authoritative answer: Name: python.org Address: 104.130.43.121
Here, we determined that the python.org host has the IP address 104.130.42.121. DNS distributes the work of looking up hostnames by using an hierarchical system of caching servers. When connecting to a network, your network device will be given a local DNS server through either DHCP or manually, and it will query this local server when doing DNS lookups. If that server doesn't know the IP address, then it will query its own configured higher tier server, and so on until an answer can be found. ISPs run their own DNS caching servers, and broadband routers often act as caching servers as well. In this example, my device's local server is 192.168.0.4.
A device's operating system usually handles DNS, and it provides a programming interface, which applications use to ask it to resolve hostnames and IP addresses. Python provides an interface for this, which we'll discuss in Chapter 6, IP and DNS.
The Internet Protocol is a member of the set of protocols that make up the Internet protocol suite. Each protocol in the suite has been designed to solve specific problems in networking. We just saw how IP solves the problems of addressing and routing.
The core protocols in the suite are designed to work together within a stack. That is, each protocol occupies a layer within the stack, and the other protocols are situated above and below that layer. So, it is layered just like a cake. Each layer provides a specific service to the layers above it, while hiding the complexity of its own operation from them, following the principle of encapsulation. Ideally, each layer only interfaces with the layer below it in order to benefit from the entire range of the problem solving powers of all the layers below.
Python provides modules for interfacing with different protocols. As the protocols employ encapsulation, we typically only need to work with one module to leverage the power of the underlying stack, thus avoiding the complexity of the lower layers.
The TCP/IP Suite defines four layers, although five layers are often used for clarity. These are given in the following table:
|
Layer |
Name |
Example protocols |
|---|---|---|
|
5 |
Application layer |
HTTP, SMTP, IMAP |
|
4 |
Transport layer |
TCP, UDP |
|
3 |
Network layer |
IP |
|
2 |
Data-link layer |
Ethernet, PPP, FDDI |
|
1 |
Physical layer |
- |
Layers 1 and 2 correspond to the first layer of the TCP/IP suite. These two bottom layers deal with the low level network infrastructure and services.
Layer 1 corresponds to the physical media of the network, such as a cable or a Wi-Fi radio. Layer 2 provides the service of getting the data from one network device to another, directly connected network device. This layer can employ all sorts of layer 2 protocols, such as Ethernet or PPP, as long as the Internet Protocol in layer 3 can ask it to get the data to the next device in the network by using any type of available physical medium.
We don't need to concern ourselves with the two lowest layers, since we will rarely need to interface with them when using Python. Their operation is almost always handled by the operating system and the network hardware.
Layer 3 is variously called the Network layer and the Internet layer. It exclusively employs the Internet Protocol. As we have already seen, it has been tasked primarily with internetwork addressing and routing. Again, we don't typically directly interface with this layer in Python.
Layers 4 and 5 are more interesting for our purposes.
Layer 4 is the first layer that we may want to work with in Python. This layer can employ one of two protocols: the Transmission Control Protocol (TCP) and the User Datagram Protocol (UDP). Both of these provide the common service of end-to-end transportation of data between applications on different network devices.
Although IP facilitates the transport of data from one network device to another, it doesn't provide us with a way of letting the destination device know what it should do with the data once it receives it. One possible solution to this would be to program every process running on the destination device to check all of the incoming data to see if they are interested in it, but this would quickly lead to obvious performance and security problems.
TCP and UDP provide the answer by introducing the concept of ports. A port is an endpoint, which is attached to one of the IP addresses assigned to the network device. Ports are claimed by a process running on the device, and the process is then said to be listening on that port. Ports are represented by a 16-bit number, so that each IP address on a device has 65,535 possible ports that the processes can claim (port number 0 is reserved). Ports can only be claimed by one process at a time, even though a process can claim more than one port at a time.
When a message is sent over the network through TCP or UDP, the sending application sets the destination port number in the header of the TCP or UDP packet. When the message arrives at the destination, the TCP or UDP protocol implementation running on the receiving device reads the port number and then delivers the message payload to the process that is listening on that port.
Port numbers need to be known before the messages are sent. The main mechanism for this is convention. In addition to managing the IP address space, it is also the responsibility of IANA to manage the assignment of port numbers to network services.
A service is a class of application, for example a web server, or a DNS server, which is usually tied to an application protocol. Ports are assigned to services rather than specific applications, because it gives service providers the flexibility to choose what kind of software they want to use to provide a service, without having to worry about the users who would need to look up and connect to a new port number simply because the server has started using Apache instead of IIS, for example.
Most operating systems contain a copy of this list of services and their assigned port numbers. On Linux, this is usually found at /etc/services, and on Windows this is usually found at c:\windows\system32\drivers\etc\services. The complete list can also be viewed online at http://www.iana.org/assignments/port-numbers.
TCP and UDP packet headers may also include a source port number. This is optional for UDP, but mandatory for TCP. The source port number tells the receiving application on the server where it should send replies to when sending data back to the client. Applications can specify the source port that they wish to use, or if a source port has not been specified for TCP, then one is assigned randomly by the operating system when the packet is sent. Once the OS has a source port number, it assigns it to the calling application and starts listening on it for a reply. If a reply is received on that port, then the received data is passed to the sending application.
So, both TCP and UCP provide an end-to-end transport for the application data through the provision of ports, and both of them employ the Internet Protocol to get the data to the destination device. Now, let's look at their features.
UDP is documented as RFC 768. It is deliberately uncomplicated: it provides no services other than those that we described in the previous section. It just takes the data that we want to send, packetizes it with the destination port number (and optional source port number), and hands it off to the local Internet Protocol implementation for delivery. Applications on the receiving end see the data in the same discrete chunks in which it was packetized.
Both IP and UDP are what are called connectionless protocols. This means that they attempt to deliver their packets on a best effort basis, but if something goes wrong, then they will just shrug their metaphorical shoulders and move on to delivering the next packet. There is no guarantee that our packets will reach their destinations, and no error notification if a delivery fails. If the packets do make it, then there is no guarantee that they will do so in the same order as they were sent. It's up to a higher layer protocol or the sending application to determine if the packets have arrived and whether to handle any problems. These are protocols in the fire-and-forget style.
The typical applications of UDP are internet telephony and video streaming. DNS queries are also transported using UDP.
We'll now look at UDP's more dependable sibling, TCP, and then discuss the differences, and why applications may choose to use one or the other.
The Transmission Control Protocol is documented as RFC 761. As opposed to UDP, TCP is a connection based protocol. In such a protocol, no data is sent until the server and the client have performed an initial exchange of control packets. This exchange is called a handshake. This establishes a connection, and from then on data can be sent. Each data packet that is received is acknowledged by the receiving party, and it does so by sending a packet called an ACK. As such, TCP always requires that the packets include a source port number, because it depends on the continual two-way exchange of messages.
From an application's point of view, the key difference between UDP and TCP is that the application no longer sees the data in discrete chunks; the TCP connection presents the data to the application as a continuous, seamless stream of bytes. This makes things much simpler if we are sending messages that are larger than a typical packet, however it means that we need to start thinking about framing our messages. While with UDP, we can rely on its packetization to provide a means of doing this, with TCP we must decide a mechanism for unambiguously determining where our messages start and end. We'll see more about this in Chapter 8, Client and Server Applications.
TCP provides the following services:
In-order delivery
Receipt acknowledgment
Error detection
Flow and congestion control
Data sent through TCP is guaranteed to get delivered to the receiving application in the order that it was sent in. The receiving TCP implementation buffers the received packets on the receiving device and then waits until it can deliver them in the correct order before passing them to the application.
Because the data packets are acknowledged, sending applications can be sure that the data is arriving and that it is okay to continue sending the data. If an ACK is not received for a sent packet, then within a set time period the packet will be resent. If there's still no response, then TCP will keep resending the packet at increasing intervals, until a second, longer timeout period expires. At this point, it will give up and notify the calling application that it has encountered a problem.
The TCP header includes a checksum of the header data and the payload. This allows the receiver to verify whether a packet's contents have been modified during the transmission.
TCP also includes algorithms which ensure that traffic is not sent too quickly for the receiving device to process, and these algorithms also infer network conditions and regulate the transmission rate to avoid network congestion.
Together these services provide a robust and reliable transport system for application data. This is one of the reasons many popular higher level protocols, such as HTTP, SMTP, SSH, and IMAP, depend on TCP.
Given the features of TCP, you may be wondering what the use of a connectionless protocol like UDP is. Well, the Internet is still a pretty reliable network, and most of the packets do get delivered. The connectionless protocols are useful where the minimum transfer overhead is required, and where the occasional dropped packet is not a big deal. TCP's reliability and congestion control comes at the cost of needing additional packets and round-trips, and the introduction of deliberate delays when packets are lost in order to prevent congestion. These can drastically increase latency, which is the arch-nemesis of real-time services, while not providing any real benefit for them. A few dropped packets might result in a transient glitch or a drop in signal quality in a media stream, but as long as the packets keep coming, the stream can usually recover.
UDP is also the main protocol that is used for DNS, which is interesting because most DNS queries fit inside a single packet, so TCP's streaming abilities aren't generally needed. DNS is also usually configured such that it does not depend upon a reliable connection. Most devices are configured with multiple DNS servers, and it's usually quicker to resend a query to a second server after a short timeout rather than wait for a TCP back-off period to expire.
The choice between UDP and TCP comes down to the message size, whether latency is an issue, and how much of TCP's functionality the application wants to perform itself.
Finally we come to the top of the stack. The application layer is deliberately left open in the IP protocol suite, and it's really a catch-all for any protocol that is developed by application developers on top of TCP or UDP (or even IP, though these are rarer). Application layer protocols include HTTP, SMTP, IMAP, DNS, and FTP.
Protocols may even become their own layers, where an application protocol is built on top of another application protocol. An example of this is the Simple Object Access Protocol (SOAP), which defines an XML-based protocol that can be used over almost any transport, including HTTP and SMTP.
Python has standard library modules for many application layer protocols and third-party modules for many more. If we write low-level server applications, then we will be more likely to be interested in TCP and UDP, but if not, then application layer protocols are the ones we'll be working with, and we'll be looking at some of them in detail over the next few chapters.
The power of the layer model of network protocols is that a higher layer can easily build on the services provided by the lower layers and this enables them to add new services to the network. Python provides modules for interfacing with protocols at different levels in the network stack, and modules that support higher-layer protocols follow the aforementioned principle by using the interfaces supplied by the lower level protocols. How can we visualize this?
Well, sometimes a good way to see inside something like this is by breaking it. So, let's break Python's network stack. Or, more specifically, let's generate a traceback.
Yes, this means that the first piece of Python that we're going to write is going to generate an exception. But, it will be a good exception. We'll learn from it. So, fire up your Python shell and run the following command:
>>> import smtplib >>> smtplib.SMTP('127.0.0.1', port=66000)
What are we doing here? We are importing smtplib, which is Python's standard library for working with the SMTP protocol. SMTP is an application layer protocol, which is used for sending e-mails. We will then try to open an SMTP connection by instantiating an SMTP object. We want the connection to fail and that is why we've specified the port number 66000, which is an invalid port. We will specify the local host for the connection, as this will cause it to fail quickly, rather than make it wait for a network timeout.
On running the preceding command, you should get the following traceback:
Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/usr/lib/python3.4/smtplib.py", line 242, in __init__ (code, msg) = self.connect(host, port) File "/usr/lib/python3.4/smtplib.py", line 321, in connect self.sock = self._get_socket(host, port, self.timeout) File "/usr/lib/python3.4/smtplib.py", line 292, in _get_socket self.source_address) File "/usr/lib/python3.4/socket.py", line 509, in create_connection raise err File "/usr/lib/python3.4/socket.py", line 500, in create_connection sock.connect(sa) ConnectionRefusedError: [Errno 111] Connection refused
This was generated by using Python 3.4.1 on a Debian 7 machine. The final error message will be slightly different from this if you run this on Windows, but the stack trace will remain the same.
Inspecting it will reveal how the Python network modules act as a stack. We can see that the call stack starts in smtplib.py, and then as we go down, it moves into socket.py. The socket module is Python's standard interface for the transport layer, and it provides the functions for interacting with TCP and UDP as well as for looking up hostnames through DNS. We'll learn much more about this in Chapter 7, Programming with Sockets, and Chapter 8, Client and Server Applications.
From the preceding program, it's clear that the smtplib module calls into the socket module. The application layer protocol has employed a transport layer protocol (which in this case is TCP).
Right at the bottom of the traceback, we can see the exception itself and the Errno 111. This is an error message from the operating system. You can verify this by going through /usr/include/asm-generic/errno.h (asm/errno.h on some systems) for the error message number 111 (on Windows the error will be a WinError, so you can see that it has clearly been generated by the OS). From this error message we can see that the socket module is calling down yet again and asking the operating system to manage the TCP connection for it.
Python's network modules are working as the protocol stack designers intended them to. They call on the lower levels in the stack to employ their services to perform the network tasks. We can work by using simple calls made to the application layer protocol, which in this case is SMTP, without having to worry about the underlying network layers. This is network encapsulation in action, and we want to make as much use of this as we can in our applications.
Before we start writing code for a new network application, we want to make sure that we're taking as much advantage of the existing stack as possible. This means finding a module that provides an interface to the services that we want to use, and that is as high up the stack as we can find. If we're lucky, someone has already written a module that provides an interface that provides the exact service we need.
Let's use an example to illustrate this process. Let's write a tool for downloading Request for Comments (RFC) documents from IETF, and then display them on screen.
Let's keep the RFC downloader simple. We'll make it a command-line program that just accepts an RFC number, downloads the RFC in text format, and then prints it to stdout.
Now, it's possible that somebody has already written a module for doing this, so let's see if we can find anything.
The first place we look should always be the Python standard library. The modules in the library are well maintained, and well documented. When we use a standard library module, the users of your application won't need to install any additional dependencies for running it.
A look through the Library Reference at https://docs.python.org doesn't seem to show anything directly relevant to our requirement. This is not entirely surprising!
So, next we will turn to third-party modules. The Python package index, which can be found at https://pypi.python.org, is the place where we should look for these. Here as well, running a few searches around the theme of RFC client and RFC download doesn't seem to reveal anything useful. The next place to look will be Google, though again, the searches don't reveal anything promising. This is slightly disappointing, but this is why we're learning network programming, to fill these gaps!
There are other ways in which we may be able to find out about useful third-party modules, including mailing lists, Python user groups, the programming Q&A site http://stackoverflow.com, and programming textbooks.
For now, let's assume that we really can't find a module for downloading RFCs. What next? Well, we need to think lower in the network stack. This means that we need to identify the network protocol that we'll need to use for getting hold of the RFCs in text format by ourselves.
The IETF landing page for RFCs is http://www.ietf.org/rfc.html, and reading through it tell us exactly what we want to know. We can access a text version of an RFC using a URL of the form http://www.ietf.org/rfc/rfc741.txt. The RFC number in this case is 741. So, we can get text format of RFCs using HTTP.
Now, we need a module that can speak HTTP for us. We should look at the standard library again. You will notice that there is, in fact, a module called http. Sounds promising, though looking at its documentation will tell us that it's a low level library and that something called urllib will prove to be more useful.
Now, looking at the urllib documentation, we find that it does indeed do what we need. It downloads the target of a URL through a straightforward API. We've found our protocol module.
Now we can write our program. For this, create a text file called RFC_downloader.py and save the following code to it:
import sys, urllib.request
try:
rfc_number = int(sys.argv[1])
except (IndexError, ValueError):
print('Must supply an RFC number as first argument')
sys.exit(2)
template = 'http://www.ietf.org/rfc/rfc{}.txt'
url = template.format(rfc_number)
rfc_raw = urllib.request.urlopen(url).read()
rfc = rfc_raw.decode()
print(rfc)We can run the preceding code by using the following command:
$ python RFC_downloader.py 2324 | less
On Windows, you'll need to use more instead of less. RFCs can run to many pages, hence we use a pager here. If you try this, then you should see some useful information on the remote control of coffee pots.
Let's go through our code and look at what we've done so far.
First, we import our modules and check whether an RFC number has been supplied on the command line. Then, we construct our URL by substituting the supplied RFC number. Next, the main activity, the urlopen() call will construct an HTTP request for our URL, and then it will contact the IETF web server over the Internet and download the RFC text. Next, we decode the text to Unicode, and finally we print it out to screen.
So, we can easily view any RFC that we like from the command line. In retrospect, it's not entirely surprising that there isn't a module for this, because we can use urllib to do most of the hard work!
But, what if HTTP was brand new and there were no modules, such as urllib, which we could use to speak HTTP for us? Well, then we would have to step down the stack again and use TCP for our purposes. Let's modify our program according to this scenario, as follows:
import sys, socket
try:
rfc_number = int(sys.argv[1])
except (IndexError, ValueError):
print('Must supply an RFC number as first argument')
sys.exit(2)
host = 'www.ietf.org'
port = 80
sock = socket.create_connection((host, port))
req = (
'GET /rfc/rfc{rfcnum}.txt HTTP/1.1\r\n'
'Host: {host}:{port}\r\n'
'User-Agent: Python {version}\r\n'
'Connection: close\r\n'
'\r\n'
)
req = req.format(
rfcnum=rfc_number,
host=host,
port=port,
version=sys.version_info[0]
)
sock.sendall(req.encode('ascii'))
rfc_raw = bytearray()
while True:
buf = sock.recv(4096)
if not len(buf):
break
rfc_raw += buf
rfc = rfc_raw.decode('utf-8')
print(rfc)The first noticeable change is that we have used socket instead of urllib. Socket is Python's interface for the operating system's TCP and UDP implementation. The command-line check remains the same, but then we will see that we now need to handle some of the things that urllib was doing for us before.
We have to tell socket which transport layer protocol that we want to use. We do this by using the socket.create_connection() convenience function. This function will always create a TCP connection. You'll notice that we have to explicitly supply the TCP port number that socket should use to establish the connection as well. Why 80? 80 is the standard port number for web services over HTTP. We've also had to separate the host from the URL, since socket has no understanding of URLs.
The request string that we create to send to the server is also much more complicated than the URL that we used before: it's a full HTTP request. In the next chapter, we'll be looking at these in detail.
Next, we deal with the network communication over the TCP connection. We send the entire request string to the server using the sendall() call. The data sent through TCP must be in raw bytes, so we have to encode the request text as ASCII before sending it.
Then, we piece together the server's response as it arrives in the while loop. Bytes that are sent to us through a TCP socket are presented to our application in a continuous stream. So, like any stream of unknown length, we have to read it iteratively. The recv() call will return the empty string after the server sends all its data and closes the connection. Hence, we can use this as a condition for breaking out and printing the response.
Our program is clearly more complicated. Compared to our previous one, this is not good in terms of maintenance. Also, if you run the program and look at the start of the output RFC text, then you'll notice that there are some extra lines at the beginning, and these are as follows:
HTTP/1.1 200 OK Date: Thu, 07 Aug 2014 15:47:13 GMT Content-Type: text/plain Transfer-Encoding: chunked Connection: close Set-Cookie: __cfduid=d1983ad4f7… Last-Modified: Fri, 27 Mar 1998 22:45:31 GMT ETag: W/"8982977-4c9a-32a651f0ad8c0"
Because we're now dealing with a raw HTTP protocol exchange, we're seeing the extra header data that HTTP includes in a response. This has a similar purpose to the lower-level packet headers. The HTTP header contains HTTP-specific metadata about the response that tells the client how to interpret it. Before, urllib parsed this for us, added the data as attributes to the response object, and removed the header data from the output data. We would need to add code to do this as well to make this program as capable as our first one.
What can't immediately be seen from the code is that we're also missing out on the urllib module's error checking and handling. Although low-level network errors will still generate exceptions, we will no longer catch any problems in the HTTP layer, which urllib would have done.
The 200 value in the first line of the aforementioned headers is an HTTP status code, which tells us whether there were any problems with the HTTP request or response. 200 means that everything went well, but other codes, such as the infamous 404 'not found' can mean something went wrong. The urllib module would check these for us and raise an exception. But here, we need to handle these ourselves.
So, there are clear benefits of using modules as far up the stack as possible. Our resulting programs will be less complicated, which will make them quicker to write, and easier to maintain. It also means that their error handling will be more robust, and we will benefit from the expertise of the modules' developers. Also, we benefit from the testing that the module would have undergone for catching unexpected and tricky edge-case problems. Over the next few chapters, we'll be discussing more modules and protocols that live at the top of the stack.
To round up, we're going to look at a few frequently encountered aspects of TCP/IP networks that can cause a lot of head-scratching for application developers who haven't encountered them before. These are: firewalls, Network Address Translation, and some of the differences between IPv4 and IPv6.
A firewall is a piece of hardware or software that inspects the network packets that flow through it and, based on the packet's properties, it filters what it lets through. It is a security mechanism for preventing unwanted traffic from moving from one part of a network to another. Firewalls can sit at network boundaries or can be run as applications on network clients and servers. For example, iptables is the de facto firewall software for Linux. You'll often find a firewall built into desktop anti-virus programs.
The filtering rules can be based on any property of the network traffic. The commonly used properties are: the transport layer protocol (that is, whether traffic uses TCP or UDP), the source and destination IP addresses, and the source and destination port numbers.
A common filtering strategy is to deny all inbound traffic and only allow traffic that matches very specific parameters. For example, a company might have a web server it wants to allow access to from the Internet, but it wants to block all traffic from the Internet that is directed towards any of the other devices on its network. To do so, it would put a firewall directly in front of or behind its gateway, and then configure it to block all incoming traffic, except TCP traffic with the destination IP address of the web server, and the destination port number 80 (since port 80 is the standard port number for the HTTP service).
Firewalls can also block outbound traffic. This may be done to stop malicious software that finds its way onto internal network devices from calling home or sending spam e-mail.
Because firewalls block network traffic, they can cause obvious problems for network applications. When testing our applications over a network, we need to be sure that the firewalls that exist between our devices are configured such that they let our application's traffic through. Usually, this means that we need to make sure that the ports which we need are open on the firewall for the traffic between the source and the destination IP addresses to flow freely. This may take some negotiating with an IT support team or two, and maybe looking at our operating system's and local network router's documentation. Also, we need to make sure that our application users are aware of any firewall configuration that they need to perform in their own environments in order to make use of our program.
Earlier, we discussed private IP address ranges. While they are potentially very useful, they come with a small catch. Packets with source or destination addresses in the private ranges are forbidden from being routed over the public Internet! So, without some help, devices using private range addresses can't talk to devices using addresses on the public Internet. However, with Network Address Translation (NAT), we can solve this. Since most home networks use private range addresses, NAT is likely to be something that you'll encounter.
Although NAT can be used in other circumstances, it is most commonly performed by a gateway at the boundary of the public Internet and a network that is using private range IP addresses. To enable the packets from the gateway's network to be routed on the public Internet as the gateway receives packets from the network that are destined for the Internet, it rewrites the packets' headers and replaces the private range source IP addresses with its own public range IP address. If the packets contain TCP or UDP packets, and these contain a source port, then it may also open up a new source port for listening on its external interface and rewrite the source port number in the packets to match this new number.
As it does these rewrites, it records the mapping between the newly opened source port and the source device on the internal network. If it receives a reply to the new source port, then it reverses the translation process and sends the received packets to the original device on the internal network. The originating network device shouldn't be made aware of the fact that its traffic is undergoing NAT.
There are several benefits of using NAT. The internal network devices are shielded from malicious traffic directed toward the network from the Internet, devices which use NAT devices are provided with a layer of privacy since their private addresses are hidden, and the number of network devices that need to be assigned precious public IP addresses is reduced. It's actually the heavy use of NAT that allows the Internet to continue functioning despite having run out of IPv4 addresses.
NAT can cause some problems for network's applications, if it is not taken into consideration at design time.
If the transmitted application data includes information about a device's network configuration and that device is behind a NAT router, then problems can occur if the receiving device acts on the assumption that the application data matches the IP and the TCP/UDP header data. NAT routers will rewrite the IP and TCP/UDP header data, but not the application data. This is a well known problem in the FTP protocol.
Another problem that FTP has with NAT is that in FTP active mode, a part of the protocol operation involves the client opening a port for listening on, and the server creating a new TCP connection to that port (as opposed to just a regular reply). This fails when the client is behind a NAT router because the router doesn't know what to do with the server's connection attempt. So, be careful about assuming that servers can create new connections to clients, since they may be blocked by a NAT router, or firewall. In general, it's best to program under the assumption that it's not possible for a server to establish a new connection to a client.
We mentioned that the earlier discussion is based on IPv4, but that there is a new version called IPv6. IPv6 is ultimately designed to replace IPv4, but this process is unlikely to be completed for a while yet.
Since most Python standard library modules have now been updated to support IPv6 and to accept IPv6 addresses, moving to IPv6 in Python shouldn't have much impact on our applications. However, there are a few small glitches to watch out for.
The main difference that you'll notice in IPv6 is that the address format has been changed. One of the main design goals of the new protocol was to alleviate the global shortage of IPv4 addresses and to prevent it from happening again the IETF quadrupled the length of an address, to 128 bits, creating a large enough address space to give each human on the planet a billion times as many addresses as there are in the entire IPv4 address space.
The new format IP addresses are written differently, they look like this:
2001:0db8:85a3:0000:0000:b81a:63d6:135b
Note the use of colons and hexadecimal format.
There are rules for writing IPv6 addresses in more compact forms as well. This is principally done by omitting runs of consecutive zeros. For example, the address in the preceding example could be shortened to:
2001:db8:85a3::b81a:63d6:135b
If a program needs to compare or parse text-formatted IPv6 addresses, then it will need to be made aware of these compacting rules, as a single IPv6 address can be represented in more than one way. Details of these rules can be found in RFC 4291, which is available at http://www.ietf.org/rfc/rfc4291.txt.
Since colons may cause conflicts when used in URIs, IPv6 addresses need to be enclosed in square brackets when they are used in this manner, for example:
http://[2001:db8:85a3::b81a:63d6:135b]/index.html
Also, in IPv6, it is now standard practice for network interfaces to have multiple IP addresses assigned to them. IPv6 addresses are classified by what scope they are valid in. The scopes include the global scope (that is, the public Internet) and the link-local scope, which is only valid for the local subnet. An IP address's scope can be determined by inspecting its high-order bits. If we enumerate the IP addresses of local interfaces to use for a certain purpose, then we need to check if we have used the correct address for the scope that we intend to work with. There are more details in RFC 4291.
Finally, with the mind-boggling cornucopia of addresses that are available in IPv6, the idea is that every device (and component, and bacterium) can be given a globally unique public IP address, and NAT will become a thing of the past. Though it sounds great in theory, some concerns have been raised about the implications that this has for issues like user privacy. As such, additions designed for alleviating these concerns have been made to the protocol (http://www.ietf.org/rfc/rfc3041.txt). This is a welcome progression; however, it can cause problems for some applications. So reading through the RFC is worth your while, if you're planning for your program to employ IPv6.
In the first part of this chapter, we looked at the essentials of networking with TCP/IP. We discussed the concept of network stacks, and looked at the principle protocols of the Internet protocol suite. We saw how IP solves the problem of sending messages between devices on different networks, and how TCP and UDP provide end-to-end transport between applications.
In the second section, we looked at how network programming is generally approached when using Python. We discussed the general principle of using modules that interface with services as far up the network stack as we can manage. We also discussed where we might find those modules. We looked at examples of employing modules that interface with the network stack at different layers to accomplish a simple network task.
Finally, we discussed some common pitfalls of programming for TCP/IP networks and some steps that may be taken to avoid them.
This chapter has been heavy on the networking theory side of things. But, now it's time to get stuck into Python and put some application layer protocols to work for us.
