The term Next Generation Network (NGN) has been around for over 20 years, and refers to the current state-of-the-art network equipment, protocols, and features. In this chapter, we will discuss networking concepts such as hyperscale networking, software-defined networking, network hardware, and software design along with a litany of network design ideas utilized in NGN.
A big driver in NGN is the constant newer, better, and faster forwarding ASICs coming out of companies such as Barefoot, Broadcom, Cavium, and Nephos (MediaTek). The advent of commodity networking chips has shortened the development time for generic switches, allowing hyperscale networking end users to build equipment upgrades into their network designs.
At the time of writing, multiple companies have announced 6.4 Tbps switching chips. In layman terms, a 6.4 Tbps switching chip can handle 64x100 GbE of evenly distributed network traffic without losing any packets. To put the number in perspective, the entire internet in 2004 was about 4 Tbps, so all of the internet traffic in 2004 could have crossed this one switching chip without any issues (internet traffic 1.3 EB/month; http://blogs.cisco.com/sp/the-history-and-future-of-internet-traffic).
At the start of the commercial internet age (1994), software routers running on minicomputers such as BBNs PDP-11-based IP routers designed in the 1970s were still in use and hubs were simply dumb hardware devices that broadcast traffic everywhere.
At that time, the state of the art in networking was the Cisco 7000 series router, introduced in 1993. The next generation router was the Cisco 7500 (1995), while the Cisco 12000 series (gigabit) routers and the Juniper M40 were only concepts.
In this book, we will cover the current and near future of networking. When we say next generation, we are speaking of the current state of the art and the near future of networking equipment and software. For example, 100 GB Ethernet is the current state of the art, while 400 GB Ethernet is in the pipeline.
The definition of a modern network is that it is a network that contains one or more of the following concepts:
- Software-defined Networking (SDN)
- Network design concepts
- Next generation hardware
- Hyperscale networking
- Open networking hardware and software
- Network Function Virtualization (NFV)
- Highly configurable traffic management
Both open and closed network hardware vendors have been innovating at a high rate of speed with the help of and due to hyperscale companies such as Google, Facebook, and others who have the need for next generation high speed network devices. This provides the network architect with a reasonable pipeline of equipment to be used in designs.
Google and Facebook are both companies with hyperscale networks. A hyperscale network is one where the data stored, transferred, and updated on the network grows exponentially. Hyperscale companies deploy new equipment, software, and configurations weekly or even daily to support the needs of their customers. These companies have needs that are outside of the normal networking equipment available, so they must innovate by building their own next generation network devices, designing multi-tiered networks (like a three-stage Clos network), and automating the installation and configuration of the next generation networking devices.
The need for hyperscalers is well summed up by Google's Amin Vahdat in a 2014 Wired article: "We couldn't buy the hardware we needed to build a network of the size and speed we needed to build."
In this chapter, we will cover the basics of modern or next generation networking. When you are done with this chapter, you will have a good grasp of the following:
- Network protocols
- Next generation networking concepts
- Network design planning
- Open networking hardware and software
- Proprietary networking hardware and software
- Open source software controllers
- Closed source software controllers
- Network function virtualization
- Traffic engineering concepts
- Tools we will use in this book
Here you will find the definition of terms that we will use in this book. They have been broken into groups of similar concepts.
In network devices and network designs, there are many important concepts to understand. Here we'll begin with the way data is handled. The easiest way to discuss networking is to look at the OSI layer and point out where each device sits.
The OSI layer with respect to routers and switches is as follows:
- Layer 1 (Physical): This layer includes cables, hub, and switch ports. This is how all of the devices connect to each other, including copper cables (CatX), fiber optics, and Direct Attach Cable (DAC), which connect SFP ports without fiber.
- Layer 2 (Data link layer): This layer includes the raw data sent over the links and manages the Media Access Control (MAC) addresses for Ethernet.
- Layer 3 (Network layer): This layer includes packets that have more than just layer 2 data, such as IP, IPX (Novell Networks protocol), and AFP (Apple's protocol).
In a network, you will have equipment that switches and/or routes traffic. A switch is a networking device that connects multiple devices, such as servers, provides local connectivity, and provides an uplink to the core network. A router is a network device that computes paths to remote and local devices, providing connectivity to devices across a network. Both switches and routers can use copper and fiber connections to interconnect. There are a few parts to a networking device: the forwarding chip, the TCAM, and the network processor. Some newer switches have Baseboard Management Controllers (BMCs) which manage the power, fans, and other hardware, lessening the burden on the Network Operating System (NOS) to manage these devices.
Currently, routers and switches are very similar as there are many layer 3 forwarding capable switches and some layer 2 forwarding capable routers. Making a switch layer 3 capable is less of an issue than making a router layer 2 forwarding as the switch already is doing layer 2 and adding layer 3 is not an issue. However, a router does not do layer 2 forwarding in general, so it has to be modified to allow for ports to switch rather than route.
The control plane is where all of the information about how packets should be handled is kept. Routing protocols live in the control plane and are constantly scanning information received to determine the best path for traffic to flow. This data is then packed into a simple table and pushed down to the data plane.
A Virtual Local Area Network (VLAN) is a way of creating separate logical networks within a physical network. VLANs are generally used to separate/combine different users or network elements such as phones, servers, and workstations. You can have up to 4,096 VLANs on a network segment.
Virtual Extensible LAN (VXLAN) was created for large, dynamic isolated logical networks, virtualized networks, and multiple tenant networks. You can have up to 16 million VXLANs on a network segment versus 4,096 VLANs.
A VXLAN Tunnel Endpoint (VTEP) is a set of two logical interfaces: inbound, which encapsulates incoming traffic into VXLANs, and outbound, which removes the encapsulation of outgoing traffic from VXLANs back to its original state.
Network design requires the knowledge of the physical structure of the network so that the proper design choices are made. For example, in a data center, you would have a local area network; if you have multiple data centers near each other, they would be considered a metro area network.
A LAN is generally considered to be within the same building. These networks can be bridged (switched) or routed. In general, LANs are segmented into areas to avoid large broadcast domains.
A MAN is generally defined as multiple sites in the same geographic area or city, that is, a metropolitan area. A MAN generally runs at the same speed as a LAN, but is able to cover larger distances.
A WAN is essentially everything that is not a LAN or MAN. WANs generally use fiber optic cables to transmit data from one location to another. WAN circuits can be provided via multiple connections and data encapsulations, including MPLS, ATM, and Ethernet.
Most large network providers utilize Dense Wavelength Division Multiplexing (DWDM) to put more bits on their fiber networks. DWDM puts multiple colors of light onto the fiber, allowing up to 128 different wavelengths to be sent down a single fiber.
DWDM has just entered open networking with the introduction of Facebook's Voyager system.
In a leaf-spine network design, there are leaf switches (that connect to the servers), sometimes called Top of Rack (ToR) switches, connected to a set of spine switches (that connect leaves), sometimes called End of Rack (EoR) switches:
A Clos network is one of the ways to design a multi-stage network. Based on the switching network design by Charles Clos in 1952, a three-stage Clos is the smallest version of a Clos network. It has an ingress, a middle, and an egress stage. Some hyperscale networks use a five-stage Clos, where the middle is replaced with another three-stage Clos. In a three-stage Clos, there is an ingress, a middle ingress, a middle, a middle egress, and an egress stage. All stages are connected to their neighbor, so in the example shown here, Ingress 1 is connected to all four of the middle stages just as Egress 1 is connected to all four of the middle stages.
A Clos network can be built in odd numbers starting with three, so a five, seven, and so on stage Clos is possible. For even-numbered designs, Benes designs are usable:
Here we will discuss the concepts of network controllers. Every networking device has a controller, whether built in or external to manage the forwarding of the system.
A controller is a computer that sits on the network and manages one or more network devices. A controller can be built into a device, like the Cisco Supervisor module, or be standalone, like an OpenFlow controller.
The controller is responsible for managing all of the control plane data and deciding what should be sent down to the data plane.
Generally, a controller will have a Command-line Interface (CLI) and more recently a web configuration interface. Some controllers will even have an Application Programming Interface (API).
An OpenFlow controller, as it sounds, is a controller that uses the OpenFlow protocol to communicate with network devices. The most common OpenFlow controllers that people hear about are OpenDaylight and ONOS. People who are working with OpenFlow would also know of Floodlight and RYU.
A route processor is a computer that sits inside of the chassis of the network device you are managing. Sometimes, the route processor is built in to the system, while at other times, it is a module that can be replaced/upgraded. Many vendor multislot systems have multiple route processors for redundancy.
An example of a removable route processor is the Cisco 9500 series Supervisor module. There are multiple versions available, including revision A, with a 4-core processor and 16 GB of RAM, and revision B, with a 6-core processor and 24 GB of RAM.
Previous systems such as the Cisco Catalyst 7600 had options such as the SUP720 (Supervisor Module 720) of which they offered multiple versions: the standard SUP720 had a limited number of routes that it could support (256k) versus the SUP720 XL which could support up to 1 M routes:
In Juniper terminology, the controller is called a Route Engine (RE). These are similar to the Cisco Route Processor/Supervisor modules. Unlike Cisco Supervisor modules, which utilize special CPUs, Juniper's REs generally use common x86 CPUs. Like Cisco, Juniper multislot systems can have redundant processors.
Juniper has recently released the information about the Next Generation Route Engines (NG-REs). One example is the new RE-S-X6-64G, a 6-core x86 CPU-based routing engine with 64 GB DRAM and 2x64 GB SSD storage available for MX240/MX480/MX960. These NG-REs allow for containers and other virtual machines to be run directly.
When looking at single Rack Unit (RU) or pizza box design switches, there are some important design considerations. Most 1 RU switches do not have redundant processors or field replaceable route processors. In general, the Field Replaceable Units (FRUs) that the customer can replace are power supplies and fans. If the failure is outside of the available FRUs, the entire switch must be replaced in the event of a failure. With white-box switches, this can be a simple process as white-box switches can be used in multiple locations of your network, including the customer edge, provider edge, and core. Sparing (keeping a spare switch) is easy when you have the same hardware in multiple parts of the network.
Recently, commodity switch fabric chips have come with built-in low power ARM CPUs that can be used to manage the entire system, leading to cheaper and less power-hungry designs.
The Facebook Wedge is different from most white-box switches as it has its controller as an add-in module, the same board that is used in some of the OCP servers. By separating the controller board from the switch, different boards can be put in place, such as higher memory, faster CPUs, and different CPU types.
A routing protocol is a daemon that runs on a controller and communicates with other network devices to exchange route information. For this section, we will use common words to demonstrate the way the routing protocol is working; these should not be construed as the actual way that the protocols talk.
BGP is a path-vector-based External Gateway Protocol (EGP) that makes routing decisions based on paths, network policies, or rules (route-maps on Cisco). Though designed as an EGP, BGP can be used as both an interior (iBGP) and exterior (eBGP) routing protocol. BGP uses keepalive packets (are you there?) to confirm that neighbors are still accessible.
BGP is the protocol that is utilized to route traffic across the internet, exchanging routing information between different Autonomous System Numbers (ASNs). An ASN comprises all of the connected networks under the control of a single entity, such as Level 3, which has Autonomous System 1 (AS1) or Sprint (AS1239).
When two different ASNs interconnect, BGP peering sessions are set up between two or more network devices that have direct connections with each other.
In an eBGP scenario, AS1 and AS1239 would set up BGP peering sessions that would allow traffic to route between their AS.
In an iBGP scenario, the same AS would peer with other routers with the same AS and transfer the routes that are defined on the system. While iBGP is used internally in most networks, iBGP is used in large corporate networks because other Interior Gateway Protocols (IGPs) may not scale.
Consider these examples:
- iBGP next-hop self: In this scenario, AS1 and AS2 are peered with each other and exchange one prefix each. AS1 advertises 192.168.1.0/24 and AS2 advertises 192.168.2.0/24. Each network has two routers, one border router, which connects to other ASNs, and one internal router, which gets its routes from the border router. The routes are advertised internally with the next-hop set as the border router. This is a standard scenario when you are not running an IGP inside, to distribute the routes for the border router external interfaces:
The conversation goes like this:
AS1 -> AS2: Hi AS2, I am AS1
AS2 -> AS1: Hi AS1, I am AS2
AS1 -> AS2: I have the following route, 192.168.1.0/24
AS2 -> AS1: I have received the route, I have 192.168.2.0/24
AS1 -> AS2: I have received the route
AS1 -> Internal Router AS1: I have this route, 192.168.2.0/24, you can reach it through me at 10.1.1.1
AS2 -> Internal Router AS2: I have this route, 192.168.1.0/24, you can reach it through me at 10.1.1.1
- iBGP next-hop unmodified: In the next scenario, the border routers are the same, but the internal routers are given a next-hop of the external (other AS) border router:
- The last scenario is where you peer with a router server, a system that handles peering, filtering the routes based on what you have specified you send. The routes are then forwarded onto your peers with your IP as the next-hop:
OSPF is a relatively simple protocol. Different links on the same router are put into the same or different areas. For example, you would use Area 1 for the interconnects between campuses, but you would use another area, such as Area 10, for the campus itself. By separating areas, you can reduce the amount of cross-talk that happens between devices.
There are two versions of OSPF, v2 and v3. The main difference between v2 and v3 is that v2 is for IPv4 networks and v3 is for IPv6 networks:
When there are multiple paths that can be taken, the cost of the links must be taken into account. In the following diagram, you can see where there are two paths, one has a total cost of 20 (5+5+10) and the other, 16 (8+8), so the traffic will take the lowest-cost link:
IS-IS is a link-state routing protocol, operating by flooding link-state information throughout a network of routers using Network Entity Titles (NETs). Each IS-IS router has its own database of the network topology, built by aggregating the flooded network information. IS-IS is used by companies who are looking for fast convergence, scalability, and rapid flooding of new information.
IS-IS uses the concept of levels instead of areas as in OSPF. There are two levels in IS-IS, Level 1 (area) and Level 2 (backbone). A Level 1 Intermediate System (IS), keeps track of the destinations within its area, while a Level 2 IS keeps track of paths to the Level 1 areas:
EIGRP is Cisco's proprietary routing protocol. It is hardly ever seen in current networks, but if you see it in yours, then you need to plan accordingly. Replacing EIGRP with OSPF is suggested so that you can interoperate with non-Cisco devices.
If RIP is being used in your network, it must be replaced during the design. Most newer routing stacks do not support RIP. It is one of the original routing protocols, using the number of hops (routed ports) between the device and remote location to determine the optimal path. RIP sends its entire routing database out every 30 seconds. When routing tables were small, many years ago, RIP worked fine. With larger tables, the traffic bursts and the resulting recomputing by other routers in the network causes routers to run at almost 100 percent CPU all the time.
Cables will be mentioned throughout the book, so we will review the major types here.
Copper cables have been around for a very long time. Originally, network devices were connected using coax cable (the same cable used for television antennas). These days, there are a few standard cables that are used. These are the different RJ45 cables:
- Cat 5 : This is a 100 MB capable cable, used for both 10 MB and 100 MB connections
- Cat 5e: This is a 1 GbE capable cable, but not suggested for 1 GbE networks (Cat 6 is better and the price difference is nominal)
- Cat 6: This is a 1 GbE capable cable, and it can be used for any speed, at or below 1 GbE, including 100 MB and 10 MB
The following is the list of fiber/hot pluggable cables and ports:
- Small Form-factor Pluggable (SFP):
- SFP: This is capable of up to 1 GbE connections
- SFP+: This is of the same size as SFP, and is capable of up to 10 Gb connections
- SFP28: This is of the same size as SFP, capable of up to 25 Gb connections
- Quad Small Form-factor Pluggable (QSFP): This is a bit wider than SFP, but capable of multiple GbE connections
- QSFP+: This is of the same size as QSFP, and is capable of 40 GbE as 4x10GbE on the same cable
- QSFP28: This is of the same size as the QSFP, capable of 100 GbE
- Direct Attach Cable (DAC): Has a built in SFP or QSFP type connector and goes directly into the port cage on the switch.
- Fiber optic cable: Needs a SFP or QSFP type connector in the port cage to connect into the switch. Some switches may have fixed fiber optic ports, but this is uncommon in current generation products.
As routers and switches continue to become more dense, where the number of ports on the front of the device can no longer fit in the space, manufacturers have moved to what we call breakout cables. For example, if you have a switch that can handle 3.2 Tbps of traffic, you need to provide 3200 Gbps of port capacity. The easiest way to do that is to use 32 100 Gb ports, which will fit on the front of a 1 U device. You cannot fit 128 10 Gb ports without using either a breakout patch panel (which will then use another few RUs) or a breakout cable.
For a period of time in the 1990s, Cisco used RJ21 connectors to provide up to 96 Ethernet ports per slot:
Network engineers would then create breakout cables to go from the RJ21 to RJ45.
These days, we have both DAC and fiber breakout cables. For example, here you can see a 1x4 breakout cable, providing 4x10 G or 25 G ports from a single 40 G or 100 G port:
There are a litany of concepts that define a modern network, from simple principles to full feature sets.
In general, a next-generation data center design enables you to move to a widely distributed non-blocking fabric with uniform chipset, bandwidth, and buffering characteristics in a simple architecture.
In one example, to support these requirements, you would begin with a true three-tier Clos switching architecture with ToR, spine, and fabric layers to build a data center network. Each ToR would have access to multiple fabrics and have the ability to select a desired path based on application requirement or network availability.
Following the definition of a modern network from the introduction, here we lay out the general definition of the parts.
Here we will discuss the concepts that build an NGN.
SDNs can be defined in multiple ways. The general definition of a SDN is one which can be controlled as a singular unit instead of on a system-by-system basis. The control plane, which would normally be in the device and uses routing protocols, is replaced with a controller. SDNs can be built using many different technologies, including OpenFlow, overlay networks, and automation tools.
Within an SDN, you will have the concept of controllers. There are four controllers that we will talk about in this book:
- OpenDaylight and ONOS, which are OpenFlow-based open source controllers
- Application Policy Infrastructure Controller (APIC) from Cisco
- NSX from VMware
As we mentioned in the introduction, 20 years ago, NGN hardware would have been the Cisco GSR (officially introduced in 1997) or the Juniper M40 (officially released in 1998). Large Cisco and Juniper customers would have been working with the companies to help come up with the specifications and determining how to deploy the devices (possibly Alpha or Beta versions) in their networks:
Today, we can look at the hyperscale networking companies to see what a modern network looks like. A hyperscale network is one where the data stored, transferred, and updated on the network grows exponentially. Technology such as 100 Gb Ethernet, SDN, open networking equipment, and software are being deployed by hyperscale companies.
Open hardware has been around for about 10 years, first in the consumer space and more recently in the enterprise space. Enterprise open networking hardware companies such as Quanta and Accton provide a significant amount of the hardware currently utilized in networks today. Companies such as Google and Facebook have been building their own hardware for many years. Facebook's routers such as the Wedge 100 and Backpack are available publicly for end users to utilize.
Some examples of open networking hardware are as follows:
- Dell S6000-ON: This is a 32x40 G switch with 32 QSFP+ ports on the front
- Quanta LY8: This is a 48x10 G + 6x40 G switch with 48 SFP+ ports and 6 QSFP+ ports
- Facebook Wedge 100: This is a 32x100 G switch with 32 QSFP28 ports on the front
To use open networking hardware, you need an operating system. The operating system manages the system devices such as fans, power, LEDs, and temperature. On top of the operating system, you will run a forwarding agent. Examples of forwarding agents are Indigo, the open source OpenFlow daemon, and Quagga, an open source routing agent.
Cisco and Juniper are the leaders in the closed hardware and software space. Cisco produces switches such as the Nexus series (3000, 7000, and 9000) with 9000 programmable by ACI. Juniper provides the MX series (480, 960, and 2020) with 2020 being the highest end forwarding system they sell.
Cisco has multiple NOSes including IOS, NX-OS, and IOS-XR. All Cisco NOSes are closed source and proprietary to the system that they run on. Cisco has what the industry call an industry standard CLI, which is emulated by many other companies.
Juniper ships a single NOS, Junos, which can install on multiple different systems. Junos is a closed source BSD-based NOS. The Junos CLI is significantly different from IOS and is more focused on engineers who program.
Network virtualization, not to be confused with NFV, is the concept of recreating the hardware interfaces that exist in a traditional network in software. By creating a software counterpart to the hardware interfaces, you decouple the network forwarding from the hardware.
There are a few companies and software projects that allow the end user to enable network virtualization. The first one is NSX, which comes from the same team that developed Open vSwitch (OVS) called Nicira, which was acquired by VMware in 2012. Another project is Big Cloud Fabric by Big Switch Networks, which utilizes a heavily modified version of Indigo, an OpenFlow controller.
NFV can be summed up by the statement, due to recent network focused advancements in PC hardware, any service able to be delivered on proprietary, application specific hardware should be able to be done on a virtual machine, essentially, on routers, firewalls, load balancers, and other network devices, all running virtually on commodity hardware.
While Open-source software (OSS) has been around for decades, the concept of open networking hardware has not. By combining OSS with open networking hardware, end users are able to create their own network devices that provide the connectivity and services that are necessary for them.
One of the first companies to come out with open networking hardware was Quanta Computer. In 2009, Pronto started to provide open networking switches including the LB4G and LB9(A). These switches were used by the OpenFlow team at Stanford to develop OpenFlow on hardware. Sold under the Pronto Networks name (now, Pica8), these switches were used by companies such as Google for their SDN projects.
In 2011, Facebook started the Open Compute Project (OCP). The goal of the OCP was to provide a place where companies could share hardware and software designs. These designs are used by multiple hardware manufacturers to build OCP specification hardware. In 2013, the OCP introduced the networking project, where networking vendors could submit open hardware designs for network switches.
Companies such as Big Switch Networks, Cumulus Networks, and Pluribus Networks utilize open switching hardware built by companies such as Accton, Dell, and Quanta to create fully open and malleable networks. OSS projects such as Open Network Linux, OpenSwitch, and OS10 provide a open software base for these devices on which end users can build their own tools.
One of the most important software tools is Open Network Install Environment (ONIE), which is a small Linux image that allows end users to install a NOS onto a network device such as a switch.
Some examples of open source networking software are as follows:
- Facebook's FBOSS, a Thrift-based daemon that manages the forwarding of the switch by interacting with Broadcom's OpenNSL. FBOSS has no routing capabilities of its own and requires all information to be provided via a configuration file and Thrift API calls.
- Microsoft's Software for Open Networking in the Cloud (SONiC) uses a Quagga-based routing daemon talking to Switch Abstraction Interface (SAI) and runs on a few open hardware switches including the Dell S6000-ON and the Mellanox SN2700.
- Google have also designed their own switches since 2004, but have not released the designs or software information. In 2012, one of their switches was accidentally shipped to the wrong location and appeared on the internet.
Accton has been quite active in the open hardware space, providing a multitude of designs, including some designed by Facebook, such as the Wedge. The current generation Facebook Wedge is the Wedge 100, providing 32 ports of 100 G. Accton also has its own switches, such as the AS7716, that provide 32 ports of 100 G:
Most open networking hardware designs are based around switching ASICs from Broadcom, but over the past few years, other companies such as Barefoot Networks, Cavium, and Mellanox have brought out more open designs. Barefoot is a good example of a fully open design, where they utilize a specific language, called P4, to program the forwarding hardware.
The Facebook designed switches are focused on Facebook's own design needs. The Facebook Wedge 100 runs a standard Linux image with drivers for the Broadcom switching chips. On top of the software stack, Facebook uses an OSS project called FBOSS to control the switches via a Thrift API. This allows Facebook to manage their switches the same way they manage their servers.
Facebook contracted with both Accton and Quanta to build the Wedge 40. The Wedge 40 is built from commodity components and are reused from other Facebook systems. The CPU complex and Board Management Controllers (BMC) come from the Facebook servers.
The next generation networking devices that have come from the needs of hyperscale networking companies have a few commonalities:
- In general, the configuration and operation of these devices have been designed to be automated or managed from a central controller
- Automation is done via tools that use everything from screen scraping to utilizing Thrift or REST APIs
- Most of these hyperscale-focused next generation networking devices have one or more parts that are traditionally found on servers, such as a BMC, powerful Intel processors, and large solid state storage drives
The equipment used in hyperscale networks can be from established vendors, such as Cisco and Juniper, or from open networking companies, such as Edgecore and Quanta. Dell is a special case as they offer both closed and open versions of their switching hardware, designated with a -ON at the end, for example, the S6000-ON and a 32x40G switch. Mellanox, which started as a storage network vendor, has been building open networking switches, including the SN2700, a 32x100 G switch, and the SN2100, a 16x100 G switch.
Many open networking designs come out of specific needs of the hyperscale companies and some even come from the hyperscale companies. Facebook have open sourced five switches, all designed to meet their needs:
- Wedge 40: This is a 16x40 G switch with a BMC: running FBOSS
- 6-pack: This is a 128x40 G modular switch with multiple BMCs: running FBOSS
- Wedge 100: This is a 32x100 G switch with a BMC: running FBOSS
- Backpack: This is a 128x100 G modular switch with multiple BMCs: running SnapRoute
- Voyager: This is a open transponder for DWDM networks, which includes both 12x100 G Ethernet and 4x200 G DWDM ports: running an FBOSS-like daemon:
Software is the heart of any network: no matter how sophisticated hardware gets, software is necessary to utilize the hardware. Here we will discuss the software components behind open hardware initiatives.
In order to use open hardware, there needs to be an installation environment. Currently, ONIE is the standard. ONIE was developed by Cumulus Networks in coordination with Big Switch Networks, and it provides a GRUB or U-Boot installable miniature Linux environment from which NOS can be installed on the system.
The Bootloader is a small bootable software image that is flashed to Cisco hardware. The Bootloader initializes the system and brings up the devices necessary to load the main software, IOS.
Open source projects such as OpenBMC have been released to provide the software to run on the BMC and system processor.
Networking companies such as RTBrick and SnapRoute have been formed to provide API manageable networking stacks. SnapRoute provides an entire forwarding infrastructure including L2/L3 and forwarding chip drivers. SnapRoute have written their project in Go, Google's language of choice.
Software-defined products such as Cisco's Application Centric Infrastructure (ACI) and VMware's NSX have come out of large companies such as Cisco and VMware along with products such as Big Cloud Fabric (BCF) from Big Switch Networks.
Open NOS companies such as Cumulus Networks and Pica8 have released software for open switches. Pica8 also provides a full solution, selling switches with PicOS installed. Pica8 originally provided switches under the Pronto name, but now sells them under the Pica8 name, for example, the P-5401: 32x40 G switch.
Software from companies such as Cisco and Juniper are considered closed source as they do not include access to the source code. Cisco has multiple operating systems, including their original Internetwork Operating System (IOS), not to be confused with Apple's recent use for their iDevices.
IOS is a binary blob operating system that is loaded into memory on boot of Cisco devices. IOS is easy to upgrade since all of the configuration information is kept separate and the IOS filesystem is immutable. Installing a new IOS version simply requires that you upload it to the device and point the configuration to load it. IOS uses a CLI that is considered to be the standard interface and replicated by many other vendors.
Once you have your hardware picked out, you need an NOS. An NOS is what runs on your switch and allows it to forward packets. Companies such as Cumulus Networks and Pica8 sell full NOSes with L2 and L3 forwarding capability. In the OSS world, there are a few choices, including Open Network Linux (ONL), OpenSwitch, and SONiC.
ONL is a project started by Rob Sherwood, previously of Big Switch Networks and now at Facebook. The goal was to provide a simple, clean Linux-based open source network operating system. ONL provides a ONIE compatible installable NOS, on which the user can install their own forwarding agents. At the time of writing, ONL supported ~35 switches from Alpha Networks, Dell, DNI, Edgecore, Mellanox, Quanta, and others.
OpenSwitch is a project started by HP (now HPE) to provide a full NOS using Quagga as the base and creating a full layer 2 / layer 3 platform on top of a Linux base.
The project used a central database based on OVSDB and required all data to be exchanged through the database rather than directly between themselves. The design was complex and eventually supported ~3 switches directly, rebranded HP versions of Edgecore switches, and unofficially supported ~4 more that were ported by end user or vendors.
OpenSwitch started pivoting at the time of this book. The Quagga design with OVSDB was replaced with SnapRoute, an API-driven routing stack running on Dell's OS10 Open Edition, a Debian-based NOS.
SONiC is a project that Microsoft started to run inside their own network on white-box switches. It uses Quagga for forwarding and Redis as a database to store information and exchange data between processes. SONiC runs on top of Debian 8 and can be run on Dell OS10 or ONL.
At the time of writing, SONiC supported more switches than OpenSwitch, including some Edgecore, Arista, and Mellanox switches.
If you need to run a forwarding agent on a Linux-based NOS, there are a few options, of which most, other than Mellanox's SwitchDev implementation, have proprietary / closed source parts. Broadcom provides both OF-DPA and OpenNSL publicly as binary-only options for programming their ASICs. Cavium offers an SAI interface to their SDK, but it is not publicly available at the time of writing.
To quote the Linux kernel documentation, The Ethernet switch device driver model (switchdev) is an in-kernel driver model for switch devices which offload the forwarding (data) plane from the kernel. Essentially, it is an open Netlink listener that allows for the offloading of forwarding information to hardware. It can also be used for soft switches such as OVS and offload network interface cards using SR-IOV.
From the open switching side, currently, only Mellanox supports SwitchDev for their 10/25/40/50/100G switches. Broadcom only supports its consumer-grade switches such as those in access points and home routers.
Indigo is a OpenFlow-based forwarding agent that runs on the NOS to provide forwarding. Introduced in 2008 by Stanford University, Indigo is the base for Big Switch Networks OpenFlow daemon, the ON.Lab CORD project, and Indigo is integrated into OF-DPA, the Broadcom OpenFlow driver.
The concept of OpenFlow is simple, program forwarding tables in hardware and software switches using a standardized interface. We will cover OpenFlow in detail later in this book.
While specifically designed for the Facebook Wedge switches, FBOSS provides a Thrift-based API with integration into Broadcom's Open Network Switch Layer (OpenNSL). FBOSS allows for static configuration of interfaces and forwarding entries. FBOSS also has a Netlink listener available, which allows for the end user to run programs such as Quagga or FRR on the system.
SwitchD is the Cumulus Networks SDK programmer. It is available with the purchase of Cumulus Linux, a Linux-based NOS.
There are two types of software controllers. The first type uses OpenFlow to manage the hardware in the network. OpenFlow is an open standard that is easily utilized to manage the forwarding of traffic. The second type uses a proprietary API to manage the hardware in the network. The API can be open or closed, but will not be generic.
An OpenFlow controller manages multiple networking devices by programming switches using the OpenFlow protocol. OpenFlow-based network devices run a OpenFlow daemon such as Indigo, which translates the OpenFlow commands into switch forwarding data.
There are branded and unbranded versions of OpenDaylight available. Brocade makes the Brocade Virtual Controller (BVC). The OpenDaylight foundation releases unbranded versions of the software. You can also obtain OpenDaylight from Cisco.
The following diagram shows a test setup for OpenDaylight using two switches:
A good example of a open source, open API design is what the company called SnapRoute is doing with their product FlexSwitch. FlexSwitch uses a REST-based API to program the L2/L3 forwarding entries.
In my 25 or so years of networking, I have dealt with a lot of different networking technologies, each iteration (supposedly) better than the last. Starting with Thinnet (10BASE2), moving through ARCNET, 10BASE-T, token ring, ATM to the Desktop, Fiber Distributed Data Interface (FDDI), and onward. Generally, the technology improved for each system until it was swapped out. A good example is the change from a literal ring for token ring to a switching design, where devices hung off of a hub (as in 10BASE-T).
ATM to the Desktop was a novel idea, providing up to 25 Mbps to connected devices, but the complexity of configuring and managing it was not worth the gain.
Today, almost everything is Ethernet, as shown by the Facebook Voyager DWDM system, which uses Ethernet over both traditional SFP ports and the DWDM interfaces. Ethernet is simple, well supported, and easy to manage.
During late 1996 and early 1997, the Exodus Network used FDDI rings to connect the main routers together at 100 Mbps. As the network grew, we had to decide between two competing technologies, FDDI switches and Fast Ethernet (100Base-T), both providing 100 Mbps. FDDI switches from companies such as DEC (FDDI Gigaswitch) were used in most of the Internet Exchange Points (IXPs) and worked reasonably well with one minor issue, Head-of-Line Blocking (HoLB), which also impacted other technologies. HoLB occurs when a packet is destined for an interface that is already full, so a queue is built; if the interface continues to be full, eventually, the queue will be dropped.
While we were testing the DEC FDDI Gigaswitches, we were also in deep discussions with Cisco about the availability of Fast Ethernet (FE) and working on designs. Because FE was new, there were concerns about how it would perform and how we would be able to build a redundant network design.
In the end, we decided to use FE, connect the main routers in a full mesh, and use routing protocols to manage failover.
During the high-growth period at Exodus Communications, there was a request to connect a new data center to the original one and allow customers to put servers in both locations using the same address space. To do this, we chose LAN Emulation (LANE), which allows an ATM network to be used like a LAN. On paper, LANE looked like a great idea, the ability to extend the LAN so that customers could use the same IP space in two different locations. In reality, it was very different.
For hardware, we were using Cisco 5513 switches, which provided a combination of Ethernet and ATM ports.
There were multiple issues with this design:
- First, the customer is provided with an Ethernet interface, which runs over an ATM optical interface. Any error on the physical connection between switches or the ATM layer would cause errors on the Ethernet layer.
- Second, monitoring was very hard, when there were network issues, you had to look in multiple locations to determine where the errors were happening.
After a few weeks, we did a midnight swap, putting Cisco 7500 routers in to replace the 5500 switches and moving customers onto new blocks for the new data center.
When designing a new network, some of the following factors might be important to you:
- Simple, focused, yet non-blocking IP fabric
- Multistage parallel fabrics based on the Clos network concept
- Simple merchant silicon
- Distributed control plane with some centralized controls
- Wide multi-path (ECMP)
- Uniform chipset, bandwidth, and buffering
- 1:1 oversubscribed (non-blocking fabric)
- Minimizing the hardware necessary to carry east—west traffic
- Ability to support a large number of bare metal servers without adding an additional layer
- Limiting fabric to a five-stage Clos within the data center to minimize lookups and switching latency
- Support host attachment at 10 G, 25 G, 50 G, and 100G Ethernet
- Traffic management
In a modern network, one of the first decisions is whether you will use a centralized controller or not. If you use a centralized controller, you will be able to see and control the entire network from one location. If you do not use a centralized controller, you will need to either manage each system directly or via automation. There is a middle space where you can use some software-defined network pieces to manage parts of the network, such as an OpenFlow controller for the WAN or VMware NSX for your virtualized workloads.
Once you know what the general management goal is, the next decision is whether to use open, proprietary, or a combination of both open and proprietary networking equipment. Open networking equipment is a concept that has been around less than a decade and started when very large network operators decided that they wanted better control of the cost and features of the equipment in their networks. Google is a good example. In the following figure, you can see how Facebook used both their own hardware, 6-pack/Backpack, and legacy vendor hardware for their interoperability and performance testing:
Google wanted to build a high-speed backbone, but were not looking to pay the prices that the incumbent proprietary vendors such as Cisco and Juniper wanted. Google set a price per port (1 G/10 G/40 G) that they wanted to hit and designed equipment around that. Later, companies such as Facebook decided to go in the same direction and contracted with commodity manufacturers to build network switches that met their needs.
Proprietary vendors can offer the same level of performance or better using their massive teams of engineers to design and optimize hardware. This distinction even applies on the software side, where companies such as VMware and Cisco have created SDN tools such as NSX and ACI.
With the large amount of networking gear available, designing and building a modern network can appear to be a complex concept. Designing a modern network requires research and a good understanding of networking equipment. While complex, the task is not hard if you follow the guidelines listed in this section.
These are a few of the stages of planning that need to be followed before the modern network design is started:
- The first step is to understand the scope of the project (single site, multisite, multicontinent, and multiplanet).
- The second step is to determine whether the project is a green field (new) or brown field deployment (how many of the sites already exist and will/will not be upgraded?).
- The third step is to determine whether there will be any SDN, NGN, or open networking pieces.
- Finally, it is key that the equipment to be used is assembled and tested to determine whether the equipment meets the needs of the network.
The project scope is one of the most important pieces of information needed. The project scope can go from a single device in one location to hundreds or even thousands of devices across multiple continents. Understanding the project scope provides a guideline on which to base the network design and hardware/software needs.
If the network is being designed for internal use, then looking at other locations should give information about the best practices of the company the network is being designed for. If the network is being designed for an external company, then it is useful to ask for documentation, hardware lists, and even a tour of a current site so that the concept can be understood.
This is not meant to be an exhaustive list, but there are a few things that need to be understood when designing the network:
- Is the network all internal?
- Does the network have a DMZ?
- Does the network have multiple internet connections?
- Does the network have storage and compute separate or together?
- Does the network need to support iSCSI or other SAN protocols?
- Does the network use MPLS, SD-WAN, or other tunneling technologies?
- Does the network have multiple Points of Presence (POP), and how large is a POP?
- Does the network use containers? If so, does it have a container-specific network?
At the end of the book, you will find a generic check sheet to scope the project.
A greenfield network is a site where there is no networking equipment currently. For a greenfield deployment, there are a lot of options, but the needs of the network musts be clearly understood. In a perfect situation, the site would be completely malleable, allowing for power, cooling, and infrastructure to be built out to meet the needs of the design. Since a perfect situation is not always possible, taking inventory of the infrastructure is necessary before a design can be committed.
At the end of the book, you will find a generic check sheet that provides an overview of what should be understood about the site.
If you plan to use any next generation hardware, you will need to do some research and show due diligence. Since next generation hardware generally means equipment that has not been out for very long (or may not be out in the public), there will be little to no public information or testing of the equipment. You will want to schedule a Proof of Concept (PoC) to be done with the hardware you expect to use.
If you are using open next generation hardware, you may be able to use reference customers to understand what designs are being used and what features.
While PC-based network devices have been available since the 80s, they were generally used by small companies and networking enthusiasts who didn't or couldn't afford to buy a commercial-based solution. In the last few years, many drivers have brought PC-based networking devices back into the limelight, including: Ethernet as the last mile, better network interface cards, and Intel's focus on networking processing in its last few generation of chips.
Today, many vendors are producing PC-based network devices with advancements in packet handling within Intel's processors, allowing processor cores to be re-programmed into network processors, and allowing PC-based network devices to push tens or even hundreds of Gbps.
Some of the values of the NFV concept are speed, agility, and cost reduction. By centralizing designs around commodity server hardware, network operators can do the following:
- Do a single PoP/site design based on commodity compute hardware:
- Avoiding designs involving one-off installations of appliances that have different power, cooling, and space needs simplifies planning
- Utilize resources more effectively:
- Virtualization allows providers to allocate only the necessary resources needed by each feature/function
- Deploy network functions without having to send engineers to each site:
- Truck rolls are costly both from a time and money standpoint
- Achieve reductions in OpEX and CapEX
- Achieve reduction of system complexity
Traffic engineering and traffic shaping is the concept of detecting and prioritizing different types of network traffic. Once prioritized, different bandwidth allocations can be provided to the traffic. Prioritization can be strict or loose and as a set amount or a variable amount (percentage).
Traffic engineering can be done in a few different ways, including MPLS TE tunnels, Virtual Circuits (VCs), and Quality of Service (QoS).
There are many tools that we will discuss and utilize in this book, tools to monitor networks, tools to configure networks, and everything in between.
Monitoring the network is highly important and has been the basis for quite a few great open-source tools such as Nagios, Monit, Sensu, and Zabbix.
Nagios is one of the older and most mature open source monitoring tools, providing a core infrastructure and a set of plugins for different devices:
The generic Nagios display shows alerts and details about the network status.
The following tools will help you in network configuration.
While we are working with the network, it will be useful to keep logs of changes in configurations. Really Awesome New Cisco confIg Differ (RANCID) is a free tool that can log in to many different types of systems, not just Cisco (though it started out that way).
Note
You can find more about RANCID at http://www.shrubbery.net/rancid/.
Postman is a program that works with APIs. It is available at http://www.getpostman.com. We will utilize Postman when dealing with REST-based APIs:
In this chapter, we discussed many different concepts that tie NGN together. These concepts will be discussed more in forthcoming chapters, so use this chapter as a reference if you need an explanation of any terms. Some takeaways from the chapter are as follows.
The term NGN refers to the latest and near term networking equipment and designs. We looked at networking concepts such as local, metro, and wide area networks, network controllers, routers, and switches as well as routing protocols such as BGP, IS-IS, OSPF, and RIP.
Now, what defines a modern network? There are many pieces, which are used either singularly or together, that create a modern network: SDN using building blocks including OpenFlow, Cisco ACI, and VMware NSX; next generation hardware and hyperscale networking including open networking hardware from Accton, Quanta, and Facebook along with software from Big Switch Networks, Cumulus, and Pica8; proprietary NGN hardware and software from Cisco, Juniper and Arista along with internal use only hardware from Google; open source software controllers such as OpenFlow controllers such as Floodlight, ONOS, and OpenDaylight, along with closed source controllers such as Cisco APIC and VMware NSX, NFV, traffic engineering using QoS, and OpenFlow; and network design planning including the scoping of the current network/site. A list of tools we will use in this book such as Git, Nagios, Postman, and RANCID.
In the next chapter, we will go deeper into networking hardware and software including, the Open Compute Project and its goals for hyperscale networks.
