Chapter 4. Networking Nuts and Bolts

Basic IP configuration has been addressed in the Chapter 3, Sysadmin Configuration Differences. This chapter covers advanced network configurations possible in Solaris 11, along with a more in-depth overview of the changes.

Networking re-architected

Solaris 11 has a vastly redesigned networking infrastructure, both at the kernel level, and at the sysadmin level. It is crucial to understand this redesign.

Therefore, I shall start this chapter with some orientation material.

The following topics are touched on in this chapter:

• Kernel redesign

• Orientation to new Solaris 11 networking

• Interface naming

• NWAM – automatic networking configuration

• IPMP (IP multipathing), tunneling, and VLAN management

• Network resource management (per-interface, flows, and IP QoS)

• Other changes

The changes in Solaris 11 improve both performance, and general manageability. In the manageability department, changes you make with the new tools stay across reboots.

Kernel redesign

Prior to Solaris 11, many, but not all, network drivers took advantage of a common driver framework called the Generic LAN Driver (GLD) framework.

Now in Solaris 11, virtually all drivers use GLD Version 3. This has allowed a redesign of the kernel-to-user interface to bring...

Orientation to new Solaris 11 networking

Networking functionality has now been split into layers, almost similar in some ways to how the old simple method of accessing disks through disk slices, got split up by ZFS into pools, mirrors, and the like.

Previously, almost all network configuration in Solaris was done by the physical interface, xyz0, and some permutation of the ifconfig command. Now, however, the multiple personalities of ifconfig, have been divided up into separate commands and layers.

The basic layers can be grouped as follows:

Physical, readonly (dladm)	Data link (dladm)	IP interface (ipadm)	IP "addrobj" (ipadm)
bge0, nxge0, e1000g0, and so on	net0, vlan0, vnic0, aggr0, stub0	net0, (simple case usually should match data link object name), ipmp0	net0/labelhere

To configure an IP address generally requires one item from each column, building step-by-step, from the left-most column, until you get to the right-hand side.

The left-most column is only visible from a global...

Interface naming and IP labels

By default, all network interface names visible at the general sysadmin level, are now simply called net0, net1, and so on. The inconsistency of names such as bge0, e1000g0, and ngxe0 are now hidden from view.

If you have different types of physical interface, and you need to know which netX name maps to which specific physical name, you can view mappings with:

dladm show-link

This command will show you all dladm level devices, including Virtual NICs and aggregation links. However, if you would like to narrow it down to just the ones associated with physical interfaces, you should use the following command:

dladm show-phys

Beyond that, what you most likely care even more about, after the initial setup, is which interface is being used for what purpose. Even more in this era of virtual services and zones, it is nice to know this information per IP address, rather than per interface. To that end, Solaris 11 associates IP address with user-named strings. Now if...

NWAM – NetWork AutoMagic

Someone on the Solaris engineering team decided to name this feature, Network AutoMagic. But from a server sysadmin perspective, it might perhaps be better named "Never Wake A Monster".

NWAM is primarily useful for people running Solaris on a laptop, who have to deal with wireless in different locations. If you are the console user (that is, physically in front of the machine), and within the GNOME-based desktop, you will have a fairly nice tool that allows you to easily configure wireless, and even swap between location-based profiles for other IP needs. In that context, it is fairly useful. If, on the other hand, your systems are servers which ignore DHCP, it is best avoided.

NWAM gets in the way of doing almost anything intelligent or fancy with the normal network control tools dladm or ipadm. It will cause complaints about "Persistent operation on temporary object". Technically, you can add the -t flag to dladm or ipadm, to make your operation temporary as well...

IPMP – IP multipathing

IP multipathing is a configuration that allows an IP address to automatically failover between two or more physical network interfaces. Prior versions of Solaris did already have IPMP support, but it was a bit hacky. It involved using ifconfig to assign two or more interfaces to a group, and then basically hoping the kernel did a good job from there. It was then up to the kernel to plumb and unplumb a secondary IP address (such as bge0:1) from one physical interface or another.

IPMP support in Solaris 11 is much more refined. The floating IP is given its own persistent, virtual object, at the same level as any other IP object at the ipadm layer. This then allows you to apply persistent tuning options to it.

There is also a new tool specifically written to monitor IPMP; predictably, it is called ipmpstat.

Setting up IPMP

Before you can set up a multipath IP object, you first have to set up underlying regular IP objects and interfaces with their own IP addresses (so that...

Link aggregation

Link aggregation, aka "trunking", is similar, yet different to IP multipathing (IPMP). Both actually let you increase outbound bandwidth by spreading it across multiple links. Both give you a measure of automatic failover support. However, only link aggregation lets you spread inbound bandwidth across multiple links.

That's the good news. The bad news is link aggregation has some important limitations that, while they may not be a problem in "small to midsize" shops, can be a problem in enterprise level endeavors.

Link aggregation uses the IEEE 802.3ad protocol, sometimes known as LACP. This makes it an Ethernet-only solution. An even stronger limitation is that all the links that are aggregated together must be on the same switch.

In contrast, IPMP, can be used across truly redundant, separate switches. Furthermore, as it is an IP-level solution, rather than a link-level solution, it can be used on top of alternative transports such as InfiniBand.

The bonus good news, for...

VNIC – Virtual NIC

Prior to Solaris 11, it was always possible to add a virtual, or secondary IP address to a network interface, such as bge0:1 on top of bge0.

Solaris 11 takes things an important step further, by allowing you to encapsulate an IP address within a virtual network interface object. This object can be treated as a first class network entity in its own right. It is possible to run snoop on it. It is also possible to assign it to a zone, and allow the zone to have full access to it, without compromising the security of the physical device, or other VNICs on that device.

You can name a VNIC almost anything you want, as long as you put a number at the end of it. For ease of comprehension, however, you may want to stick to the standard of vnicX.

Sample usage is as follows:

dladm create-vnic -l net0 vnic1
ipadm create-ip vnic1
ipadm create-addr -T static -a local=1.2.3.9/24 vnic1/extraIP
snoop -d vnic1

As mentioned, snoop will pick up traffic related to 1.2.3.9, and to broadcast traffic...

VLAN tagging

VLAN tagging is a network layer 2 standard designed to allow separate Ethernet broadcast domains to coexist safely on a single physical interface or network fabric. Configuring a virtual IP address or virtual network interface with a specific VLAN ID, or tag, is the way to let the network hardware know which broadcast domain to assign a packet to.

Solaris 10 supported use of VLAN tagging for interfaces; however, the administrative interface was highly ugly. It involved using ifconfig to plumb virtual interfaces with mandatory 6-digit numbers.

Solaris 11's unified network interface allows for much cleaner VLAN tagging usage. You can now create a VLAN tagged virtual interface that is a top-level network entity in its own right. Basically, it is a VNIC with a fixed extra attribute, which is the VLAN tag ID. You can give it a custom name to denote that it is not just an ordinary VNIC, but after that, you can treat it as any other VNIC level device. For example:

dladm create-vlan...

IP tunneling

In Solaris 10 and earlier, it was possible to create IP tunnels manually, using ifconfig. In Solaris 11, you use the dladm and ipadm commands. The bad news is the process involves a few more steps. The good news is it is automatically persistent across reboots, and also is more readable, by virtue of the fact that you can name the tunnels.

To create an IPv4 tunnel between your own host at 1.2.3.4 and a remote gateway at 5.6.7.8, use the following sequence of commands:

dladm create-iptun -T ipv4 -a local=1.2.3.4,remote=5.6.7.8 tunnelname0
ipadm create-ip tunnelname0
ipadm create-addr -T static -a local=1.2.3.4,remote=5.6.7.8 tunnelname0/usefulnamehere

As you can see, the syntax is fairly similar to VLAN creation. First, the creation of a base-layer kernel object is required, through dladm. After that, the IP-level setup via ipadm is almost identical to other IP administrative tasks.

Bridging

Bridging is the term used when a device transparently forwards network layer 2 packets from one broadcast domain to another. In essence, it can make the Solaris server act like an Ethernet switch (or in some cases, other layer 2 protocol switches).

Note

Bridging makes the involved physical interfaces to always run in promiscuous mode, which has the effect of degrading effective interface throughput.

Normally, I would consider bridging on a Solaris box to not be used enough to include here, but the improvements in Solaris 11 are so impressive that they are worth mentioning.

The old way of making a Solaris machine act as a bridge was to set ip_forwarding=1 on the interfaces you wished, then telling other machines to use the Solaris machine as a bridge, and blindly hope for the best.

The new way is much improved. There are real administrative commands instead of ndd, and the OS has been enhanced to properly support the following standard bridging protocols:

Spanning Tree Protocol (STP)...

Network resource management

Now, there are three main ways to handle resource management in Solaris 11: by flows, by interface, and by custom Quality of Service rules.

The QoS methods are the most flexible, but also highly complex, and perhaps best avoided unless absolutely necessary.

Related to resource management is knowing just how much data is flowing through the resource. At a coarse level, the dlstat command will tell you the amount of data that has been flowing through a low-level physical or virtual interface. Similar to the iostat command, it can show you total number of packets since reboot, or you can tell it to report over regular intervals.

At a little more refined level, is the flowstat command, mentioned in the Flow-based resource management section.

Per-interface management

There are two types of resource management on an dladm level interface that are of interest:

• CPU based

• Bandwidth based

Limits are set through interface properties. You could consider them somewhat similar to old...

Other changes

Other things that have changed in Solaris 11 networking, that we won't go into detail about here, but are worth mentioning briefly:

• The netcat utility is now included in Solaris. This is useful for a simple user-level network port redirector. Wireshark is also just a pkg install away.

• The supported DHCP server for Solaris is now the ISC DHCP server (the most common open source one).

  Note

  Since this DHCP server is widely known and documented, the only things worth mentioning here are the Solaris specific bits:

  New DHCP SMF service FMRIs:

  svc:/network/dhcp/server:ipv4

  svc:/network/dhcp/server:ipv6

  New DHCP server configuration files:

  /etc/inet/dhcpd4.conf

  /etc/inet/dhcpd6.conf

• Solaris 11 supports Virtual Router Redundancy Protocol (VRRP). However, since most readers will probably not want to turn their Solaris box into a router, all that needs to be mentioned here is that there is a single administrative command, predictably named vrrpadm.

Summary

Solaris 11 networking administration has changed so much that it is almost like using a different operating system. There are clear benefits from the new admin interfaces, as well as the performance benefits of the kernel rearchitecture. The most important takeaways for the new networking paradigm are to understand what new capabilities are now available and to know which toolsets are most important to use.

First and foremost, remember the following parallel paths for tool chains:

• NWAM profile-oriented configuration: netcfg and netadm. Despite the generic names, these are only for NWAM.

• Standard sysadmin tools: ipadm, dladm, flowadm, flowstat, and dlstat. Remember that ipadm and dladm only work normally if NWAM is disabled.

Secondly, remember the virtualization and stackability of the new VNIC, VLAN, and aggr objects. You can treat any of these types as a first class object, where you can assign exclusive use of them to a zone, including snoop and IP address control. You can also stack...

Interface naming and IP labels

By default, all network interface names visible at the general sysadmin level, are now simply called net0, net1, and so on. The inconsistency of names such as bge0, e1000g0, and ngxe0 are now hidden from view.

If you have different types of physical interface, and you need to know which netX name maps to which specific physical name, you can view mappings with:

dladm show-link

This command will show you all dladm level devices, including Virtual NICs and aggregation links. However, if you would like to narrow it down to just the ones associated with physical interfaces, you should use the following command:

dladm show-phys

Beyond that, what you most likely care even more about, after the initial setup, is which interface is being used for what purpose. Even more in this era of virtual services and zones, it is nice to know this information per IP address, rather than per interface. To that end, Solaris 11 associates IP address with user-named strings. Now if...

NWAM – NetWork AutoMagic

Someone on the Solaris engineering team decided to name this feature, Network AutoMagic. But from a server sysadmin perspective, it might perhaps be better named "Never Wake A Monster".

NWAM is primarily useful for people running Solaris on a laptop, who have to deal with wireless in different locations. If you are the console user (that is, physically in front of the machine), and within the GNOME-based desktop, you will have a fairly nice tool that allows you to easily configure wireless, and even swap between location-based profiles for other IP needs. In that context, it is fairly useful. If, on the other hand, your systems are servers which ignore DHCP, it is best avoided.

NWAM gets in the way of doing almost anything intelligent or fancy with the normal network control tools dladm or ipadm. It will cause complaints about "Persistent operation on temporary object". Technically, you can add the -t flag to dladm or ipadm, to make your operation temporary as well...

IPMP – IP multipathing

IP multipathing is a configuration that allows an IP address to automatically failover between two or more physical network interfaces. Prior versions of Solaris did already have IPMP support, but it was a bit hacky. It involved using ifconfig to assign two or more interfaces to a group, and then basically hoping the kernel did a good job from there. It was then up to the kernel to plumb and unplumb a secondary IP address (such as bge0:1) from one physical interface or another.

IPMP support in Solaris 11 is much more refined. The floating IP is given its own persistent, virtual object, at the same level as any other IP object at the ipadm layer. This then allows you to apply persistent tuning options to it.

There is also a new tool specifically written to monitor IPMP; predictably, it is called ipmpstat.

Setting up IPMP

Before you can set up a multipath IP object, you first have to set up underlying regular IP objects and interfaces with their own IP addresses (so that...

Link aggregation

Link aggregation, aka "trunking", is similar, yet different to IP multipathing (IPMP). Both actually let you increase outbound bandwidth by spreading it across multiple links. Both give you a measure of automatic failover support. However, only link aggregation lets you spread inbound bandwidth across multiple links.

That's the good news. The bad news is link aggregation has some important limitations that, while they may not be a problem in "small to midsize" shops, can be a problem in enterprise level endeavors.

Link aggregation uses the IEEE 802.3ad protocol, sometimes known as LACP. This makes it an Ethernet-only solution. An even stronger limitation is that all the links that are aggregated together must be on the same switch.

In contrast, IPMP, can be used across truly redundant, separate switches. Furthermore, as it is an IP-level solution, rather than a link-level solution, it can be used on top of alternative transports such as InfiniBand.

The bonus good news, for...

VNIC – Virtual NIC

Prior to Solaris 11, it was always possible to add a virtual, or secondary IP address to a network interface, such as bge0:1 on top of bge0.

Solaris 11 takes things an important step further, by allowing you to encapsulate an IP address within a virtual network interface object. This object can be treated as a first class network entity in its own right. It is possible to run snoop on it. It is also possible to assign it to a zone, and allow the zone to have full access to it, without compromising the security of the physical device, or other VNICs on that device.

You can name a VNIC almost anything you want, as long as you put a number at the end of it. For ease of comprehension, however, you may want to stick to the standard of vnicX.

Sample usage is as follows:

dladm create-vnic -l net0 vnic1
ipadm create-ip vnic1
ipadm create-addr -T static -a local=1.2.3.9/24 vnic1/extraIP
snoop -d vnic1

As mentioned, snoop will pick up traffic related to 1.2.3.9, and to broadcast traffic...

VLAN tagging

VLAN tagging is a network layer 2 standard designed to allow separate Ethernet broadcast domains to coexist safely on a single physical interface or network fabric. Configuring a virtual IP address or virtual network interface with a specific VLAN ID, or tag, is the way to let the network hardware know which broadcast domain to assign a packet to.

Solaris 10 supported use of VLAN tagging for interfaces; however, the administrative interface was highly ugly. It involved using ifconfig to plumb virtual interfaces with mandatory 6-digit numbers.

Solaris 11's unified network interface allows for much cleaner VLAN tagging usage. You can now create a VLAN tagged virtual interface that is a top-level network entity in its own right. Basically, it is a VNIC with a fixed extra attribute, which is the VLAN tag ID. You can give it a custom name to denote that it is not just an ordinary VNIC, but after that, you can treat it as any other VNIC level device. For example:

dladm create-vlan...

IP tunneling

In Solaris 10 and earlier, it was possible to create IP tunnels manually, using ifconfig. In Solaris 11, you use the dladm and ipadm commands. The bad news is the process involves a few more steps. The good news is it is automatically persistent across reboots, and also is more readable, by virtue of the fact that you can name the tunnels.

To create an IPv4 tunnel between your own host at 1.2.3.4 and a remote gateway at 5.6.7.8, use the following sequence of commands:

dladm create-iptun -T ipv4 -a local=1.2.3.4,remote=5.6.7.8 tunnelname0
ipadm create-ip tunnelname0
ipadm create-addr -T static -a local=1.2.3.4,remote=5.6.7.8 tunnelname0/usefulnamehere

As you can see, the syntax is fairly similar to VLAN creation. First, the creation of a base-layer kernel object is required, through dladm. After that, the IP-level setup via ipadm is almost identical to other IP administrative tasks.

Bridging

Bridging is the term used when a device transparently forwards network layer 2 packets from one broadcast domain to another. In essence, it can make the Solaris server act like an Ethernet switch (or in some cases, other layer 2 protocol switches).

Note

Bridging makes the involved physical interfaces to always run in promiscuous mode, which has the effect of degrading effective interface throughput.

Normally, I would consider bridging on a Solaris box to not be used enough to include here, but the improvements in Solaris 11 are so impressive that they are worth mentioning.

The old way of making a Solaris machine act as a bridge was to set ip_forwarding=1 on the interfaces you wished, then telling other machines to use the Solaris machine as a bridge, and blindly hope for the best.

The new way is much improved. There are real administrative commands instead of ndd, and the OS has been enhanced to properly support the following standard bridging protocols:

Spanning Tree Protocol (STP)...

Network resource management

Now, there are three main ways to handle resource management in Solaris 11: by flows, by interface, and by custom Quality of Service rules.

The QoS methods are the most flexible, but also highly complex, and perhaps best avoided unless absolutely necessary.

Related to resource management is knowing just how much data is flowing through the resource. At a coarse level, the dlstat command will tell you the amount of data that has been flowing through a low-level physical or virtual interface. Similar to the iostat command, it can show you total number of packets since reboot, or you can tell it to report over regular intervals.

At a little more refined level, is the flowstat command, mentioned in the Flow-based resource management section.

Per-interface management

There are two types of resource management on an dladm level interface that are of interest:

• CPU based

• Bandwidth based

Limits are set through interface properties. You could consider them somewhat similar to old...

Other changes

Other things that have changed in Solaris 11 networking, that we won't go into detail about here, but are worth mentioning briefly:

• The netcat utility is now included in Solaris. This is useful for a simple user-level network port redirector. Wireshark is also just a pkg install away.

• The supported DHCP server for Solaris is now the ISC DHCP server (the most common open source one).

  Note

  Since this DHCP server is widely known and documented, the only things worth mentioning here are the Solaris specific bits:

  New DHCP SMF service FMRIs:

  svc:/network/dhcp/server:ipv4

  svc:/network/dhcp/server:ipv6

  New DHCP server configuration files:

  /etc/inet/dhcpd4.conf

  /etc/inet/dhcpd6.conf

• Solaris 11 supports Virtual Router Redundancy Protocol (VRRP). However, since most readers will probably not want to turn their Solaris box into a router, all that needs to be mentioned here is that there is a single administrative command, predictably named vrrpadm.

Summary

Solaris 11 networking administration has changed so much that it is almost like using a different operating system. There are clear benefits from the new admin interfaces, as well as the performance benefits of the kernel rearchitecture. The most important takeaways for the new networking paradigm are to understand what new capabilities are now available and to know which toolsets are most important to use.

First and foremost, remember the following parallel paths for tool chains:

• NWAM profile-oriented configuration: netcfg and netadm. Despite the generic names, these are only for NWAM.

• Standard sysadmin tools: ipadm, dladm, flowadm, flowstat, and dlstat. Remember that ipadm and dladm only work normally if NWAM is disabled.

Secondly, remember the virtualization and stackability of the new VNIC, VLAN, and aggr objects. You can treat any of these types as a first class object, where you can assign exclusive use of them to a zone, including snoop and IP address control. You can also stack...