VMware vCloud Director Cookbook

By Daniel Langenhan
  • Instant online access to over 7,500+ books and videos
  • Constantly updated with 100+ new titles each month
  • Breadth and depth in over 1,000+ technologies
  1. Setting Up Networks

About this book

VMware vCloud Director is an enterprise software solution that enables the building of secure, private clouds by pooling together infrastructure resources into virtual data centers. The tool enables self-service via a web interface to reduce the management overhead and offers amazing possibilities for production and development environments. Thus, the tool will ensure efficient management of resources with data center efficiency and business agility.

"VMWare VCloud Director Cookbook" will cover a lot of ground, ranging from easy to complex recipes. It will not only dive into networks, data-stores, and vApps, but also cover vCloud design improvements, troubleshooting, and the vCloud API.

"VMWare VCloud Director Cookbook" is split into different sections, each of which deals with a special topic in vCloud - from networks, to vApps, to storage and design. This book contains over 80 recipes with the difficulty levels ranging from simple to very advanced. You will learn how to automate vCloud easily and quickly with the API, and also learn how to isolate a vApp and still fully access it without risking the network. Design considerations that need to be addressed while deploying the vCloud and more will also be looked into.

"VMWare VCloud Director Cookbook" will make your life as an admin a lot easier by providing you with some good recipes that have been proven to work in small to large enterprises.

Publication date:
October 2013


Chapter 1. Setting Up Networks

In this chapter, we will see how to set up the various network resources that we will use in the next chapter. We will cover the following recipes:

  • Setting up an External Network

  • Creating 1,000 isolated networks without VXLANs

  • Making VXLANs work

  • Integrating Cisco 1000v into vCD

  • Giving your networks an Edge

  • Doing it all(most) without a Distributed Switch



Network virtualization is what makes vCloud Director such an awesome tool. However, before we go full out in the next chapter, we need to set up network virtualization, and it is what we will be focusing on here.

When we talk about isolated networks, we are talking about vCloud Director making use of different methods of the Network layer 3 encapsulation (OSI/ISO model). Basically, it's the same concept that was introduced with VLANs. VLANs split up the network communication in a network in different totally-isolated communication streams. vCloud makes use of these isolated networks to create networks in Organizations and vApps.

vCloud Director has three different network items listed as follows:

  • External Network: This is a network that exists outside vCloud, for example, a production network. It is basically a port group in vSphere that is used in vCloud to connect to the outside world. An External Network can be connected to multiple Organization Networks. External Networks are not virtualized and are based on existing port groups on vSwitch or a Distributed Switch (also called a vNetwork Distributed Switch or vNDS).

  • Organization Network: This is a network that exists only inside one organization. You can have multiple Organization Networks in an organization. Organizational networks come in three different types:

    • Isolated: An isolated Organization Network exists only in this organization and is not connected to an External Network; however, it can be connected to vApp Networks or VMs. This network type uses network virtualization and its own network settings.

    • Routed Network (Edge Gateway): An Organization Network connects to an existing Edge Device. An Edge Gateway allows defining firewall, NAT rules, DHCP services, Static Routes, as well as VPN connections and the load balance functionality. Routed Gateways connect External Networks to vApp Networks and/or VMs. This network uses virtualized networks and its own network settings.

    • Directly connected: This Organization Network is an extension of an External Network into the organization. They directly connect External Networks to vApp Networks or VMs. These networks do NOT use network virtualization and they make use of the network settings of an External Network.

  • vApp Network: This is a virtualized network that only exists inside a vApp. You can have multiple vApp Networks inside one vApp. A vApp Network can connect to VMs and to Organization Networks. It has its own network settings. When connecting a vApp Network to an Organization Network, you can create a router between the vApp and the Organization Network, which lets you define DHCP, firewall, NAT rules, and Static Routing.

To create isolated networks, vCloud Director uses Network Pools. Network Pools are a collection of VLANs, port groups, and VLANs that can use layer 2 in the layer 3 encapsulation. The content of these pools can be used by Organizations and vApp Networks for network virtualization.

Network Pools

There are four kinds of Network Pools that can be created:

  • Virtual eXtensible LANs (VXLAN): VXLAN networks are layer 2 networks that are encapsulated in layer 3 packets. VMware calls this Software Defined Networking (SDN). VXLANs are automatically created by vCloud Director (vCD); however, they don't work out of the box and require some extra configuration in vCloud Network and Security (refer to the Making VXLANs work recipe).

  • Network isolation-backed: These have basically the same concept as VXLANs; however, they work out of the box and use MAC-in-MAC encapsulation. The difference is that VXLANs can transcend routers whereas Network isolation-backed networks can't (refer to the Creating isolated networks without 1,000 VXLANs recipe).

  • vSphere port groups-backed: vCD uses pre-created port groups to build the vApp or Organization Networks. You need to pre-provision one port group for every vApp/Organization Network you would like to use.

  • VLAN-backed: vCD uses a pool of VLAN numbers to automatically provision port groups on demand; however, you still need to configure the VLAN trunking. You will need to reserve one VLAN for every vApp/Organization Network you would like to use.

VXLANs and Network isolation-backed networks solve the problems of pre-provisioning and reserving a multitude of VLANs, which makes them extremely important. However, using a port group or VLAN Network Pools can have additional benefits that we will explore later.

So let's get started!


Setting up an External Network

Let's start with something very simple, such as setting up an External Network.

Getting ready

Creating an External Network requires an existing port group in vSphere. This port group can be on a vSwitch, a Distributed vSwitch, or a Cisco 1000v Distributed Switch. The port group can be supported by a VLAN or a physical network.

How to do it...

  1. Log in to vCloud Director with a system administrator (SysAdmin) role.

  2. Click on Manage & Monitor.

  3. Click on External Networks.

  4. Click on the green plus icon (+). Now, the New External Network wizard starts.

  5. Select the vCenter that contains the port group and then select the port group you want the External Network connected to. If you have many networks, there is a filter just on the right above the list of the networks, as seen in the following screenshot:

  6. Add a subnet definition that contains at least the Gateway address, Network mask, and a Static IP pool by clicking on Add, as shown in the following screenshot:

  7. Enter a name for this network and close the wizard.

The External Network will now be created and is ready to be used.

How it works...

An External Network is just a connection between vCloud Director and a port group on vSphere. vCloud Director adds IP management to the port group. When creating an External Network, you have to define a pool. This pool is used to automatically assign IP addresses to VMs, Edge Gateways, or vApp routers attached to this External Network. A Static IP Pool has to contain a minimum of one IP, but can contain the maximum available IPs minus the gateway address. vCloud Director will manage all the IPs assigned though Organization Networks and Edge devices. The IP assignments can be seen by right-clicking on the External Network and selecting IP Allocations as shown in the following screenshot:

Using only one IP in an External Network Static IP Pool is interesting only if all IPs for VMs are assigned manually and no Edge or vApp router is used. If this is not the case, one should assign at least 5 to 10 IPs to the Network Pool. We will make excessive use of the External Network and we will use its IP pool for load balancing, VPNs, and much more.

There's more...

You can assign more than one IP range to an External Network, making it possible to create more than one IP range that can be used. However, IP allocation happens automatically and you are not able to control which IP from what range will be allocated to which specific VM. Creating multiple IP network ranges in External Networks is preferable when used together with IP suballocation in Edge devices.

When a VM is destroyed or undeployed, the IP will be released back to the pool. The setting of the default time for the IP release is set by navigating to Administration | General | IPaddress release timeout. The default value is 0 seconds. This setting specifies how long discarded IP addresses should be held before they can be reused. Think about your ARP tables and how long you have set your router's refresh time. If IP addresses are reallocated to new MAC addresses, a router might not be able to route it properly.


Creating 1,000 isolated networks without VXLANs

Network Pools are essential for network virtualization. If you are not sure about VXLAN networks, here is how you create 1,000 networks using only one VLAN.

Getting ready

As I have already mentioned, we need one VLAN that is trunked to a Distributed Switch. The VLAN doesn't need to be routed. The only other requirement is that the network gear can accommodate a higher MTU.

How to do it...

  1. In the system organization, we click on Manage & Monitor and then on Network pools.

  2. Now click on the green plus (+) icon to add a Network Pool.

  3. Now click on Network Isolation-backed as shown in the following screenshot:

  4. Define how many networks you would like to create. The maximum is 1,000:

  5. Type in the VLAN number you would like to use for the Network Pool.

  6. Now select the vCenter and the Distributed Switch you want to use.

  7. Give the Network Pool a name.

  8. After clicking on Next, you will see all the values you have entered.

  9. Click on Finish in order to create the Network Pool.

  10. Now we need to set the MTU for this VLAN to a minimum of 1524.

    A safer choice is 1600, as this makes sure you have enough room for additional encapsulations down the track. Make sure that your physical switching infrastructure can use a higher MTU than the default 1500.

  11. Click on the created Network Pool and select Properties.

  12. Click on Network pool MTU and set the MTU to 1600 as shown in the following screenshot:

  13. Click on OK.

How it works...

Network isolation-backed networks actually don't use layer 2 and layer 3 encapsulations, but they use the MAC-in-MAC encapsulation. When a new vApp or Organization Network is created, vCD will create a new port group and will then use this port group to encapsulate the traffic on a MAC basis. The same technique was used in VMware Lab Manager, which was then called Host Spanning Networks. This doesn't come without cost. Because of the additional encapsulation, another 24 bits are required for each package, meaning that the MTU should be increased to a minimum of 1524. If you don't change the MTU, you will have a network frame fragmentation.

The good thing is that Network isolation-backed Network Pools are quite fast and easy to configure and set up. They provide you with 1,000 isolated networks for each VLAN. You can define more than one Network isolation-backed network. However, you can only assign one Network Pool to an Organizational virtual Datacenter (OvDC), as there is a one-to-one relationship between them. You cannot create isolated networks before you assign a Network Pool to an OvDC.

To assign a Network Pool to an OvDC:

  1. Navigate to Manage & Monitor | Organizational VDC.

  2. Right-click on the OvDC you want the pool assigned to and select Properties.

  3. Click on Network pool & Services.

  4. Select the network pool you like to assign as shown in the following screenshot:

  5. Select the number of networks you would like to assign to the OvDC as shown in the following screenshot.

One of the disadvantages is that the networks are isolated, meaning we can't use them for anything other than vCloud Director.

See also

  • We will work extensively with vApp and Organization Networks in Chapter 2, vCloud Networks


Making VXLANs work

VXLANs are great, but they don't work out of the box. In the following sections, we discuss how to set them up.

Getting ready

As you already have vCloud set up, you must have a vCloud Network and Security appliance (vCNS) deployed (formally known as vShield), and it should be configured to use your vCenter. For this recipe, you will need to be able to log in to the vCNS appliance with an administrator account.

Additionally, we need a VLAN on which the VXLANs will exist, and having a DHCP in that VLAN makes things easier. If no DHCP is accessible on this VLAN, you will need to provide one IP address per ESXi server in this VLAN.

The Segment ID you have to enter in step 14 in the How to do it… section is rather important, especially when you have multiple vCNS or vCloud installations (not multiple cells). Each of these installations should have a different range. If this is your first VXLAN installation, just use the range that is supplied in the steps.

Last but not least, you should have a multicast address range (see http://en.wikipedia.org/wiki/Multicast_address); this is best arranged with the network administrator. If you can't figure out what to use, the range given in the steps will work fine for a VXLAN that exists only in one location.


If you are using the Cisco 1000v, please check out the Integrating the Cisco 1000v into vCD recipe before continuing here.

How to do it...

  1. Open a browser and browse to the vCNS appliance https://[ip of vCNS].

  2. Log in to the appliance (the default username is admin and the password is default).

  3. Make sure that you have switched to the Host & Clusters view.

  4. Expand the Datacenters folder.

  5. Click on your data center.

  6. On the right side of the screen, you should now find multiple menus; one of them says Network virtualization, so click on it.

  7. Now select Preparation and then Connectivity as shown in the following screenshot:

  8. Click on Edit.

  9. Select your Cluster.

  10. Select the Distributed Switch as well as the VLAN ID for the VXLAN that you want to use and click on Next as shown in the following screenshot:

  11. Now select a Teaming policy (for example, Fail Over) and its MTU (for example, 1600) as shown in the following screenshot:

  12. Click on Finish.

  13. Wait until the agents are installed on all the ESXi servers. The status should then show Normal (you might need to refresh).

  14. Click on Segment ID and then on Edit as shown in the following screenshot:

  15. You have to now enter a range for the Segment IDs (for example, 5000-6000).

  16. Now enter the Multicast address range (for example,

  17. Click on Finish.

  18. We are now done with vCNS and can leave the rest to vCloud Director.

  19. Log in to vCloud Director and click on Network pools.

  20. The VXLAN pool, which is automatically created with your PvDC, should now show a green tick. If this is not the case, right-click on it and select Repair.

How it works...

VXLANs were created by VMware together with Cisco. The idea was to solve the problems of modern data centers. Typically, these problems relate to the inflexibility of VLAN and Switching boundaries due to too much subnetting, IP, and VLAN management. The idea behind VXLANs is to create virtualized networking that is used on top of the common networking layer. They are in use just like the Network isolation-backed Network Pools we discussed in the other recipe; however, VXLANs have the benefit of being routable, flexible, and can transcend to different locations. This makes them extremely flexible and elegant to use.

VXLANs are actually like VLANs; the main difference is that VLANs (802.1q) have a 12-bit namespace whereas VXLANs have a 24-bit one, which increases the number of VLANs from 4,096 to more than 16 million unique namespaces.

VXLANs use layer 2 in layer 3 encapsulation. This means they use the Internet Protocol (IP, layer 3) to propagate the networks (from layer 2 upward), making them routable and far more flexible across network borders. One could envision VXLANs as a tunnel between two endpoints where additional networks exist.

There's more...

VXLANs don't really exist in vCloud Director; VXLANs are defined in the vCNS appliance. vCNS creates the VXLAN tunnel endpoints and manages the VXLANs for the whole virtual infrastructure. For all this to happen, vCNS must install an agent on each ESXi server. This is done when you click on Finish, as explained in step 12 in the How to do it… section of this recipe. These agents provide the connection between VMs and the VXLANs. As the VXLAN packages are bigger than the common network packages, we have to adjust the MTU to avoid frame fragmentation. A safe setting is 1600. Each agent will be deployed and connected to a new vmknic. The IP for the vmknic is assigned via DHCP; however, this can be changed in vCNS for each ESXi server.

The Fail Over policy that you set on the Distributed Switches depends on what the physical switching architecture can do. If EtherChannels are set up, choose them. Link Aggregation Control Protocol (LACP) can be chosen either in active or passive mode. LACP in an active mode sends out packages to talk to LACP-activated devices, whereas in a passive mode it waits until a LACP-activated device talks to it. If in doubt about all of this, please involve your network team.

If you want to configure VXLANs with Static IPs for each vmknic, follow the ensuing procedure:

  1. Log in to vCNS as an administrator.

  2. Navigate to Hosts & Clusters View | Datacenter | [your datacentre] | Network Virtualization | Connectivity as shown in the following screenshot:

  3. Write down which vmknic is used for VXLANs.

  4. Log in to vCenter as an administrator.

  5. Set a fixed IP for each of the vmknics on each ESXi server.

See also

The following are some links for further reading:


Integrating the Cisco 1000v into vCD

The Cisco 1000v Distributed Switch is an alternative to the VMware Distributed Switch; however, getting it working with vCloud is a challenge. In the following sections, we will see how to overcome it.

Getting ready

First and foremost, we need the Cisco 1000v installed (refer to the How it works section in this recipe). The next important thing is to hook the Cisco 1000v up to vCenter (refer to the See also section links in this recipe). After all this is done, you can use the Cisco 1000v in vSphere and we can now start linking it up to vCloud.

How to do it...

  1. Log on to the Cisco 1000v Virtual Supervisor Module (VSM).

  2. Run the following commands to activate the REST interface of the VSM:

    1kv# conf t
    1kv (Config)# feature network-segmentation-manager
    1kv (Config)# feature segmentation
    1kv (Config)# exit
    1kv # copy running-config startup-config
    [########################################] 100%
    1kv# exit
  3. Log out of the Cisco 1000v.

  4. Log in to the vCNS (vShield) as an administrator.

  5. Click on Settings & Reports.

  6. Click on Networking and then on Add Switch Provider as shown in the following screenshot:

  7. Now enter the Cisco VSM IP or hostname and the service API URL https://[VSM IP] /n1k/services/NSM.

  8. Enter the admin credentials for the VSM and click on OK.

And that's it. Now you can use the Making VXLANs work recipe to create VXLANs using the Cisco 1000v.

How it works...

The Cisco 1000v replaced the VMware Distributed Switch. The benefit of this is that you can use the Cisco tools and Cisco language to configure it. For all intents and purposes, it acts and behaves like a Cisco physical switch. One of the drawbacks is that there can be maximal 63 hosts (63 hosts and one Cisco v1000) connected to one Cisco 1000v and the integration between Cisco and VMware isn't as smooth as it may appear to be.

Cisco 1000v consists of two different items. The VSM is the management console and is basically a VM where the Cisco OS runs. The Virtual Ethernet module (VEM) is a plugin into the ESXi server that replaces the functionality of the VMware Distributed Switch.

There are basically two different designs for the Cisco 1000v installation. The first and easiest is to install the Cisco 1000v VSMs (Manager) as a VM on vSphere. This has considerable drawbacks as the Cisco 1000v VSMs can't be used with the Cisco 1000v itself, meaning you need to install it using normal vSwitches, which in return calls for a separate management cluster. The other better but more costly alternative is to use Cisco 1010. The Cisco 1010 is nothing else but a physical server that can contain up to six VSMs.

Flow for a Cisco 1000v installation is shown in the following diagram:

vCloud Director talks to vCenter for management (create VMs and so on), and talks to vCNS to create and manage complex networks (for example, Edge). The vCNS communicates with vCenter to create port groups and push out configurations to Distributed Switches. vCenter communicates with ESXi servers to facilitate management and to push network configurations out to the ESXi servers.

When the Cisco 1000v enters the picture, things change. Instead of vCNS talking to vCenter (the dotted line in the preceding figure), it now talks to the Cisco 1000v VSM. The VSM will then talk to vCenter and the VEM installed in the ESXi server to facilitate networking.

See also

The following are some useful links for the Cisco 1000v:


Giving your networks an Edge

For the recipes in the next chapter, we need an Edge. So, let's see how it works.

Getting ready

To create an Edge device in an organization, we need:

  • One External Network, with some free IP addresses

  • One organization

  • One OvDC connected to one Network Pool with at least one free isolated network

How to do it...

  1. Log in to vCloud (not the organization) as a SysAdmin.

  2. Click on Manage & Monitor and select Organization.

  3. Double-click on the organization you want to create the Edge in.

  4. The Organization Network should now have opened as a separate tab.

  5. Click on Administration.

  6. Double-click on the OvDC your Network Pool is associated with.

  7. Click on Edge Gateways as shown in the following screenshot:

  8. To create a new Edge device, click on the green plus (+) sign.

The Edge wizard opens up and the first page lets you choose some very basic and important settings. I marked the settings that I will use for this installation so that it matches the need of the Edge device in the next chapter. All of them can be configured later too.

Perform the following steps for configuring the Edge Gateway:

  1. Of the two Edge gateway configuration options, Compact and Full, select Compact. This option basically decides the resources that should be allocated to the gateway. Compact can be later upgraded to Full.

  2. Uncheck the Enable High Availability option; if chosen, the gateway is protected against faults in ESXi hosts.

  3. Check the Configure IP Settings option; it is used to manually configure the IP setting for the Edge.

  4. Check the Sub-Allocate IP Pools option; it makes the IPs from the External Network available to the Edge.

  5. Uncheck Configure Rate Limits; it is used to reduce the inbound and outbound bandwidth, as shown in the following screenshot:

Perform the following steps for configuring the External Networks:

  1. Select the External Networks that the Edge should be connected to and click on Add.

  2. If you select more than one External gateway, specify which one will be the default gateway.

  3. Select the Use default gateway for DNS Relay option as it allows DNS forwarding for the Edge.

Perform the following for configuring the IP settings:

  1. For each External Network, select if the gateway IP should be automatically taken from the pool or assigned manually.

  2. Clicking on Change IP Assignment will open up a window where you can assign the IP manually.

Perform the following steps to suballocate IP pools:

  1. Select the External Network you would like to create a suballocation for.

  2. Type the range you want to suballocate.

  3. Click on Add.

  4. Name the Edge device and click on Next to see the summary, or Finish to start the deploy of the Edge device.

  5. The Edge device is now deploying, which can take a moment. If you want, check out vCenter and see what's happening.

  6. We now need an organization network connecting to the Edge. Click on Org VDC Networks.

  7. Click on the green plus (+) sign to create a new Organization Network.

  8. Select Create a routed network…, select the Edge device you have created, and click on Next.

Perform the following steps for configuring the network:

  1. Type in the addresses in the Gateway address and Network mask options.

  2. The Edge can forward your DNS requests. If you don't want that and have your own DNS server in this Organization Network later configured, you can switch it off.

  3. Specify a Static IP pool for this network.

  4. Give the network a name and description.

  5. Click on Next for a summary or Finish to create the Organization Network.as shown in the following screenshot:

How it works...

The Edge is essentially a router with extras. It provides you with the ability to route between different networks, create firewall rules, create a DHCP service for Organization Networks, create load balancers, and also define a VPN network here.

The Edge devices are only accessible from the organization they were created in. The Edge has two sides: the northbound facing (External Networks) and the southbound facing (Organization Networks). For each Organization Network, the Edge provides a gateway address that can also act as a DNS forwarder. All communication through the Edge has to pass the firewall and can be controlled that way.

Each Organization Network can be configured with D-NAT and S-NAT, as well as DHCP and Static Routing. Load balancing can be configured and we will do that in the next chapter. As the Edge can be connected to multiple External Networks, it can serve as a hub for all kinds of connections, for example, direct Internet connections and local networks.

There's more...

I would strongly encourage you to take a trip into the vCNS and look at the Edge we created, and compare its vCNS options to the options vCloud Director presents you.

  1. Log in to vCNS.

  2. Expand Datacenter and click on your data center.

  3. Select Network Virtualization and then Edges.

  4. Double-click on your Edge device as shown in the following screenshot:

  5. The settings for the vCloud Edge can be found when you right-click on Edge and then click on Properties as shown in the following screenshot:

    We will work with the Edge in the next chapter and find out what we can do with it


Doing it all(most) without a Distributed Switch

So you want vCloud, but you don't have a Distributed Switch; here is how you can enjoy vCloud using vSwitches.

Getting ready

Surprisingly, you don't need much, just a lot of port groups. Make sure that these port groups are created exactly the same on all ESXi hosts (case sensitive).

How to do it...

We discuss the changes required for each of the different networks:

  • External Networks: Instead of using Distributed Switches, we will use normal vSwitches with port groups. No big drawbacks here.

  • Network Pools: The only Network Pool type you can use is the port group-backed one. This also means that you will need to have one port group per isolated network. The drawback is that you can also only assign one Network Pool to each OvDC, making it a one-to-one relationship between the port group, OvDC, and isolated network.

How it works...

Using vSwitches instead of Distributed Switches with vCloud Director is a possibility, but normally this method is only used when VMware Enterprise Plus Licensing is not available. However, with the introduction of vCloud Suite (which comes with vSphere Enterprise Plus), it is not really needed anymore. Still, in some cases this might provide a solution.

About the Author

  • Daniel Langenhan

    Daniel Langenhan is a Virtualisation expert with formidable skills in Architecture, Design and Implementation for large multi-tier systems. His experience and knowledge of process management, enterprise-level storage, Linux and Windows operation systems has made him and his business a highly sought after international consultancy in the Asia-Pacific and European regions for multinational clientele in the areas of Finance, Communication, Education and Government. Daniel has been working with VMware products since 2002 and is directly associated with VMWare since 2008. His proven track record of successful integrations of Virtualisation into different business areas while minimizing cost and maximizing reliability and effectiveness of the solution for his clients.

    Currently, Daniel is operating in the Europe and Asia-Pacific region with his company vLeet GmbH and Melbourne Business Boosters Pty Ltd.

    Daniel's expertise and practical approach to VMWare has resulted in the publication of the following books:

    • Instant VMware vCloud Starter, Packt Publishing
    • VMware View Security Essentials, Packt Publishing
    • VMware vCloud Director Cookbook, Packt Publishing
    • VMware vRealize Orchestrator Cookbook, Packt Publishing
    • VMware vRealize Orchestrator Essentials, Packt Publishing

    He has also lent his expertise to many other publishing projects as a Technical Editor.

    Browse publications by this author
Book Title
Access this book, plus 7,500 other titles for FREE
Access now