Home Cloud & Networking VMware vSphere Troubleshooting

VMware vSphere Troubleshooting

books-svg-icon Book
eBook $35.99 $24.99
Print $54.99
Subscription $15.99 $10 p/m for three months
$10 p/m for first 3 months. $15.99 p/m after that. Cancel Anytime!
What do you get with a Packt Subscription?
This book & 7000+ ebooks & video courses on 1000+ technologies
60+ curated reading lists for various learning paths
50+ new titles added every month on new and emerging tech
Early Access to eBooks as they are being written
Personalised content suggestions
Customised display settings for better reading experience
50+ new titles added every month on new and emerging tech
Playlists, Notes and Bookmarks to easily manage your learning
Mobile App with offline access
What do you get with a Packt Subscription?
This book & 6500+ ebooks & video courses on 1000+ technologies
60+ curated reading lists for various learning paths
50+ new titles added every month on new and emerging tech
Early Access to eBooks as they are being written
Personalised content suggestions
Customised display settings for better reading experience
50+ new titles added every month on new and emerging tech
Playlists, Notes and Bookmarks to easily manage your learning
Mobile App with offline access
What do you get with eBook + Subscription?
Download this book in EPUB and PDF formats, plus a monthly download credit
This book & 6500+ ebooks & video courses on 1000+ technologies
60+ curated reading lists for various learning paths
50+ new titles added every month on new and emerging tech
Early Access to eBooks as they are being written
Personalised content suggestions
Customised display settings for better reading experience
50+ new titles added every month on new and emerging tech
Playlists, Notes and Bookmarks to easily manage your learning
Mobile App with offline access
What do you get with a Packt Subscription?
This book & 6500+ ebooks & video courses on 1000+ technologies
60+ curated reading lists for various learning paths
50+ new titles added every month on new and emerging tech
Early Access to eBooks as they are being written
Personalised content suggestions
Customised display settings for better reading experience
50+ new titles added every month on new and emerging tech
Playlists, Notes and Bookmarks to easily manage your learning
Mobile App with offline access
What do you get with eBook?
Download this book in EPUB and PDF formats
Access this title in our online reader
DRM FREE - Read whenever, wherever and however you want
Online reader with customised display settings for better reading experience
What do you get with video?
Download this video in MP4 format
Access this title in our online reader
DRM FREE - Watch whenever, wherever and however you want
Online reader with customised display settings for better learning experience
What do you get with video?
Stream this video
Access this title in our online reader
DRM FREE - Watch whenever, wherever and however you want
Online reader with customised display settings for better learning experience
What do you get with Audiobook?
Download a zip folder consisting of audio files (in MP3 Format) along with supplementary PDF
What do you get with Exam Trainer?
Flashcards, Mock exams, Exam Tips, Practice Questions
Access these resources with our interactive certification platform
Mobile compatible-Practice whenever, wherever, however you want
BUY NOW $10 p/m for first 3 months. $15.99 p/m after that. Cancel Anytime!
eBook $35.99 $24.99
Print $54.99
Subscription $15.99 $10 p/m for three months
What do you get with a Packt Subscription?
This book & 7000+ ebooks & video courses on 1000+ technologies
60+ curated reading lists for various learning paths
50+ new titles added every month on new and emerging tech
Early Access to eBooks as they are being written
Personalised content suggestions
Customised display settings for better reading experience
50+ new titles added every month on new and emerging tech
Playlists, Notes and Bookmarks to easily manage your learning
Mobile App with offline access
What do you get with a Packt Subscription?
This book & 6500+ ebooks & video courses on 1000+ technologies
60+ curated reading lists for various learning paths
50+ new titles added every month on new and emerging tech
Early Access to eBooks as they are being written
Personalised content suggestions
Customised display settings for better reading experience
50+ new titles added every month on new and emerging tech
Playlists, Notes and Bookmarks to easily manage your learning
Mobile App with offline access
What do you get with eBook + Subscription?
Download this book in EPUB and PDF formats, plus a monthly download credit
This book & 6500+ ebooks & video courses on 1000+ technologies
60+ curated reading lists for various learning paths
50+ new titles added every month on new and emerging tech
Early Access to eBooks as they are being written
Personalised content suggestions
Customised display settings for better reading experience
50+ new titles added every month on new and emerging tech
Playlists, Notes and Bookmarks to easily manage your learning
Mobile App with offline access
What do you get with a Packt Subscription?
This book & 6500+ ebooks & video courses on 1000+ technologies
60+ curated reading lists for various learning paths
50+ new titles added every month on new and emerging tech
Early Access to eBooks as they are being written
Personalised content suggestions
Customised display settings for better reading experience
50+ new titles added every month on new and emerging tech
Playlists, Notes and Bookmarks to easily manage your learning
Mobile App with offline access
What do you get with eBook?
Download this book in EPUB and PDF formats
Access this title in our online reader
DRM FREE - Read whenever, wherever and however you want
Online reader with customised display settings for better reading experience
What do you get with video?
Download this video in MP4 format
Access this title in our online reader
DRM FREE - Watch whenever, wherever and however you want
Online reader with customised display settings for better learning experience
What do you get with video?
Stream this video
Access this title in our online reader
DRM FREE - Watch whenever, wherever and however you want
Online reader with customised display settings for better learning experience
What do you get with Audiobook?
Download a zip folder consisting of audio files (in MP3 Format) along with supplementary PDF
What do you get with Exam Trainer?
Flashcards, Mock exams, Exam Tips, Practice Questions
Access these resources with our interactive certification platform
Mobile compatible-Practice whenever, wherever, however you want
  1. Free Chapter
    The Methodology of Problem Solving
About this book
VMware vSphere is the leading server virtualization platform with consistent management for virtual data centers. It enhances troubleshooting skills to diagnose and resolve day to day problems in your VMware vSphere infrastructure environment. This book will provide you practical hands-on knowledge of using different performance monitoring and troubleshooting tools to manage and troubleshoot the vSphere infrastructure. It begins by introducing systematic approach for troubleshooting different problems and show casing the troubleshooting techniques. You will be able to use the troubleshooting tools to monitor performance, and troubleshoot issues related to Hosts and Virtual Machines. Moving on, you will troubleshoot High Availability, storage I/O control problems, virtual LANS, and iSCSI, NFS, VMFS issues. By the end of this book, you will be able to analyze and solve advanced issues related to vShpere environment such as vcenter certificates, database problems, and different failed state errors.
Publication date:
November 2015
Publisher
Packt
Pages
270
ISBN
9781783551767

 

Chapter 1. The Methodology of Problem Solving

This chapter covers a basic overview of troubleshooting skills, a complete set of troubleshooting tools for vSphere infrastructure, and tips and techniques on how these tools can be used to troubleshoot your vSphere infrastructure.

The topics covered in this chapter are as follows:

  • Troubleshooting techniques

  • Installing and configuring vMA

  • Configuring a centralized syslog server

  • Utilizing PowerCLI

  • A comprehensive reference of log files

  • Collecting logs

  • Understanding the health of vSphere hosts

 

Troubleshooting techniques


We all fix things in our daily lives, and all it takes to fix these things are troubleshooting skills. As with all skills, whether it's playing the piano, fixing a broken car, acting, or writing a computer program, some people are gifted with these skills for troubleshooting by nature. If you have a natural skill, you might assume that everyone else is also gifted. You may have learned how to ride a bike effortlessly, without knowing how much work other people may have had to put into it.

In the same way, some people have a natural talent for troubleshooting and are better at it than others. Such people quickly grasp the necessary steps and easily isolate the problem until they are able to find the root cause. Let's say your motorbike stops working and you take it to a mechanic, telling him the problems and the symptoms of your motorbike. A mechanic who is good at troubleshooting could be able to isolate the problem right away. He could also be able to explain you why your motorbike fails and what is the root cause of the problem. On the contrary, when you take your motorbike to a mechanic who isn't good in troubleshooting, you can expect more time to fix the motorbike and a higher repair bill. You may also need to go every now and then to see the mechanic to get your motorbike fixed at the earliest.

But this does not mean that if you don't have troubleshooting skills, you cannot learn them. Troubleshooting skills can be learned and mastered by anyone. For example, like many other skills, we apply certain techniques in troubleshooting as well—it does not matter whether we are gifted with this skill or not. When we start practicing, it becomes our second nature. We all want to be better troubleshooters, but we also need to be precise and fast. A good system engineer is gifted with troubleshooting skills. When we work in highly available environments where downtime is measured in dollars, we always want to have the right troubleshooting skill set to solve the problem. This requires precision, speed, comprehension, and troubleshooting skills.

Of course, it makes sense that you would prefer to go to the good mechanic who knows what it takes to fix your motorbike efficiently. Applying these scenarios will not only help you to troubleshoot in all aspects of life but also to troubleshoot vSphere in terms of identifying problems and their root causes, and understanding how to fix them.

You should consider a structured approach to troubleshooting rather than doing so without applying any methodology. The following aspects can be helpful and can teach you how to best practice troubleshooting, taking the motorbike to be repaired as an example:

Root Cause of Problem

Troubleshooting Skills Required

In the Engine

Action Needed

Not working at all

Easy

Dead battery

Problem understanding

Malfunctioning

Medium

Dashboard blinking light

Problem understanding + investigation

Malfunctioning, but the symptoms are seen in other components

Hard

Loss of power

Problem understanding + real-time investigation + correlation of events

Not working, but the problem disappeared

Requires on long analysis

Weak battery or some mechanical problems

Problem understanding + historical investigation + correlation of events

Precise communication

You should always establish good communication methods within your work environment. Communicating your problem effectively is one of the key skills required essentially for troubleshooting, especially when you are working in a collaborative environment. Lack of communication can lead to some serious and never-ending problems with increasing down-time. You might be working continuously without realizing that your other team members are working on the same problem as your are. If you've precise communication, you will always avoid the path that your other team members have already discovered.

The following communication methods can be effectively used to communicate within and outside of teams:

  • Direct conversation: You can communicate the problem directly, in person, with your team members

  • Voice/Video chats: Voice and video chats are very common now a days and enable a geographically distributed team to conduct regular meetings

  • Web sessions: Web sessions can be used to access remote systems, conducting presentations and sharing whiteboards

  • Email/Text chat: Email is the most common tool to used now a days for all kind of office communication

Creating a knowledge base of identified problems and solutions

While working on any system, you will face many common problems again and again. You should always create a knowledge base of these common problems, which includes identifying the problem, its symptoms, and the solution to be applied, along with a Root Cause Analysis (RCA) of the problem. Documenting and creating a knowledge repository of these problems and steps taken to troubleshoot them will save you a lot of work in the future. This will also help you to share the knowledge of troubleshooting with all your team members at one place. In addition, it will help you transfer knowledge to your newly hired team members and allow them to use a smarter and more methodological approach towards troubleshooting.

You might be able to fix the issue with no understanding of the root cause, but you cannot completely prevent it. You should always isolate and find the correct root cause in order to avoid problems in the future. If you know the root cause, you can easily assign the problematic issue to the correct team to resolve it accordingly. Sometimes you can come across very complex problems, where you may find the root cause, but sometimes that changes several times in the procedure. Highly available environments also have high stress and require your full concentration, excellent troubleshooting skills, and the correct domain knowledge. This becomes more crucial when it costs your organization money at every single second.

Obtaining the required knowledge of the problem space

For highly available environments, where every second of down time can cost you dollars, you would always have the right people in the right place in order to make sure your investment has been made at the right place. The value you will get by having the right people for the right job would save you not only in terms of Return on Investment (ROI) but also in terms of your reputation. If the required knowledge is missing, you should conduct training: first educate yourself and then transfer the knowledge to your team members. A technical team equipped with the knowledge of the problem space is highly desirable at all times.

Isolating the problem space

Whenever you face a critical problem, you should always try to divide the problem into smaller issues and try to divide it among your team members. If your team has only one member, you can still divide the problem into smaller ones. This approach does not only enable you to solve the problem quickly but also engages your team members to concentrate on different areas of the problem. Obviously, you should avoid working on the same problem that your other team members are working on. Thus, you should always make sure you have divided the problem space appropriately.

Documenting and keeping track of changes

You should always encourage your team members to log all their problems, their solutions, and the steps that were taken to reach to the solutions. You could centralize such information using a Knowledgebase or a local Wiki within your organization. Once you have your Knowledgebase in place with records of problems and their troubleshooting solutions, you can start testing the solutions. This will assure you that the solutions in your knowledge base are robust and well tested. You can use some kind of document version control so that as the problem evolves, your documentation can keep track of all of these changes.

When you are working in a data center, where you need to work together with other members of a team, this documentation process enables the entire team to solve the problem more easily. If you document the solutions in your organization, you truly enable your junior team members to learn new things and solve problems without involving senior team members.

Troubleshooting with power tools

In VMware vSphere troubleshooting, we will discuss and troubleshoot problems with different vSphere hosts, virtual machines, and vCenter Server. In simple walkthroughs, we will identify the problems and fix those problems by applying our knowledge. You will see how to isolate vSphere-related technical issues and how to apply troubleshooting techniques to those issues. We will discuss different VMware power tools to mange a vSphere infrastructure in centralized way, which includes VMware vSphere Management Assistant (vMA), EXCLI, vSphere PowerCLI, ESXTOP, resxotop, performance monitoring charts, and many other tools. These tools will be introduced step by step in the upcoming chapters.

 

Configuring the vSphere management agent


VMware vMA is a SUSE Linux-based virtual appliance that is shipped with vSphere SDK for Perl and vSphere command line interface. You can use vMA to manage your entire vSphere infrastructure from a central service console by executing different service scripts, creating and analyzing log bundles, monitoring performance, and much more. You can also use vSphere VMA to act as a centralized log server to receive logs from all of your vSphere hosts. Let's look at the various configuration parameters of our first VMware power tool, vSphere VMA.

Installation

VMware vMA requires a minimum of 3 GB of disk space and 600 MB of RAM. The Open Virtual Machine Format (OVF) template is based on SUSE Linux 64-bit architecture. vMA supports vSphere 4.0 Update 2 to vSphere 6.0 and vCenter 5.0 and upward. vMA can be used to target vCenter 5.0 or later, ESX/ESXi3.5 Update 5, and vSphere ESXi 4.0 Update 2 or later systems. A single vMA appliance can support a different number of targets, depending on how it is being used at runtime. You will require a user name and password to download the vMA application. It can be downloaded from https://my.vmware.com/group/vmware/details?productId=352&downloadGroup=VMA550.

We will deploy the new vMA from the vSphere Client tied to a vCenter Server 5.0 or vCenter Server 4.x. It can be deployed on the following vSphere releases:

  • vCenter Server 5.0

  • vCenter Server 6.0

The virtualized hosts that can be managed from the vMA are:

  • ESXi 3.5 Update 5

  • ESXi 4.0 Update 2

  • vSphere ESXi 4.1 and 4.1 Update 1

  • vSphere ESXi 5.0

  • vSphere ESXi 6.0

Installation steps

To install VMware vMA, perform the following steps:

  1. Once you are done with downloading the appliance, extract the vMA zip file into a directory.

  2. Log in to your vCenter or vSphere Client. From your vCenter client, you can select any vSphere host to which you would like to deploy vMA.

  3. To start the OVF appliance deployment wizard, choose the option Deploy OVF Template from the file menu.

  4. Select the Deploy from a file or URL option.

  5. Then, browse the folder where you have already extracted your vMA appliance. Click on the vMA OVF template to select it.

  6. Next, accept the vMA license agreement.

  7. Give an FQDN to your vMA appliance; I have given mine as vma.linxsol.com. The default name is also acceptable.

  8. Choose the appropriate folder to store your appliance for inventory.

  9. From your vCenter Server, choose the resource pool to allocate resources for the vMA appliance. If you do not select any resource pool, the wizard will place your appliance in the highest level of resource pool, which is selected by default.

  10. Choose the storage where you would like to store your vMA appliance; it could be a local data store, iSCSI, FC SAN, or NFS data store.

  11. Next, choose Disk Format options. I usually choose Thin.

  12. For the network, you can configure DHCP for your vMA appliance to obtain a dynamic IP address or you can configure the IP address manually. Make sure that your vMA appliance is part of the management subnet in order to access your vSphere hosts and the vCenter Server.

  13. By clicking Next, you will be asked to review the information. Once you find that the information is correct, you may proceed to click on Finish.

  14. The wizard will take a while and will then deploy the vMA appliance to one of the vSphere hosts.

Tip

In case you don't remember the vi-admin password, it is possible to reset it from the GRUB boot loader screen. Choose the very first option SUSE Linux Enterprise Server 11 SP1 for VMware and press e on your keyboard to edit the line. Move down your cursor to choose the line starting with "kernel /vmlinuz.." and press e again to edit it. Add init=/bin/bash to the end of the line. Then, press Enter and press b to boot the kernel. The vMA will boot into a bash shell prompt. Next, type the following command to reset the vi-admin password:

# passwd vi-admin
 

VMware vMA features


The vMA is an appliance based on SUSE Linux. It is designed to consolidate vSphere administrative tasks. Here is a brief introduction of vMA features:

  • vSphere SDK and CLI: You can use CLI to add vCenter Server and vSphere hosts as vMA targets to perform different kinds of operations by running scripts and programs. Adding a target can authenticate you, so you do not require to authenticate against vCenter Server or vSphere hosts when you run an agent or a vSphere CLI command on any of the targets. Do not confuse CLI with PowerCLI, the vSphere PowerShell implementation.

  • Using vSphere SDK API: You can use the vSphere APIs shipped with vMA to program and to connect to vMA targets programmatically. VMware vMA provides the VmaTargetLib library that supports utilization of the API using Perl and Java. You can run agent code using vMA on different software modules and on different hardware supported by VMware ESX. At the time of writing this book, the code can be run only in the CLI of existing vSphere hosts. This agent code can be modified and can be utilized in the vMA appliance by calling the vSphere API.

  • Authentication with vi-admin and vi-user: The vMA appliance can run agents or scripts that otherwise interact with vCenter and ESX(i) servers without repeated authentication. You can reutilize vMA service console scripts that are presently used for the vSphere host's administration. However, slight changes to the scripts are usually required. vMA comes with two users by default, named vi-admin and vi-user. To perform all the administrative tasks, for example, addition or removal of hosts, running vSphere CLI commands, agents on the added targets, you will require the user vi-admin. To run vSphere CLI commands and agents with read-only privileges on the added targets, you will use vi-user.

  • Active directory single-sign-on: vMA is also capable of joining the MS Active Directory (AD) domain, and you can use AD user to log in to vMA. This allows you to assign consistent and fine granular privileges to users on the vCenter Server system or the vSphere host, thus enabling users to run the commands accordingly.

  • Vi-logger: The vMA can collect logs from each of these server types for analysis. This is through a component on the vMA called vi-logger.

The vMA consists of the following components:

  • SUSE Linux Enterprise Server 11 SP1 64-bit: vMA has recently moved to SUSE Linux. Previous versions of vMA were all built on top of Red Hat—either Red Hat Enterprise Linux or CentOS—but with the release of vSphere 5, all virtual appliances have been migrated to SUSE Linux Enterprise Server 11.

  • VMware Tools

  • vSphere CLI

  • vSphere SDK for Perl

  • Java JRE 1.6

  • vi-fastpass: This refers to the authentication component of vMA.

Powering-on vMA

You must configure vMA when it boots for the very first time. To do so, power on the virtual appliance. Right-click on the vMA appliance and click on Open Console.

You will be presented with a screen prompting for network configuration. To configure network options, you must answer the prompts.

You can also specify the host name for your vMA appliance using one of the prompts. vMA allows you to have 64 alphanumeric characters in the host name.

You must configure a password for vi-admin. Answering the password prompt, you must enter your old password first; it will then prompt you to type in a new password. The new password must be able to comply with the vMA password policy, that is, password should be at least eight characters long. It must contain one upper case character, one lower case character, one numeric character, and one symbol.

Note

The default password of vi-admin for VMware vMA is vmware.

AD integration

By default, a vMA appliance comes with PowerBroker Identity Services – Open Edition, formerly known as Likewise Open, to support Active Directory integration. PowerBroker Identity Services uses Pluggable Authentication Modules (PAM) and Name Service Switch (NSS). It supports Kerberos, NTLM, and SPNEGO authentication. You can type the following to join your vMA with an AD domain controller:

sudo domainjoin-cli join FQDN domain-admin-user
sudo domainjoin-cli join linxsol.com zeeshan

The preceding command uses the Likewise Open's domainjoin-cli script using the join flag, followed by the MS AD controller FQDN and the user name of the user who has administrative rights to join computers to AD. Once you enter this, it will prompt you for a password. Enter the password, and you will see a success message appearing in your console. You can also check the status of your server to see if it has integrated with a domain controller by running the following script:

./lw-get-status

The script can be found in the /opt/likewise/bin directory. You can also check the status of your vMA appliance and AD integration by typing the following command in the console:

sudo domainjoin-cli query

Either you can append the full path before running the script or you can go to the preceding directory and run the script. The likewise identity service ships up with a lot of different scripts to manage AD integration. You can remove vMA from the domain by running the following command in the console:

sudo domainjoin-cli leave

The vMA console displays a message stating whether vMA has left the AD domain.

The BeyondTrust website maintains excellent documentation and a community wiki about PowerBroker Identity Services. For more information, please visit http://www.beyondtrust.com/Resources/OpenSourceDocumentation/.

Tip

The vMA host name can be changed anytime. Changing the host name is similar changing it in a Linux host: you only need to modify the /etc/HOSTNAME and /etc/hosts files. You can also change it from the vMA console by typing the command 'sudo hostname new-name'.

AD unattended access

We will use the ktpass tool to configure the principal name of the vMA appliance for the service in Active Directory Domain Services (AD DS). The process will create a .keytab file containing the shared secret key of the service. The .keytab file that is generated by the ktpass tool is based on the Massachusetts Institute of Technology (MIT) implementation of the Kerberos authentication protocol. The Ktpass command-line tool authorizes Linux- or UNIX-based services that support Kerberos authentication to use the interoperability features provided by the Kerberos Key Distribution Center (KDC) service.

In the subsequent example, you will learn to create a Kerberos .keytab file called machine.keytab in your current working directory for the user Sample1. (You will merge this file with the Krb5.keytab file on a host computer that is not running the Windows operating system.) The Kerberos .keytab file will be created for all supported encryption types for the general principal type.

To generate a .keytab file for a vMA appliance, use the following steps to map the principal to the account and set the host principal password:

  1. Use the Active Directory Users and Computers snap-in to create a user account for a service on a computer that is not running the Windows operating system. For example, create an account with the name User1.

  2. Use Ktpass to set up an identity map for the user account by typing the following in the command prompt:

    ktpass /princ host/vma.linxsol.com@linxsol.com /mapuser User1 /pass MyPas$w0rd /out User1.keytab /crypto all /ptype KRB5_NT_PRINCIPAL /mapop set 
    

    Here, linxsol.com is the name of the domain and User1 is the user who has permissions for the vCenter administration. Now you have a file called User1.keytab file. Copy the file to /home/local/linxsol.com/User1. You can use WinSCP and log in as user linxsol.com\User1 to move the file.

  3. You can type the following in console to make sure that the user linxsol.com\User1 on vMA has the ownership of the User1.keytab file:

    ls -l /home/local/linxsol.com/User1/User1.keytab
    
    chown 'linxsol.com\User1' /home/local/linxsol.com/User1/User1.keytab
    

    You should mind the quotes around linxsol.com\User1 so that bash interprets them as a string.

  4. Create a cron job for the .keytab file so that it can renew the ticket every hour for User1@linxsol.com. On vMA, create a script in /etc/cron.hourly/kticket-renew with the following contents:

    #!/bin/sh
    
    su – 'linxsol.com\User1' -c '/usr/bin/kinit -k -t /home/local/linxsol.com/User1/User1.keytab User'
    
  5. You can also add the preceding script to a service in /etc/init.d to refresh the tickets when vMA is booted.

vMA web UI

The web UI allows you to manage the vMA appliance. It does not enable you to manage the vCenter and vSphere hosts from the web interface. You can access the web UI by pointing your browser to https://<vma_address_or_hostname:5480 and logging in as vi-admin. The web interface enables you to perform a system reboot or shutdown. You can check the status of the vMA appliance, set its time zone, and update it to the latest version. In previous versions, you were able to use vma-update, but now this functionality has been migrated to the web interface. You can also use the web interface to update the network address setting (IP address, HTTP Proxy) of the vMA appliance.

vi-user

The vMA appliance comes with a built-in user called vi-user. This user cannot be used until you reset its password. By default, it does not have any password. The vi-user user has read-only privileges on the target systems. It also exists on all the target systems by default, regardless of whether you enable it in your vMA appliance or not. You can log in to your vMA appliance by using vi-user, but you will only be able to run the commands on target systems that do not need administrative permissions. The vi-user user is limited to run commands only against the vSphere hosts that set up with vi-fastpass authentication (we will discuss fpauth and adauth later in the chapter). The vi-user user cannot be used to run commands against systems authenticated against AD. It is also unable to run any commands as sudo. You can change its password as you change it in Linux normally, by typing the following command in vMA console:

sudo passwd vi-user

Type the new password for vi-user and confirm it once prompted.

Configuring vMA as a syslog server

A centralized syslog server can save you a lot of troubleshooting efforts. For example, if a remote system crashes, you might lose all the important logs within that remote system, which could help you to troubleshoot issues with the remote system. If you log into a centralized logging system, it can provide you with the most recent logs of that remote system before the system crashed.

We will now walk through how to configure the vMA appliance as a syslog server to centralize logging for vSphere hosts. When vMA collects the logs from your vSphere host, sometimes the logs have the vSphere host timestamp, and sometimes they will have the vMA Localtime timestamp.vSphere host, which uses UTC as its time zone while time stamping the logs. You can avoid the issue of timestamp difference in the logs by changing the local time on the vMA to UTC, with the following command:

sudo rm /etc/localtime

sudo ln -s /usr/share/zoneinfo/UTC /etc/localtime

You can also set up NTP servers in your vMA appliance to sync your environment's time. To do so, run the following commands in the shell:

sudo zypper in ntp yast2-ntp-client  #It will install ntp client

sudo vi /etc/ntp.conf 

Add in your NTP servers under the heading # Use public servers from the pool.ntp.org project. Configure ntpd to start on reboot:

sudo /sbin/chkconfig ntpd on

Now you can start the ntpd service:

sudo /sbin/service ntpd restart

Make sure your NTP servers are reachable:

sudo ntpq -p

Log files usually grow large in size; you should always consider well your disk space requirements. It is always recommended that you place all of your logs on a separate disk drive.

You should add an additional disk to the vMA appliance where the logs will be stored. If your VMware infrastructure consists of a large number of servers, you should allocate a big enough disk for that.

After hot-adding the disk to the VM, rescan the SCSI bus of the OS in the usual GNU/Linux way to see the disk. You need to become root to perform this action; otherwise you will get Permission Denied error:

vma:/home/vi-admin # echo "- - -" > /sys/class/scsi_host/host0/scan
 vma:/home/vi-admin # fdisk /dev/sdb
 Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
 Building a new DOS disklabel with disk identifier 0x03e0767d.
 Changes will remain in memory only, until you decide to write them.
 After that, of course, the previous content won't be recoverable.
 
 Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)
 
 Command (m for help): n
 Command action
    e   extended
    p   primary partition (1-4)
 p
 Partition number (1-4, default 1): 1
 First sector (2048-209715199, default 2048):
 Using default value 2048
 Last sector, +sectors or +size{K,M,G} (2048-209715199, default 209715199):
 Using default value 209715199
 
 Command (m for help): p
 
 Disk /dev/sdb: 107.4 GB, 107374182400 bytes
 255 heads, 63 sectors/track, 13054 cylinders, total 209715200 sectors
 Units = sectors of 1 * 512 = 512 bytes
 Sector size (logical/physical): 512 bytes / 512 bytes
 I/O size (minimum/optimal): 512 bytes / 512 bytes
 Disk identifier: 0x03e0767d
 
    Device Boot      Start         End      Blocks   Id  System
 /dev/sdb1            2048   209715199   104856576   83  Linux
 
 Command (m for help): w
 The partition table has been altered!
 
 Calling ioctl() to re-read partition table.
 Syncing disks.
 vma:/home/vi-admin # mkfs.
 mkfs.bfs     mkfs.cramfs  mkfs.ext2    mkfs.ext3    mkfs.ext4    mkfs.minix
 vma:/home/vi-admin # mkfs.ext4 /dev/sdb1
 mke2fs 1.41.9 (22-Aug-2009)
 Filesystem label=
 OS type: Linux
 Block size=4096 (log=2)
 Fragment size=4096 (log=2)
 6553600 inodes, 26214144 blocks
 1310707 blocks (5.00%) reserved for the super user
 First data block=0
 Maximum filesystem blocks=4294967296
 800 block groups
 32768 blocks per group, 32768 fragments per group
 8192 inodes per group
 Superblock backups stored on blocks:
         32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
         4096000, 7962624, 11239424, 20480000, 23887872
 
 Writing inode tables: done
 Creating journal (32768 blocks): done
 Writing superblocks and filesystem accounting information: done
 
 This filesystem will be automatically checked every 28 mounts or
 180 days, whichever comes first.  Use tune2fs -c or -i to override.

Create a directory where the logs will be stored. I used /var/log/esxi-syslog/ and mounted the fresh partition to this location: vma:/home/vi-admin # mkdir /var/log/esxi-syslogvma:/home/vi-admin # mount /dev/sdb1 /var/log/esxi-syslog. To automatically remount the partition at boot time, add the following line to /etc/fstab: vma:/home/vi-admin # echo "/dev/sdb1 /var/log/esxi-syslog ext4 acl,user_xattr,noatime 1 1" >> /etc/fstab.

If you do not configure the preceding line in your fstab file, you will always need to mount your drive manually after rebooting your host.

The vMA syslog configuration file can be found at /etc/syslog-ng/syslog-ng.conf. The default options usually satisfy most of the requirements for logging. The file is self-explanatory, and you can read it to tune your log server configuration.

sudo service syslog restart

Creating a logrotate file

It is a common practice to rotate logs on Linux systems. You can create a logrotate file to compress and rotate the log files. Use vi or a text editor of your choice to create a file in /etc/logrotate.d to rotate and compress logs from /var/log/esxi.

vma:/home/vi-admin# vi /etc/logrotate.d/esxi-log.conf
var/log/esxi/*.log {         weekly         missingok         rotate 6         compress                 delaycompress         notifempty         nocreate         sharedscripts         postrotate                 /etc/init.d/syslog reload         endscript }

The configuration file is self-explanatory: the first line tells logrotate utility to rotate the log weekly. The second line tells it to keep the six log files. The compression will be for all files except the last rotated log.

The vMA authentication mechanism

The policy found in vMA is a credentials caching mechanism that allows us to connect to ESX(i) or vCenter servers. The mechanisms are of the following two types:

  • Fast Pass Authentication (fpauth)

  • Active Directory Authentication (adauth)

The fpauth essentially allows us to manage a vSphere host or vCenter Server under vMA by using a vi-admin and vi-user account. The vMA appliance uses a XOR cipher to obfuscate the passwords of both accounts. Once you are authenticated against target vSphere hosts or vCenter, you can start managing targets and execute either vCLI or vSphere SDK for Perl scripts without specifying credentials every time. That is why it becomes much easier to run a single command against a large number of hosts. In previous releases of vMA, adauth was used. You can authenticate against vSphere hosts and vCenter using your Windows AD credentials. I have previously described how we can join vMA with an AD attended or unattended. It requires your vSphere target hosts and vCenter to be already members of AD domain.

As we have already configured it, we only need to add target hosts to our vMA appliance to execute vCLI cmdlets or the perl scripts provided by vSphere SDK. These credentials remain saved within vMA until you log out or reboot your vMA appliance. Using adauth is much more secure than using fpauth and I would recommend you to use adauth whenever it is possible for you.

We are ready to start adding hosts in our newly installed vMA appliance. I will describe instructions for setting up and verifying both standard fpauth and adauth.

Accessing systems from vMA

Let's add our first vSphere host to our vMA appliance. We will use adauth to add a vCenter Server system as a vMA target.

Log in to vMA as vi‑admin. Add a server as a vMA target by running the following command:

vifp addserver crimv1vcs001.linxsol.com --authpolicy adauth --username linxsol\\zeeshan

The command is self-explanatory here. The addserver directive requires the vSphere host or vCenter name to be added, and the authpolicy directive requires the authentication mechanism to be used. In this case, I have used AD Authentication. I will show you later how to use fastpass to add target servers. If you do not use authpolicy directive in the command, vMA appliance uses the fastpass authentication by default. The username directive is optional that takes an authorized AD user name to authenticate. If you do not specify it, vMA will prompt you for the AD authorized user name for the vCenter.

We can verify whether the vCenter system has been correctly added to the vMA appliance by running the following command:

vifp listservers --long
crimv1vcs001.linxsol.com                vCenter     adauth

Let's add one of the vSphere hosts using fpauth to the vMA appliance:

vifp addserver crimv3esxi002.linxsol.com --authpolicy fpauth

It will prompt you for the root user password of that vSphere host. Let's verify our hosts in the vMA appliance:

vifp listservers --long
crimv1vcs001.linxsol.com              vCenter   adauth
crimv1esxi001.linxsol.com             ESXi       adauth
crimv1esx002.linxsol.com              ESXi       fpauth
crimv1esx003.linxsol.com              ESXi       adauth

Before we can run the vCLI commands against any of the hosts, we need to set it up as a target. Run the following command to set the target server (vifptarget --set | -s <server>):

vifptarget --set crimv1esx001.linxsol.com

Verify that you can run a vSphere CLI command without authentication by running a command on one of the vSphere hosts. The following command will not ask you for the credentials; instead, it will use the authentication mechanism to verify against the AD Domain:

esxcli --server <VC_server> --vihost <esx_host> network nic list

You can easily remove the target by using the following command:

vifp removeserver crimv1esx001.linxsol.com

Now you are ready to use the vMA appliance to manage VMware infrastructure. I will cover more practical examples in the coming chapters.

vMA scripts samples

You can find different scripts written in Perl and Java in your vMA appliance. These scripts are easy-to-use examples that show you how you can modify VmaTarget.login() method according to your target host. You can find these scripts in /opt/vmware/vma/samples. For code samples in Java, you can browse /opt/vmware/vma/samples/java, and for Perl, you can browse /opt/vmware/vma/samples/perl. In the perl directory, you can find three scripts and a README file: bulkAddServers.pl, listTargets.pl, and mcli.pl. The README file contains all the information you are required to run these scripts. I will walk you through a brief description of what these scripts can do for you. The bulkAddServers.pl can add multiple vSphere hosts to your vMA appliance in bulk. The script can read the host names from a text file provided by you or you can pass the vCenter Server host name in the arguments when executing the script. The listTagets.pl script can collect different information about your targets, for example, versioning. The last script mcli.pl can be used to run a single command on different vMA target hosts. You can provide a text file or pass the host name to the script in an argument.

 

PowerCLI


VMware vSphere PowerCLI is a powerful CLI that you can use to perform almost all of your daily administration tasks quickly. A basic reference has been provided in the Appendix C, Power CLI - A Basic Reference, section of the book to set it up and run the basic command. It can be used to set up a syslog server; it can also be used to download a vc-support or vm-support log bundle from VMware vSphere vCenter Server and/or ESX/VSphere hosts.

Connecting to vCenter Server or an ESX/vSphere host with PowerCLI

To run specific vSphere PowerCLI cmdlets and perform administration or monitoring tasks, you must connect to vCenter Server or a VSphere host, and then follow these steps:

  1. Launch vSphere PowerCLI.

  2. In the vSphere PowerCLI console window, establish a connection to a VSphere host or a vCenter Server using the following command:

    Connect-VIServer -Server crimv1vcs001.linxsol.com
    
  3. The output appears similar to as follows:

    Name                    Port     User
    ----                    ----     ----
    crimv1vcs001.linxsol.com          443      linxsol\zeeshan
    

Note

If the certificate is not trusted, a warning display appears. Depending on your security policy, these warnings can be ignored. Once it is done, it will ask you for a user name and password.

Setting up a syslog server using PowerCLI

We will set up a central syslog for our vSphere hosts using the PowerCLI:

Set-VMhostSyslogServer –SysLogServer 'vma.linxsol.com:514' –VMHost crimv3esxi001.linxsol.com

You can also remove the SysLogServer function by typing the following command:

Set-VMhostSyslogServer –SysLogServer $null –VMHost crimv3esxi001.linxsol.com

CMDLETS reference: https://www.vmware.com/support/developer/PowerCLI/PowerCLI41U1/html/Set-VMHostSysLogServer.html.

Setting up a sysLog server manually

Let's configure our vSphere host manually to use a syslog server as part of a post-installation script. You can run the following command in the console:

vim-cmd hostsvc/advopt/update Syslog.Remote.Hostname string vma.linxsol.com

You can also set this in the vSphere Client by clicking on a vSphere host and then navigate to Configuration | Advanced Settings. Here, expands syslog in the tree and enter the syslog server details in the Remote field.

Tip

vSphere host Firewall Exception for Syslog Ports

Note

You may need to manually open the Firewall rule set for syslog when redirecting logs. It seems that for UDP traffic, this firewall rule has no effect in vSphere host5.0 build 456551, and the UDP port 514 traffic flows regardless.

To open outbound traffic via the vSphere host Firewall on UDP port 514, TCP port 514 and 1514, use these commands:

esxcli network firewall ruleset set --ruleset-id=syslog --enabled=true

esxcli network firewall refresh
 

A comprehensive reference of log files


You should always configure your log files properly. Log files are the best way to get invaluable information in the detection of a problem and in troubleshooting issues. In the upcoming sections, you will find a comprehensive reference about your VMware vCenter and vSphere host infrastructure components.

vSphere log files – vSphere host 5.1 and later

When you troubleshoot different issues within the virtual environment, vSphere log files become the most important troubleshooting tool one can have. Not only can you use the log files to find and fix the problems, but you can also use them to avoid these problems from occurring in the future.

Logs for vSphere host 5.1 or later are grouped according to the source component (for more detailed information, please visit the VMware Knowledgebase):

vsphere Log files

Description

/var/log/auth.log

vSphere host shell authentication success and failure

/var/log/esxupdate.log

vSphere host patch and update installation logs

/var/log/hostd-probe.log

Host management service responsiveness checker

/var/log/usb.log:

USB device arbitration events, such as discovery and pass-through to virtual machines

/var/log/vmkernel.log

Core VMkernel logs, including device discovery, storage and networking device and driver events, and virtual machine startup

/var/log/lacp.log

Link aggregation control protocol logs

/var/log/hostd.log

Host management service logs, including virtual machine and host task and events, communication with the vSphere Client and vCenter Server vpxa agent, and SDK connections

/var/log/Xorg.log

Video acceleration

/var/log/dhclient.log

DHCP client service, including discovery, address lease requests, and renewals.

/var/log/rhttpproxy.log

HTTP connections proxied on behalf of other VSphere host webservices

/var/log/shell.log

vSphere host shell usage logs, including enable/disable and every command entered

/var/log/sysboot.log

Early VMkernel startup and module loading

/var/log/boot.gz

A compressed file that contains boot log information and can be read using zcat /var/log/boot.gz|more

/var/log/syslog.log

Management service initialization, watchdogs, scheduled tasks, and DCUI use

/var/log/vobd.log

VMkernel observation events, similar to vob.component.event

/var/log/vmkwarning.log

A summary of warning and alert log messages excerpted from the VMkernel logs

/var/log/vmksummary.log

A summary of VSphere host startup and shutdown, and an hourly heartbeat with uptime, the number of virtual machines running, and service resource consumption

Note

For information about sending logs to another location (such as a datastore or remote syslog server), see Configuring syslog on ESXi 5.0 (2003322).

Logs from vCenter Server components on vSphere host 5.1, 5.5, and 6.0

When vSphere host 5.1/5.5/6.0 is managed by vCenter Server 5.1, 5.5, and 6.0, two components are installed, each with their own logs, as described in the following table:

vsphere related vcenter Log Files

Description

/var/log/vpxa.log

vCenter Server vpxa agent logs, including communication with vCenter Server and the Host Management hostd agent

/var/log/fdm.log

vSphere host high availability logs, produced by the fdm service

vCenter log files

The vCenter Server logs are placed in a different directory on disk depending on the vCenter Server version and the deployed platform. Sometimes, logs are pointed to store in a drive other than the system drive. As each log grows, it is rotated over a series of numbered component-nnn.log files. On some platforms, the rotated logs are compressed.

vcenter Log Files

Description

%ALLUSERSPROFILE%\Application Data\VMware\VMware VirtualCenter\Logs\

vCenter Server 5.x and earlier versions on Windows XP, 2000, 2003

C:\ProgramData\VMware\VMware VirtualCenter\Logs\

vCenter Server 5.x and earlier versions on Windows Vista, 7, 2008

/var/log/vmware/vpx/

vCenter Server Appliance 5.x

/var/log/vmware/vami

vCenter Server Appliance 5.x UI

C:\ProgramData\VMware\VMware VirtualCenter\Logs\

vpxd.log

Main vCenter Server logs, consisting of all vSphere Client and WebServices connections, internal tasks and events, and communication with the vCenter Server Agent (vpxa) on managed ESX/VSphere hosts

C:\ProgramData\VMware\VMware VirtualCenter\Logs\

vpxd-profiler.log

C:\ProgramData\VMware\VMware VirtualCenter\Logs\profiler.log

C:\ProgramData\VMware\VMware VirtualCenter\Logs\scoreboard.log

Profiled metrics for operations performed in vCenter Server; used by VPX Operational Dashboard (VOD) accessible at https://VCHostnameOrIPAddress/vod/index.html

C:\ProgramData\VMware\VMware VirtualCenter\Logs\ vpxd-alert.log

Non-fatal information logged about the vpxd process

C:\ProgramData\VMware\VMware VirtualCenter\Logs\cim-diag.log

C:\ProgramData\VMware\VMware VirtualCenter\Logs\vws.log

CIM monitoring information, including communication between vCenter Server and managed hosts' CIM interface

C:\ProgramData\VMware\VMware VirtualCenter\Logs\drmdump\cluster.xxx\proposeAction.dump.gz

Actions proposed and taken by VMware Distributed Resource Scheduler (DRS), grouped by the DRS-enabled cluster managed by vCenter Server—these logs are compressed and are inside the cluster folder named proposeAction.dump.gz

C:\ProgramData\VMware\VMware VirtualCenter\Logs\ls.log

Health reports for the Licensing Services extension and connectivity logs to vCenter Server

C:\ProgramData\VMware\VMware VirtualCenter\Logs\vimtool.log

Dump of string used during the installation of vCenter Server with hashed information for DNS, username and output for JDBC creation

C:\ProgramData\VMware\VMware VirtualCenter\Logs\stats.log

Information about the historical performance data collection from the ESXi/ESX hosts

C:\ProgramData\VMware\VMware VirtualCenter\Logs\sms.log

Health reports for the Storage Monitoring Service extension, connectivity logs to vCenter Server, the vCenter Server database, and the xDB for vCenter Inventory Service

C:\ProgramData\VMware\VMware VirtualCenter\Logs\eam.log

Health reports for the ESX Agent Monitor extension, connectivity logs to vCenter Server

C:\ProgramData\VMware\VMware VirtualCenter\Logs\catalina.<date>.log

C:\ProgramData\VMware\VMware VirtualCenter\Logs\localhost.<date>.log

The connectivity information and status of the VMware Web Management Services

C:\ProgramData\VMware\VMware VirtualCenter\Logs\jointool.log

The health status of the VMwareVCMSDS service and individual ADAM database objects, internal tasks and events, and replication logs between linked-mode vCenter Servers

C:\ProgramData\VMware\VMware VirtualCenter\Logs\Additional log files:

manager.<date>.log

C:\ProgramData\VMware\VMware VirtualCenter\Logs\host-manager.<date>.log

Additional log files for the VMware vCenter Server

Tip

If the service is running under a specific user, the logs may be located in the profile directory of that user instead of %ALLUSERSPROFILE%.

vCenter inventory service log files

The vCenter inventory service logs are placed in a different directory on a disk depending on the vCenter Server version and the deployed platform. For vCenter Server 5.x and earlier versions on Windows XP, 2000, 2003:

 %ALLUSERSPROFILE%\Application Data\VMware\Infrastructure\Inventory Service\Logs 

The default location for vCenter Server 5.x Linux Virtual Appliance is /var/log/vmware/vpx/inventoryservice.

Note

If the vCenter Server inventory service is running under a specific user, the logs may be located in the profile directory of that user instead of %ALLUSERSPROFILE%.

As each log grows, it is rotated over a series of numbered component-nnn.log files. On some platforms, the rotated logs are compressed.

To collect the vSphere 5.1 vCenter Server inventory service logs, navigate to Start | All Programs | VMware | Generate Inventory Service log bundle.

Following are the log file locations for vCenter Server 5.x and earlier versions on Windows Vista, 7, 2008:

vcenter Log Files

Description

C:\ProgramData\VMware\Infrastructure\Inventory Service\Logs\ds.log

The main vCenter Inventory Service logs, consisting of all vCenter Server and Single Sign-On connections, internal tasks and events, and information about the xDB

C:\ProgramData\VMware\Infrastructure\Inventory Service\Logs\vim-is-install.log

Information about the installation of Inventory Service including computer name, operating system revision, the date of installation, and the number of revisions that have been installed or upgraded on the system

C:\ProgramData\VMware\Infrastructure\Inventory Service\Logs\ wrapper.log

Information about the status of the Java runtime environment

vSphere Profile-Driven Storage log files

The vSphere Profile-Driven Storage logs are placed in a different directory on disk depending on the vCenter Server version and the deployed platform. vCenter Server 5.x and earlier versions on Windows XP, 2000, 2003:

%ALLUSERSPROFILE%\Application Data\VMware\Infrastructure\Profile-Driven Storage\Logs 

The default location for vCenter Server 5.x Linux Virtual Appliance is /var/log/vmware/vpx/sps.

Note

If the service is running under a specific user, the logs may be located in the profile directory of that user instead of %ALLUSERSPROFILE%.

vSphere Profile-Driven Storage logs are grouped by component and purpose. As each log grows, it is rotated over a series of numbered component-nnn.log files. On some platforms, the rotated logs are compressed.

You cannot view vSphere Profile-Driven Storage logs using vSphere Client or vSphere Web Client. To export these logs, see Collecting Diagnostic Information in the upcoming part of the chapter.

vCenter Server 5.x and earlier versions on Windows Vista, 7, 2008:

vcenter Log Files

Description

C:\ProgramData\VMware\Infrastructure\Profile-Driven Storage\Logs\ sps.log

The main Profile-Driven Storage logs, consisting of all vCenter Server and Management Webservices connections, internal tasks and events, and information about the storage profile integrity

C:\ProgramData\VMware\Infrastructure\Profile-Driven Storage\Logs\vim-sps-install.log

Information about the installation of Profile-Driven Storage including computer name, operating system revision, the date of installation, and the number of revisions that have been installed or upgraded on the system

C:\ProgramData\VMware\Infrastructure\Profile-Driven Storage\Logs\wrapper.log

Information about the state of the Java runtime environment

Configuring logs and collecting logs

There are different ways to collect logs from vCenter Server and vSphere hosts. You should configure logs using CLI or vMA, or you can use Host Profiles to configure syslog functionality for a cluster or for similar hosts. (We are not covering Setting up Host Profiles to enable logging). You can use the following ways to gather all the diagnostic information from your VMware infrastructure:

  • vSphere Client

  • vSphere Web Client

  • PowerCLI

  • vm-support

  • vm-support from vMA

Using vSphere Client

vSphere host 5.x diagnostic information can be gathered using the vSphere Client connected to the VSphere host or to vCenter Server. To gather diagnostic data using the VMware vSphere Client, follow these steps:

  1. Open the vSphere Client and connect to vCenter Server or directly to an vSphere host 5.x.

  2. Log in using an account with administrative privileges or with the Global.Diagnostics permission.

  3. Select a vSphere host, cluster, or datacenter in the inventory.

  4. Navigate to File | Export | Export System Logs.

  5. If a group of vSphere hosts is available in the selected context, select the host or group of hosts from the source list.

  6. Click Next.

  7. In the System Logs pane, select the components for which the diagnostic information must be obtained. To collect diagnostic information for all the components, click Select All.

  8. If required, select the Gather performance data option and specify a duration and interval.

  9. Specify the download location in the next screen to download the logs, and then click Next, and on the last screen click Finish to finish the wizard. On the last screen, you will be presented with a summary of logs you are going to download.

  10. Once the wizard completes collecting logs, it will store them in the location you have specified. The log bundle is named with the current date and time, for example, VMware-vCenter-support-yyyy-mm-dd@HH-MM-SS.zip.

Using vSphere Web Client

You can gather the diagnostic information of vSphere host 5.0 and higher version by using the vSphere Web Client. The following steps need to be performed to collect diagnostic data:

  1. Log in to the vSphere Web Client using your credentials.

  2. Click on the Hosts and Cluster.

  3. Select a vSphere host, cluster, or datacenter you want to collect the diagnostic information.

  4. Click on Actions.

  5. Choose All vCenter Actions and then click on the Export System Logs.

  6. Click Next.

  7. You can select the different components of diagnostic logs to be exported in the System Logs pane. You can choose Select All if you want to export all the logs.

  8. You can also select the Gather performance data option and specify a duration and interval.

  9. Then click on Generate Log Bundle and then Download Log Bundle.

Using the vm-support tool

VMware has provided a useful tool in order to collect diagnostics information and troubleshooting problems. It is a script-based tool that can also collect the state information of the virtual machines. You can run the vm-support in vSphere hosts or from the vMA appliance. Some of the logs that can be collected using vm-support are vmkernel, host, CIM, virtual machines, security, vpxa, cronjobs, dmesg, update logs, configuration information of the NICs, switches, storage adapters, NAS mounts, multi-path setup.

Running vm-support in a console session on vSphere hosts

Let's use vm-support tool to generate log bundle in VSphere hosts:

  1. Log in to the vSphere host using a Secure Shell Session (SSH).

  2. Type the following command and hit Enter:

    vm-support
    
  3. The vm-support command will generate a compressed bundle of logs and will save it in a file with a .tgz extension in one of the following locations:

    • /var/tmp/

    • /var/log/

    • The current working directory

Once this is done, you can download the logs.

Tip

To export the log bundle to a shared vmfs datastore, use this command: vm-support -w /vmfs/volumes/VMFS_DATASTORE_NAME.

Generating logs on stdout

You can also use vm-support to output logs on standard output stream. For example, you can get all the logs over an SSH connection without saving anything locally on the host.

  1. From a Linux machine, log in to the vSphere host and type the following command:

    ssh root@vSphereHostname Or IPAddress vm-support -s > logbundle.tgz
    

    The –s flag tells vm-support utility to stream the logs on STDOUT, following which you can specify the path and the file name you would like to save your logs in. The vm-support utility will generate the logs and store everything into the specified file.

    Note

    This requires entry of the password for the root account, and cannot be used with the lockdown mode.

  2. Let's do that to generate log bundle and store into a data store:

    ssh root@vSphereHostnameOrIPAddress 'vm-support -s > /vmfs/volumes/datastore001/logbundle.tgz'
    

Using vm-support in vMA to collect logs

VMware has provided a nicely written BASH script to collect vSphere host logs from the vMA appliance. You can download the script from the following link: http://goo.gl/e1R4Zu.

The script uses vi-logger that is now deprecated to collect the logs from vSphere hosts.

Follow these steps to collect the logs from vMA 4.0 or vMA 4.1:

  1. Once 1035911_vMA-vilogger-gatherer.txt is downloaded to your vMA appliance, rename it as log_collector.sh by typing the following command:

    mv 1035911_vMA-vilogger-gatherer.sh log_collector.sh
    
  2. Give it executable permission:

    chmod +x log_collector.sh
    
  3. Run this command to execute the script:

    /vMA-vilogger-gatherer.sh
    

    An archive of log files is created within the current folder.

We can also collect logs from vMA without using the preceding script:

  1. Create a directory to store the logs:

    DIRNAME=log_collector 
    
  2. Now manually copy the log files by running the cp command:

    cp -r /var/log/vmware/* log_collector/
    
  3. You can compress the log files by running the following command:

    tar cvzf log_collector_bundle.tgz log_collector
    

For the 4.x version, please visit http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1024122to configure vMA to collect logs.

Using PowerCLI to collect the log bundle

PowerCLI is a very powerful and easy-to-use command-line tool to manage vSphere infrastructure. It lets you have complete control by automating, monitoring, and managing your vSphere infrastructure.

Collecting log bundles from vCenter Server

We will now use PowerCLI to collect diagnostic log bundles from vCenter Server. It is very simple and a one-line command to execute. Run the following in your PowerCLI shell:

Get-Log -Bundle -DestinationPath D:\vCenter_logs\logs

The output appears similar to the following:

Data
----
D:\vCenter_logs\logs\vc-support-nnnnnnnn-nnnn-nnnn-nnnn-nnnnnnnnnnnn.tgz

Collecting log bundles from a vSphere host

Let's download a vm-support diagnostic log bundle from a vSphere host managed by vCenter Server. Enter the following command:

Get-VMHost HostNameOrIP | Get-Log -Bundle -DestinationPath D:\vCenter_logs\logs

The output appears similar to the following:

Data
----
D:\vCenter_logs\logs\vm-support-nnnnnnnn-nnnn-nnnn-nnnn-nnnnnnnnnnnn.tgz

Collecting log bundles from the vSphere log browser

You can access the log browser directly from the VMware vSphere 5.1 web. The client.Log browser is a plugin that works directly with vSphere 5.1. The log browser can be very beneficial and easy-to-use tool when you are troubleshooting problems. Following are some of the benefits of log browser:

  • Log comparison

  • Searching logs

  • Finding and highlighting keywords

The log browser offers a friendly GUI that you can use to take a snapshot of specified host/vCenter logs. Once you retrieve the logs, you can search, compare, highlight, and categorize the logs based on some specific keywords. You can refresh the snapshots of retrieved and stored logs. The search term then can be highlighted for fast navigation. The admin gets a fast UI with the possibility to have the searched word to appear in color. Once you are done, you can export the logs to a file or as a VMware log bundle.

In the troubleshooting lifecycle of VMware infrastructure, log browser is a very handy and useful tool that you can directly access from the vSphere Client. It installs when you install vSphere, and you do not need to look for any other methods to collect the logs. The only problem is that the log browser is only available in vSphere 5.1.

Let's have a quick walkthrough of the log browser:

  1. Log in to your vSphere 5.1 web client using administrator credentials.

  2. On the left pane, click on the Log Browser option and in the View pane on your right, click on the Select Object option. This will open up a new window from where you can either choose vSphere hosts or any vCenter in your environment.

  3. Once the host is selected, you can choose the log file you would like to view from the Type dropdown menu.

  4. From here, you can browse different log files from different objects.

  5. You can also click on Refresh for the latest logs.

Exporting logs

You can export log files using the log browser by performing the following steps:

  1. Navigate to the log browser and select an object to browse.

  2. Select Action and then click on Export.

  3. Select the type of file that you would like to export.

  4. Once it is done, click on Export. When a new browser window appears, browse a location to save the log bundle.

 

Understanding the hardware health of vSphere hosts


VMware vSphere hosts use the Common Information Model (CIM) instead of installing the hardware agents in the vSphere host Service console. CIM provides some management functions for reporting health information or driver updates. For different installed hardware in the server, VMware provides you with the different CIM providers, for example, HBA, Network cards, Raid Controllers and more. The CIM Broker is used to receive monitoring status from the CIM providers. Once the CIM Broker collects all the status information, it becomes ready to publish all the information, which then can be accessed by APIs.

VMware vCenter Server is capable of presenting this information through the Hardware Status tab, where it provides all the hardware information:

  1. From the vSphere Client, click on Inventory and choose Hosts and Clusters.

  2. Choose a vSphere host you would like to monitor the Hardware Status tab and click on it.

  3. Browse to the Hardware Status tab; it will present you with all the information about memory, CPU, and temperature. You can choose to view Sensors, Alerts and warnings, and System event log from the dropdown menu.

  4. You can export all of this information by clicking on Export on the right top corner of the tab.

The host health monitoring tool enables you to monitor the health of many hardware components, including memory, temperature, CPUs, voltage, power, network, battery, software, watchdog, storage, and so on.

 

Miscellaneous tools


I will cover some of the very important tools for troubleshooting vSphere infrastructure components in the coming chapters, where we will also go through some other general troubleshooting tools, for example, ESXTOP, resxtop, power GUI, memory dump collector, network dump collector, and so on.

 

Summary


In this chapter, we went through some very basic vSphere troubleshooting techniques that everyone has and how they can keep improving them. We saw some very great tools that let us troubleshoot and solve problems easily. We walked through a step-by-step installation of vMA appliances and how we could utilize it as a syslog server. We also saw how vMA (VMware Management Assistant) could be used to run vSphere API calls and to run perl scripts. Then, we used the PowerCLI to collect logs and configure syslog server configuration in the vSphere hosts. The chapter also covered a comprehensive reference of vSphere infrastructure log files and their locations.

In the following chapter, you will learn how to monitor and troubleshoot vSphere hosts and how to fine-tune your virtual machine performance, and you will obtain a basic understanding of key performance metrics of vSphere.

Latest Reviews (2 reviews total)
Very useful book whether you read through or just use for reference.
VMware vSphere Troubleshooting
Unlock this book and the full library FREE for 7 days
Start now