Troubleshooting Citrix XenApp??

As with many software nowadays, XenApp requires minimal configuration and installation decisions, and an experienced administrator can configure the infrastructure in a matter of hours.

Particularly because the installation is a simple process, it is the troubleshooting that sometimes becomes difficult.

It is important to note that a solid grasp of XenApp components, interaction, and workflow is needed before performing troubleshooting.

Most times troubleshooting can be easy, either the solution is straightforward, perhaps because the administrator has experienced this problem in the past, or a simple Internet search for the particular error message will yield a Citrix knowledge-based article or blog post for that particular problem.

In all other cases, troubleshooting needs to be performed in an organized fashion so the solution is reached in the shortest amount of time possible, since many times the problem could involve downtime for a large number of users.

Although seemingly unimportant, one of the most important aspects of troubleshooting is producing a comprehensible problem statement:

Without clear answers to these questions, an ambiguous problem can undermine efforts for a solution.

Consider the fact that most of the time an issue is generally logged by a service desk or call center (first line of support), who might escalate it to a desktop support team (second line of support), and who will in turn escalate it to a Citrix team (third line of support).

If any piece of information is misunderstood by the analyst logging the incident, this in turn can be propagated to the Citrix team with the information being completely irrelevant to the troubleshooting process or even incorrect.

Consider the following scenario: a user working in the finance department calls the helpdesk and complains that an accounting application stopped working in Citrix. The application was working fine last week. The help desk agent performs a series of basic troubleshooting steps and escalates the problem to the next line of support without requesting additional information.

Consider the following questions:

While each question in itself might not directly lead to a solution, it can narrow down the problem considerably.

For instance, a positive answer to the first question might indicate this is a server or network issue as it affects multiple users.

A positive answer to the third question might indicate this is a network error; the next logical step would be to check whether there are any networking restrictions applied to subnets or IP addresses in the current location.

The fifth question is meant to check whether the issue is specific to a user, machine, or location.

When troubleshooting difficult cases, after making sure you have understood the problem (and that the information provided is correct and relevant), you must ensure a systematic approach to problem solving.

One strategy that can be used is divide and conquer, where you break down a problem into individual, easily solvable problems.

Considering the previous example where a user calls the helpdesk, one way of breaking down the problem is testing each sub-system individually, for example:

For instance, in the case of XenApp 7.5/7.6, the following components can be considered:

Going back to our example, one or more components can be causing a problem. For instance, there might be a problem with the Virtual Delivery Agent (VDA) on the server/servers hosting the finance application. This prevents the controller from being able to use the broker agent part of the VDA to communicate with the server.

Another possibility is that the issue is related to authentication. The StoreFront or the NetScaler Gateway (if the user is outside the corporate network) might have problems authenticating users to site resources.

It is important to quickly rule out as many components as possible. For instance, we could quickly test if the Citrix web page is accessible internally (where only the StoreFront component is used) and externally (where we might be reaching a NetScaler Gateway first). If the webpage is accessible internally but not externally, we would need to focus our attention on the NetScaler Gateway.

Alternatively, if, in both scenarios, the webpage does not load, we might focus our attention on the actual servers and/or delivery controllers.

Let's take another example: several users complain that during the day, applications published in XenApp start to become slow every morning.

The users mention that the slowness has been happening for some time, but it has only started to impact them recently.

Consider the following questions:

Answers to the these questions can be tremendously important when dealing with performance-related issues. For example, it is important to establish whether the performance is affected during specific hours/days (to help to isolate whether a scheduled operation is causing the issue) and whether it is consistent (for example, happens every day of the week or happens only on specific dates/days).

Further breaking down the problem could consist of:

Before describing how resolution testing should be done by administrators when troubleshooting a XenApp environment, there are two terms that need explaining. In software development terms, resolution testing is known as the process of retesting a bug once the development team has released a fix.

Regression testing is another methodology where test cases are re-executed for previously successful test cases.

Both testing methods are an important part of testing a software solution, as sometimes fixing one bug can cause regressions in other parts of the solution leading to new bugs.

Citrix administrators need to think in the same manner as testers do. Once the problem has been understood and a fix has been identified, then the fix or workaround can be applied. Once the fix is applied, the next step is to attempt to reproduce the initial issue. If this is not successful, it would generally mean the initial issue is resolved and most of the time that is the case.

However, besides testing for the initial issue, a Citrix administrator should also perform a number of tests to ensure that the fix does not negatively affect the XenApp infrastructure in another manner, for example, another application might stop working.

Once the problem has been correctly understood, and a fix applied and tested, the next step would be to determine the root cause and apply corrective actions if needed.

The Root Cause Analysis and Corrective Actions (RCCA) is the final step in troubleshooting a problem and involves determining the root cause of the issue and outlining any suggestions and recommendations for actions that can be implemented to prevent the reoccurrence of the underlying issue.

Most of the problems encountered in the Citrix world can be grouped into three categories:

Most root cause analysis reveal either a performance issue or an incorrect configuration.

Where a root cause is deemed to be performance related, tackling them usually requires improvements in the infrastructure—bigger bandwidth, more servers, faster disks, and so on. The real challenge is determining how much to scale the infrastructure so that performance falls back within acceptable parameters without spending a large amount of money.

Preventive steps for these types of problems could be:

Incorrect configurations are usually self-evident; for example, if an administrator performs a change that negatively affects the Citrix infrastructure, again usually almost immediately. The root cause analysis, therefore, focuses on the following questions:

Changes where the risks have been appropriately highlighted ("Changing X setting has the risk of bringing down the Citrix site for 15 minutes"), where the change is performed out of hours (minimizing risks) and has a proper rollback plan in place are perfectly acceptable.

Most changes have the potential of causing downtime, but if the proper change management process is followed, the risks are minimized and the potential outage reduced.

Preventive steps for this type of problems could be:

Finally, during troubleshooting, a number of changes might need to be done before the final fix is found. It is, therefore, a good idea to keep a track of these changes while the troubleshooting process is actively ongoing.

Once the correct fix has been identified, a retroactive change request should be logged in the IT system. Although, in this instance, the change hasn't followed the standard change management approval process, it is still useful to have changes logged in the system in case they need to be looked up in the future as part of troubleshooting previous changes.

In this chapter, we covered the basic methodologies of troubleshooting. We've described troubleshooting as first understanding the problem, breaking down the problem into its affected components, and finally, testing. The problems are solved once the fix or workaround is identified.

We highlighted the fact that sometimes, problems can be traced back to scheduled changes in the infrastructure, and that keeping track of changes is important as it can help in identifying the problem and mitigating or resolving it.

Finally, we discussed the root cause analysis, the process of determining the root cause of the issue (not just the fix/workaround) and preventive steps to minimize the reoccurrence of the issue.

In the next chapter, we will cover the Citrix XenApp/XenDesktop components, identifying and describing each one briefly. We will then talk about how the components interact and which communication channels are used during the interaction.