In order to successfully troubleshoot Configuration Manager, there are a number of tools that are recommended to always keep in your troubleshooting toolkit. These include a mixture of Microsoft tools, third-party tools, and some community developed tools. Best of all is that they are free. As could be expected with the broad scope of functionality within Configuration Manager, there are also quite a variety of different utilities out there, so we need to know where to use the right tool for the problem. We are going to take a look through some commonly used tools and some not so commonly used tools, then look at what they do and where we can use them. These are not necessarily the be-all and end-all but they will certainly help us get on the way to solving problems and undoubtedly save some time. In this chapter, we are going to cover the following topics:
Microsoft System Center Configuration Manager Toolkit
Microsoft System Center 2012 Configuration Manager Support Center
WMI Tools
Registry Editor
Group Policy Tools
CMTrace/Trace32/Notepad
Error code lists
PowerShell
Telnet
Wireshark
Operations Manager Management Pack
Community Tools
OK, so this one might be a bit of a cheat as the 2012 R2 version is actually 15 tools rolled into 1. It is, however, a brilliant set of free tools from Microsoft for finding out what is going on under the hood both client side and server side. For those who have never seen this one before, following are the tools that we can find in the download (https://www.microsoft.com/en-us/download/details.aspx?id=50012):
These tools are useful when troubleshooting Configuration Manager servers, specifically the following ones:
DP Job Queue Manager: This one gives us a nice clear view of what jobs a distribution point has on its to-do list. It is very useful when there are content distribution issues.
Collection Evaluation Viewer: Listed as CEViewer, this will show us behind the scenes of collection evaluations. If there are any problems with collection limiting and delays in evaluation, then this is the tool that will help.
Content Library Explorer: This tool can be used to look at content distribution and validation of content on a distribution point. It allows us to perform distribution and validation of content in one simple console.
Security Configuration Wizard: Intended for Server 2008 R2, this is designed to help us run the minimum required server roles and features in order to run Configuration Manager 2012 R2, therefore reducing the potential risk for malicious behavior. In a secure environment with compliancy requirements, this is ideal.
Content Library Transfer: If we are in the position where we have distribution point content on a disk that has filled, then this tool will help move all distribution point content from one disk to another.
Content Ownership Tool: This tool lets us move content between Configuration Manager sites. In a multiple site hierarchy, this can be ideal.
Role-Based Administration Modeling and Auditing Tool: This is a useful tool for planning role-based access to Configuration Manager and, therefore, save troubleshooting time later down the line.
Run Metering Summarization Tool: If we need to run a software metering summarization on an ad hoc basis and cannot wait for the next scheduled run, then this is the tool to use.
These tools are useful when troubleshooting Configuration Manager clients, specifically the following ones:
Client Spy: This utility allows us to check software distribution, inventory, and software metering on a client, locally or remotely, and we can also export to a file for further analysis.
Configuration Manager Trace Log Viewer: More commonly known as CMTrace, we will cover this in more detail later in this chapter, but when we want to view log files, this is a great choice.
Deployment Monitoring Tool: This is nice little tool for ad hoc troubleshooting on a client and it gives us a current status for client properties, deployments, and software updates, so we know exactly what client data Configuration Manager is working with.
Policy Spy: This is a frequently used utility for analyzing the policy on a Configuration Manager client and making policy requests. This one can help us figure out why a client isn't doing what we expect it to be doing.
Power Viewer Tool: This tool will help us troubleshoot the many different options available when it comes to power management on a Configuration Manager client. It is also useful for testing power policy changes.
Send Schedule Tool: If we want to force ad hoc evaluation of a configuration baseline on a client, then this tool will help us do this. There are other ways but having this handy tool can save us lots of time.
Wakeup Spy: If Wake on LAN (WoL) is enabled in an environment, then we will undoubtedly get to use this at some point. This tool lets us view information about client wakeup state and listens for wakeup requests.
We can download Microsoft System Center 2012 Configuration Manager Support Center at https://www.microsoft.com/en-us/download/details.aspx?id=42645.
This one is often a particular favorite for technical support or consultants who are asked to troubleshoot other Configuration Manager environments unknown to them. The free tool was released for public use by the Microsoft support team and is used by them to obtain a good overall picture of a Configuration Manager infrastructure from the client perspective. The Support Center Tool is made up of the following three tools:
Configuration Manager Support Center
Configuration Manager Support Center Viewer
Log Viewer
The Configuration Manager Support Center can perform several functions. The first is data collection. Information about the client is bundled together and can then be sent off to someone else to investigate or simply allow us to collate lots of useful information for our own investigations. The information gathered is as follows:
Log files
Policy
Certificates
Client configuration collector
Client registry
Client WMI
Troubleshooting
Debug dumps
Operating system
All this information is bundled together into a ZIP file that contains logs, debug dumps, and various XML files depending on the options we choose. This file can then be opened in the Configuration Manager Support Center Viewer, which we will come to shortly.
The Configuration Manager Support Center also allows us to load client information and control the agent service from one easy console. We can load, request, and evaluate policy, and listen for policy events in real time. Further functions enable us to work with, monitor, and manipulate client content cache for applications, software updates, and windows installers. There are also similar utilities for troubleshooting the various inventories available, software metering, and discovery data, and, finally, some log evaluation and common troubleshooting tasks. We will be revisiting some of these tools in more detail later in this book. In brief summary, this is a great one stop shop for client troubleshooting.
The Configuration Manager Support Center Viewer is the tool to use to open up the support bundle produced in the Configuration Manager Support Center. It splits all the gathered information into the respective categories and allows us or a completely remote person to browse the Configuration Manager logs, debug dumps, WMI repository, registry, policy, any gathered certificates, and the results of the troubleshooting tasks. Last but not least is the Log Viewer, which is used by default when we open any of the logs captured by the Configuration Manager Support Center, and it can be used as a generic log file viewer too. The feature set in the Log Viewer includes live updating, auto-scroll, a detail pane, wrapping of text, and a filter function on top of the standard facilities we would expect from any log viewer. All round, it is a really useful piece of software as it can make what can be often very detailed log files somewhat easier to read.
Windows Management Instrumentation (WMI) is rooted in Configuration Manager and the product has been using WMI since its early days when Configuration Manager was previously known as System Management Server (SMS). WMI was introduced in SMS 2.0 and has been used in the product for client and server functions in every version since. For this very reason, it is highly likely that we will need to take a look at WMI at some point on our troubleshooting journey. So with this in mind, it's important that we learn to use one of the various tools available to us. A simple example of WMI usage in Configuration Manager would be collection membership query rules that are made from WMI Query Language (WQL), which is a similar syntax to SQL. Effectively, what we are doing in these commands is querying a WMI database in much the same way as we might with a SQL database. For example, if we need to check a specific value in a client inventory, then we can do this with WMI. Another useful example is when working with configuration baselines as we can check specific values ad hoc and predict or confirm expected compliance or non-compliance. We shall mention just a few tools but this really is a personal preference, what it is used for and how you like to work.
Firstly, the Windows Management Instrumentation Command Line (WMIC) is a command line utility that can be run from a Command Prompt or PowerShell console that allows us to view and manipulate WMI from a command line, which means we can use it in scripts and automation products, and perform easy remote commands. It is a popular option with many as it offers additional scripting functionality which makes it particularly helpful outside of troubleshooting too. A basic example of the syntax used with WMIC would be the following query that will return the Manufacturer, Model, and Name of a machine from WMI:
Wmic computersystem get manufacturer,model,name
This would return something like the following when ran against, for example, a Microsoft Surface Pro:
Manufacturer Model Name Microsoft Corporation Surface with Windows 8 Pro Surface-Pro
This is a very basic common usage example of WMIC when trying to obtain the model number of a machine and is commonly used for a task sequence step condition to apply model specific drivers. We will reference WMIC later in this book and show where else this tool can be useful.
WBEMTest is another WMI tool but this time with a graphical user interface that again—like WMIC—allows us to dig deep into WMI and look at the classes, instances, and namespaces, as well as perform amendments. It can be started from the Run dialog or from a Command Prompt by simply typing:
Wbemtest
We are then presented with the Windows Management Instrumentation Tester window, as shown in the following screenshot, and we can make a connection into our preferred namespace:
From here we can perform several tasks such as opening, enumerating, creating, and deleting classes and instances or performing queries. If we are performing ad hoc checks or simply prefer a point and click experience, then this could be the tool to use. However, along with the other WMI tools, we should know what we are looking for before getting too click happy, otherwise there is the potential to cause more damage than harm and give us more troubleshooting than we started with.
Finally, a brief introduction to the Microsoft mini-suite of WMI tools called WMI Administrative Tools: this is freely available in the download center and includes WMI CIM Studio, WMI Object Browser, WMI Event Registration Tool, and WMI Event Viewer. These tools mostly give away their function and, with the exception of the event viewer, offer us a browser-based view of WMI. These can be a good alternative to WMIC and WBEMTest as they can offer a wider view of WMI, which is often particularly useful for someone not overly familiar with the finer details of WMI.
Also worth a mention is the Registry Editor that is built into Microsoft Windows on both server and client operating systems. Most IT administrators know this as regedit.exe and it is the default tool of choice for making any changes to or just simply viewing the contents of a registry key or value. Many of the Configuration Manager roles and the clients allow us to make changes to enable features such as extended logging or manually changing policy settings by using the registry to do so. It should be noted that changing the registry is not something that should be taken lightly as making incorrect changes can result in creating more problems not just in Configuration Manager but also in the operating system as a whole. If we stick to the published settings though we should be fine and this can be a fine tool when troubleshooting oddities and problems in a Configuration Manager environment.
A Configuration Manager is a client management tool; there are certain features and settings on a client such as software updates that may conflict with settings defined in Group Policy. Particularly in larger organizations, it can often be useful to compare and contrast the settings that may conflict between Group Policy and Configuration Manager. Using integrated tools, such a Resultant Set of Policy (RSoP) and Group Policy Result (gpresult.exe), or the Group Policy management console as part of the Remote Server Administration Tools (RSAT) can help identify where and why clients are not functioning as expected. We can then move forward and amend group policies as and where required using the Group Policy Object Editor. Used in combination, these tools can prove essential when dealing with Configuration Manager clients in particular.
Those who have spent any time at all working with Configuration Manager will know that it contains quite a few log files, literally hundreds. We will go through the log files in more detail in the next chapter, but we will need to use something to read the logs. We can use something as simple as Notepad and to an extent there are some advantages with using this as it is a no-nonsense text reader. Having said that, generally speaking, most people want a little more when it comes to reading Configuration Manager logs as they can often be long, complex, and frequently refreshed. We have already seen one example of a log viewer as part of the Configuration Manager Support Center, but Configuration Manager includes its own log file viewer that is tailored to the needs of troubleshooting the product logs. In Configuration Manager 2012 versions, we are provided with CMTrace.exe; previous versions provided us with Trace32.exe or SMSTrace.exe. They are very similar tools but we will highlight some of the features of CMTrace, which is the more modern of the two. To begin with, we can typically find CMTrace at the following locations:
%ProgramFiles%\Microsoft Configuration manager\Tools\CMTrace.exe<INSTALLATION MEDIA>\SMSSETUP\TOOLS\CMTrace.exe
Those who are running Configuration Manager 2012 R2 and up also have CMTrace available out of the box in WinPE when running Operating System Deployments. We can simply hit F8 if we have command support enabled in the WinPE image and type CMTrace. This can also be added to the later stages of a task sequence when running in the full operating system by copying the file onto the hard disk. The single biggest advantage of using CMTrace over a standard text reader is that it is a tail reader that by default is refreshed every 500 milliseconds, or, in others words, it will update the window as new lines are logged in the log file; we also have the functionality to pause the file too. Other functionality of CMTrace is to allow filtering of the log based on certain conditions and there is also a highlight feature that can highlight a whole line in yellow if a word we are looking for is found on the line. The program automatically highlights lines if certain words are found such as error or warning, which is useful but can also be a red herring at times, so this is something to be aware of if we come across logs with these keywords. We can also merge log files, and this is particularly useful when looking at time critical incidents as we can analyze data from multiple sources in the order they happened and understand the flow of information between the different components.
When working with Microsoft technologies, we sometimes see the same error message used in different products. This makes a lot of sense because some products will use code from others and some will interact with others. Often overlooked is taking an error message from Configuration Manager and finding an unrelated example from another product. Let me give an example of HTTP. Many of the Configuration Manager roles are based on HTTP communication, and HTTP has a standard set of error messages. Add in some certificates and HTTPS communications and it is not unusual for people to start to get lost in the logs. What is often found is that if we take the error out of its Configuration Manager context for a moment and look at what that error means specifically in Internet Information Services (IIS), then this can help us understand what is happening, which can be applied back to our specific problem. This also works when searching on the Internet for error codes as it is something we all do as there is a breadth of information out there. We may well find that if we search for an error code on the Web, we can see the same error elsewhere; don't be put off and read it through as we might find it gives us that eureka moment.
With this in mind, we are not going to list every error code as this would be exhaustive; however, it is worthwhile bookmarking these standard lists that are recommended as being useful. Many of these error codes are also embedded into CMTrace and can be looked up through Tools | Errors Lookup.
Windows Installer Errors Reference: This will help us understand the return codes from any Windows application or package deployments which is available at http://msdn.microsoft.com/en-us/library/aa368371.aspx.
HTTP Status Codes: These are the standard status codes used by IIS and can prove particularly useful when troubleshooting the availability of IIS-based roles. More information is available at http://support.microsoft.com/kb/943891.
Custom Err or Codes for Configuration Manager 2007: Don't worry, this is not a typo; these are reference error codes for an old product version. This is because there is no formal published list for 2012 or current branch versions; however, you will find that the custom codes have changed very little, so this is still a great list to have. More information is available at http://technet.microsoft.com/en-us/library/bb632794.aspx.
PowerShell is here to stay. A phrase often heard recently is learn PowerShell or learn golf. Like it or not we cannot get away from the emphasis on this homemade product from Microsoft. This is evident in just about all current products as PowerShell is so deeply embedded. Configuration Manager is no exception to this, and although we cannot quite do everything we can in the console, there are an increasing number of cmdlets becoming available, more than 500 at the time of writing. So the question we may ask is "where does this come into troubleshooting?" Well for the uninitiated in PowerShell maybe it won't be the first tool they turn to but with some experience we can soon find that performing things like WMI queries and typical console tasks can be made quicker and slicker with PowerShell. If we prefer, we can also read log files from PowerShell and make remote changes to machines. PowerShell can be a one-stop shop for our troubleshooting needs if we spend time to pick up the skills.
When we refer to networking tools, we are referring to everything ranging from a ping to a packet capture. When we break Configuration Manager down to its simplest form, we are trying to get data from point A to B, or server to client in other words. If we don't have a clear line of communication, then we are going to hit problems. Sounds simple right? We all know that it isn't always that easy in every organization. Often there are complex networks in place with routers and firewalls in between A and B, so it may not be clear what the problem can be. Add this to the combination of ports that Configuration Manager uses and it can be a recipe for confusion. The good thing about using network tools is that it is often a true or false scenario. A packet capture, for example, doesn't lie, it shows us exactly what is coming into and out of our network interface, which can be really powerful when looking into problems such as failed deployments, failed distribution, or Configuration Manager role installation. For these reasons, I think it is essential that we add some of the following utilities to our troubleshooting toolkit.
Ping: This is almost too obvious and isn't guaranteed due to firewalling and devices dropping ICMP requests, but it is the first line in communication checks and often overlooked as a basic check and can also help check for any name resolution problems.
Tracert: If we can run a trace route from point A to B, then this helps us rule this out or indeed rule in the point at which communications drop. Again, it is not a guaranteed result but certainly not worth forgetting about.
Telnet: If we know there is a clear line of communication, then we can go a level further and try a Telnet to test the TCP ports out. Using our preferred Telnet client, we can simply run the following:
telnet CMServer1 445This will test a connection from our source to our destination
CMServer1over TCP445.If we want to use the default Microsoft Windows Telnet client, then don't forget we will need to enable this as a feature in modern operating systems.
Microsoft Message Analyzer: Taking things a level deeper again, we can use this to capture packets at both sides of the communication chain and see what is or isn't being received and over what ports. This is the successor to Microsoft Network Monitor, and if we can master a tool like this, then it is often the last line port of call for communication issues as we can present the results to our network or firewall administrator and show them exactly what we need versus what we have. Another similar popular tool worth mentioning is Wireshark, previously known as Ethereal. There are several other tools out there to choose from, but this is a must have in our toolkit.
I have not specifically mentioned any of the other Command Prompt related tools that you can use but there are a handful of others that you may find useful. There is no steep learning curve involved with these, so if you are not already familiar, then take a little time out and see what they can do for you.
Pathping.exeNbtstat.exeNetstat.exe
Note
For a full list of ports used by Configuration Manager, we should always refer to the TechNet library documentation as this is maintained by Microsoft and is definitive for a default installation of the product. The documentation is available at https://technet.microsoft.com/en-gb/library/hh427328.aspx.
You can find System Center Configuration Manager Management Pack at https://www.microsoft.com/en-gb/download/details.aspx?id=29267.
This one is not going to be applicable to everyone, but nonetheless I had to include it as a top troubleshooting tool. If you are not familiar with System Center Operations Manager, it is essentially the performance and monitoring tool in the System Center suite. Operations Manager is another Microsoft System Center product that will monitor servers, clients, network devices, and other things such as storage hardware and blade chassis with the use of management packs. This is not intended to be a lesson in Operations Manager, but essentially the management pack contains lots of detailed information about the ins and outs of a given application or service that will automatically discover and analyze the Configuration Manager infrastructure and highlight issues where necessary. This helps a lot not only with reactive scenarios as it can point out exactly where our problems lie, but also in proactive maintenance of our Configuration Manager environment as it can highlight potential failures before they happen and alert us accordingly. Management packs are often developed by the original software developer but there are often third-party versions available and with some experience we can even write our own. With regard to Configuration Manager then, what could be better than a management pack written by Microsoft that contains lots of low-level information about their own product? If this option is available, then it is highly recommended. Following is a summary of the monitoring we can find in the Configuration Manager Management pack:
Replication health
Configuration of replication
Backup and recovery status monitor
Component availability monitoring
Service availability
Server role availability monitoring
Compliance rate of baseline deployment monitoring
Discovery backlog monitoring
General health monitoring
Database connection monitoring
Inventory backlog monitoring
Software metering backlog monitoring
Software update synchronization status monitoring
Distribution point configuration monitoring
In addition, the following diagram shows us the monitored components and how they roll up to produce a health state in Operations Manager:
Having this management pack available to us can save so much time, both in the short and long term, when experiencing problems so again this has to go into the top picks for our troubleshooting toolkit.
Finally, as user group community leaders, we couldn't leave this section out of the troubleshooting toolkit. Configuration Manager has such a great collection of community contributors that have likely to have been through our troubleshooting pain before us and either blog about it, post it on a forum, or create a fix for it. There is such an array of free tools out there that people share that we cannot ignore them. Outside of troubleshooting specifically, some of the best add-ons available for Configuration Manager are community contributions, whether that be from individuals or businesses. There are so many utilities that are ever evolving and not all will suit your needs, but if we browse the Microsoft TechNet galleries, Codeplex, and GitHub, you are sure to find a great resource to meet your requirements. Why not get involved with a user group too? In terms of troubleshooting, this is probably one of the best things I personally could recommend. It gives access to a network of people who work on the same product as us and are often using them in the same way, so it is quite likely that someone has seen our problem before and can fast forward us to a solution.
Microsoft TechNet Galleries: https://gallery.technet.microsoft.com/
Codeplex: https://www.codeplex.com/
GitHub: https://github.com/
Configuration Manager Community: http://go.microsoft.com/fwlink/p/?LinkId=626544
In summary, our troubleshooting toolkit contains a variety of applications, all of which are free, and many of these tools will not take a great amount of time to get to grips with but will undoubtedly save us time so that is a great trade-off. Also included is something that isn't strictly a tool but can be just as important when trying to solve problems—product knowledge. It goes without saying that if we already understand how the product works, then our troubleshooting will generally be quicker and more successful. There aren't any training courses that will teach us how to deal with the unexpected that can sometimes occur in any software product, so fast-tracking that knowledge gain by getting involved with people in the know seems only logical. It can not only help us get over problems but also avoid them by improving our practices in the product to prevent problems in the future. In the next chapter, we will a look at the log files that underpin the troubleshooting of Configuration Manager and which files we would commonly use during everyday troubleshooting of this, sometimes overwhelming, product.
