Troubleshooting a failed server or client deployment can be a daunting task, particularly for a novice user. A vast number of users do not fall into the typical system administration role, and they are either hobbyists or just venturing into virtual networking and cryptography. By the end of this chapter, the tools' key to identifying and correcting problems will be illustrated, and their utility will be demonstrated.
The general concepts of troubleshooting apply not only to fixing a problematic OpenVPN client or server process but also to nearly everything encountered in day-to-day work. At its most basic level, the idea of divide and conquerÂ is the phrase of the day. Separating components that are functional from those that are broken will quickly absolve the administrator from needless work and investigation.
The book is structured progressively, and it aims to help you find solutions quickly. This book will cover the followingÂ key topics surrounding fixing, identifying, and resolving OpenVPN problems:
A troubleshooting toolkit is a key to gleaning all the data needed to focus on a problem and resolve it.
Common OpenVPN issues are explored. Years of help supporting OpenVPN administrators on the forums and in IRC have provided a plethora of data, and the most common issues faced in the field are identified and solutions are provided.
The OpenVPN installation on various operating systems isÂ covered. The best client for each operating system is identified, including mobile device options. Where to go to obtain the installers and files needed.
Log files are the primary source of troubleshooting data. When to use what verbosity level and how to search for the data within the log is explained.
Startup and shutdown of the OpenVPN process for both the client and the server can cause stress and anguish. OpenVPN packaging is explained, where to go for help is shown, and how to troubleshoot those startup routines will be demonstrated.
Routing and networking can be a difficult concept to comprehend. This is likely the number one area of difficulty for beginning OpenVPN administrators.
We will discuss performance issues and how to correct performance-related issues. Also, cipher suite, key size, compression, and routing optimization will be illustrated.
Finally, problems external to OpenVPN will be explored. Such things include local LAN address collisions and incorrect firewall filtering, both locally and at an ISP.
There are a number of common utilities needed to investigate network andÂ public key infrastructure (PKI) issues. The samples within this book will be from a variety of operating systems. The server will be on FreeBSD 10.2, and we will show macOS X and Windows 7 and 10.
The majority of diagnostics will be done at the server side of the connection, but there are useful things to glean from client-side utilities. The tools listed here will be demonstrated, but this book isn't a manual for their use. For full documentation, refer to the documentation links provided.
Both the FreeBSD project and GNU have web interfaces for browsing man pages. The main page for these can be found at the following paths:
Detailed logging is available from OpenVPN on both the client and server sides, which allows configuration issues to be identified quickly. Having the ability to search these logs for the pertinent information is vital to successfully correcting problems and verifying a functional service. The utilities identified hereÂ will aid in these search tasks.
grep utility is likely to be one of the first utilities learned by an aspiring Unix user. Finding strings or keywords within a file or a set of files quickly is the first step in tracking down entries in a log file or a configuration directive.
grep allows you to search and highlight specific lines, context around those lines, filenames, line numbers, and more. In addition to finding lines of text,
grep can also omit lines you do not want to see.
The #openvpn support channel on Freenode (irc.freenode.net) IRC as well as on the OpenVPN forum (http://forums.openvpn.net), for example, request that users seeking support omit comments and empty lines with the following command:
grep -vE '^#|^;|^$' server.conf
Take a sample config file:
[email protected]:~-> cat foo.conf # this is a comment ; this is also a comment # the line above is empty config argument ; another comment
If we pipe that through our
[email protected]:~-> grep -vE '^#|^;|^$' foo.conf config argument
Paging applications are a common feature of Unix and Unix-like operating systems. These tools allow the user to view a large amount of content, typically text, to be viewed one page at a time. In general, there are three such common tools,
more utility is the most ubiquitous of the three, being installed by default on every Unix, Linux, or other similar system I have used for the past 20 years. Being the first paging utility, the
more utility's general functionality is limited. When output from a file or pipe contained more content than what could be displayed on a single screen, the content would be paged.
Scrolling down through the content was possible either a line at a time, using a down arrow key press, or a full page/window at a time with a press of the spacebar. Scrolling back up was not supported:
In 1983, Mark Nudelman authored the
less utility specifically for backward scroll capability. ItÂ was released in May, 1985, via the newsgroup net.sources. Many features have been added to
less, including pattern match highlighting and vi-like movement through the stream. To date, there have been over 450 released updates.
Modern Unix and Linux systems typically ship just the
less utility now, with
more being a hard-link to the
less binary. When executed this way,
less operates in a compatibility mode similar to
more. This behavior can also be evoked by setting the environment variable
The final pager of note is
most, which operates similar to
less, but adds the capability for multiple windows within a single terminal session. The
most pager also appears to support color escape sequences better than
less. The following screenshot shows
most displaying two windows, one with the
less man page and the other with the
most man page:
There are packages for
most available for FreeBSD, macOS X, and Linux, but the latest release of
most was in 2007, and the development seems to have stalled entirely. The windowed features can be replaced with other tools such asÂ tmux and screen, which fall outside the scope of this book.
Regular expressionÂ (regex) is a syntax that can be leveraged with string or pattern matching. There are already troves of other books and online guides about constructing quality regular expressions, but some basic syntax here will get you started in your troubleshooting endeavors.
This book will primarily use regular expressions in conjunction with the
grep utility described earlier. Coupling regex with
grep will allow us to specifically grab or omit lines from a log file. This is particularly useful when looking for specific client errors, or omitting a slew of noisy log entries from the view.
Regular expressions are composed of a sequence of pattern matching characters and character classes. Character classes are simply groups of characters or character types.
Some syntax characters to note are as follows:
Description and use
Line must start with
Denotes the start of the line. Inside a character class, denotes character exclusion.
Line must end with
Denotes the end of the line.
Signifies the following character should be interpreted literally. To match a
Start and end of a group.
Start and end of a character class.
Matches numeric characters. Same as
Matches alphanumeric characters including digits, letters, and the underscore. Same as
Â Matches any word, character, or space character one time.
Matches space and tab, respectively.
A period matches any character.
Any digit must appear exactly three times, does not have to be the same digit.
Specifies the minimum and maximum of the previous character or group. When only a single quantity is defined, indicates an exact count.
The previous character or group may or may not appear.
Matches any word character one or more times.
Indicates the previous item (group, character class, or character) must appear at least once, or more.
A separator, like a logical
There are a few online tools that can be used to validate and test your regular expression syntax. This is a good idea as they will demonstrate, graphically, how the changes to your pattern affect what is matched within a string or series of strings. Some of the online tools available online are as follows:
For additional reading, I strongly suggest the book commonly referred to as the camel book Programming Perl, 4th Edition, by Larry Wall. When I am stumped or need to understand how a regular expression is functioning, I find it an invaluable resource and a common reference.
You can also find tutorials and reading by navigating to http://www.regular-expressions.info.
There will be times when log files and OpenVPN output alone are not enough to identify a problem. It is possible that the issue resides outside of the OpenVPN process or the configuration therein. This could mean that there is a protocol error for some program being encapsulated within the tunnel or there is some upstream issue not readily apparent.
The tools listed hereÂ will provide an insight to the protocols and environment around and consuming your OpenVPN setup.
The ultimate command-line network diagnostic tool is the venerable
tcpdump is used to capture network traffic on an interface, and it provides an interface to filter-specific traffic, including unique destination addresses, ports, packet types, protocols, and more. This tool can be used at a very low level to determine maximum transmission unitÂ (MTU) issues, protocol issues, and many others.
Depending on your level of networking experience, this tool may or may not be directly useful, but packet captures can be sent to more experienced people.
The following screenshot shows a simple single ping from a test host to Google's
18.104.22.168 DNS resolver IP. We had to use
sudo as the packet capture requires root privileges on the network interface. Our first command line included option
-A, which specifies ASCII output and is the unintelligible at the end of each packet info line. The second example shows the same ping without the
-A option (same screenshot):
A much more detailed introduction to
tcpdump is available by going to Daniel Miessler's blog at https://danielmiessler.com/study/tcpdump/.
On Linux, BSD, and macOS X,
traceroute, or on Windows
tracert, knowing the path to assorted destinations is a crucial tool. You can quickly ascertain whether traffic is departing the default gateway or a VPN connection. As a bonus, response time to each hop along the path is calculated, which may indicate slow points along the route.
Contrary to the popular belief, these commands are not for hacking or seeing how many people are using a website; you will not improve your K/D ratio in Call of DutyÂ®. These are legitimate network diagnostic tools.
Check out the YouTube video by NextGenHacker101 for a quick laugh at https://www.youtube.com/watch?v=SXmv8quf_xM.
For a quick change of pace, here is a screenshot of the
tracert command from Windows 8. From the output, we can see that there are eight hops between my test Windows 8 system and Google's resolver:
mtrÂ is a utility that combines the functionality of
traceroute. This tool can help illustrate where along a network path latency or packet loss occurs. I still prefer to use
tracerouteÂ individually at times, but use
mtr to quickly identify network connectivity issues.
tracert will stop, by default, after the last hop or a maximum of 30 has been reached.
mtr, on the other hand, will continue cycling until quit with a Ctrl + C.
ping on *nix system functions in a similar manner of pinging indefinitely.
Here is a sample output from
mtr between my test system and the GoogleÂ website:
Snt column and that all list
20 apart from hop seven. I pressed Ctrl + CÂ just as the cycle hit hop seven, so the twentieth packet was never sent.
ping.Â This is usually the first tool in the network troubleshooter's toolbox. This is generally the quickest way to determine if a remote system is alive or not. This tool is very much cross-platform, and it is available on Linux, Unix, and Windows systems by default. Only the most hostile or ill-managed corporate networks block this. The following screenshot shows an example of
Coupled with the
tcpdump utility, and sometimes on its own, Wireshark is arguably the most powerful tool in our network troubleshooting toolbox. This tool provides a relatively easy-to-use graphical interface toÂ navigate packet captures. In addition, it provides a filtering interface that allows you to isolate specific streams, protocols, and destinations.
One particular trick Wireshark can do is to decrypt TLS and SSL traffic, given the private and public keys of a web server or server/client pair. This is analogous to the features of the latest next-generation firewalls that do decryption at the border for corporate networks.
The following screenshot shows a short eight-packet transaction for a short IPv6 ping:
In my experience in the scope of OpenVPN, Wireshark is primarily used along side
tcpdump. Many OpenVPN servers and clients will have
tcpdump readily available, already installed, or easily installed when needed. Wireshark requires X11 or other graphical tools and libraries that may not be as easy to install.
It is easy to take a packet capture using
tcpdump, save the capture to a file (see the
-w option for
tcpdump), and transfer that capture to a system with Wireshark installed. The observant reader may have noted the window title in the earlier screenshot: the packet capture here is actually being read-in from a file.
Cryptography and PKI are often difficult to understand and much more difficult to resolve issuesÂ with. There are primarily two utilities, OpenSSL and Wireshark, that can aid with peering into the cryptographic components of OpenVPN.
OpenVPN also supports PolarSSL (recently known as ARMÂ®Â mbedâ¢) as a replacement for OpenSSL. The latest package, 2.2.1, includes some rudimentary programs for certification creation, but it does not includeÂ
s_client and other utilities included with OpenSSL. More information can be found on their website at http://tls.mbed.org.
OpenSSL is the ubiquitous library for X.509 certificate PKI. OpenVPN has supported the use of X.509 certificates for TLS connections since before 2002. The OpenSSL command-line utilities allow certificate chain verification, outputting certificate details for inspection, build Diffie-Hellman parameters, and even substantiating an SSL/TLS server or client instance.
I have used the
s_client subcommand to fetch the full SSL certification chain for the GoogleÂ website. All three certificates are listed: the GeoTrust CA root certificate, the Google Intermediate CA (they get to sign their own certificates), and the server certificate their intermediate CA issued. See the following code:
[email protected]:~-> openssl s_client -showcerts -connect openvpn.net:443
With this command, I manually copied each certificate block and saved them to individual files,
GoogleSrv.crt (certificate 0),
GoogleCA.crt (certificate 1), and
GeoTrustCA.crt (certificate 2).
A certificate block looks like the following:
-----BEGIN CERTIFICATE----- MIIDfTCCAuagAwIBAgIDErvmMA0GCSqGSIb3DQEBBQUAME4xCzAJBgNVBAYTAlVT [snip] NhGc6Ehmo21/uBPUR/6LWlxz/K7ZGzIZOKuXNBSqltLroxwUCEm2u+WR74M26x1W b8ravHNjkOR/ez4iyz0H7V84dJzjA1BOoa+Y7mHyhD8S -----END CERTIFICATE-----
Wireshark was discussed previously, but this is where that utility will demonstrate its capability. In addition to the ability to decode and illustrate various (nearly all) protocol streams, given the private and public keys available to a VPN admin, it can decipher SSL and TLS encrypted streams, including OpenVPN streams.
To demonstrate the ability to decrypt a TLS session, we will use the OpenSSL
s_server command to create a generic HTTPS server. I have created a very simple web page that simply reads, This content is encrypted. I used the following command to create the server. Note that to start the daemon on port
443, you need to use root or
sudo. To avoid escalating privileges, you can use a port
1024, such asÂ
To begin, create a certificate/key pair:
[email protected]:~-> openssl req -x509 -newkey rsa:2048 -keyout key.pem -out cert.pem -days 365 -nodes
Then, we start our server:
[email protected]:~-> sudo openssl s_server -key key.pem -cert cert.pem -WWW -accept 443
s_server process will use the current working directory for its web root, so I placed our web content there as
The preceding command used
sudoÂ because it opened a listening port on a privileged port. All TCP/UDP ports numbered
1024 and lowerÂ are considered privileged,Â and they require root or administrator permissions to open.
Now, I will start Wireshark and set it to capture traffic on the
loopback interface. Because we are going to connect to the
localhost address (
::1), the traffic will use this interface. If we connect to the actual system IP address, then capture traffic on the real interface.
Now, open a web browser to the system. In my case, this is the local machine. The URL I will use
https://localhost/index.html, if you changed the port, add it to the URL such as
If all the steps mentioned earlierÂ were performed correctly, you should have a browser window with a simple message and a Wireshark window with approximately 25 packets captured:
In the packet capture, you will see some protocol data that is indicative of what is happening. We will touch on the protocol exchanges later, but you can clearly see the TLS handshake and cipher exchange taking place:
The lines in the capture have a black background, indicating the transmission carried encrypted payload data. Next, we will take the certificate and the key we created earlier and import those into Wireshark. This will allow us to view the transaction.
Before we do that, we will examine packets 17 and 19. Both of these are labeled with the generic phrase
Application Data and contain our actual HTML. These packets are encrypted, and they examine them by clicking on them.
By going to Preferences | Protocols | SSL, Wireshark provides a way to import the TLS key we created earlier. On macOS X, the dialog resembles the following screenshot. You can specify the port here, but it is optional. In my case, I simply listed the IP
127.0.0.1 and the key file:
If we go back and inspect our packets now, we can see a new tab in the payload pane. The first is labeled
Frame, and the second is
Decrypted SSL data:
Click on the second tab for packet 19, and we can actually see the decrypted page content:
<html> <head> <title> Hello! </title> </head> <body> <p>This content is encrypted.</p> </body> </html>
The ability to decrypt the OpenVPN TLS streams is significant enough that the Wireshark wiki itself has a page specifically demonstrating this capability: https://wiki.wireshark.org/OpenVPN.
The concept of breaking apart a problem in any system, whether it be electronic, software, physical, or even behavioral, is a common principle. The phrase, divide and conquer is often seen, and true to reality.
Readers of this book are likely familiar with the common light bulb. You may not realize it, but there is a series of automatic troubleshooting steps performed.
Imagine the followingÂ scenario:
You walk into the office, many are already at work. You step into your office and flip on the light, nothing happens. You flip the switch back and forth a couple times before sitting down and turning on the computer in the dark.
You then pick up the handset on your Cisco IP phone, calling building maintenance. You speak with someone at the other end, exclaiming that the bulbs are out in your office.
What just happened?
A large number of things occurred that weren't directly acknowledged. Most of these steps happened automatically without realization:.
Walking into the office. Nothing is out of the ordinary.
In reality, everyone else was working. There wasn't an uncomfortable silence or notable lack of work or exceptional amount of generalized confusion.
You walk in and flip the light switch in your office; more than once.
You've tried to turn on the light. After the first failure to exhibit illumination, you've automatically tested the switch by flipping it a couple times. Sure, it is not overly scientific, but it's a general functional test.
You start working at your own computer, contacting support on your VOIP phone.
Power works in your office. The computer works, and network PoE is functioning.
You've ascertained the only thing not working is the light in your office.
What's neat about this generic situation is many people do so without realizing it. Some have cars and do this when it starts or doesn't start, maybe on a cold morning. Maybe after leaving the light on over night. We need to apply this concept and method to OpenVPN or anything really.
This chapter touched on some of the most common tools used to identify and resolve configuration or network issues within the scope of OpenVPN. Some subjects, such asÂ regular expressions, were identified; however, that may not be obviously useful to a novice administrator.
Some extremely powerful capabilities were also demonstrated with Wireshark's ability to decrypt SSL streams and
tcpdump able to capture packets. Although these are most useful to an experienced network administrator, they can still be leveraged by a new or novice administrator. Having the ability to extract troubleshooting data is most critical when seeking help from others.