It is a common experience amongst all of us that from time to time our computers don't always work as we expect and we need to hunt down and resolve problems with them. Troubleshooting can be time consuming especially if you have limited experience with the software or system you are using. This article by Gary Sims, is a guide on how to solve problems with your FreeNAS server. It covers where to look for information about the problem as well as how to hunt down problems by being methodical. It also looks at the common problems people have with their FreeNAS servers including networking problems and problems with RAID.

Where to Look for Log Information

The first place to head whenever you have a configuration problem with FreeNAS is to the related configuration section and check that it is configured as expected. If, having double checked the settings, the problem persists, the next port of call is the log and information files in the Diagnostics: section of the web interface.

Keep Diagnostics Section Expanded
By default, the menu tree in the Diagnostics section of the web interface is collapsed, meaning the menu items aren't visible. To see the menu items, you need to click the word Diagnostics and the tree will expand. During initial setup and if you are doing lots of troubleshooting, you can save yourself a click by having the Diagnostics section permanently expanded. To set this option, go to System: Advanced and click on the Navigation - Keep diagnostics in navigation expanded tick box.

The Diagnostics sections has five sections, the first two are logs and information pages about the status of the FreeNAS server. The other three are networking diagnostic tools and information.

Diagnostics: Logs

This section collates all the different log files that are generated by the FreeNAS server into one convenient place. There are several tabs, one for each different service to log file type. Some of the information can be very technical, especially in the System tab. However, with some key information they can become more readable. The tabs are as follows:

What is a Daemon?
In UNIX speak, a Daemon is a system service. It is a program that runs in the background performing certain tasks. The Daemons in FreeNAS don't work with the users in an interactive mode (via the monitor, mouse, and keyboard) and as such need a place to log the results (or problems)of their actives. The FreeNAS Daemons are launched automatically by FreeBSD when it boots and some are dependent on being enabled in the web interface.

Understanding Diagnostics Logs: System

The most complicated of all the log pages is the System log page. Here, FreeBSD logs information about the system, its hardware, and the startup process. At first, this page can seem intimidating but with a little help, this page can be very helpful particularly in tracking down hardware or driver related problems.

50 Log Entries Might Not be Enough
The default number of log entries shown on the Diagnostics: Logs page is 50. For most situations, this will be sufficient but there can be times when it is not enough. For example in the Diagnostics: Logs: System tab, the total number of log entries made during the boot up process is more than 50. If you want to see how much system memory has been recognized by FreeBSD, you won't find it within the standard 50 entries. The solution is to increase the Number of log entries to show parameter on the Diagnostics: Logs: Setting tab.

The best way to learn to read the Diagnostics: Logs: System page is by example, below are several different log entry examples including logs about the CPU, memory, disks, and disk controllers:

kernel: FreeBSD 6.2-RELEASE-p11 #0: Wed Mar 12 18:17:49 CET 2008

This first entry shows the heritage of the FreeNAS server. It is based on FreeBSD and in this particular case, we see that this version of FreeNAS is using FreeBSD 6.2. There are plans (which may have already become reality) to use FreeBSD version 7.0 as the base for FreeNAS.

kernel: CPU: Intel(R) Xeon(TM) CPU 1.70GHz (1680.52-MHz 686-class CPU)

Here, the type of CPU that was detected by the FreeBSD is displayed. In this case, it is an Intel Xeon CPU running at 1.7GHz.

kernel: FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs

If your system has more than one CPU or is a dual core machine then you will see an entry in the log file (like the one above) recognizing the second CPU. If your machine has Hyper-threading technology, then the second logical processor will be reported like this: Logical CPUs per core:2

Apr 1 11:06:00 kernel: real memory = 268435456 (256 MB)
Apr 1 11:06:00 kernel: avail memory = 252907520 (241 MB)

These log entries show how much memory the system has detected. The difference in size between real memory and available memory is the difference between the amount of RAM physically installed in the computer and the amount of memory left over after the FreeBSD kernel is loaded.

kernel: atapci0: <Intel PIIX4 UDMA33 controller> port 0x1f0-
0x1f7,0x3f6,0x170-0x177,0x376,0x1050-0x105f at 
 device 7.1 on pci0
 kernel: ata0: <ATA channel 0> on atapci0
 kernel: ata1: <ATA channel 1> on atapci0

For disks to work on your FreeNAS server, a disk controller is needed and it will be either a standard ATA/IDE controller, a SATA controller or a SCSI controller. Above are the log entries for a standard ATA controller built into the motherboard. You can see that it is an Intel controller and that two channels have been seen (the primary and the secondary).

kernel: atapci1: <SiS 181 SATA150 controller> irq 17 at device 5.0 on pci0
kernel: ata2: <ATA channel 0> on atapci1

kernel: ata3: <ATA channel 1> on atapci1

Like the ATA controller listed a moment ago, SATA controllers are all recognized at boot up. Here is a SiS 181 SATA 150 controller with two channels. They are listed as devices ata2 and ata3 as ata0 and ata1 are used by the standard ATA/IDE controller.

kernel: mpt0: <LSILogic 1030 Ultra4 Adapter> irq 17 at device 16.0 on pci0

Like IDE and SATA controllers, all recognized SCSI drivers are listed in the boot up system log. Here, the controller is an LSILogic 1030 Ultra4.

kernel: ad0: 476940MB <WDC WD5000AAJB-00YRA0 12.01C02> at ata0-master UDMA100

kernel: ad4: 476940MB <Seagate ST3500320AS SD04> at ata2-master SATA150

Once the disk controllers are recognized by the system, FreeBSD can search to see which disks are attached. Above is an example of a Western Digital 500GB hard drive using the standard ATA100 interface at 100MB/s. There is also a 500GB Seagate drive connected using the SATA interface.

acd0: CDROM <TOSHIBA CD-ROM XM-7002B/1005> at ata1 as master UDMA33

When the CDROM (which is normally attached to an ATA/IDE controller) is recognized, it will look like the above.

kernel: da0 at ahd0 bus 0 target 0 lun 0
kernel: da0: <MAXTOR ATLAS10K4_73WLS DFL0> Fixed Direct Access SCSI-3 device
kernel: da0: 320.000MB/s transfers (160.000MHz, offset 127, 16bit),
 Tagged Queueing Enabled

kernel: da0: 70149MB (143666192 512 byte sectors: 255H 63S/T 8942C)

SCSI addressing is a little more complicated than that of ATA/IDE. In SCSI land, you have a controller, a channel (bus), a disk (target), and the Logical Unit Number (LUN). The example above shows that a disk (which has been assigned the device name da0) is found on the controller ahd0 on bus 0, as target 0 with the LUN 0. SCSI controllers can have multiple buses and multiple targets. Further down, you can see that the disk is a MAXTOR 73GB SCSI-3 disk.

kernel: da0 at umass-sim0 bus 0 target 0 lun 0
kernel: da0: <Verbatim Store 'n' Go 1.30> Removable Direct Access SCSI-2 device
kernel: da0: 40.000MB/s transfers

kernel: da0: 963MB (1974271 512 byte sectors: 64H 32S/T 963C)

If you are using a USB flash disk for storing the configuration information, it will most likely appear in the log file as a type of SCSI disk. The above example shows a 1GB Verbatim Store 'n' Go disk.

kernel: lnc0: <PCNet/PCI Ethernet adapter> irq 18 at device 17.0 on pci0

kernel: lnc0: Ethernet address: 00:0c:29:a5:9a:28

Another important device that needs to work correctly on your system is the network interface card. Like disk controllers and disks, it will be logged in the log file when FreeBSD recognizes it. Above is an example of an AMD Lance/PCNet-based Ethernet adapter. Each Ethernet card has a unique address know as the Ethernet address or the MAC address. It is made up of 6 numbers specified using a colon notation. Once found, FreeBSD queries the card to find its MAC address and logs the result. In the above example, it is "00:0c:29:a5:9a:28".

Converting between Device Names and the Real World

In the SCSI example above, the SCSI controller listed is ahd0. The trick to understanding these log entries better is to know how to interpret the device name ahd0. First of all ahd0 means it is a device using the ahd driver and it is the first one in the system (with numbering starting from 0).

So what is a ahd? The first place to look is further up in the log file. There should be an entry like:

kernel: ahd0: <Adaptec 39320 Ultra320 SCSI adapter> irq 11 at device 1.0 on pci2

This shows that the particular device is an Adaptec 39320 SCSI 3 controller. You can also find out more about the the ahd driver (and all FreeBSD drivers) at: http://www.freebsd.org/releases/6.2R/hardware-i386.html Search for ahd and you will find which controllers this driver supports (in this case, they are all controllers from Adaptec. If you click on the link provided, you will be taken to a specific help page about this driver. When FreeNAS moves to FreeBSD 7, then the relevant web page will be: http://www.freebsd.org/releases/7.0R/hardware.html.

Configure and manage a network attached storage solution Turn a PC into a Network Attached Storage server with FreeNAS Configure, manage, and troubleshoot your FreeNAS installation Up to date with the latest FreeNAS release Includes a comprehensive troubleshooting section  http://www.packtpub.com/learning-freenas/book

Networking Problems

Having reliable networking with Network Attached Storage is, of course, essential. This section is aimed at those who are having networking problems.

In general, networking problems can be categorized into two: Physical problems and software problems. In the physical category are such problems as broken cables and faulty hardware and in the software category problems range from device driver problems to network configuration errors.

On the surface, most network problems appear the same, the network doesn't work, you can't connect. To hunt down the problem there are different things you can try.

General Connection Problems

Having booted your FreeNAS server, the first step is to configure the networking. This is covered in detail in tutorial 2. Assuming you have configured the networking, the next step is to connect to the FreeNAS server using a web browser. If the web browser connection fails, the web browser (Firefox in this case) will show something like this:

There are two machines involved in this operation, one is the FreeNAS server and the other is the computer with the web browser. First, test that the machine with the web browser is working normally. Make sure it can access the Internet as well as other machines or devices on your network. If it can't, you need to start there!

If the web browser machine is working then the problem could be with the FreeNAS server, it is time to find the problem. Use this checklist to make sure everything is as it should be:

• Check that the cable is plugged into the network card of the FreeNAS server. It might sound simple, but I have had cases where the cable didn't click in quite correct and so the networking didn't work. Assuming that the other end of the cable is also plugged in correctly, check that the link LED lights on the FreeNAS server and also lights on the switch or hub where it is connected. If you get green lights all round then things are good.
• Check that the network was recognized during the boot up of FreeNAS server. As you can't access the web interface you will need to check this on the console. You are looking for something like this:

  

  What If My Network Card Isn't Found?
  This probably means that the network card in your machine isn't supported by FreeNAS or more specifically by FreeBSD. Check the FreeBSD hardware compatibility page for more information: http://www.freebsd.org/releases/6.2R/hardware-i386.html.

• If the cabling is right and the network is found, it is time to double check the network settings. The most common mistake is in setting the subnet mask. The subnet mask determines which network a machine is on. When you set the subnet mask using the console menu system, if the subnet mask isn't the same as the machine with the web browser, then connectivity can be lost. Also, incorrect subnet masks can cause odd behavior, for example the machine with web browser might be able to connect to the FreeNAS server but the server may not be able to reply. A typo of 25 (instead of the desired 24) can cause connection problems.
• Is the IP address unique? If you have set a static IP address, have you set the right one? If two machines are assigned the same IP address, strange things can happen. With packets of data destined for one machine arriving at the other and so on. Ensure the IP address is correct and that it is unique.
• If you are using DHCP, ensure that the DHCP server is up and running and it has free addresses to issue. If you see that the FreeNAS server has the address 0.0.0.0, it means that it was unable to get an IP address from the DHCP server.

  

Using Ping

Most operating systems (including Windows, Linux, and OS X) have a tool called ping that allows you to test the low level connectivity between two machines. To test the connection to the FreeNAS machine, you can ping it and also you can get the FreeNAS server to ping other machines.

On Windows XP to run ping, click Run... and then type cmd in the dialog box. Press Enter and a black command prompt will appear. The command to ping is simply ping followed by the IP address. So to ping the FreeNAS server on its standard address of 192.168.1.250 you would type:

ping 192.168.1.250

If all is well, then the FreeNAS server will reply. The output should look something like this:

If the PC can't contact the FreeNAS server then it will report "Request timed out" errors.

You can also use the FreeNAS server to ping other machines on the network. From the console choose option 5. You will then be asked to enter an IP address. The IP address this time won't be that of the FreeNAS server, as this command is running from the FreeNAS server, but rather of another machine on the network. Enter the address and press enter. The output should be something like this:

Discovering the IP Address of an XP Machine
To find out the IP address of an XP machine: Click Start then Run... In the dialog box type cmd and press ENTER. In the command prompt window, type ipconfig /all. The resulting output will reveal the IP address along with the subnet mask and so on.

Using Ping from within the Web Interfaces

If you have access to the FreeNAS server's web interface and you want to ping another machine to ensure that the FreeNAS server can contact it, then go to Diagnostics: Ping. Enter the IP address of the machine you wish to ping in Host and click Ping.

The output for a successful ping will look like this:

PING 192.168.1.110 (192.168.1.110): 56 data bytes
 64 bytes from 192.168.1.110: icmp_seq=0 ttl=64 time=1.148 ms
 64 bytes from 192.168.1.110: icmp_seq=1 ttl=64 time=0.697 ms
 64 bytes from 192.168.1.110: icmp_seq=2 ttl=64 time=0.638 ms
 
 --- 192.168.1.110 ping statistics ---
 3 packets transmitted, 3 packets received, 0% packet loss
 round-trip min/avg/max/stddev = 0.638/0.828/1.148/0.228 ms

A failed ping will look like this:
 PING 192.168.1.100 (192.168.1.100): 56 data bytes
 
 --- 192.168.1.100 ping statistics ---
 3 packets transmitted, 0 packets received, 100% packet loss

Notice the 100% packet loss, which means that none of the pings got to their destination.

If you have multiple network cards in your machine, you can select which card to use in the Interface drop down box. This can be very useful when you want to check the connectivity on a second Ethernet interface card.

If you want to see which route the network packets are taking (through which switches and gateways), you can use the traceroute tool which is on the Traceroute.

Using ARP Tables to Solve Network Problems

Deep down in the murky depths of internals of Ethernet is a thing known as ARP. The Address Resolution Protocol (ARP) is the standard method for finding a host's hardware address (or MAC address) when only its IP address is known. To send a packet of information to an IP address, the networking software needs to know the physical address of the network card which is serving that IP address. To do this, it sends out and ARP request and gets an ARP response with the MAC address in it.

On the Diagnostics: ARP tables page, there is a table of all the MAC addresses known to the FreeNAS server along with the corresponding IP address. This table is useful as it allows you to see if the networking layers of the FreeNAS server where we are able to at least discover the MAC address of the remote server/PC in question. If the MAC address is known but ping fails, then the problem could be a firewall.

This table is also useful to make sure that two machines don't have the same IP address by mistake. You can tally the IP address with the MAC address to be certain that the right server has been contacted.

What is a MAC Address?
A Media Access Control address (MAC address) or Ethernet address is a globally unique identifier attached to most network adapters by the manufacturer. It can be likened to the postal address of your house, which is unique, so that all things sent to that address are received there by you. The address is made up of 6 numbers specified using a colon notation. An example MAC address is "00:0c:29:a5:9a:28". http://www.coffer.com/mac_find/ is a useful website to find out which manufacturer (or vendor as they are known) created any given MAC address.

Gigabit Transfers are Slow

Gigabit networking isn't the silver bullet that some people think it is. Unfortunately, you are never going to get 125MB/s (1000 divided by 8) transfer rates between FreeNAS and your PC. There are several reasons for this, some are theoretical and some practical. To get the best transfers rates over Gigabit Ethernet, to and from your FreeNAS server, you need to consider the following:

• First of all, make sure that you have enabled the tuning of some kernel variables. You will find this option on the System: Advanced: Advanced Setup page. Tick the Tuning box and save the settings.
• Make sure your Gigabit Ethernet switch and your Gigabit Ethernet card can handle Jumbo frames. Without Jumbo frame support, the increases over 100Mb/s networking will be marginal.
• Use a PCI Express Gigabit network card when possible.
• Ensure that your CPU is able to handle the load placed on it by the networking and the disk activity.
• Configure your disks in the optimum configuration for speed throughput. This almost certainly means you need to be using some kind of RAID, either hardware or software.

Problems Connecting to Shares (via CIFS)

One of the most popular ways to connect to the FreeNAS server is via the Windows protocol CIFS. In the majority of environments, people will connect to FreeNAS from a Windows machine and probably mount the 'share' as a network drive on the machine.

There are many versions of Microsoft Windows in use today with some people still using Windows 98 or Millennium while others are using Windows 2000, Windows XP or Windows Vista. Each version of Windows has slightly different ideas about networking and a setup which works with Windows 2000 or Windows 98 might not work with Windows Vista.

Here are some problems might be encountered when using Microsoft Windows with FreeNAS.

Windows Vista Asks for My Username and Password for Anonymous Shares

This can happen when the Null passwords field is enabled on the Services: CIFS/SMB page (in the advanced settings section). To resolve the problem, disable the Null password field.

There are Two FreeNAS Servers on the Network, but Windows Can only See One

During the configuration of these two FreeNAS servers, you probably made sure that each machine had its own IP address and hostname. You also need to make sure that the default name for the CIFS server in FreeNAS isn't the same for both of your servers. Although you can connect to their web interfaces without any problems when you try to use Windows networking, only one machine will be seen.

To solve this, go to Services: CIFS/SMB and set the NetBiosName name to something unique for each server.

Turning On Logging to Help Solve Windows Networking Problems

If you find that you have a Windows networking (CIFS) problem that you can't solve then try enabling more logging for the Windows networking server (called Samba) and see if the log files reveal any hints to the problem could be.

To enable logging, go to Services: CIFS/SMB and set the Log Level to Normal rather than Minimal. The logging created by Samba will be visible on the Diagnostics: Logs page under the Daemon tab. Entries starting with smbd are about the Windows networking server and those starting with nmdb are about the Windows networking name service.

If the Normal logging level doesn't help then you can try the Full level and even the Debug level but these will provide lots of output that you might find difficult to comprehend.

If the logging is producing lots of log entries, see the "50 Log entries might not be enough" tip in the Understanding Diagnostics: Logs: System section earlier in this tutorial. Also remember that you can clear the logs at any time by pressing the "Clear log" button on any of the Diagnostics: Logs pages.

Don't forget to turn the logging back to Minimal when you are done.

Configure and manage a network attached storage solution Turn a PC into a Network Attached Storage server with FreeNAS Configure, manage, and troubleshoot your FreeNAS installation Up to date with the latest FreeNAS release Includes a comprehensive troubleshooting section  http://www.packtpub.com/learning-freenas/book

Diagnostics: Information

As well as the Diagnostics: Logs page, there is also an information page in the Diagnostics: section. The Logs concentrates on the output from the various system services in the FreeNAS server where as the information page aims to provide status and configuration information about a variety of sections of the FreeNAS server.

To access this page, go to Diagnostics: Information. At the top, there are lots of tabs. Here is what they mean:

Replacing a Failed Hard Drive in a RAID Set

The key feature of RAID is the capability of the RAID set to continue working when one of the drives fails. When a drive fails, the RAID set enters a DEGRADED state, meaning the system is working but one of the drives has failed. This drive needs to be replaced as soon as possible.

To check the status of your RAID disks, go to Disks: Software RAID. Click on all the RAID types you have on your FreeNAS server and check that they are in the COMPLETE state. If a RAID set is in the DEGRADED state, you need to replace a disk.

Here is an example of a RAID 1 set in the degraded state:

The first thing you need to know is which disk to replace. Go to Disk: Management page and check the status of each disk. If the disk is healthy, it should have the status of ONLINE. Once you have found the broken disk, you need to replace it. Some disk controllers have the ability to hot swapping, which means that the disk can be unplugged and a new disk added while the system is running. This is often a feature (but not exclusively) of SATA controllers. If you are unsure, then shutdown the FreeNAS server and replace the disk with server switched off.

Rebuilding a RAID 1 Array After Disk Failure

Now that you have replaced the disk, it is time to bring the system up to normal operations levels. This involves 2 steps that are all performed in the tools section of the RAID 1 tab on the Disks: Software RAID page.

The first step is to tell the FreeNAS server to forget any devices that were in this RAID set but are now no longer connected.

1. Go to Disks: Software RAID and click on the RAID 1 tab.
2. Click on the Tools tab.
3. Select the name of the degraded array in the Volume Name field and select the Command forget. The Disk field isn't used with the forget command.
4. Click Send Command! and the array will be told to forget about non-connected devices.
5. The output from the command should read Done.

The second step is to insert the new disk back into the array. Remaining in the Tools tab of the RAID 1 array:

1. Select the degraded array in the Volume Name field.
2. Select the new disk from the Disk drop down box.
3. Select the insert command and click Send Command!
4. The output from the command should read Done.

The array will now start rebuilding. You can check that it is rebuilding on the Diagnostics: Information: Software RAID page (watch the Synchronized: field for percentage completed) and the Manage RAID tab on the Disks: Software RAID: RAID1: page will show COMPLETE when the RAID is rebuilt.

Rebuilding a RAID 5 Array After Disk Failure

Having replaced the disk, you are ready to synchronize it with the other disks in the RAID set. This involves a single step that is performed in the tools section of the RAID 5 tab on the Disks: Software RAID page.

You need to insert the new disk back into the array. In the Tools tab of the RAID 1 array:

• Select the degraded array in the Volume Name field.
• Select the new disk from the Disk drop down box.
• Select the insert command and click Send Command!
• The output from the command should read Done.

The array will now start rebuilding. You can check that it is rebuilding on the Diagnostics: Information: Software RAID page (watch the Synchronized: field for percentage completed) and the Manage RAID tab on the Disks: Software RAID: RAID 5: page will show COMPLETE when the RAID is rebuilt.

Where to Go for More Help

If you find yourself really stuck and are not able to track down your problem then the best place for support is the online FreeNAS Help Forums.

These are found at the SourceForge.net site and you need to sign up for a free SourceForge.net account to be able to post to the forums.

To find the support forum, go to the FreeNAS website at http://freenas.org and click on the Support link. From there, go to the Help forum.

Summary

In this article, we have looked at some of the tools that are available to help track down problems with the configuration of the FreeNAS server. We have looked at networking problems and possible solutions along with how to handle RAID failures.

If you have read this article you may be interested to view :

• Installation of FreeNAS
• Local User Management in FreeNAS

Configure and manage a network attached storage solution Turn a PC into a Network Attached Storage server with FreeNAS Configure, manage, and troubleshoot your FreeNAS installation Up to date with the latest FreeNAS release Includes a comprehensive troubleshooting section  http://www.packtpub.com/learning-freenas/book

About the Author

Gary Sims is a freelance Linux/FreeBSD consultant and writer from the UK and has been working with open-source software since the mid 1990s. He first saw Linux while completing his degree in Business Information Systems at Portsmouth University. Then while working for Digital Equipment Corp he came in to contact with DEC's Ultrix and later Digital UNIX (formerly OSF/1). While developing enterprise software for DEC on its UNIX platforms he became more and convinced of the benefits of open source and open-source Unix-like operating systems like Linux and FreeBSD. After leaving DEC he became a software contractor and after moving to Romania in 2003 he became a freelance Linux/FreeBSD consultant and writer and started publishing articles for the Open Source Technology Group (owners of Linux.com and SourceForge.net). This then led to him writing his first book with Packt.