Chapter 5. Sitemaps

As smart as the Google spider is, it's possible for them to miss pages on your site. Maybe you've got an orphaned page that isn't in your navigation anymore. Or, perhaps you have moved a link to a piece of content so that it's not easily accessible. It's also possible that your site is so big that Google just can't crawl it all without completely pulling all your server's resources—not pretty!

The solution is a sitemap. There are three different kinds of sitemaps that we're going to cover in this chapter, each with a different purpose:

• XML sitemaps are designed to be easily used by search engines

• URL-list sitemaps simply list every URL in your site

• Visitor-facing sitemaps create a nice interface for your site visitors to help them easily find the content on your site that they're looking for

In this chapter, we shall cover:

• What sitemaps are and why you should use them

• How to install sitemaps on your Drupal site

• How to submit the XML sitemaps to Google

• Other things you can...

XML sitemaps

In the early 2000s, Google started supporting XML sitemaps. Soon after Yahoo came out with their own standard and other search engines started to follow suit. Fortunately, in 2006, Google, Yahoo, Microsoft, and a handful of smaller players all got together and decided to support the same sitemap specification. That made it much easier for site owners to make sure every page of their web site is crawled and added to the search engine index. They published their specification at http://sitemaps.org. Shortly thereafter, the Drupal community stepped up and created a module called (surprise!) the XML sitemap module. This module automatically generates an XML sitemap containing every node and taxonomy on your Drupal site. Actually, it was written by Matthew Loar as part of the Google Summer of Code. The Drupal 6 version of the module was developed by Kiam LaLuno. Finally, in mid-2009, Dave Reid began working on a version 2.0 of the module to address performance, scalability, and reliability...

Specifying the XML sitemap priority for nodes

Now that you have the XML sitemap module properly installed and configured, you can start defining the priority of the content on your site—by default, the priority is .5. However, there are times when you may want Google to visit some content more often and other times when you may not want your content in the sitemap at all (like the comment or contact us submission forms).

Each node now has an XML sitemap section that looks like the following screenshot:

You can adjust the priority on a node-by-node basis by changing the default. You can even omit nodes from the sitemap by selecting Not in site map.

Submitting your XML sitemap to Google

Carry out the following steps in order to submit your XML sitemap to Google:

1. 1. If you have not already done so, you need to verify your web site with Google Webmaster Tools. Refer to Chapter 1, The Tools You'll Need, for details.

2. 2. Now point your browser to Google's Webmaster Tools at http://www.google.com/Webmasters/. Click on Sign in to Webmaster Tools, as shown in the following screenshot:

1. 3. You should see a list of your sites. Click on the Add link in the Sitemap column, located to the right of your site link, as shown in the following screenshot:

1. 4. Double-check that your sitemap is working.

2. 5. Copy and paste your sitemap URL (http://www.yourDrupalsite.com/sitemap.xml) into the blank space provided and click on Submit Sitemap. NOTE: If you get an error in Google, try tweaking your URL by adding ?q= after the /, as follows:

   http://www.yourDrupalsite.com/?q=sitemap.xml)

1. 6. You should see a confirmation message that looks like this:

1. 7. Now wait for...

Google News XML Sitemap

Google has created one of the most popular news sources on the Internet just by collecting and organizing news articles from other sites. It's called Google News and if you are running a news web site then you know how powerful it can be for picking up your comment. One front page article can generate 50,000 or more visitors in an hour or two. To be listed in Google News takes more than luck. You need to write great content and proactively seek to create timely and news-worthy content. If you've done that and you're still not showing up in Google news then it's time to create a Google News XML Sitemap. The Google News sitemap generator module was originally created by Adam Boyse at Webopius and is being maintained by Dave Reid. Thanks to both of you!

Setting up the Google News sitemap generator module

1. 1. Download the Google News sitemap module from http://drupal.org/project/googlenews and install it just like a normal Drupal module. See Chapter 1, The Tools You'll...

URL list

The XML Sitemap is the ideal choice because it allows you to specify a lot of information about the content of your site. But, say for some reason that you can't install an XML Sitemap. Maybe there's a conflict with another module that you just have to have. Perhaps your server doesn't have the power to handle the large overhead that an XML sitemap needs for large sites. Or, possibly you want to submit a sitemap to a search engine that doesn't support XML yet.

Well, there is an alternative. It's not as robust but it is a functional, albeit rudimentary, solution. Just make a list of every URL in your site and put the list in one big text document with one URL on each line. Too much work, you say? Good thing there is a Drupal module that does all the work for you. It's called the URL list module. It's maintained by David K. Norman. Thank you, David!

Setting up a URL list sitemap

1. 1. Download the Sitemap module from http://drupal.org/project/urllist and install it just like a normal Drupal...

Visitor-facing sitemaps

XML Sitemaps are great for search engines but as you can see, they're not user-friendly at all. Some of your site visitors will want to see all of the pages or sections available to them on your web site. That's where a Visitor-facing sitemap comes in handy. Fortunately, there is a Drupal module that will do that for you automatically! It's called the Site map module. Not only does it show you a nice overview of your site but it can show the RSS feeds too. Everybody raise a glass to Nic Ivy and Fredrik Jonsson, respectively the original author and current maintainer of this module. Cheers, gentlemen!

Setting up a visitor-facing sitemap

Carry out the following steps to set up a visitor-facing sitemap:

1. 1. Download the Sitemap module from http://drupal.org/project/ site_map and install it just like a normal Drupal module. See Chapter 1, The Tools You'll Need, for step-by-step module installation instructions.

2. 2. Go to http://www.yourDrupalsite.com/admin/settings/sitemap...

Summary

In this chapter, we discussed the origin of sitemaps and how they're used to make sure your entire site is crawled by the search engines. We also talked about how to make a user-friendly sitemap for your site visitors. In this chapter, we have covered the following topics:

• XML sitemaps

• URL-list sitemap

• Google News XML sitemaps

• Visitor-facing sitemaps

In the next chapter, it's time to get technical with your SEO. Get ready to face the robots!