Extending Search Engine Optimization using Sitemaps in Drupal 6

Ben Finklea

September 2009

We have discussed some of the techniques earlier in Search Engine Optimization using Sitemaps in Drupal 6 article which mainly covered XML sitemaps.

The Google spider is quite smart, but then the possibility of them missing pages on your site is also quite high. You may have pages that are not in navigation anymore or you have moved a link to a piece of content so that it's not easily accessible. A possibility is also that your site is very big for Google to just crawl it all without completely pulling all your server's resources. The solution to this is sitemaps.

Google News XML Sitemap

Google has created one of the most popular news sources on the Internet just by collecting and organizing news articles from other sites. It's called Google News and if you are running a news web site then you know how powerful it can be for picking up your comment. One front page article can generate 50,000 or more visitors in an hour or two. To be listed in Google News takes more than luck. You need to write great content and proactively seek to create timely and news-worthy content. If you've done that and you're still not showing up in Google news then it's time to create a Google News XML Sitemap. The Google News sitemap generator module was originally created by Adam Boyse at Webopius and is being maintained by Dave Reid. Thanks to both of you!

Setting up the Google News sitemap generator module

Carry out the following steps to set up the Google News sitemap generator module:

  1. Download the Google News sitemap module from http://drupal.org/project/googlenews and install it just like a normal Drupal module.
  2. Go to http://www.yourDrupalsite.com/admin/settings/googlenews or go to your admin screen, and click the Administer | Site Configuration | Google News sitemap feed link. You'll see a screen similar to the following screenshot:

    Drupal 6 Search Engine Optimization

  3. Select the content types that you wish to show up in the news feed. If all your story content types are newsworthy, pick Story. Your blog or page content types are probably not a good fit and selecting them may hurt the chances of your content being approved by Google.
  4. Click on Save configuration.
  5. Point your browser to http://www.yourDrupalsite.com/googlenews.xml and double check that you can see the sitemap, as shown in the following screenshot:

    Drupal 6 Search Engine Optimization

Submitting your Google News sitemap to Google News

Once you've assembled your new articles for a single publication label, submit them to Google News sitemaps by carrying out the following steps:

  1. Check Google News to see if your site is already included. If not, you can request inclusion by visiting the following link, http://www.google.com/support/news_pub/bin/request.py?ctx=answer. The inclusion process may take up to a few weeks, and you'll only be able to submit a News sitemap once this process is complete.
  2. If your site is already showing up in Google News then proceed. If not, you should wait a couple of weeks and try again.
  3. Log in to Google Webmaster Tools by pointing your browser to http://www.google.com/webmasters/tools/.
  4. On the Webmaster Tools Dashboard, click on Add next to the site you want to submit.
  5. From the Choose type drop-down menu, select News sitemap, and then type the sitemap URL, in this case http://www.yourDrupalsite.com/googlenews.xml.
  6. In the list, select the publication label for the articles. You can select only one label for each sitemap.
  7. Click on OK.

URL list

The XML Sitemap is the ideal choice because it allows you to specify a lot of information about the content of your site. But, say for some reason that you can't install an XML Sitemap. Maybe there's a conflict with another module that you just have to have. Perhaps your server doesn't have the power to handle the large overhead that an XML sitemap needs for large sites. Or, possibly you want to submit a sitemap to a search engine that doesn't support XML yet.

Well, there is an alternative. It's not as robust but it is a functional, albeit rudimentary, solution. Just make a list of every URL in your site and put the list in one big text document with one URL on each line. Too much work, you say? Good thing there is a Drupal module that does all the work for you. It's called the URL list module. It's maintained by David K. Norman. Thank you, David!

Setting up a URL list sitemap

  1. Download the Sitemap module from http://drupal.org/project/urllist and install it just like a normal Drupal module.
  2. Go to http://www.yourDrupalsite.com/admin/settings/urllist or go to your admin screen and click the Administer | Site Configuration | URL list link. You'll see the URL list screen, as shown in the following screenshot:

    Drupal 6 Search Engine Optimization

  3. You can adjust the settings to keep track of who accessed the URL list to submit your site to Yahoo! and to help you authenticate with Yahoo! However, you can leave all these settings untouched for now.
  4. Point your browser to http://www.yourDrupalsite.com/urllist.txt (http://www.yourDrupalsite.com/?q=urllist.txt if you don't have clean URLs installed) and you'll see your URL list sitemap, as shown in the following screenshot:

    Drupal 6 Search Engine Optimization

You can submit this sitemap to Google, Yahoo!, and many other search engines in lieu of an XML sitemap. Just follow the same steps as defined in the Submit your XML Sitemap to Google section above but use # http://www.yourDrupalsite.com/urllist.txt as the URL.

Remember to use http://www.yourDrupalsite.com/?q=urllist.txt if Google has problems with your URL.

Visitor-facing sitemaps

XML Sitemaps are great for search engines but as you can see, they're not user-friendly at all. Some of your site visitors will want to see all of the pages or sections available to them on your web site. That's where a Visitor-facing sitemap comes in handy. Fortunately, there is a Drupal module that will do that for you automatically! It's called the Site map module. Not only does it show you a nice overview of your site but it can show the RSS feeds too. Everybody raise a glass to Nic Ivy and Fredrik Jonsson, respectively the original author and current maintainer of this module. Cheers, gentlemen!

Setting up a visitor-facing sitemap

Carry out the following steps to set up a visitor-facing sitemap:

  1. Download the Sitemap module from http://drupal.org/project/site_map and install it just like a normal Drupal module.
  2. Go to http://www.yourDrupalsite.com/admin/settings/sitemap or go to your admin screen and click the Administer | Site configuration | Site Map link. You'll see the Site map, as shown on the following screenshot:

    Drupal 6 Search Engine Optimization

  3. In the Site map message box, put a nice message that describes your site map. This is a great place to put a couple of keywords that generally describe your site.
  4. The Site map content option will consist of the following:
    • Show front page: This option allows you to decide whether or not to display the front page of your site in the sitemap. It's a personal preference and doesn't have much affect on your SEO.
    • Show active blog authors: Shows or hides the ten most active blogs. Should be turned on for most sites that have blogs. If you don't have any blogs on your site, turn this option off.
    • Books to include in the site map: If you're using the Book module then you can include your book sections in your sitemap. If you do have some books on your Drupal site then you should select them.
    • Menus to include in the site map: Gives you control over which menus you should include in your sitemap. Be sure that only menus that you want visitors to see are selected here as you don't want to expose links to your admin or other non-visitor sections of your site. You probably should not turn on the Navigation menu.
    • Show FAQ content: If you're using the FAQ module, this will show the content of your FAQ. Turn it on.
    • Categories to include in the site map: This will allow you to show taxonomy terms in your sitemap. A great idea for most sites.
  5. Categories settings: It's a good idea to turn on as much content in the sitemap as possible. However, try to find a balance between lots of good links into your site for your visitors and overwhelming them with too much information. Keep in mind that this sitemap is primarily for visitors, not search engines.
    • Show books expanded: Shows all the sections and child pages of your book.
    • Show node counts by categories: This shows a number next to each taxonomy term showing how many items are in that term. This can be helpful information to your visitors.
    • Categories depth: Specify the number of subcategories that should be included under each category. Enter all to include all subcategories, 0 for no subcategories, or -1 to not append the depth of each category. This can be confusing at first so you may want to experiment with this setting to get it working just how you want it.
  6. The RSS settings will consist of the following:
    • Include RSS links: This setting will show little RSS icons next to any pages that have RSS feeds. This is a great idea if you want people to be able to subscribe to your site's content.
    • RSS feed depth: This setting allows you to specify how many subcategories should display the RSS symbol. Dozens or hundreds of RSS symbols can get annoying so use your best judgment based on your site's content. Not sure? Just leave it on all.
  7. Click on Save configuration.
  8. Double check to make sure that your sitemap looks correct, by pointing your browser to http://www.yourDrupalsite.com/sitemap. Your sitemap should look similar to the following screenshot:

    Drupal 6 Search Engine Optimization

  9. By default, your sitemap is only visible to the admin. You need to give permission to anonymous users to view the sitemap.
    • Visit the following link, http://www.yourDrupalsite.com/admin/user/permissions or navigate to Administer | User management | Permissions.
    • Next to access site map, check the box under anonymous user and authenticated user.

      Drupal 6 Search Engine Optimization

    • Click on Save permissions.

Use the Devel module
I used the Devel module to generate some dummy data to see how the Site map module would look. Don't use Devel on a live site unless you really know what you're doing! However, when developing your site on a test server, the Devel module can be invaluable. Check it out:http://drupal.org/project/devel. Devel is maintained by the seemingly-ubiquitous Moshe Weitzman. He's a true Drupal Rockstar. Thanks, Moshe!


In this article, we continued our discussion on how should the sitemaps be used to make sure your entire site is crawled by the search engines. We also talked about how to make a user-friendly sitemap for your site visitors. To summarize, we have covered the following types of sitemaps:

  • Google News XML sitemaps
  • URL-list sitemap
  • Visitor-facing sitemaps

If you have read this article you may be interested to view :

You've been reading an excerpt of:

Drupal 6 Search Engine Optimization

Explore Title