 |
|
| Using the Sitemap Generator |  |  | Contents Before you begin
The Google Sitemap Generator is a Python script that creates a Sitemap for your site using the Sitemap Protocol. This script can create Sitemaps from URL lists, web server directories, or from access logs. In order to use this script:
If you aren't sure about any of this, you can check with your web hosting company. Now you're ready to get started. Here's an overview of what you'll need to do.
If you are unable to use the Sitemap Generator, you can add a Sitemap to your Google webmaster tools account in another format, such as a simple text file.You can also find links to third-party programs that support Google Sitemaps here. For News Sitemaps: The Sitemap Generator is not recommended for use in creating Google News Sitemaps at this time, due to the special requirements of News Sitemaps. News Sitemaps are intended to be dynamic lists of only the most recently published news articles (rather than the entire website), and they are updated frequently.
The Sitemap Generator files are available in ZIP and GZ archive formats from the following location: http://sourceforge.net/project/showfiles.php?group_id=137793&package_id=153422 Once you download the archive, extract it into a local directory. Locate the following files:
This section provides step-by-step instructions for creating a configuration file. It also provides a complete reference of the options available. If you are creating Mobile Sitemaps, see the additional mobile guidelines. In order to create a configuration file for your site, you must have the following information:
Create the configuration file as follows:
<site base_url="http://www.example.com/" store_into="/var/www/docroot/sitemap.xml.gz" verbose="1"> <!-- ** MODIFY or DELETE ** "url" nodes specify individual URLs to include in the map. <br> Required attributes: href - the URL Optional attributes: lastmod - timestamp of last modification (ISO8601 format) changefreq - how often content at this URL is usually updated priority - value 0.0 to 1.0 of relative importance in your site --> <url href="http://www.example.com/stats?q=name" /> <url href="http://www.example.com/stats?q=age" lastmod="2004-11-14T01:00:00-07:00" changefreq="yearly" priority="0.3" /> This section gives two examples: the first includes only the required attribute and the second includes the required attribute as well as the optional attributes. Use this format for each of the URLs you want to include. The changefreq attribute gives Google a general idea of how often the URL is updated. This helps Google know how often to visit the page for new content. The priority attribute gives Google information about the relative importance of this page compared to the other pages of your site. This attribute has no effect on how Google compares your page to pages on other sites, it just helps Google know which pages of your site that you think are most important. Locate the following section: <!-- ** MODIFY or DELETE ** "urllist" nodes name text files with lists of URLs. An example file "example_urllist.txt" is provided. Required attributes: path - path to the file Optional attributes: encoding - encoding of the file if not US-ASCII --> <urllist path="example_urllist.txt" encoding="UTF-8" /> Use this format to point to the path and name of the text file that contains your list of URLs. You can use the provided example_urllist.txt file as a template for that text file. You can specify either a relative or complete path to your web server. For instance, if the Sitemap Generator and urlist.txt file are located in the same directory, you can simply specify the filename of the .txt file, If you create a text file with an encoding other than UTF-8, you can use the encoding attribute to indicate this encoding. If you have multiple .txt files, you can use wildcards. For instance: <urllist path="example_urllist*.txt" encoding="UTF-8" /> For each URL you include in the text file, you can specify the last modification date, change frequency, and priority. See the URLlist text file reference section for complete information about the structure of this file. Locate the following section: <!-- ** MODIFY or DELETE ** "directory" nodes tell the script to walk the file system and include all files and directories in the Sitemap. Required attributes: path - path to begin walking from url - URL equivalent of that path Optional attributes: default_file - name of the index or default file for directory URLs --> <directory path="/var/www/icons" url="http://www.example.com/images/" /> This section gives two examples. If all of your pages are contained in subdirectories of one path, then you only need to include one entry. However, if you have multiple paths to pages on your site, include an entry for each. Remember that each URL must begin with the base URL you specified in step 3. For instance, the examples given in the example_config.xml file both have URLs that begin with http://www.example.com/. Therefore, both URLs are valid. Replace the example entries with entries for your site. Many sites will only have one entry that points to the base URL. Ensure that path value is the complete path to the directory on your web server. Ensure that the url value is the complete URL, including the protocol (such as http) and a trailing slash, if required. You can use the default_file parameter to specify the filename that your server uses as the default page for a directory. In the above example, /var/www/docroot resolves to http://www.example.com/index.html. You are not required to specify this. However, if you do, the Sitemap Generator will include the page that maps to each subdirectory only once (rather than list both the directory URL and filename URL) and will use the last modified date of the file (rather than the directory) to extract the lastmod attribute for that page. Locate the following section: <!-- ** MODIFY or DELETE ** "accesslog" nodes tell the script to scan webserver log files to extract URLs on your site. Both Common Logfile Format (Apache's default logfile) and Extended Logfile Format (IIS's default logfile) can be read. Required attributes: path - path to the file Optional attributes: encoding - encoding of the file if not US-ASCII --> <accesslog path="/etc/httpd/logs/access.log" encoding="UTF-8" /> <accesslog path="/etc/httpd/logs/access.log.0" encoding="UTF-8" /> <accesslog path="/etc/httpd/logs/access.log.1.gz" encoding="UTF-8" /> This section gives three examples. You should replace these entries and include an entry for each log file. Ensure that the path value is the complete path and filename on your web server. If the log files are not encoded as US-ASCII or UTF-8, then use the optional encoding attribute to specify the encoding. Rather than list each log file, you can use wildcards. For instance, in the above example, you could include the following entry that would include all three log files: <accesslog path="/etc/httpd/logs/access.log*" encoding="UTF-8" /> The Sitemap Generator assigns priority to URLs it finds in the logs based on how often each URL is accessed. For instance, a URL that has been accessed 100 times will be given a higher priority than a URL that has been accessed twice. The actual priority assignment is relative and depends on each URL as compared to other URLs in the site. Locate the following section: <!-- ** MODIFY or DELETE ** This section gives one example. You should replace this entry and include an entry for each Sitemap you want to include. Ensure that the path value is the complete path and filename on your web server. You can list gzipped Sitemaps as well, as long as they have a .gz extension. Rather than list each Sitemap, you can use wildcards. For instance, the following entry would include any Sitemaps that begin with the word "sitemap" and have an .xml extension: <sitemap path="/var/www/docroot/subpath/sitemap*.xml" /> The Sitemap Generator extracts all URLs and the optional data listed for each URL for every Sitemap you list and creates one Sitemap with this information. At this time, we can't guarantee that this method will work Sitemaps created with tools other than the Sitemap Generator. <!-- ******************************************************** FILTERS Filters specify wild-card patterns that the script compares against all URLs it finds. Filters can be used to exclude certain URLs from your Sitemap, for instance if you have hidden content that you hope the search engines don't find. Filters can be either type="wildcard", which means standard path wildcards (* and ?) are used to compare against URLs, or type="regexp", which means regular expressions are used to compare. Filters are applied in the order specified in this file. An action="drop" filter causes exclusion of matching URLs. An action="pass" filter causes inclusion of matching URLs, shortcutting any other later filters that might also match. If no filter at all matches a URL, the URL will be included. Together you can build up fairly complex rules. The default action is "drop". The default type is "wildcard". You can MODIFY or DELETE these entries as appropriate for your site. However, unlike above, the example entries in this section are not contrived and may be useful to you as they are. ********************************************************* --> <!-- Exclude URLs that end with a '~' (IE: emacs backup files) --> <filter action="drop" type="wildcard" pattern="*~" /> <!-- Exclude URLs within UNIX-style hidden files or directories --> <filter action="drop" type="regexp" pattern="/\.[^/]*" /> You can use filtering to exclude specific URLs from the generated Sitemap. You might want to do this to create a cleaner list, to reduce redundant listings, or to keep certain URLs from being indexed. Note that if you use a robots.txt file to keep URLs from being indexed, then even if the URLs are included in your Sitemap, Google will not search or index them. You can use any or all of the filtering methods. You can delete the entries you don't need and can create additional entries, if desired. Below are sample usages. <filter action="drop" type="wildcard" pattern="*.jpg" /> This filter excludes URLs that end in .jpg. You might want to include a similar filter if all of your site's images are embedded within HTML pages and should not be accessed as standalone URLs. <filter action="pass" type="wildcard" pattern="*.htm*" /> <filter action="drop" type="wildcard" pattern="*" />This filter includes all .htm* files but excludes everything else. Config File Syntax Reference A complete explanation of the config file syntax is below. Each tag begins with a code sample, followed by a description of the attributes. site <site base_url="http://www.example.com/" store_into="/var/www/html/sitemap.xml.gz" verbose="1" supress_search_engine_notify="1" default_encoding="UTF-8">
url <url href="http://www.example.com/stats?q=age" lastmod="2004-11-14T01:00:00-07:00" changefreq="yearly" priority="0.3" />
urllist <urllist path="/var/www/html/urllist.txt" encoding="UTF-8" />
The urllist.txt file is a simple text file containing a list of URLs to map. You can also include optional attributes for each URL. Attributes are entered on the same line as the URL and are separated by a single space. For example: http://www.example.com/abc/something http://www.example.com/abc/xyy.pdf lastmod=2001-12-31T14:05:06+00:00 http://www.example.com/abc/def?x=12&y=23 changefreq=weekly priority=0.3
directory --> <directory path="/var/www/icons" url="http://www.example.com/images/" />
accesslog <accesslog path="/etc/httpd/logs/access-0.log" encoding="UTF-8"/>
sitemap <sitemap path="/var/www/docroot/subpath/sitemap.xml" />
filter <filter action="drop" type="wildcard" pattern="*/internal/*" />
Files referenced by your configuration file, either URL lists or web server logs, can use encodings other than the default UTF-8. You can specify alternate encodings in config.xml to affect how the Sitemap Generator reads your files. Some common encodings are:
You create a configuration file for a Mobile Sitemap in the same way as for a non-mobile Sitemap. However, you must create a separate config file for each markup language and run the Sitemap Generator with each config file separately so that you create a separate Sitemap for each. Each config file must:
Examples of filtering Below are some examples of how you can use extension-based filters to generate Mobile Sitemaps for different markup languages. The specific filtering you use should be based on the types of markup languages used in your site, and how you specify each type. If you have implemented the details of your site differently (for instance, you may organize URLs with different markup languages in separate folders), you should filter based on the specifics of your site implementation. Remember that filters are applied in the order you list them in the config file. So, the first filter you should list is a "pass" action that specifies the URLs you want to include in the Sitemap. To create a Sitemap for WML (WAP 1.2) content: <filter action="pass" type="wildcard" pattern="*.wml" /> <filter action="drop" type="wildcard" pattern="*.*" /> To create a Sitemap for XHTML mobile profile (WAP 2.0) content: <filter action="pass" type="wildcard" pattern="*.xhtml" /> <filter action="drop" type="wildcard" pattern="*.*" />
You should upload the following files to your web server in a location you can access from a command line:
The method you use to upload these files depends on your environment. Common methods include FTP and SCP. For more information, contact your web host.
In order to run the Sitemap Generator, you'll need to connect to your web server. The method you use to connect depends on your environment. For instance, you can generally access a UNIX-based server using SSH. For more information on connecting to your web server and running scripts, talk to your web host. Once you have copied the files to your web server, you'll need to run the Sitemap Generator script. Connect to your web server and run the following command (replace <path/config.xml> with the path to and filename of your configuration file; if you have uploaded this file to the same location as the Python script, you can exclude the path): python sitemap_gen.py --config=<path/config.xml> For instance, a UNIX-based command line might look similar to this:
A MS-DOS-based command shell might look similar to this:
Tip: If you're testing your configuration and are not ready to submit your Sitemap, the following syntax will prevent Sitemap Generator from contacting Google:
You'll see the status of your request in the command prompt: Reading configuration file: /path/config.xml Opened URLLIST "/path/urllist.txt" Walking DIRECTORY "/var/www/html/dir" Walking DIRECTORY "/var/www/html/dir2" Opened ACCESSLOG "/etc/httpd/logs/access-0.log" Sorting and normalizing collected URLs. Writing Sitemap file "/path/sitemap.xml.gz" with 1092 URLs Notifying search engines. Notifying www.google.com Count of file extensions on URLs: 208 .html 574 .jpg ... Number of errors: 0 Number of warnings: 0 If you don't see very much output like this, remember that the verbose setting in your configuration file affects how much information is printed on the screen. This example is representative of setting verbose to "1". Any errors in the file will also be returned. For instance, if you leave the url= attribute off a directory entry, the script will output the following: [ERROR] Directory entries must have both "path" and "url" attributes Number of errors: 1 Correct any errors in your config.xml file and re-run the script. If no errors are present, the Sitemap Generator will create a new sitemap.xml.gz file in the location you specified in the config file.
The Sitemap Generator creates a sitemap.xml.gz file in the location you specified in the config file. Once this file is successfully created, make sure it is accessible through a web browser. Then, add it to your Google Sitemaps account. This enables Google to provide you with useful status and statistical information. If Google reports problems with your Sitemap, you can correct the problems and resubmit it. You only have to add the Sitemap manually once. After that, you can use an HTTP request to notify Google about changes to your Sitemap (although you can also resubmit it through your Google webmaster tools account).
We suggest setting up Sitemap Generator to run as often as your content is changed, to a maximum frequency of once per hour. Webmasters with a UNIX web server may consider setting this up as a cron job. Webmasters using other platforms should contact their system administrator for help in configuring recurring scripts. You may also benefit from peer advice in the Google Sitemaps Group on Google Groups. You can use an HTTP request to let Google know about changes to your Sitemap. However, please make sure that you log in to Google webmaster tools with your Google Account once to manually add your Sitemap to your Google webmaster tools account.
In this section we talk about some common questions or issues that some people run into while using Sitemap Generator, and what you can do if you come across one of these. Web-accessible Issue: Sitemap Generator returns the following error and warning: [ERROR] When attempting to access your generated Sitemap at the following URL: http://www.example.com/sitemap.xml.gz we failed to read it. Please verify the store_into path you specified in your configuration file is web-accessible. Consult the FAQ for more information. [WARNING] Proceeding to notify with an unverifiable URL. What just happened? Sitemap Generator created the file where you specified, then attempted to retrieve it using HTTP just as a search engine would, and failed. It went ahead and notified search engines anyway, but it's outputting the error and warning to let you know that your Sitemap may be not readable by search engines. Sitemap Generator creates your Sitemap file at the path specified in the store_into attribute of your config.xml file. Sitemap Generator then builds a URL to that file using the base_url attribute, and reports the URL to search engines. For instance, if you set the configuration to: <site base_url="http://www.example.com/" store_into="/var/www/html/sitemap.xml.gz"> Sitemap Generator will notify search engines to look for your Sitemap at: http://www.example.com/sitemap.xml.gz The file must be accessible through this URL. If the file can not be retrieved using this URL, search engines have no way of finding your Sitemap. What can you do? You need to verify that your config.xml is specifying the correct base_url and store_into paths for your Sitemap. You also need to confirm that web browsers can retrieve the Sitemap file off of the base_url. If you find errors in the path or URL, you may need to re-run Sitemap Generator, or just move the Sitemap file to the correct place and notify search engines manually through their web sites. Support for XML Issue: Sitemap Generator gives the following error: Some installs of Python 2.2 do not include complete support for XML. Please try upgrading your version of Python and re-running the script. What just happened? As the message says, some platforms have a version of Python (the language the Sitemap Generator script is written in) that is missing support libraries needed for processing XML files. This script requires full XML support in order to run. What can you do? Try upgrading your installed version of Python to a newer version. You will probably need to contact your system administrator to do this. Note that this just affects the Sitemap Generator, not Sitemaps overall. If you have another method or tool for creating Sitemaps, you can certainly use it and submit your Sitemaps to search engines. What are all the extra files in the .gz or .zip? When you extract the sitemap_gen.py script, you'll probably see quite a few more files than we refer to above. The full file list looks closer to this: AUTHORS ChangeLog COPYING example_config.xml example_urllist.txt PKG-INFO README setup.py sitemap_gen.py test_sitemap_gen.py The extra files tend to be information on the package and licensing terms. You are encouraged to look through these files. The one exception is test_sitemap_gen.py, which is a unit-test script that other developers may find useful if they wish to contribute to this open source project. If you intend to use Sitemap Generator without modifying any of the source code - this is the expected case for nearly everyone - you probably don't need this test script. If you are interested in helping with this project, please visit http://sourceforge.net/projects/goog-sitemapgen. You can also find links to some third party programs that support Google Sitemaps here.Issue: Your site domain name or URLs within it contain non-ASCII characters. Generally, non-ASCII URLs should be encoded using UTF-8 before being percent-escaped. However, some webservers only respond correctly if URLs are encoded using an encoding other than UTF-8. All URLs within your Sitemap, as well as the URL of the Sitemap itself must be encoded for readability by the web server on which they are located. Within the site definition section, use the optional default_encoding attribute to specify the encoding used by your webserver. If you don't use this tag and your webserver uses an encoding other than UTF-8, we can't know which encoding to use. If your URLs contain non-ASCII characters, we recommend that you run the Sitemap Generator script using Python 2.3 or higher. This version of Python has increased non-ASCII support. If your domain name contains non-ASCII characters, you must use Python 2.3 or later, as Internationalizing Domain Names in Applications (IDNA) support wasn't added until this version. Without IDNA support, the Sitemap Generator can't correctly encode a non-ASCII domain name. Last modified: 15 April 2006 | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
 | | |
--
Zhipeng Zhang (Alan) BCompSc MInfoTech MACS(Prov)
"You must be the change you want to see in the world."
"Begin at the beginning and go on till you come to the end; then stop."
-- Lewis Carroll, Alice in Wonderland
No comments:
Post a Comment