How web servers works

http://computer.howstuffworks.com/web-server4.htm

 

 

How web servers works

 

 

Inside This Article

1.  Introduction to How Web Servers Work

2.  The Basic Process

3.  Behind the Scenes

4.  The Internet

5. Clients and Servers

6.  IP Addresses

7.  Domain Names

8.  Name Servers

9.  Ports

10.  Protocols

11.  Putting It All Together

12.  Extras: Security

13.  Extras: Dynamic Pages

14.  Lots More Information

15. See all Internet articles

 

 

 

Introduction to How Web Servers Work

 

Have you ever wondered about the mechanisms that delivered this page to you? Chances are you are sitting at a computer right now, viewing this page in a browser. So, when you clicked on the link for this page, or typed in its URL ( uniform resource locator), what happened behind the scenes to bring this page onto your screen?

If you've ever been curious about the process, or have ever wanted to know some of the specific mechanisms that allow you to surf the Internet, then read on. In this article, you will learn how Web servers bring pages into your home, school or office. Let's get started!

 

The Basic Process

Let's say that you are sitting at your computer, surfing the Web, and you get a call from a friend who says, "I just read a great article! Type in this URL and check it out. It's at http://www.howstuffworks.com/web-server.htm." So you type that URL into your browser and press return. And magically, no matter where in the world that URL lives, the page pops up on your screen.

At the most basic level possible, the following diagram shows the steps that brought that page to your screen:

Your browser formed a connection to a Web server, requested a page and received it.

On the next page, we'll dig a bit deeper.

Behind the Scenes

If you want to get into a bit more detail on the process of getting a Web page onto your computer screen, here are the basic steps that occurred behind the scenes:

• The browser broke the URL into three parts:
  1. The protocol ("http")
  2. The server name ("www.howstuffworks.com")
  3. The file name ("web-server.htm")

• The browser communicated with a name server to translate the server name "www.howstuffworks.com" into an IP Address , which it uses to connect to the server machine.

• The browser then formed a connection to the server at that IP address on port 80. (We'll discuss ports later in this article.)

• Following the HTTP protocol, the browser sent a GET request to the server, asking for the file "http://www.howstuffworks.com/web-server.htm." (Note that cookies may be sent from browser to server with the GET request -- see How Internet Cookies Work for details.)

• The server then sent the HTML text for the Web page to the browser. (Cookies may also be sent from server to browser in the header for the page.)

• The browser read the HTML tags and formatted the page onto your screen.

If you've never explored this process before, that's a lot of new vocabulary. To understand this whole process in detail, you need to learn about IP addresses, ports, protocols... The following sections will lead you through a complete explanation.

The Internet

So what is "the Internet"? The Internet is a gigantic collection of millions of computers, all linked together on a computer network. The network allows all of the computers to communicate with one another. A home computer may be linked to the Internet using a phone-line modem, DSL or cable modem that talks to an Internet service provider ( ISP). A computer in a business or university will usually have a network interface card (NIC) that directly connects it to a local area network (LAN ) inside the business. The business can then connect its LAN to an ISP using a high-speed phone line like a T1 line. A T1 line can handle approximately 1.5 million bits per second, while a normal phone line using a modem can typically handle 30,000 to 50,000 bits per second.

ISPs then connect to larger ISPs, and the largest ISPs maintain fiber-optic "backbones" for an entire nation or region. Backbones around the world are connected through fiber-optic lines, undersea cables or satellite links (see An Atlas of Cyberspaces for some interesting backbone maps). In this way, every computer on the Internet is connected to every other computer on the Internet.

 

Clients and Servers

In general, all of the machines on the Internet can be categorized as two types: servers and clients. Those machines that provide services (like Web servers or FTP servers) to other machines are servers. And the machines that are used to connect to those services are clients. When you connect to Yahoo! at www.yahoo.com to read a page, Yahoo! is providing a machine (probably a cluster of very large machines), for use on the Internet, to service your request. Yahoo! is providing a server. Your machine, on the other hand, is probably providing no services to anyone else on the Internet. Therefore, it is a user machine, also known as a client. It is possible and common for a machine to be both a server and a client, but for our purposes here you can think of most machines as one or the other.

A server machine may provide one or more services on the Internet. For example, a server machine might have software running on it that allows it to act as a Web server, an e-mail server and an FTP server. Clients that come to a server machine do so with a specific intent, so clients direct their requests to a specific software server running on the overall server machine. For example, if you are running a Web browser on your machine, it will most likely want to talk to the Web server on the server machine. Your Telnet application will want to talk to the Telnet server, your e-mail application will talk to the e-mail server, and so on...

IP Addresses

To keep all of these machines straight, each machine on the Internet is assigned a unique address called an IP address. IP stands for Internet protocol, and these addresses are 32-bit numbers, normally expressed as four "octets" in a "dotted decimal number." A typical IP address looks like this:

          216.27.61.137 

The four numbers in an IP address are called octets because they can have values between 0 and 255, which is 2⁸ possibilities per octet.

Every machine on the Internet has a unique IP address. A server has a static IP address that does not change very often. A home machine that is dialing up through a modem often has an IP address that is assigned by the ISP when the machine dials in. That IP address is unique for that session -- it may be different the next time the machine dials in. This way, an ISP only needs one IP address for each modem it supports, rather than for each customer.

If you are working on a Windows machine, you can view a lot of the Internet information for your machine, including your current IP address and hostname, with the command WINIPCFG.EXE (IPCONFIG.EXE for Windows 2000/XP). On a UNIX machine, type nslookup at the command prompt, along with a machine name, like www.howstuffworks.com -- e.g. "nslookup www.howstuffworks.com " -- to display the IP address of the machine, and you can use the command hostname to learn the name of your machine. (For more information on IP addresses, see IANA.)

As far as the Internet's machines are concerned, an IP address is all you need to talk to a server. For example, in your browser, you can type the URL http://209.116.69.66 and arrive at the machine that contains the Web server for HowStuffWorks. On some servers, the IP address alone is not sufficient, but on most large servers it is -- keep reading for details.

Domain Names

Because most people have trouble remembering the strings of numbers that make up IP addresses, and because IP addresses sometimes need to change, all servers on the Internet also have human-readable names, called domain names. For example, www.howstuffworks.com is a permanent, human-readable name. It is easier for most of us to remember www.howstuffworks.com than it is to remember 209.116.69.66.

The name www.howstuffworks.com actually has three parts:

1. The host name ("www")
2. The domain name ("howstuffworks")
3. The top-level domain name ("com")

Domain names within the ".com" domain are managed by the registrar called VeriSign. VeriSign also manages ".net" domain names. Other registrars (like RegistryPro, NeuLevel and Public Interest Registry) manage the other domains (like .pro, .biz and .org). VeriSign creates the top-level domain names and guarantees that all names within a top-level domain are unique. VeriSign also maintains contact information for each site and runs the "whois" database. The host name is created by the company hosting the domain. "www" is a very common host name, but many places now either omit it or replace it with a different host name that indicates a specific area of the site. For example, in encarta.msn.com, the domain name for Microsoft's Encarta encyclopedia, "encarta" is designated as the host name instead of "www."

 

Name Servers

The whois CommandOn a UNIX machine, you can use the whois command to look up information about a domain name. You can do the same thing using the whois form at VeriSign. If you type in a domain name, like "howstuffworks.com," it will return to you the registration information for that domain, including its IP address.

A set of servers called domain name servers (DNS) maps the human-readable names to the IP addresses. These servers are simple databases that map names to IP addresses, and they are distributed all over the Internet. Most individual companies, ISPs and universities maintain small name servers to map host names to IP addresses. There are also central name servers that use data supplied by VeriSign to map domain names to IP addresses.

If you type the URL "http://www.howstuffworks.com/web-server.htm" into your browser, your browser extracts the name " www.howstuffworks.com," passes it to a domain name server, and the domain name server returns the correct IP address for www.howstuffworks.com. A number of name servers may be involved to get the right IP address. For example, in the case of www.howstuffworks.com, the name server for the "com" top-level domain will know the IP address for the name server that knows host names, and a separate query to that name server, operated by the HowStuffWorks ISP, may deliver the actual IP address for the HowStuffWorks server machine.

On a UNIX machine, you can access the same service using the nslookup command. Simply type a name like "www.howstuffworks.com" into the command line, and the command will query the name servers and deliver the corresponding IP address to you.

So here it is: The Internet is made up of millions of machines, each with a unique IP address. Many of these machines are server machines, meaning that they provide services to other machines on the Internet. You have heard of many of these servers: e-mail servers, Web servers, FTP servers, Gopher servers and Telnet servers, to name a few. All of these are provided by server machines. 

 

Ports

Any server machine makes its services available to the Internet using numbered ports, one for each service that is available on the server. For example, if a server machine is running a Web server and an FTP server, the Web server would typically be available on port 80, and the FTP server would be available on port 21. Clients connect to a service at a specific IP address and on a specific port.

Each of the most well-known services is available at a well-known port number. Here are some common port numbers:

• echo 7
• daytime 13
• qotd 17 (Quote of the Day)
• ftp 21
• telnet 23
• smtp 25 (Simple Mail Transfer, meaning e-mail)
• time 37
• nameserver 53
• nicname 43 (Who Is)
• gopher 70
• finger 79
• WWW 80

If the server machine accepts connections on a port from the outside world, and if a firewall is not protecting the port, you can connect to the port from anywhere on the Internet and use the service. Note that there is nothing that forces, for example, a Web server to be on port 80. If you were to set up your own machine and load Web server software on it, you could put the Web server on port 918, or any other unused port, if you wanted to. Then, if your machine were known as xxx.yyy.com, someone on the Internet could connect to your server with the URL http://xxx.yyy.com:918. The ":918" explicitly specifies the port number, and would have to be included for someone to reach your server. When no port is specified, the browser simply assumes that the server is using the well-known port 80. 

 

Protocols

Once a client has connected to a service on a particular port, it accesses the service using a specific protocol. The protocol is the pre-defined way that someone who wants to use a service talks with that service. The "someone" could be a person, but more often it is a computer program like a Web browser. Protocols are often text, and simply describe how the client and server will have their conversation.

Perhaps the simplest protocol is the daytime protocol. If you connect to port 13 on a machine that supports a daytime server, the server will send you its impression of the current date and time and then close the connection. The protocol is, "If you connect to me, I will send you the date and time and then disconnect." Most UNIX machines support this server. If you would like to try it out, you can connect to one with the Telnet application. In UNIX, the session would look like this:

%telnet web67.ntx.net 13 Trying 216.27.61.137... Connected to web67.ntx.net. Escape character is '^]'. Sun Oct 25 08:34:06 1998 Connection closed by foreign host. 

 

On a Windows machine, you can access this server by typing "telnet web67.ntx.net 13" at the MSDOS prompt.

In this example, web67.ntx.net is the server's UNIX machine, and 13 is the port number for the daytime service. The Telnet application connects to port 13 (telnet naturally connects to port 23, but you can direct it to connect to any port), then the server sends the date and time and disconnects. Most versions of Telnet allow you to specify a port number, so you can try this using whatever version of Telnet you have available on your machine.

Most protocols are more involved than daytime and are specified in Request for Comment (RFC) documents that are publicly available (see http://sunsite.auc.dk/RFC/ for a nice archive of all RFCs). Every Web server on the Internet conforms to the HTTP protocol, summarized nicely in The Original HTTP as defined in 1991 . The most basic form of the protocol understood by an HTTP server involves just one command: GET. If you connect to a server that understands the HTTP protocol and tell it to "GET filename," the server will respond by sending you the contents of the named file and then disconnecting. Here's a typical session:

%telnet www.howstuffworks.com 80 Trying 216.27.61.137... Connected to howstuffworks.com. Escape character is '^]'. GET http://www.howstuffworks.com/ <html> <head> <title>Welcome to How Stuff Works</title>   ... </body> </html> Connection closed by foreign host. 

 

In the original HTTP protocol, all you would have sent was the actual filename, such as "/" or "/web-server.htm." The protocol was later modified to handle the sending of the complete URL. This has allowed companies that host virtual domains, where many domains live on a single machine, to use one IP address for all of the domains they host. It turns out that hundreds of domains are hosted on 209.116.69.66 -- the HowStuffWorks IP address.

 

Putting It All Together

Now you know a tremendous amount about the Internet. You know that when you type a URL into a browser, the following steps occur:

• The browser breaks the URL into three parts:
  1. The protocol ("http")
  2. The server name ("www.howstuffworks.com")
  3. The file name ("web-server.htm")

• The browser communicates with a name server to translate the server name, "www.howstuffworks.com," into an IP address, which it uses to connect to that server machine.

• The browser then forms a connection to the Web server at that IP address on port 80.

• Following the HTTP protocol, the browser sends a GET request to the server, asking for the file "http://www.howstuffworks.com/web-server.htm." (Note that cookies may be sent from browser to server with the GET request -- see How Internet Cookies Work for details.)

• The server sends the HTML text for the Web page to the browser. (Cookies may also be sent from server to browser in the header for the page.)

• The browser reads the HTML tags and formats the page onto your screen.

 

Extras: Security

You can see from this description that a Web server can be a pretty simple piece of software. It takes the file name sent in with the GET command, retrieves that file and sends it down the wire to the browser. Even if you take into account all of the code to handle the ports and port connections, you could easily create a C program that implements a simple Web server in less than 500 lines of code. Obviously, a full-blown enterprise-level Web server is more involved, but the basics are very simple.

Most servers add some level of security to the serving process. For example, if you have ever gone to a Web page and had the browser pop up a dialog box asking for your name and password, you have encountered a password-protected page. The server lets the owner of the page maintain a list of names and passwords for those people who are allowed to access the page; the server lets only those people who know the proper password see the page. More advanced servers add further security to allow an encrypted connection between server and browser, so that sensitive information like credit card numbers can be sent on the Internet.

That's really all there is to a Web server that delivers standard, static pages. Static pages are those that do not change unless the creator edits the page.

 

Extras: Dynamic Pages

But what about the Web pages that are dynamic? For example:

• Any guest book allows you to enter a message in an HTML form, and the next time the guest book is viewed, the page will contain the new entry.

• The whois form at Network Solutions allows you to enter a domain name on a form, and the page returned is different depending on the domain name entered.

• Any search engine lets you enter keywords on an HTML form, and then it dynamically creates a page based on the keywords you enter.

In all of these cases, the Web server is not simply "looking up a file." It is actually processing information and generating a page based on the specifics of the query. In almost all cases, the Web server is using something called CGI scripts to accomplish this feat. CGI scripts are a topic unto themselves, and are described in the HowStuffWorks article How CGI Scripting Work.

For more information on Web servers and related topics, check out the links on the next page.

 

Lots More Information

Related HowStuffWorks Articles

• How Internet Infrastructure Works
• How Domain Name Servers Work
• How ASPs Work
• How E-mail Works
• How File Sharing Works
• How Internet Cookies Work
• How Network Address Translation Works
• How CGI Scripting Works
• Where are all the Internet domain names registered and maintained?
• Why do some Web sites include "www" in the URL while others don't?

More Great Links

• Whatis.com - great dictionary of Internet terms
• The Original HTTP as defined in 1991
• Firewalls FAQ
• IANA
• Network Solutions
• RFC 1340 - Assigned numbers
• Setting up your own DNS
• Web Security
• HTTP overview

 

 

--
Zhipeng Zhang (Alan)   BCompSc  MInfoTech MACS(Prov)

"You must be the change you want to see in the world."

"Begin at the beginning and go on till you come to the end; then stop."
                                                                                       -- Lewis Carroll, Alice in Wonderland

Cannot Access shared resources on a LAN computer

http://www.softwaretipsandtricks.com/forum/windows-xp/17296-cannot-access-shared-resources-computer.html

 

Q:

 

Two computers A and B are on a network having Windows XP. From Computer A, I can see the shared resources in Computer B. But I cannot see the shared resources of Computer A from Computer B. I can ping Computer B from Computer A and also the same form Computer A to Computer B. So the network seems to work fine. What could be the problem. Can any one help me?

Regards,
Satish

 

 

A:

 

If you have win xp home edition, this is the only way to get over the problem as the local settings & sedurity policy are not available in Administration Tools.
As far as I know, user rights policies applies to WinXP Home as well, you just don't have a builtin GUI tool to see/change them.

You should be able to set/remove those privileges with the Windows 2003 resource kit command line tool ntrights.exe.

Ntrights.exe is in the free Win2k3 resource kit:

Windows Server 2003 Resource Kit Tools
http://go.microsoft.com/fwlink/?LinkId=4544
or
http://www.microsoft.com/downloads/d...displaylang=en
The kit will install on WinXP or later.

After installation, click on: Start, All Programs, Windows Resource Kit Tools, Command Shell

Then enter the following commands. (Attention: they are case sensitive.)

net user guest /active:yes
ntrights +r SeNetworkLogonRight -u Guest
ntrights -r SeDenyNetworkLogonRight -u Guest

The first command enables network access for Guest, the two subsequent ones change two different policies to allow network access for Guest.

This has been confirmed to work by several users.

 

 

 

Technology

php
javascript
Vbscript
Mysql
Apache
xml
Ajax

Google Webmaster | sitemap-generator

https://www.google.com/webmasters/tools/docs/en/sitemap-generator.html

 

Webmaster Tools

About Webmaster Tools Using the Sitemap Protocol Using the Sitemap Generator Google Webmaster Central			Contents Before you begin Downloading the Sitemap Generator program files Creating a configuration file Uploading the files to your web server Running the Sitemap Generator script Submitting your Sitemap to Google Troubleshooting Before you begin The Google Sitemap Generator is a Python script that creates a Sitemap for your site using the Sitemap Protocol. This script can create Sitemaps from URL lists, web server directories, or from access logs. In order to use this script: You must be able to connect to and run scripts on your web server. Your web server must have Python 2.2 or later installed. You must know the command that launches Python. (Generally, this is python, but may vary by installation. For instance, if the web server has two versions of Python installed, the earlier version may be invoked by the command python and the later version may be invoked by the command python2.) You must know the directory path to your site. If your web server hosts one site, this may be a path such as var/www/html. If you have a virtual server that hosts multiple sites, this may be a path such as home/virtual/site1/fst/var/www/html. You must be able to upload files to your web server (for instance, using FTP). If you will be generating a list of URLs based on access logs, you must know the encoding used for those logs and the complete path to them. If you aren't sure about any of this, you can check with your web hosting company. Now you're ready to get started. Here's an overview of what you'll need to do. Download the Sitemap Generator program files. Extract the files to a local directory. Create a configuration file for your site using the provided example_config.xml file as a template. Modify this file as needed for your site and save it. Upload the necessary files to your web server. Run sitemap_gen.py. Add the generated Sitemap to your Google webmaster tools account. Set up a recurring script. (optional) If you are unable to use the Sitemap Generator, you can add a Sitemap to your Google webmaster tools account in another format, such as a simple text file.You can also find links to third-party programs that support Google Sitemaps here. For News Sitemaps: The Sitemap Generator is not recommended for use in creating Google News Sitemaps at this time, due to the special requirements of News Sitemaps. News Sitemaps are intended to be dynamic lists of only the most recently published news articles (rather than the entire website), and they are updated frequently. 1. Downloading the Sitemap Generator program files The Sitemap Generator files are available in ZIP and GZ archive formats from the following location: http://sourceforge.net/project/showfiles.php?group_id=137793&package_id=153422 Once you download the archive, extract it into a local directory. Locate the following files: README —contains the latest information about this tool sitemap_gen.py —the python script that generates your Sitemap example_config.xml —the template configuration file you'll use to specify the configuration for your site example_urllist.txt —the template URL list you can use if you wish to create a Sitemap based on a set of URLs that you specify 2a. Creating a configuration file This section provides step-by-step instructions for creating a configuration file. It also provides a complete reference of the options available. If you are creating Mobile Sitemaps, see the additional mobile guidelines. In order to create a configuration file for your site, you must have the following information: The base URL for your site (such as http://www.example.com/). Only URLs that begin with this base URL can be included in the Sitemap. Ensure that you include the protocol (such as http://). For instance, http://www.google.com is a valid base url, but www.google.com is not. The web server path to the location where you want to store the Sitemap. Generally, this is the path to the base URL as the Sitemap cannot contain URLs that are in a higher-level directory from the location of the Sitemap. When you run the Google Sitemap Generator, it creates the Sitemap and places it in the location you specify. The methods you want the Sitemap Generator to use to create your Sitemap. You can use any combination of methods. The following methods are available: URL —list individual URLs in this section of the configuration file, along with information about each URL. You would generally use this method in conjunction with another method to manually include additional URLs that other methods wouldn't pick up. URL list —point the configuration file to a text file that contains a list of URLs. You might want to use this method if this text file already exists or if you use a script to generate a list of URLs. Directory paths —specify the directory paths for your site and corresponding URLs to those paths. The Sitemap Generator will create a list of URLs based on the contents of those directories. You might want to use this method if your site consists of static HTML files. Access logs —point to the path to your log files. The Sitemap Generator will create a list of URLs based on the URLs included in the logs. You might want to use this method if your site consists of dynamic pages. Sitemap —point to existing Sitemaps that you have created with the Sitemap Generator. The Sitemap Generator will create a single Sitemap that includes the URLs contained in each Sitemap. You could use this method if you have already created several smaller Sitemaps that you want to combine into one larger Sitemap. Create the configuration file as follows: Open the example_config.xml file in a text editor. Save it as a new file (such as config.xml or mysite_config.xml). Locate the site definition section: <site base_url="http://www.example.com/" store_into="/var/www/docroot/sitemap.xml.gz" verbose="1"> Change the base_url value to the URL for your site. Change the store_into value to the path on your web server where you want to store the Sitemap and the filename you want to use for the Sitemap. Generally, this is the path to the base URL since Google can only accept URLs that are at the same level as or subdirectories of the directory that holds the Sitemap. You can specify a relative path from the directory where you upload the script or a complete path from the root of your web server. If you upload the script to your base URL directory, you can simply specify the filename. Locate the generation method sections that begin with ** MODIFY or DELETE **. Each of these sections corresponds to a method for generating a Sitemap. Delete the sections for the methods you aren't going to use. Follow the instructions below for the methods you are going to use. URL Locate the following section: <!-- ** MODIFY or DELETE ** "url" nodes specify individual URLs to include in the map. <br> Required attributes: href - the URL Optional attributes: lastmod - timestamp of last modification (ISO8601 format) changefreq - how often content at this URL is usually updated priority - value 0.0 to 1.0 of relative importance in your site --> <url href="http://www.example.com/stats?q=name" /> <url href="http://www.example.com/stats?q=age" lastmod="2004-11-14T01:00:00-07:00" changefreq="yearly" priority="0.3" /> This section gives two examples: the first includes only the required attribute and the second includes the required attribute as well as the optional attributes. Use this format for each of the URLs you want to include. The changefreq attribute gives Google a general idea of how often the URL is updated. This helps Google know how often to visit the page for new content. The priority attribute gives Google information about the relative importance of this page compared to the other pages of your site. This attribute has no effect on how Google compares your page to pages on other sites, it just helps Google know which pages of your site that you think are most important. URL list Locate the following section: <!-- ** MODIFY or DELETE ** "urllist" nodes name text files with lists of URLs. An example file "example_urllist.txt" is provided. Required attributes: path - path to the file Optional attributes: encoding - encoding of the file if not US-ASCII --> <urllist path="example_urllist.txt" encoding="UTF-8" /> Use this format to point to the path and name of the text file that contains your list of URLs. You can use the provided example_urllist.txt file as a template for that text file. You can specify either a relative or complete path to your web server. For instance, if the Sitemap Generator and urlist.txt file are located in the same directory, you can simply specify the filename of the .txt file, If you create a text file with an encoding other than UTF-8, you can use the encoding attribute to indicate this encoding. If you have multiple .txt files, you can use wildcards. For instance: <urllist path="example_urllist*.txt" encoding="UTF-8" /> For each URL you include in the text file, you can specify the last modification date, change frequency, and priority. See the URLlist text file reference section for complete information about the structure of this file. Directory paths Locate the following section: <!-- ** MODIFY or DELETE ** "directory" nodes tell the script to walk the file system and include all files and directories in the Sitemap. Required attributes: path - path to begin walking from url - URL equivalent of that path Optional attributes: default_file - name of the index or default file for directory URLs --> <directory path="/var/www/icons" url="http://www.example.com/images/" /> <directory path="/var/www/docroot" url=" http://www.example.com/" default_file="index.html" /> This section gives two examples. If all of your pages are contained in subdirectories of one path, then you only need to include one entry. However, if you have multiple paths to pages on your site, include an entry for each. Remember that each URL must begin with the base URL you specified in step 3. For instance, the examples given in the example_config.xml file both have URLs that begin with http://www.example.com/. Therefore, both URLs are valid. Replace the example entries with entries for your site. Many sites will only have one entry that points to the base URL. Ensure that path value is the complete path to the directory on your web server. Ensure that the url value is the complete URL, including the protocol (such as http) and a trailing slash, if required. You can use the default_file parameter to specify the filename that your server uses as the default page for a directory. In the above example, /var/www/docroot resolves to http://www.example.com/index.html. You are not required to specify this. However, if you do, the Sitemap Generator will include the page that maps to each subdirectory only once (rather than list both the directory URL and filename URL) and will use the last modified date of the file (rather than the directory) to extract the lastmod attribute for that page. Access logs Locate the following section: <!-- ** MODIFY or DELETE ** "accesslog" nodes tell the script to scan webserver log files to extract URLs on your site. Both Common Logfile Format (Apache's default logfile) and Extended Logfile Format (IIS's default logfile) can be read. Required attributes: path - path to the file Optional attributes: encoding - encoding of the file if not US-ASCII --> <accesslog path="/etc/httpd/logs/access.log" encoding="UTF-8" /> <accesslog path="/etc/httpd/logs/access.log.0" encoding="UTF-8" /> <accesslog path="/etc/httpd/logs/access.log.1.gz" encoding="UTF-8" /> This section gives three examples. You should replace these entries and include an entry for each log file. Ensure that the path value is the complete path and filename on your web server. If the log files are not encoded as US-ASCII or UTF-8, then use the optional encoding attribute to specify the encoding. Rather than list each log file, you can use wildcards. For instance, in the above example, you could include the following entry that would include all three log files: <accesslog path="/etc/httpd/logs/access.log*" encoding="UTF-8" /> The Sitemap Generator assigns priority to URLs it finds in the logs based on how often each URL is accessed. For instance, a URL that has been accessed 100 times will be given a higher priority than a URL that has been accessed twice. The actual priority assignment is relative and depends on each URL as compared to other URLs in the site. sitemap Locate the following section: <!-- ** MODIFY or DELETE ** "sitemap" nodes tell the script to scan other Sitemap files. This can be useful to aggregate the results of multiple runs of this script into a single Sitemap. Required attributes: path - path to the file --> <sitemap path="/var/www/docroot/subpath/sitemap.xml" /> This section gives one example. You should replace this entry and include an entry for each Sitemap you want to include. Ensure that the path value is the complete path and filename on your web server. You can list gzipped Sitemaps as well, as long as they have a .gz extension. Rather than list each Sitemap, you can use wildcards. For instance, the following entry would include any Sitemaps that begin with the word "sitemap" and have an .xml extension: <sitemap path="/var/www/docroot/subpath/sitemap*.xml" /> The Sitemap Generator extracts all URLs and the optional data listed for each URL for every Sitemap you list and creates one Sitemap with this information. At this time, we can't guarantee that this method will work Sitemaps created with tools other than the Sitemap Generator. Locate the filter definition section: <!-- ******************************************************** FILTERS Filters specify wild-card patterns that the script compares against all URLs it finds. Filters can be used to exclude certain URLs from your Sitemap, for instance if you have hidden content that you hope the search engines don't find. Filters can be either type="wildcard", which means standard path wildcards (* and ?) are used to compare against URLs, or type="regexp", which means regular expressions are used to compare. Filters are applied in the order specified in this file. An action="drop" filter causes exclusion of matching URLs. An action="pass" filter causes inclusion of matching URLs, shortcutting any other later filters that might also match. If no filter at all matches a URL, the URL will be included. Together you can build up fairly complex rules. The default action is "drop". The default type is "wildcard". You can MODIFY or DELETE these entries as appropriate for your site. However, unlike above, the example entries in this section are not contrived and may be useful to you as they are. ********************************************************* --> <!-- Exclude URLs that end with a '~' (IE: emacs backup files) --> <filter action="drop" type="wildcard" pattern="*~" /> <!-- Exclude URLs within UNIX-style hidden files or directories --> <filter action="drop" type="regexp" pattern="/\.[^/]*" /> You can use filtering to exclude specific URLs from the generated Sitemap. You might want to do this to create a cleaner list, to reduce redundant listings, or to keep certain URLs from being indexed. Note that if you use a robots.txt file to keep URLs from being indexed, then even if the URLs are included in your Sitemap, Google will not search or index them. You can use any or all of the filtering methods. You can delete the entries you don't need and can create additional entries, if desired. Below are sample usages. <filter action="drop" type="wildcard" pattern="*.jpg" /> This filter excludes URLs that end in .jpg. You might want to include a similar filter if all of your site's images are embedded within HTML pages and should not be accessed as standalone URLs. <filter action="pass" type="wildcard" pattern="*.htm*" /> <filter action="drop" type="wildcard" pattern="*" /> This filter includes all .htm* files but excludes everything else. Once you have made all the changes for your site, save the file. Config File Syntax Reference A complete explanation of the config file syntax is below. Each tag begins with a code sample, followed by a description of the attributes. site Required tag at the beginning of each config file. <site base_url="http://www.example.com/" store_into="/var/www/html/sitemap.xml.gz" verbose="1" supress_search_engine_notify="1" default_encoding="UTF-8"> base_url required The HTTP path of the base of your website - only URLs that begin with this base can be included in the Sitemap store_into required The web server path to the desired output file. The script will create this file - there's no need to create the file before running the script. verbose optional Enter a number from 0-3, with higher numbers corresponding to increased debug information suppress_search_engine_notify optional Disable search engine notification by entering "1" for testing purposes default_encoding optional Specify a character encoding to be applied to file system paths and URLs url Optional tag that you can use to list each URL in your site. <url href="http://www.example.com/stats?q=age" lastmod="2004-11-14T01:00:00-07:00" changefreq="yearly" priority="0.3" /> href required The HTTP path of the base of your website - only URLs that begin with this base can be included in the Sitemap lastmod optional The time the URL was last modified in W3C Datetime format (YYYY-MM-DDThh:mm:ss+00:00). You may omit the time portion. Examples:   "2005-02-21T18:00:15+00:00"   "2005-02-21" changefreq optional The frequency with which the URL is likely to change. This is considered a hint and not a command. The value must be one of "always", "hourly", "daily", "weekly", "monthly", "yearly", or "never". priority optional The priority of this page relative to other pages on the same site. The value is a number between 0.0 and 1.0, where 0.0 is the lowest priority and 1.0 is the highest priority. The priority can affect the order that search engines select URLs to explore on your site. Since the priority is relative, it is only used to select between URLs within your own site; the priority of your pages will not be compared to the priority of pages on other sites. urllist Optional tag that you can use to point to a text file that contains a list of the URLs in your site. <urllist path="/var/www/html/urllist.txt" encoding="UTF-8" /> path required The path and filename of the .txt file. You can specify either a relative or complete path. encoding optional The encoding of the file, if not UTF-8. The urllist.txt file is a simple text file containing a list of URLs to map. You can also include optional attributes for each URL. Attributes are entered on the same line as the URL and are separated by a single space. For example: http://www.example.com/abc/something http://www.example.com/abc/xyy.pdf lastmod=2001-12-31T14:05:06+00:00 http://www.example.com/abc/def?x=12&y=23 changefreq=weekly priority=0.3 lastmod optional The time the URL was last modified in W3C Datetime format (YYYY-MM-DDThh:mm:ss+00:00). You may omit the time portion. Examples:   "2005-02-21T18:00:15+00:00"   "2005-02-21" changefreq optional The frequency with which the URL is likely to change. This is considered a hint and not a command. The value must be one of "always", "hourly", "daily", "weekly", "monthly", "yearly", or "never". priority optional The priority of this page relative to other pages on the same site. The value is a number between 0.0 and 1.0, where 0.0 is the lowest priority and 1.0 is the highest priority. The priority can affect the order that search engines select URLs to explore on your site. Since the priority is relative, it is only used to select between URLs within your own site; the priority of your pages will not be compared to the priority of pages on other sites. directory Optional tag that you can use to specify directories in your site so the Sitemap Generator can create a list of URLs from the files found in those directories. --> <directory path="/var/www/icons" url="http://www.example.com/images/" /> <directory path="/var/www/docroot" url=" http://www.example.com/" default_file="index.html" /> path required States the initial path. Sitemap Generator will traverse this directory and all subdirectories. url required Specifies the URL equivalent of the path value. default_file optional Specifies the default file for a directory on the server. accesslog Optional tag that you can use to specify the path and filename of IIS and Apache-style access logs so the Sitemap Generator can automatically pick up URLs from them. <accesslog path="/etc/httpd/logs/access-0.log" encoding="UTF-8"/> path required States the path to the file. encoding optional Specifies encoding of the file, if not UTF-8. sitemap Optional tag that you can use to specify the path and filename of existing Sitemaps that you have created with the Sitemap Generator. The Sitemap Generator will create a single Sitemap that includes the URLs contained in each Sitemap. <sitemap path="/var/www/docroot/subpath/sitemap.xml" /> path required States the path to the Sitemap file. filter Optional tag that you can use to build rules that include or exclude specific files. Filters are obeyed in the order in which they appear in the config.xml file. However, intermixing filter entries and input entries (url, urllist, directory, or accesslog) has no additional effect - every URL the Sitemap Generator adds to the Sitemap is first compared against every filter. If no filter matches a URL, the default is to include the URL in the Sitemap. <filter action="drop" type="wildcard" pattern="*/internal/*" /> action optional The action the filter should take. Valid values are: drop - excludes matching URLs. This is the default action, so if no action is specified, the generator assumes "drop". pass - includes matching URLs. type optional The type of filtering. Valid values are: wildcard - standard path wildcards (? and *) are used to compare against URLs. This is the default type, so if no type is specified, the generator assumes "wildcard". regexp - regular expressions are used. pattern required Specifies the pattern to match against. Encodings Files referenced by your configuration file, either URL lists or web server logs, can use encodings other than the default UTF-8. You can specify alternate encodings in config.xml to affect how the Sitemap Generator reads your files. Some common encodings are: encoding="utf-8" is the assumed default encoding="ascii" is a subset of UTF-8 so you don't have to specify it encoding="iso-8859-1" is common for many west European languages 2b. Additional information for creating a mobile configuration file You create a configuration file for a Mobile Sitemap in the same way as for a non-mobile Sitemap. However, you must create a separate config file for each markup language and run the Sitemap Generator with each config file separately so that you create a separate Sitemap for each. Each config file must: Specify a different filename for the store_into value. Use filters to specify the URLs to exclude and include for the markup language. Remember that each Sitemap should include URLs for only one markup language. This means that the same URL may be included in multiple Sitemaps, if those URLs serve multiple markup languages. Examples of filtering Below are some examples of how you can use extension-based filters to generate Mobile Sitemaps for different markup languages. The specific filtering you use should be based on the types of markup languages used in your site, and how you specify each type. If you have implemented the details of your site differently (for instance, you may organize URLs with different markup languages in separate folders), you should filter based on the specifics of your site implementation. Remember that filters are applied in the order you list them in the config file. So, the first filter you should list is a "pass" action that specifies the URLs you want to include in the Sitemap. To create a Sitemap for WML (WAP 1.2) content: <filter action="pass" type="wildcard" pattern="*.wml" /> <filter action="drop" type="wildcard" pattern="*.*" /> To create a Sitemap for XHTML mobile profile (WAP 2.0) content: <filter action="pass" type="wildcard" pattern="*.xhtml" /> <filter action="drop" type="wildcard" pattern="*.*" /> 3. Uploading the files to your web server You should upload the following files to your web server in a location you can access from a command line: config.xml —this is the configuration file you just created using example_config.xml. sitemap_gen.py —this is the Python script that generates your Sitemap. urllist.txt —this file is optional; you only need to include it if you used the text file method of generating a Sitemap. The method you use to upload these files depends on your environment. Common methods include FTP and SCP. For more information, contact your web host. 4. Running the Sitemap Generator script (sitemap_gen.py) In order to run the Sitemap Generator, you'll need to connect to your web server. The method you use to connect depends on your environment. For instance, you can generally access a UNIX-based server using SSH. For more information on connecting to your web server and running scripts, talk to your web host. Once you have copied the files to your web server, you'll need to run the Sitemap Generator script. Connect to your web server and run the following command (replace <path/config.xml> with the path to and filename of your configuration file; if you have uploaded this file to the same location as the Python script, you can exclude the path): python sitemap_gen.py --config=<path/config.xml> For instance, a UNIX-based command line might look similar to this: A MS-DOS-based command shell might look similar to this: Tip: If you're testing your configuration and are not ready to submit your Sitemap, the following syntax will prevent Sitemap Generator from contacting Google: $ python sitemap_gen.py --config= config.xml --testing You'll see the status of your request in the command prompt: Reading configuration file: /path/config.xml Opened URLLIST "/path/urllist.txt" Walking DIRECTORY "/var/www/html/dir" Walking DIRECTORY "/var/www/html/dir2" Opened ACCESSLOG "/etc/httpd/logs/access-0.log" Sorting and normalizing collected URLs. Writing Sitemap file "/path/sitemap.xml.gz" with 1092 URLs Notifying search engines. Notifying www.google.com Count of file extensions on URLs: 208 .html 574 .jpg ... Number of errors: 0 Number of warnings: 0 If you don't see very much output like this, remember that the verbose setting in your configuration file affects how much information is printed on the screen. This example is representative of setting verbose to "1". Any errors in the file will also be returned. For instance, if you leave the url= attribute off a directory entry, the script will output the following: [ERROR] Directory entries must have both "path" and "url" attributes Number of errors: 1 Correct any errors in your config.xml file and re-run the script. If no errors are present, the Sitemap Generator will create a new sitemap.xml.gz file in the location you specified in the config file. 5. Submitting your Sitemap to Google The Sitemap Generator creates a sitemap.xml.gz file in the location you specified in the config file. Once this file is successfully created, make sure it is accessible through a web browser. Then, add it to your Google Sitemaps account. This enables Google to provide you with useful status and statistical information. If Google reports problems with your Sitemap, you can correct the problems and resubmit it. You only have to add the Sitemap manually once. After that, you can use an HTTP request to notify Google about changes to your Sitemap (although you can also resubmit it through your Google webmaster tools account). 6. Setting up a recurring script We suggest setting up Sitemap Generator to run as often as your content is changed, to a maximum frequency of once per hour. Webmasters with a UNIX web server may consider setting this up as a cron job. Webmasters using other platforms should contact their system administrator for help in configuring recurring scripts. You may also benefit from peer advice in the Google Sitemaps Group on Google Groups. You can use an HTTP request to let Google know about changes to your Sitemap. However, please make sure that you log in to Google webmaster tools with your Google Account once to manually add your Sitemap to your Google webmaster tools account. Troubleshooting In this section we talk about some common questions or issues that some people run into while using Sitemap Generator, and what you can do if you come across one of these. Web-accessible Issue: Sitemap Generator returns the following error and warning: [ERROR] When attempting to access your generated Sitemap at the following URL: http://www.example.com/sitemap.xml.gz we failed to read it. Please verify the store_into path you specified in your configuration file is web-accessible. Consult the FAQ for more information. [WARNING] Proceeding to notify with an unverifiable URL. What just happened? Sitemap Generator created the file where you specified, then attempted to retrieve it using HTTP just as a search engine would, and failed. It went ahead and notified search engines anyway, but it's outputting the error and warning to let you know that your Sitemap may be not readable by search engines. Sitemap Generator creates your Sitemap file at the path specified in the store_into attribute of your config.xml file. Sitemap Generator then builds a URL to that file using the base_url attribute, and reports the URL to search engines. For instance, if you set the configuration to: <site base_url="http://www.example.com/" store_into="/var/www/html/sitemap.xml.gz"> Sitemap Generator will notify search engines to look for your Sitemap at: http://www.example.com/sitemap.xml.gz The file must be accessible through this URL. If the file can not be retrieved using this URL, search engines have no way of finding your Sitemap. What can you do? You need to verify that your config.xml is specifying the correct base_url and store_into paths for your Sitemap. You also need to confirm that web browsers can retrieve the Sitemap file off of the base_url. If you find errors in the path or URL, you may need to re-run Sitemap Generator, or just move the Sitemap file to the correct place and notify search engines manually through their web sites. Support for XML Issue: Sitemap Generator gives the following error: Some installs of Python 2.2 do not include complete support for XML. Please try upgrading your version of Python and re-running the script. What just happened? As the message says, some platforms have a version of Python (the language the Sitemap Generator script is written in) that is missing support libraries needed for processing XML files. This script requires full XML support in order to run. What can you do? Try upgrading your installed version of Python to a newer version. You will probably need to contact your system administrator to do this. Note that this just affects the Sitemap Generator, not Sitemaps overall. If you have another method or tool for creating Sitemaps, you can certainly use it and submit your Sitemaps to search engines. What are all the extra files in the .gz or .zip? When you extract the sitemap_gen.py script, you'll probably see quite a few more files than we refer to above. The full file list looks closer to this: AUTHORS ChangeLog COPYING example_config.xml example_urllist.txt PKG-INFO README setup.py sitemap_gen.py test_sitemap_gen.py The extra files tend to be information on the package and licensing terms. You are encouraged to look through these files. The one exception is test_sitemap_gen.py, which is a unit-test script that other developers may find useful if they wish to contribute to this open source project. If you intend to use Sitemap Generator without modifying any of the source code - this is the expected case for nearly everyone - you probably don't need this test script. If you are interested in helping with this project, please visit http://sourceforge.net/projects/goog-sitemapgen. You can also find links to some third party programs that support Google Sitemaps here. Non-ASCII characters Issue: Your site domain name or URLs within it contain non-ASCII characters. Generally, non-ASCII URLs should be encoded using UTF-8 before being percent-escaped. However, some webservers only respond correctly if URLs are encoded using an encoding other than UTF-8. All URLs within your Sitemap, as well as the URL of the Sitemap itself must be encoded for readability by the web server on which they are located. Within the site definition section, use the optional default_encoding attribute to specify the encoding used by your webserver. If you don't use this tag and your webserver uses an encoding other than UTF-8, we can't know which encoding to use. If your URLs contain non-ASCII characters, we recommend that you run the Sitemap Generator script using Python 2.3 or higher. This version of Python has increased non-ASCII support. If your domain name contains non-ASCII characters, you must use Python 2.3 or later, as Internationalizing Domain Names in Applications (IDNA) support wasn't added until this version. Without IDNA support, the Sitemap Generator can't correctly encode a non-ASCII domain name. Last modified: 15 April 2006
			©2007 Google - Privacy Policy - Terms and Conditions - About Google

--
Zhipeng Zhang (Alan)   BCompSc  MInfoTech MACS(Prov)

"You must be the change you want to see in the world."

"Begin at the beginning and go on till you come to the end; then stop."
                                                                                       -- Lewis Carroll, Alice in Wonderland

How do I add my site to Google's search results?

http://www.google.com/support/webmasters/bin/answer.py?answer=34397&ctx=sibling

 

Webmaster Help Center Change Language: Dansk Deutsch English English (UK) Español Français Italiano Nederlands Norsk Polski Português Română Suomi Svenska Русский Türkçe 中文（简体） 中文（繁體） 日本語 한국어 Google Help > Help Center Home > My site and Google > My site in the Google index

Documentation Webmaster Guidelines How do I add my site? About Google webmaster tools Webmaster Central Tools Google webmaster tools Submit your content to Google Google services and tools

How do I add my site to Google's search results? Inclusion in Google's search results is free and easy; you don't even need to submit your site to Google. Google is a fully automated search engine that uses software known as "spiders" to crawl the web on a regular basis and find sites to add to our index. In fact, the vast majority of sites listed in our results aren't manually submitted for inclusion, but found and added automatically when our spiders crawl the web. To determine whether your site is currently included in Google's index, just perform a search for your site's URL. For example, a search for [ site:www.google.com ] returns the following results: http://www.google.com/search?hl=en&q=site%3Awww.google.com+ Although Google crawls billions of pages, it's inevitable that some sites will be missed. When our spiders miss a site, it's frequently for one of the following reasons: The site isn't well connected through multiple links to other sites on the web. The site launched after Google's most recent crawl was completed. The design of the site makes it difficult for Google to effectively crawl its content. The site was temporarily unavailable when we tried to crawl it or we received an error when we tried to crawl it. You can use Google webmaster tools to see if we received errors when trying to crawl your site. Our intent is to represent the content of the internet fairly and accurately. To help make this goal a reality, we offer guidelines as well as tips for building a crawler-friendly site. While there's no guarantee that our spiders will find a particular site, following these guidelines should increase your site's chances of showing up in our search results. Consider creating and submitting a detailed site map of your pages. Google Sitemaps is an easy way for you to submit all your URLs to the Google index and get detailed reports about the visibility of your pages on Google. With Google Sitemaps, you can automatically keep us informed of all of your current pages and any updates you make to those pages. Please note that submitting a Sitemap doesn't guarantee that all pages of your site will be crawled or included in our search results. You may also be interested in... What are Google's webmaster guidelines Tips for creating a Google-friendly site Was this information helpful? Yes   No

Search Help Center

©2007 Google

--
Zhipeng Zhang (Alan)   BCompSc  MInfoTech MACS(Prov)

"You must be the change you want to see in the world."

"Begin at the beginning and go on till you come to the end; then stop."
                                                                                       -- Lewis Carroll, Alice in Wonderland