Frequently Asked Questions


What is robot.txt and its use ?

It is an exclusion protocol used to prevent the web spiders to access certain pages of a site. This is carried out in the following format.

  1. Allows web robots to visit all files

    User-agent:*
    Disallow:

  2. Allows all web robots out

    User-agent:*
    Disallow: /

  3. By specifying the directory name between two slashes in Disallow, denies the web spiders to access to those specified directory.

    User-agent:*
    Disallow: /cgi-bin/
    Disallow: /images/


Web Hosting By HIOXINDIAFree Online Site Map Generator Tool