TheJoe.it Into the (open) source

27Jun/102

Exclude files and directories from indexing using the file “robots.txt”

spider_miniatura

Exist in the network of standards of behavior for crawler (the offered, or even spider) by theContent indexing. I am not referring to the file ".htaccess", that is used to configure the webserver, I'm talking about the file "robots.txt".

The file "robots.txt" is one of configuration file simple that there are, and unlike ".htaccess" should be placed uniquely only in directory radice Site. This file communicates to the search engines that index our site indexing or less determined file the directory, and the operation is very simple: