There are standards of behavior for crawlers in the network (the bot, or even spider) for content indexing. I am not referring to the file “.htaccess“, that is used to configure the webserver, I'm talking about the file “robots.txt“. The file “robots.txt” it is one of the simplest configuration files that exist, and Read more…