What's this Google[Bot] user...

[dexter]

I have seen some users like Google[Bot],Yahoo[Bot] etc., in this forum...to be frank i have seen them in several phpbb3 forums...

http://img204.imageshack.us/img204/3788/57522823qv0.jpg

I know that these are not humans but what are these .....?
what they do....?
i have seen that they are not online every time but only during certain time....
Does it come with default phpbb3 installation or we have to do some modifications...?

BTW there is no QnA section in this forum...... :roll:

[ravi]

These boots are default user of phpbb3 for SEO.

[Jessenokian95]

Can U tell me what are they in detail

[Gigacore]

Google Bot is a web crawler. It will crawl through your websites/blogs/forums and will index ur pages and will store it in google's database. It will make a sitemap in its db. Whenever someone searches for something and if it is related to the content on ur site then ur site will appear on the result page.

Crawler helps in getting more speeds, no need to worry. And each and every site will be crawled automatically.

For more info, read this:

http://en.wikipedia.org/wiki/Googlebot

[aravindian]

And each site's root will contain a text file called
robots.txt

It sets the permissions for the Search Engine Bots.

It is mainly used to prevent crawling by search engines so as to save the server load and resources and bandwidth.

Examples:
http://9zap.com/robots.txt
http://orkut.com/robots.txt

You can understand better if you read the comments given in the file of 9zap:
# This file is to prevent the crawling and indexing of certain parts
# of your site by web crawlers and spiders run by sites like Yahoo!
# and Google. By telling these "robots" where not to go on your site,
# you save bandwidth and server resources.
#
# This file will be ignored unless it is at the root of your host:
# Used: http://example.com/robots.txt
# Ignored: http://example.com/site/robots.txt