Serach Engine..Crawler & Bots !!??

[Nneel]

Hello..

Does Any one facing any problem with Search Engine Bots ?
Does the bots be a reason of server load at Bluehost?

like we got Yahoo!Slurp - Always Crawling our site ,always 10+ Yahoo
google adsense,Google, Became Bot, worldindexer..

Approx 18+ Search Engine Bot/Crawler always on our site..

Does any one facing any problem?

Our Site is vbulletin 3.6.2 with vbseo and some essential add-ons .

please let us know..

Thanx in advance.

[retokmeier]

You can check the bandwidth used by search bots on Awstats (Cpanel). If load from search bots is an issue, you can excluse some/all search bots using a robots.txt file. Also check www.google.com/support/webmasters/ for advice and to contact Google if they are creating too much load on your site.

retokmeier

[Nneel]

yes i know....

but im really curious any existing user facing this problem?

[retokmeier]

Not me. I have half a dozen search engines crawling my site daily.

[fedor]

Well just buy that other BH package and they will block crawlers and bots

[C.R.T.]

I have had problem with Baiduspider choking my site. My site is now operational after I blocked it from .htaccess.