Asked By
boompy
210 points
N/A
Posted on - 04/21/2011
Hi guys,
I'm new to this kind of stuff so I need your help. I have published website and manage to get it indexed in Google and Bing. However, Bing bot is crawling private sectors of my site and i want to know how to stop it. It also keeps on over-crawling my site and I'm afraid that it might eventually cause an increase on my server load. Please help, any advice will be really helpful! TIA
Answered By
eyronpol
0 points
N/A
#123874
Help! Bingbot is crawling sectors of my site too much!
Hello Boomboompau,
There are several possible causes why your site is being over crawled by Bing bot. However, the most common is the robots.txt. Have you tried configuring your robots.txt file?
Help! Bingbot is crawling sectors of my site too much!
Thank you for replying. Like what I said, I'm still new in this webmaster role so I guess the answer is NO. I haven't configured the robots.txt file yet. Any advice? TIA
Answered By
eyronpol
0 points
N/A
#123876
Help! Bingbot is crawling sectors of my site too much!
Okay here's the thing, if Bing bot determines that your website has a slow connection, it automatically adjusts how often it crawls your website. However, you can increase the delay between crawls by using the Crawl-delay parameter in the Robots.txt file to specify a minimum frequency in seconds. Since you mentioned that you're new to this business, try asking for assistance from a SEO or other webmaster on how to configure this. Look at the example below:
User-agent: MSNBot/Bingbot
Crawl-delay: 120
Individual crawler sections override the settings that are specified in * sections. If you've specified Disallow settings for all crawlers, you must add the Disallow settings to the MSNBot/Bingbot section you create in the Robots.txt file. For example, your Robots.txt file might have the following:
User-agent: *
Disallow: /private/
If you add an MSNBot/Bingbot section, you must add any Disallow settings to the MSNBot/Bingbot section. For example:
User-agent: MSNBot/Bingbot
Crawl-delay: 120
Disallow: /private/
This is are the same robots.txt configuration we use on our website so I hope this helps.
Help! Bingbot is crawling sectors of my site too much!
Wow! Thanks a lot eyronpol. I will definitely try your suggestion. if ever it doesn't work, I will definitely be back to ask more questions.
