Help! Bingbot is crawling sectors of my site too much!

Asked By 210 points N/A Posted on -
qa-featured

Hi guys,

I'm new to this kind of stuff so I need your help. I have published website and manage to get it indexed in Google and Bing. However, Bing bot is crawling private sectors of my site and i want to know how to stop it. It also keeps on over-crawling my site and I'm afraid that it might eventually cause an increase on my server load. Please help, any advice will be really helpful! TIA

SHARE
Best Answer by eyronpol
Answered By 0 points N/A #123874

Help! Bingbot is crawling sectors of my site too much!

qa-featured

Hello Boomboompau,

There are several possible causes why your site is being over crawled by Bing bot. However, the most common is the robots.txt. Have you tried configuring your robots.txt file?

Answered By 0 points N/A #123875

Help! Bingbot is crawling sectors of my site too much!

qa-featured

Thank you for replying. Like what I said, I'm still new in this webmaster role so I guess the answer is NO. I haven't configured the robots.txt file yet. Any advice? TIA

Best Answer
Best Answer
Answered By 0 points N/A #123876

Help! Bingbot is crawling sectors of my site too much!

qa-featured

Okay here's the thing, if Bing bot determines that your website has a slow connection, it automatically adjusts how often it crawls your website. However, you can increase the delay between crawls by using the Crawl-delay parameter in the Robots.txt file to specify a minimum frequency in seconds. Since you mentioned that you're new to this business, try asking for assistance from a SEO or other webmaster on how to configure this. Look at the example below:

User-agent: MSNBot/Bingbot
Crawl-delay: 120

Individual crawler sections override the settings that are specified in * sections. If you've specified Disallow settings for all crawlers, you must add the Disallow settings to the MSNBot/Bingbot section you create in the Robots.txt file. For example, your Robots.txt file might have the following:

User-agent: *
Disallow: /private/

If you add an MSNBot/Bingbot section, you must add any Disallow settings to the MSNBot/Bingbot section. For example:

User-agent: MSNBot/Bingbot
Crawl-delay: 120
Disallow: /private/

This is are the same robots.txt configuration we use on our website so I hope this helps.

Answered By 0 points N/A #123877

Help! Bingbot is crawling sectors of my site too much!

qa-featured

Wow! Thanks a lot eyronpol. I will definitely try your suggestion. if ever it doesn't work, I will definitely be back to ask more questions.

Login/Register to Answer

Related Questions