Help! Bingbot is crawling sectors of my site too much!

Asked By boompy 210 points N/A Posted on - 04/21/2011

Hi guys,

I'm new to this kind of stuff so I need your help. I have published website and manage to get it indexed in Google and Bing. However, Bing bot is crawling private sectors of my site and i want to know how to stop it. It also keeps on over-crawling my site and I'm afraid that it might eventually cause an increase on my server load. Please help, any advice will be really helpful! TIA

• Status: Open
• Question Views: 3327
• Answer Count: 4
• Vote Up 20 Vote Down

• Answer Accepted: Yes
• Question Category: Web Marketing

Best Answer by eyronpol

Go To Solution

Answered By eyronpol 0 points N/A #123874

Help! Bingbot is crawling sectors of my site too much!

Hello Boomboompau,

There are several possible causes why your site is being over crawled by Bing bot. However, the most common is the robots.txt. Have you tried configuring your robots.txt file?

About eyronpol

Questions
0

Answers
2

Best Answers
1

• Vote Up 0 Vote Down

• Posted on - 04/21/2011
• Question Category: Web Marketing

Answered By [email protected] 0 points N/A #123875

Help! Bingbot is crawling sectors of my site too much!

Thank you for replying. Like what I said, I'm still new in this webmaster role so I guess the answer is NO. I haven't configured the robots.txt file yet. Any advice? TIA

About [email protected]

Questions
0

Answers
2

Best Answers
0

• Vote Up 0 Vote Down

• Posted on - 04/21/2011
• Question Category: Web Marketing

Best Answer

Best Answer

Answered By eyronpol 0 points N/A #123876

Help! Bingbot is crawling sectors of my site too much!

Okay here's the thing, if Bing bot determines that your website has a slow connection, it automatically adjusts how often it crawls your website. However, you can increase the delay between crawls by using the Crawl-delay parameter in the Robots.txt file to specify a minimum frequency in seconds. Since you mentioned that you're new to this business, try asking for assistance from a SEO or other webmaster on how to configure this. Look at the example below:

User-agent: MSNBot/Bingbot
Crawl-delay: 120

Individual crawler sections override the settings that are specified in * sections. If you've specified Disallow settings for all crawlers, you must add the Disallow settings to the MSNBot/Bingbot section you create in the Robots.txt file. For example, your Robots.txt file might have the following:

User-agent: *
Disallow: /private/

If you add an MSNBot/Bingbot section, you must add any Disallow settings to the MSNBot/Bingbot section. For example:

User-agent: MSNBot/Bingbot
Crawl-delay: 120
Disallow: /private/

This is are the same robots.txt configuration we use on our website so I hope this helps.

About eyronpol

Questions
0

Answers
2

Best Answers
1

• Vote Up 0 Vote Down

• Posted on - 04/21/2011
• Question Category: Web Marketing

Answered By [email protected] 0 points N/A #123877

Help! Bingbot is crawling sectors of my site too much!

Wow! Thanks a lot eyronpol. I will definitely try your suggestion. if ever it doesn't work, I will definitely be back to ask more questions.

About [email protected]

Questions
0

Answers
2

Best Answers
0

• Vote Up 0 Vote Down

• Posted on - 04/21/2011
• Question Category: Web Marketing