Â
Scenario: Our HR has uploaded a set of documents in document library.
He does not want the document library to be crawled and index.
What would you do?
Similarly We have site and we don't want it to be crawled?
How is this be achieved
Â
If you are talking about a web page that can not or that should not be indexed and crawled by search engine robots, crawlers, or spiders, you can add a meta tag on the page where these sets of documents will be posted. To make the page somewhat like invisible to search engine robots and other web crawlers, you can add these meta tags on the page:
Meta tags are placed inside the <head>..</head> tag of the web page. The first meta tag above will tell the web crawlers when they hit the page to not index the page and don’t explore all found links. The noindex value in the content will tell web crawlers to keep the page from being indexed although the links can be explored. While the value nofollow, on the other hand, will tell web crawlers to keep the links on the page from being explored but the page can be indexed.
The second meta tag will tell web crawlers to just ignore the page.
This is also the equivalent of the first meta tag.
