Asked By
Mia Walker
30 points
N/A
Posted on - 04/06/2012
What does web crawler mean and what is its function in web development? Where can I find a web crawler and how can I use them?
What does web crawler mean?
Hi Mia,
Web crawler is a computer program that browses the web in the an automated orderly manner. It blends with the top search results from Google, Yahoo, Bing Search and other popular search engine.
Web crawlers are mainly used to create a copy of all the visited pages for later processing by a search engine that will index the downloaded pages to provide fast searches. Crawlers can also be used for automating maintenance tasks on a web site, such as checking links or validating HTML code. Also, crawlers can be used to gather specific types of information from Web pages, such as harvesting e-mail addresses (usually for sending spam).
Here are few example of web crawlers:
-
Yahoo! Slurp
-
Bingbot
-
Googlebot
You can create your own web crawler to meet your needs. Go to this link to fnd more info on how to.
https://www.makeuseof.com/tag/build-basic-web-crawler-pull-information-website-2/
I hope it helped.
What does web crawler mean?
A web crawler is also known as a web spider, a web robot, an automatic indexer, or an ant. It is an automated script or can be a program that browses the World Wide Web in a systematical way for the purpose of indexing or web indexing. This process is referred to as spidering or web crawling.
Spidering or web crawling is being used by web search engines to update their web content or the indexes of other sites’ web content. Web crawlers are capable of copying all the pages they visit so that the search engine that indexes the downloaded pages can process them later. This way, users can search them faster.
Web spiders are also capable of authenticating HTML codes and hyperlinks. A web spider begins by “seeds” which is a list of URLs that the web spider should visit. As it visits the URLs, it recognizes all the hyperlinks in the page and then includes them to the “crawl frontier” or the list of URLs to visit. Googlebot is the most famous web crawler.
