What does web crawler mean?

Asked By 30 points N/A Posted on -

What does web crawler mean and what is its function in web development? Where can I find a web crawler and how can I use them?

Best Answer by Estillero State
Best Answer
Best Answer
Answered By 0 points N/A #102366

What does web crawler mean?


Hi Mia,

 Web crawler is a computer program that browses the web in the an automated orderly manner. It blends with the top search results from Google, Yahoo, Bing Search and other popular search engine.


Web crawlers are mainly used to create a copy of all the visited pages for later processing by a search engine that will index the downloaded pages to provide fast searches. Crawlers can also be used for automating maintenance tasks on a web site, such as checking links or validating HTML code. Also, crawlers can be used to gather specific types of information from Web pages, such as harvesting e-mail addresses (usually for sending spam).

Here are few example of web crawlers:

  • Yahoo! Slurp
  • Bingbot
  • Googlebot

You can create your own web crawler to meet your needs. Go to this link to fnd more info on how to.



I hope it helped.



Answered By 573035 points N/A #102367

What does web crawler mean?


A web crawler is also known as a web spider, a web robot, an automatic indexer, or an ant. It is an automated script or can be a program that browses the World Wide Web in a systematical way for the purpose of indexing or web indexing. This process is referred to as spidering or web crawling.

Spidering or web crawling is being used by web search engines to update their web content or the indexes of other sites’ web content. Web crawlers are capable of copying all the pages they visit so that the search engine that indexes the downloaded pages can process them later. This way, users can search them faster.

Web spiders are also capable of authenticating HTML codes and hyperlinks. A web spider begins by “seeds” which is a list of URLs that the web spider should visit. As it visits the URLs, it recognizes all the hyperlinks in the page and then includes them to the “crawl frontier” or the list of URLs to visit. Googlebot is the most famous web crawler.

Login/Register to Answer

Related Questions