What does web crawler mean?

Asked By Mia Walker 30 points N/A Posted on - 04/06/2012

What does web crawler mean and what is its function in web development? Where can I find a web crawler and how can I use them?

• Status: Open
• Question Views: 1416
• Answer Count: 2
• Vote Up 1 Vote Down

• Answer Accepted: Yes
• Question Category: Web Components

Best Answer by Estillero State

Go To Solution

Best Answer

Best Answer

Answered By Estillero State 5 points N/A #102366

What does web crawler mean?

Hi Mia,

 Web crawler is a computer program that browses the web in the an automated orderly manner. It blends with the top search results from Google, Yahoo, Bing Search and other popular search engine.

 

Web crawlers are mainly used to create a copy of all the visited pages for later processing by a search engine that will index the downloaded pages to provide fast searches. Crawlers can also be used for automating maintenance tasks on a web site, such as checking links or validating HTML code. Also, crawlers can be used to gather specific types of information from Web pages, such as harvesting e-mail addresses (usually for sending spam).

Here are few example of web crawlers:

• Yahoo! Slurp
• Bingbot
• Googlebot

You can create your own web crawler to meet your needs. Go to this link to fnd more info on how to.

https://www.makeuseof.com/tag/build-basic-web-crawler-pull-information-website-2/

 

I hope it helped.

 

 

About Estillero State

Questions
7

Answers
70

Best Answers
7

• Vote Up 0 Vote Down

• Posted on - 04/15/2012
• Question Category: Web Components

Answered By Sharath Reddy 590495 points N/A #102367

What does web crawler mean?

A web crawler is also known as a web spider, a web robot, an automatic indexer, or an ant. It is an automated script or can be a program that browses the World Wide Web in a systematical way for the purpose of indexing or web indexing. This process is referred to as spidering or web crawling.

Spidering or web crawling is being used by web search engines to update their web content or the indexes of other sites’ web content. Web crawlers are capable of copying all the pages they visit so that the search engine that indexes the downloaded pages can process them later. This way, users can search them faster.

Web spiders are also capable of authenticating HTML codes and hyperlinks. A web spider begins by “seeds” which is a list of URLs that the web spider should visit. As it visits the URLs, it recognizes all the hyperlinks in the page and then includes them to the “crawl frontier” or the list of URLs to visit. Googlebot is the most famous web crawler.

About Sharath Reddy

Questions
1

Answers
14599

Best Answers
2290

• Vote Up 0 Vote Down

• Posted on - 06/28/2014
• Question Category: Web Components