How to Develop Web Crawlers in PHP

Asked By 10 points N/A Posted on -

I have developed a simple website using PHP, HTML and Mysql. What I need to do now is to develop a software that will search the web for specific information and use this to update my webpage. I have learned that web crawlers are candidates to achieve that. I would like some steps needed to develop a simple web crawler to help me achieve my objective. Can I get some to buy and later modify them? How do you modify a web crawler?

Answered By 0 points N/A #190720

How to Develop Web Crawlers in PHP



03 /* This code contains the crawler framework that is sued to scrape the links across a specified domain and take the home page as the seed page. Also, we are storing the scraped links from an array to a MySQL database using a simple query. */
05 class webCrawler
06 {
08   protected $markup = '';
10   public function __construct($uri)
11   {
12     $this->markup = $this->getMarkup($uri);
13   }
15   public function getMarkup($uri)
16   {
17     return file_get_contents($uri);
18   }
20 /*  The protected data collection method uses PCRE function preg_match_all() in order to return all tags within the markup that are accepted using our pattern of /href="([^"]*)"/'  . */
22   protected function _get_links()
23   {


Related Questions