Need to screen scrape a website from a classic asp 3.0 page

Asked By 120 points N/A Posted on -
qa-featured

I am doing a project for my final year thesis. The prototype involves accumulating search results from different search engines into one web page. For this I need to programmatically send the request to the web sites,  and then read the results into a variable. Is this possible to do this with Classic ASP 3.0?

SHARE
Best Answer by WizKid
Answered By 0 points N/A #90774

Need to screen scrape a website from a classic asp 3.0 page

qa-featured

You will need to use a technique similar to AJAX. The difference is you will require to use a server side object instead of a client side object. Microsoft has released a server side object called ServerXMLHttp object. This object is the server side equivalent of the client side XMLHttp object. The methods and properties of the object are almost similar.

Answered By 0 points N/A #90776

Need to screen scrape a website from a classic asp 3.0 page

qa-featured

You will be far better off by using the search Application Programming Interface development kits released by major search providers. They give rich functionality. They provide Web Services that you can call and retrieve the set of information that you require.

Check the search SDK provided by Google and the BOSS SDK provided by Yahoo. You may need to write a COM+ object using the SOAP toolkit as you are using the Classic ASP 3.0. You then register it in the machine and create the object using the Server. Create object method. I do not think there is direct support for using SOAP inside a Classic ASP 3.0 (at least I have never used it).

Answered By 120 points N/A #90777

Need to screen scrape a website from a classic asp 3.0 page

qa-featured

Thank you WizKid and Thank you Electrica. I more or less prefer less coding and getting the work done within the means of ASP 3.0. Writing a COM+ object is going to be an overload. WizKid, can you provide me some sample code on using the ServerXMLhttp object?

Best Answer
Best Answer
Answered By 0 points N/A #90778

Need to screen scrape a website from a classic asp 3.0 page

qa-featured

Following five lines of code should do it. The first line creates the ServerXMLHttp object. The second line then opens a connection to the report page. The example shows a POST request (as in your case). Then it sets the header and posts the data. In this case it will be your search data as key value pairs. The response is then read back into a variable.

Code:

  1. Set xmlServerRequest = Server.CreateObject("MSXML2.ServerXMLHTTP")
  2. xmlServerRequest.Open "POST","full url", False
  3. xmlServerRequest.setRequestHeader "Content-Type", "text/xml"
  4. xmlServerRequest.send dataString
  5. result = xmlServerRequest.responsexml.xml
Let me know if it works.

 

Answered By 120 points N/A #90780

Need to screen scrape a website from a classic asp 3.0 page

qa-featured

It works for Google. But it does not work to other selected search engines. The xml property returns empty. Any reason as to why this is happening?

Answered By 0 points N/A #90781

Need to screen scrape a website from a classic asp 3.0 page

qa-featured

I believe this is because the data that is return is not in XHTML format. You can alter the code and get the result of the responseText property instead of the responseXML property.

PS: This is why it is recommended that all web pages are XHTML compliant to make cross application integration and information exchange smooth and simple.

Answered By 120 points N/A #90783

Need to screen scrape a website from a classic asp 3.0 page

qa-featured

Thank you WizKid, the responseText property works. Thank you again for your help and advise!

Related Questions