November 23rd, 2012, 06:36 PM
How do I get search engine results programatically?
How do I get search engine results programatically? Some search engines use AJAX to display content and so the URL does not change. This makes it difficult to get result content from a search engine programatically.
Most search engines do not use AJAX. But two popular ones that I am interested in is YANDEX and DUCKDUCKGO. Both of these search engines use AJAX to advance beyond the first page of results. How do I find a way to programatically access these pages?
November 23rd, 2012, 07:01 PM
Yandex has an API for searches, it would need a fair bit of google translate or russian language skills to make sense of it all though. I hope that helps...I can't post links yet, but Google for Yandex api - I'm sure you'll be able to find it there somewhere
November 24th, 2012, 06:08 AM
I use the c# WebBrowser class and I get the HTML after I use this class to navagate to a URL. I was able to use yandex by using a "&p=" to navagate to a particular page.
The duckduckgo search engine does not work this way, though.
I have found out how to make a specialized query for YANDEX in a modified URL. I will ask DUCKDUCKGO for advice.
https://duckduckgo.com/api.html does not provide a solution.
Running this, http://api.duckduckgo.com/?q=stuff+to+search&format=xml , in a browser's address bar does not provide anything as http://duckduckgo.com/?q=stuff+to+search does.
This shouldn't be rocket science. I am able to see the HTML code. Shouldn't there be some sort of AJAX tag in the HTML that I can find to make use of?