December 12th, 2016, 04:27 AM
Automated internet search algorithm
I would like to know if it is technically possible to develop an automated internet search algorithm.
Let me give you a random example of what I have in mind.
For example, let’s say I am the largest disaster relief organization in the world and I want to run an automated google search every single day to see if there is any crisis/disaster happening anywhere around the world. The reason I want an automated search (not manual) because I may have many different key word combinations that I want to use (let’s say 1000 different key word combinations) and it would be hectic to do it manually. Let me give few key word examples that this disaster relief organization might want to use every day:
- “Disaster” + “need help urgently”
- “Massive disaster” + “no help”
- “People are suffering” + “earthquake”
- “Where is the UN”
- “We are trapped”
- “Africa” + “hunger” + “no food”
There might be hundreds of key word combinations that the disaster relief organization want to type every single day and they want an automated search algorithm to run all these key word combinations daily and then extract the key links in a page or table. Of course in Google you can also restrict the date – for e.g. ask Google to only give new results for the last day … since the organization will be running the automated algorithm each day and it only wants to see the new results.
The above was a random example for a disaster relief organization, but it can be for any agency or company.
My two most important questions are:
- Is this technically feasible to do?
- If it is possible, then who is the type of person that can do it? What is his “job title” or expertise or qualifications?
December 12th, 2016, 07:29 AM
There's no one answer to this. It depends on the type of content you're looking for. With your example, I would say to forget Google and go with Twitter: that's the best source of real-time information around nowadays.
But sure, as a general concept, yes you can periodically run Google searches for information and compare results over time.
Finding a developer depends on the medium. Web application? Smartphone app? Desktop tool? That's where most of the knowledge needs to be.
December 13th, 2016, 06:11 AM
Thank you for the reply.
If it is for web application, will the developer just be software developer or algorithm developer?
Originally Posted by requinix
In general, are the experts who can do this rare? Or is it simple and widely available?
It would be great if you could provide the name of companies or individuals that provide this service, and customers that implemented such service for their business.
December 13th, 2016, 06:18 AM
Originally Posted by Zainab-GR
I don't think there are any experts for it. If you're lucky you'll find someone with experience using Google's custom search engines, but for everyone else they can learn how to do it.
Originally Posted by Zainab-GR
January 27th, 2017, 01:36 PM
I will also recommend twitter instead of google for this.
You would also need to use numerous proxies since google might ban the automated searches.
These type of software are called parsers or scrapers.
I know 2 guys who can do it, but what is the budget?