#1
  1. No Profile Picture
    Registered User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Dec 2016
    Posts
    2
    Rep Power
    0

    Automated internet search algorithm


    Hi,
    I would like to know if it is technically possible to develop an automated internet search algorithm.
    Let me give you a random example of what I have in mind.

    For example, let’s say I am the largest disaster relief organization in the world and I want to run an automated google search every single day to see if there is any crisis/disaster happening anywhere around the world. The reason I want an automated search (not manual) because I may have many different key word combinations that I want to use (let’s say 1000 different key word combinations) and it would be hectic to do it manually. Let me give few key word examples that this disaster relief organization might want to use every day:

    - “Disaster” + “need help urgently”
    - “Massive disaster” + “no help”
    - “People are suffering” + “earthquake”
    - “Where is the UN”
    - “We are trapped”
    - “Africa” + “hunger” + “no food”
    - Etc.

    There might be hundreds of key word combinations that the disaster relief organization want to type every single day and they want an automated search algorithm to run all these key word combinations daily and then extract the key links in a page or table. Of course in Google you can also restrict the date – for e.g. ask Google to only give new results for the last day … since the organization will be running the automated algorithm each day and it only wants to see the new results.

    The above was a random example for a disaster relief organization, but it can be for any agency or company.

    My two most important questions are:
    - Is this technically feasible to do?
    - If it is possible, then who is the type of person that can do it? What is his “job title” or expertise or qualifications?
  2. #2
  3. Forgotten Moderator
    Devshed Supreme Being (6500+ posts)

    Join Date
    Mar 2007
    Location
    Washington, USA
    Posts
    16,042
    Rep Power
    9616
    There's no one answer to this. It depends on the type of content you're looking for. With your example, I would say to forget Google and go with Twitter: that's the best source of real-time information around nowadays.

    But sure, as a general concept, yes you can periodically run Google searches for information and compare results over time.

    Finding a developer depends on the medium. Web application? Smartphone app? Desktop tool? That's where most of the knowledge needs to be.
  4. #3
  5. No Profile Picture
    Registered User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Dec 2016
    Posts
    2
    Rep Power
    0
    Thank you for the reply.

    Originally Posted by requinix
    Finding a developer depends on the medium. Web application? Smartphone app? Desktop tool? That's where most of the knowledge needs to be.
    If it is for web application, will the developer just be software developer or algorithm developer?

    In general, are the experts who can do this rare? Or is it simple and widely available?
    It would be great if you could provide the name of companies or individuals that provide this service, and customers that implemented such service for their business.
  6. #4
  7. Forgotten Moderator
    Devshed Supreme Being (6500+ posts)

    Join Date
    Mar 2007
    Location
    Washington, USA
    Posts
    16,042
    Rep Power
    9616
    Originally Posted by Zainab-GR
    If it is for web application, will the developer just be software developer or algorithm developer?
    Web developer.

    Originally Posted by Zainab-GR
    In general, are the experts who can do this rare? Or is it simple and widely available?
    I don't think there are any experts for it. If you're lucky you'll find someone with experience using Google's custom search engines, but for everyone else they can learn how to do it.
  8. #5
  9. No Profile Picture
    Registered User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Jan 2017
    Posts
    2
    Rep Power
    0
    I will also recommend twitter instead of google for this.

    You would also need to use numerous proxies since google might ban the automated searches.

    These type of software are called parsers or scrapers.

    I know 2 guys who can do it, but what is the budget?

    Expert at:
    pointofsalesoftware.org

IMN logo majestic logo threadwatch logo seochat tools logo