#1
  1. No Profile Picture
    Registered User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Sep 2013
    Posts
    2
    Rep Power
    0

    Web development resource


    Hi community,

    I'm asking for a high level understanding of what technologies would be considered to create a website that could crawl other websites and pull their content based on specified logic.

    Its not intended to sound shady! I'd like to understanding the basic concept of how to do this and look into - however I'm struggling to grasp this.

    Anyone who'd be kind enough to give a brief overviews of the technologies/software (e.g. does WordPress allow this kind of functionality to be applied - early looking into suggests not to me?) to consider would be really appreciated!

    Regards,
    Ant
  2. #2
  3. Contributing User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Sep 2013
    Location
    Saint-Petersburg, Russia
    Posts
    240
    Rep Power
    29
    Hi!

    Shortly speaking, these are two separate programs, you tell about:
    - crawler which visits web-pages and fetches info from here;
    - web-site to display info about crawling results.

    They need not be necessarily written as the single app. For example, crawler can store its result to your website from time to time, not continuously.

    As of technologies - there is not anything special. Anything could be even written from scratch. For example web-site could be written in PHP - perhaps, using some convenient framework, like CakePHP - and crawler could be done in Python or Java for example.

    (of course web-site could be written also in java, python etc - and crawler could be created in C++ or even PHP or tons of other languages)

    As the good crawler is not a simple thing, perhaps you will prefer to take some half-done library for this. Much depends on which language you want to use. You can find a lot of crawlers by googling for "crawler java" or "crawler python" etc.
  4. #3
  5. No Profile Picture
    Registered User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Sep 2013
    Posts
    2
    Rep Power
    0
    Thanks Rodiongork!

    It may be a suprise to hear, but I'm fairly new to website creation and designing!

    When creating the website and it's actual content, is it not easier and just as good to use software such as Wordpress? As I understand it, this gives me a GUI which will automatically build the code as I create a website via the software's GUI. Does this type of website design (i.e. using software instead of self-coding) hinder a website, and how-if so?

    Can programming languages that allows bespoke functionality (like web crawling) be easily incorporated into softwares that build websites like WordPress?

    Thanks again Rodiongork.

    Regards,
    Ant

IMN logo majestic logo threadwatch logo seochat tools logo