A Product Grabbing Script - Where do i start?

Hi all,

this is my first post and although have used this website before have never jumped into the forums


I have a degree in AI / computer science and am a strong programmer in several languages although web based scripting languages are not my strong points (except for javascript)

I can however learn most languages quickly.

I need a script that will either work perfectly (unlikely i know!) or can be customised by me to do what i want.

I am in the process of designing a website that will pull together products (the same product) from various electronic websites and dynamically update the stock status. I would like to be able to enter url's for the product page and let the script periodically read the url and update say an sql database on my site.

I have never used anything like this before and so have tried searching google for scripts out there but the only ones i could find tended to revolve around the XML affiliate feeds that big shops offer (amazon etc).

However, not all the retailers use these feeds so i need something that can just pull the prices / stock status etc from the pages i choose.

So to sum up,

Can anyone think of any scripts that would be ideal for this pupose? preferably free and customisable.

What language will be best to do this? PHP?

Is this going to be a relativly simple process or quite complex?

Also, while i am here - can someone recommend a free system that can be used to create new users / allow each user a 'my account' space where they can update details etc... a bit like you get on forum software. Essentially just for a member area. I presume there are several out there?

Hope i complied with the forum rules here,

Thanks a lot,

edd

For the sites/URLs that don't have feeds you "could" write a shell script that would "wget" the pages... then using your language of choice...write a script that strips everything but the content you're after from the saved page and output it as a CSV text file...then one final script to update the sql DB from the text file...

use a cron job to run the scripts at whatever time interval you want.

I'd use PHP and MYsql... it is far and away the most popular open-source server side software


to a fluent programmer this probably seem clunky but it works for me..

good luck