Screen-scraping certain websites?!

Hi guys and thanks for reading my thread.

I wasnt sure where to post this but here goes. Ive been looking at using VBScript in ASP pages to parse web content.

So far I have created scripts, for a project that I am working on, that can parse the data I require out of the following (example) web addresses:

http://site.forum.betfair.com/jive3/betex/forums.jspa?forumID=25&schatname=#forumanchornull

and

http://horses.sportinglife.com/Meetings/

Ive been using the xmlHTTP object to get a response text, which is then parsed using a few VBScript functions to pick out the text that Id like to store. The scripts work because the data, text, numbers etc that Id like are written in the sourcecode for the page (i.e. right click page ---> view source in I.E)

Where I have come unstuck is that there are other pages that I would like data from but I have absolutely no idea how the page is generated. Take a look at this site (unfortunately I cant provide a link straight to the right page...):

http://sports.betfair.com/

On the menu on the left, navigate to the following:

All Markets --> Soccer --> English Soccer --> Barclays Premiership ---> Winner 2006/2007

It should be possible to see a betting market with premiership teams down the left side, with a table of odds/money in blue and pink squares.

I have viewed the source code for pages like this and I cant seem to work out how the page is created. There are programs and other sites "out there" that I know are getting data from this website, but my XMLresponse method obviously isnt going to work.

I wonder if someone could advise me as to what sort of method or programming language I could use to extract data from a site like this, and also how much effort it would take to write a script that would be able to read the text and numbers from the betting markets.

Thanks to anyone who can offer any help or advice on this and thanks for reading such a lengthy post.

Regards,

Porky

They're certainly going out of their way to hide the content, but then again it's their job

Have you looked for RSS feeds with this information, any reason you're targetting this particular site?

Yes they are. I think the reason is that they are trying to sell people "API applications" that are small programs that retrieve and display the data in a different way, and you have to pay for them.

The reason that I am targetting this particular site is that its a betting exchange rather than a bookies. Its important for the project that I am working on as I will need to be able to both back (I.e. bet that something will win) and lay (bet that something will lose) any selection, and this is on possible to do on a standard betting site, and the odds on betting exchanges are always slightly different to those in a bookies.

The betfair website has an RSS feed but its for market results (as in, football matches, completed horse races etc) rather than for current market prices, which is the bit that I need.

I think Im into finding someone on the betfair developers API forum who could help me out, and just find out how much money they would be expecting to make it for me :/

Hmmmmmm.