Spider that uses mysql

Dev Shed Forums Sponsor:

April 10th, 2009, 04:56 PM

ianevans

Registered User

Join Date: Jun 2004

Posts: 8

Time spent in forums: 3 m 57 sec
Reputation Power: 0

Spider that uses mysql

I'm looking for a web crawler that will spider my site and toss the urls and page content into a mysql database so that database can , in turn, be indexed by Sphinx Site Search.

Not wanting to reinvent the wheel, I'm wondering if anyone knows of a code snippet that will do this or can point me towards a tutorial.

Thanks.

April 10th, 2009, 07:27 PM

L7Sqr

Contributing User

Join Date: Jan 2004

Location: Constant Limbo

Posts: 989

Time spent in forums: 2 Weeks 2 Days 22 h 45 m 6 sec
Reputation Power: 362

Google is your friend:

linky1
linky2

__________________
True happiness is not getting what you want, it's wanting what you've already got.

My Blog

Viewing: Dev Shed Forums > Programming Languages > Ruby Programming > Spider that uses mysql

« Previous Thread | Next Thread »

Developer Shed Advertisers and Affiliates

Thread Tools	Search this Thread
Email this Page	Search this Thread: Advanced Search
Display Modes	Rate This Thread
Linear Mode Switch to Hybrid Mode Switch to Threaded Mode	Rate This Thread:

Posting Rules
You may not post new threads You may not post replies You may not post attachments You may not edit your posts vB code is On Smilies are On [IMG] code is On HTML code is Off

View Your Warnings | New Posts | Latest News | Latest Threads | Shoutbox

Forum Jump