SunQuest
           Business Help
 
Forums: » Register « |  User CP |  Games |  Calendar |  Members |  FAQs |  Sitemap |  Support | 
User Name:
Password:
Remember me
Go Back   Dev Shed ForumsWeb Site ManagementBusiness Help

Reply
Add This Thread To:
  Del.icio.us   Digg   Google   Spurl   Blink   Furl   Simpy   Y! MyWeb 
Thread Tools Search this Thread Rate Thread Display Modes
 
Unread Dev Shed Forums Sponsor:
Stop making mediocre tutorials.The best tutorials are video! Camtasia Studio makes it easy to create engaging, buzz-building screen videos at any size, in any popular format. Download the free trial!
  #1  
Old June 17th, 2006, 04:24 PM
ffoeg's Avatar
ffoeg ffoeg is offline
Self-proclaimed Googlebot
Dev Shed Novice (500 - 999 posts)
 
Join Date: Jul 2005
Location: Edmonton, Alberta, Canada, Earth, Solar System, Milky Way Galaxy, Universe
Posts: 648 ffoeg User rank is Sergeant Major (2000 - 5000 Reputation Level)ffoeg User rank is Sergeant Major (2000 - 5000 Reputation Level)ffoeg User rank is Sergeant Major (2000 - 5000 Reputation Level)ffoeg User rank is Sergeant Major (2000 - 5000 Reputation Level)ffoeg User rank is Sergeant Major (2000 - 5000 Reputation Level)ffoeg User rank is Sergeant Major (2000 - 5000 Reputation Level) 
Time spent in forums: 6 Days 2 h 51 m 4 sec
Reputation Power: 32
Send a message via MSN to ffoeg Send a message via Yahoo to ffoeg
Facebook MySpace
Spidered data: sellable?

I have been writing and running this spider app of mine for a few months and I have fully indexed 50000 web pages, files, and images with it. It has found close to a million links so far for future indexing as well.

It has the full body text of indexed web each page, stripped of all html, thumbnails of most of the images and id3 tags from the mp3s etc etc etc in a 'description' categoy. If the document had a title, that's in the 'title' row, else it's the filename. I have attempted to mark spam, commercial and porn sites as well.

The data is good for searching in but it bogs my system down too much to run an actual public search engine on my site. Is there any companies that buy this kind of stuff? I imagine email lists go well but I won't be spidering that.

Reply With Quote
  #2  
Old July 20th, 2006, 01:41 AM
joej0e joej0e is offline
Contributing User
Dev Shed Newbie (0 - 499 posts)
 
Join Date: Jul 2006
Posts: 142 joej0e User rank is Sergeant Major (2000 - 5000 Reputation Level)joej0e User rank is Sergeant Major (2000 - 5000 Reputation Level)joej0e User rank is Sergeant Major (2000 - 5000 Reputation Level)joej0e User rank is Sergeant Major (2000 - 5000 Reputation Level)joej0e User rank is Sergeant Major (2000 - 5000 Reputation Level)joej0e User rank is Sergeant Major (2000 - 5000 Reputation Level) 
Time spent in forums: 1 Day 23 h 28 m 27 sec
Reputation Power: 25
Just a gentle reminder that you could be infringing on copyright laws on some of the sites you spidered, you should definitely check on this area first before attempting to sell the stuff.

Reply With Quote
  #3  
Old July 20th, 2006, 11:25 AM
Veamon's Avatar
Veamon Veamon is offline
Blowing up trees with my mind.
Dev Shed Frequenter (2500 - 2999 posts)
 
Join Date: Feb 2005
Location: Neverland Ranch, minus the touching.
Posts: 2,712 Veamon User rank is Lieutenant General (80000 - 90000 Reputation Level)Veamon User rank is Lieutenant General (80000 - 90000 Reputation Level)Veamon User rank is Lieutenant General (80000 - 90000 Reputation Level)Veamon User rank is Lieutenant General (80000 - 90000 Reputation Level)Veamon User rank is Lieutenant General (80000 - 90000 Reputation Level)Veamon User rank is Lieutenant General (80000 - 90000 Reputation Level)Veamon User rank is Lieutenant General (80000 - 90000 Reputation Level)Veamon User rank is Lieutenant General (80000 - 90000 Reputation Level)Veamon User rank is Lieutenant General (80000 - 90000 Reputation Level)Veamon User rank is Lieutenant General (80000 - 90000 Reputation Level)Veamon User rank is Lieutenant General (80000 - 90000 Reputation Level)Veamon User rank is Lieutenant General (80000 - 90000 Reputation Level)Veamon User rank is Lieutenant General (80000 - 90000 Reputation Level)Veamon User rank is Lieutenant General (80000 - 90000 Reputation Level)Veamon User rank is Lieutenant General (80000 - 90000 Reputation Level)  Folding Points: 127259 Folding Title: Super Ultimate Folder - Level 1Folding Points: 127259 Folding Title: Super Ultimate Folder - Level 1Folding Points: 127259 Folding Title: Super Ultimate Folder - Level 1Folding Points: 127259 Folding Title: Super Ultimate Folder - Level 1Folding Points: 127259 Folding Title: Super Ultimate Folder - Level 1Folding Points: 127259 Folding Title: Super Ultimate Folder - Level 1
Time spent in forums: 3 Weeks 3 Days 23 h 43 m 24 sec
Reputation Power: 878
I think Google beat you to it.
__________________

Raid1 in XP Pro

My open source projects:
------------------------
Blobber - Add images as blobs to SQL Server
------------------------

Reply With Quote
Reply

Viewing: Dev Shed ForumsWeb Site ManagementBusiness Help > Spidered data: sellable?


Thread Tools  Search this Thread 
Search this Thread:

Advanced Search
Display Modes  Rate This Thread 
Rate This Thread:


Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is On
HTML code is Off
View Your Warnings | New Posts | Latest News | Latest Threads | Shoutbox
Forum Jump


Forums: » Register « |  User CP |  Games |  Calendar |  Members |  FAQs |  Sitemap |  Support | 
  
 





© 2003-2008 by Developer Shed. All rights reserved. DS Cluster 5 hosted by Hostway