UNIX Help
 
Forums: » Register « |  User CP |  Games |  Calendar |  Members |  FAQs |  Sitemap |  Support | 
User Name:
Password:
Remember me
Go Back   Dev Shed ForumsOperating SystemsUNIX Help

Reply
Add This Thread To:
  Del.icio.us   Digg   Google   Spurl   Blink   Furl   Simpy   Y! MyWeb 
Thread Tools Search this Thread Rate Thread Display Modes
 
Unread Dev Shed Forums Sponsor:
  #1  
Old January 27th, 2006, 02:08 PM
crustymonkey's Avatar
crustymonkey crustymonkey is offline
Contributing User
Dev Shed Newbie (0 - 499 posts)
 
Join Date: Oct 2003
Location: Minneapolis, MN
Posts: 195 crustymonkey User rank is Lance Corporal (50 - 100 Reputation Level)crustymonkey User rank is Lance Corporal (50 - 100 Reputation Level)crustymonkey User rank is Lance Corporal (50 - 100 Reputation Level) 
Time spent in forums: 1 Day 6 h 38 m 24 sec
Reputation Power: 5
Best lightweight database to use for log files

I just want to throw this question out there to see what other people's opinions are on this subject. My goal in my current project is this. I want to parse the Postfix log files and dump the contents to a database for fast lookups by our support staff. Currently the log files reach sizes on a per-day basis of approximately 12 gigs (this is for a small ISP). We frequently need to access the log files to troubleshoot email problems that customers have. What I'm looking for is something that is lightweight, simple, has a Perl API and is fast for lookups. My experience to this point has been with SQL databases such as postgres, MySQL and MSSQL. I'm thinking that these may be a slower and more feature ridden than what I'm really looking for, but then again, I may be wrong... Anyhow, I'm looking for opinions here. Incidentally, this is a Sun OS 5.8 machine.
__________________
badger badger badger badger
badger badger badger badger
MUSHROOM MUSHROOM

Reply With Quote
  #2  
Old January 27th, 2006, 03:57 PM
codergeek42's Avatar
codergeek42 codergeek42 is offline
[Insert clever comment here.]
Dev Shed God 2nd Plane (6000 - 6499 posts)
 
Join Date: Jul 2003
Location: Anaheim, CA (USA)
Posts: 6,435 codergeek42 User rank is General 3rd Grade (Above 100000 Reputation Level)codergeek42 User rank is General 3rd Grade (Above 100000 Reputation Level)codergeek42 User rank is General 3rd Grade (Above 100000 Reputation Level)codergeek42 User rank is General 3rd Grade (Above 100000 Reputation Level)codergeek42 User rank is General 3rd Grade (Above 100000 Reputation Level)codergeek42 User rank is General 3rd Grade (Above 100000 Reputation Level)codergeek42 User rank is General 3rd Grade (Above 100000 Reputation Level)codergeek42 User rank is General 3rd Grade (Above 100000 Reputation Level)codergeek42 User rank is General 3rd Grade (Above 100000 Reputation Level)codergeek42 User rank is General 3rd Grade (Above 100000 Reputation Level)codergeek42 User rank is General 3rd Grade (Above 100000 Reputation Level)codergeek42 User rank is General 3rd Grade (Above 100000 Reputation Level)codergeek42 User rank is General 3rd Grade (Above 100000 Reputation Level)codergeek42 User rank is General 3rd Grade (Above 100000 Reputation Level)codergeek42 User rank is General 3rd Grade (Above 100000 Reputation Level)codergeek42 User rank is General 3rd Grade (Above 100000 Reputation Level)  Folding Points: 39542 Folding Title: Starter FolderFolding Points: 39542 Folding Title: Starter Folder
Time spent in forums: 1 Month 1 Week 6 Days 10 h 39 m 10 sec
Reputation Power: 1158
Send a message via ICQ to codergeek42 Send a message via AIM to codergeek42 Send a message via Yahoo to codergeek42 Send a message via Google Talk to codergeek42
Quote:
Originally Posted by crustymonkey
Incidentally, this is a Sun OS 5.8 machine.
Moved from the Linux Help forum to UNIX Help...
__________________
~~ Peter ~~
( My Blog: It's exactly like normal nerdiness, but completely different. ) :: ( Supporter of the EFF & FSF ) :: ( I'm a GNU/Linux addict and Free Software Advocate. ) :: ( How to Ask Questions the Smart Way ) :: ( The Fedora Project, sponsored by Red Hat ) :: ( GNOME: The Free Software Desktop Project ) :: ( GnuPG Public Key )

Reply With Quote
  #3  
Old January 27th, 2006, 05:07 PM
SimonGreenhill's Avatar
SimonGreenhill SimonGreenhill is offline
(retired)
Dev Shed God 11th Plane (10000 - 10499 posts)
 
Join Date: Dec 2003
Location: The Laboratory
Posts: 10,101 SimonGreenhill User rank is General 5th Grade (Above 100000 Reputation Level)SimonGreenhill User rank is General 5th Grade (Above 100000 Reputation Level)SimonGreenhill User rank is General 5th Grade (Above 100000 Reputation Level)SimonGreenhill User rank is General 5th Grade (Above 100000 Reputation Level)SimonGreenhill User rank is General 5th Grade (Above 100000 Reputation Level)SimonGreenhill User rank is General 5th Grade (Above 100000 Reputation Level)SimonGreenhill User rank is General 5th Grade (Above 100000 Reputation Level)SimonGreenhill User rank is General 5th Grade (Above 100000 Reputation Level)SimonGreenhill User rank is General 5th Grade (Above 100000 Reputation Level)SimonGreenhill User rank is General 5th Grade (Above 100000 Reputation Level)SimonGreenhill User rank is General 5th Grade (Above 100000 Reputation Level)SimonGreenhill User rank is General 5th Grade (Above 100000 Reputation Level)SimonGreenhill User rank is General 5th Grade (Above 100000 Reputation Level)SimonGreenhill User rank is General 5th Grade (Above 100000 Reputation Level)SimonGreenhill User rank is General 5th Grade (Above 100000 Reputation Level)SimonGreenhill User rank is General 5th Grade (Above 100000 Reputation Level)  Folding Points: 4925 Folding Title: Novice Folder
Time spent in forums: 3 Months 3 Weeks 5 h 49 m 4 sec
Reputation Power: 1331
Facebook
MySQL is one of the lighter-weight DB's ( especially if you go for an older version like 3.2x or 4.0 - they've progressively added more features, and the latest 5.0 is noticeably slower ).

There are two others that are much lighter, but I'm not sure if they'll work on your OS. Firebird ( haven't used it myself, but I've heard very good things ) and SQLite which is very cool, and only a few hundred kb. Not sure how well it handles multiple users, but this may not be a problem for you. It can handle DB's up to 2TB, and is very fast.

--Simon

Reply With Quote
  #4  
Old January 29th, 2006, 03:38 AM
LinuxPenguin's Avatar
LinuxPenguin LinuxPenguin is offline
fork while true;
Dev Shed God 1st Plane (5500 - 5999 posts)
 
Join Date: May 2005
Location: England, UK
Posts: 5,535 LinuxPenguin User rank is General (90000 - 100000 Reputation Level)LinuxPenguin User rank is General (90000 - 100000 Reputation Level)LinuxPenguin User rank is General (90000 - 100000 Reputation Level)LinuxPenguin User rank is General (90000 - 100000 Reputation Level)LinuxPenguin User rank is General (90000 - 100000 Reputation Level)LinuxPenguin User rank is General (90000 - 100000 Reputation Level)LinuxPenguin User rank is General (90000 - 100000 Reputation Level)LinuxPenguin User rank is General (90000 - 100000 Reputation Level)LinuxPenguin User rank is General (90000 - 100000 Reputation Level)LinuxPenguin User rank is General (90000 - 100000 Reputation Level)LinuxPenguin User rank is General (90000 - 100000 Reputation Level)LinuxPenguin User rank is General (90000 - 100000 Reputation Level)LinuxPenguin User rank is General (90000 - 100000 Reputation Level)LinuxPenguin User rank is General (90000 - 100000 Reputation Level)LinuxPenguin User rank is General (90000 - 100000 Reputation Level)LinuxPenguin User rank is General (90000 - 100000 Reputation Level)  Folding Points: 11590 Folding Title: Novice Folder
Time spent in forums: 1 Month 3 Weeks 1 Day 19 h 30 m 28 sec
Reputation Power: 1008
It depends how much work it will actually be doing but by the sounds of it, i'd go with SQLite. If demand gets too big, switch to PostGresSQL (the `big gun`, if you like). Both are free to use and have lovely PERL modules for using them

Reply With Quote
Reply

Viewing: Dev Shed ForumsOperating SystemsUNIX Help > Best lightweight database to use for log files


Thread Tools  Search this Thread 
Search this Thread:

Advanced Search
Display Modes  Rate This Thread 
Rate This Thread:


Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is On
HTML code is Off
View Your Warnings | New Posts | Latest News | Latest Threads | Shoutbox
Forum Jump


Forums: » Register « |  User CP |  Games |  Calendar |  Members |  FAQs |  Sitemap |  Support | 
  
 





© 2003-2008 by Developer Shed. All rights reserved. DS Cluster 6 hosted by Hostway
Stay green...Green IT