ideas for WWW Mailing List archives

Hello,

Looking for ideas for making a mailing list available via a web page, and it needs to be searchable. Ideally it would need to be updated pretty frequently, perhaps as much as evey quarter hour?

This is a high-volume list, averaging btwn 2000-3000 messages per month, sometimes as high as 200-300 per day.

Eventually, I'd like to add some other related lists, but they are *much* lower volume, perhaps a few hundred to a thousand per month total per list.

I was wondering if something could be set up w/ a SQL database or something to keep things indexed?

I really have no prior experience w/ this sort of thing, but I'm interested in learning what I can.

TIA,

nuk

depends on where the news are stored.
if in traditional "usenet" news, you could index them once per 15 minutes into a database (databases provide the fastest search i think)

if you have much cpu & harddisk power related to the number of people accessing the search, just do a full-text search in text-files, newsgroups or whatever.

programming a search script for 200-300 mails per day should not be a problem for a server of today... unless several 10s users use it at the same time.

can you access the way the mailing list is stored? then make an index while processing new posts. it will be as much up-to-date as possible...

The messages are received as part of a mailing list, so I could potentially have them in about any format I really wanted. IIRC, there are 'raw' message archives available for past months/years. This is kind of an idea that I have running thru my head at this stage, whether or not it would be something doable on a hosted website w/ say, several hundred megs of space and perhaps PHP/MYSQL available. Right now, most of the existing / previous archives use perl and something like MHonarc, Glimpse, or HT://dig, and claim that to rebuild the index takes so long that they can only afford (system resource-wise) to re-index once a day, sometimes once a week. I have no idea what other load may have been on those particular machines.

Thanks,

nuk

the "format" i am talking about is the way you access the files. do you get them via email, usenet, text-file, ... ?
you don´t need to rebuild the whole index on every new post. just add a line to it. maybe this requires you not to use ht://dig et al, but write your own database-driven script.

Right now I get the messages via email, and they are downloaded from my ISP w/ fetchmail every 5 minutes, processed thru a series of procmail recipes, which sorts my messages according to mailing list, and then puts them in a directory; my mail client uses a MH format for handling messages (each message is a separate file). It wouldn't be hard to fork off a copy of each message into a standard mbox file using procmail, though.

Thanks,

nuk

procmail is fine. it supports piping the mail to a script that can upload it to eg. a mysqldatabase then. you can setup this in your procmailrc. i would go this way.

Do you know of anything 'pre-canned' that does this (the web/mysql part) or something similar that I could look at for learning purposes?

Thanks,

nuk

sorry, i don´t know many pre-made scripts. but i can give you some code to play with:

say php is installed on your system, it is available as executable and not only apache module.
procmail pipes the email to your script like this: |php /root/scripts/mail_in.php?subject=subject&body=body

your database "news" has one table "news":
id: int 11 unsigned auto-increment;
subject: blob;
body: blob;

then your script should do this:

to insert the mail into a database.

your search.php can look like this:


you probably need to modify a lot though...

Thanks for the ideas! It'll probably be a little while before I get up to speed enough to tackle coding the whole thing myself, but I guess it'll give me a direction to head in.

Thanks,

nuk