Help to stop hijacking my email server

I have a question i would like to ask that somebody maybe able to help me with and point me in the right direction.

I run my own server from home , Its more of just a hobby site than anything and a place for me to learn , My site is a php-nuke site and i run apserv , which consists of apache, mysql and php , and i also use qk smtp server to send the activation emails the emails get sent via mail() and in qk smtp they arrive via URL, it also sends emails from the forum when people are watching topics , now everything works fine but everynow and then, not very often it has happened about 3 times this year so far somebody has hijacked my email server to send large amounts of email, i presume they are just spam mail as the arrive with random sender , iw URL , there are mass amounts of these normally i catch it when i get up in the morning but not before about 30,000 have been sent and another 50,000 in the que before i shut the email server down and delete the que , is there anyway that i can stop this and safe guard this from happening again , is there a better email server i could use maybe , taking into account all it is used for is sending emails.
Any advice would be most helpful

Simple solution for you. Download Zone Alarm (freeware version) and BLOCK all incoming connections to the port 25 on your pc. That's it you can bid good bye to those hijackers

Another way (depending if you're a technical guy) is you can configure your email server to AUTHENTICATE all users before sending emails so that only genuine user(s) of your email server would be able to avail the SMTP facility of your email server.

But i'd suggest you go with solution no. 1, i mean why even entertain traffic from trouble makers coz they will try to hammer your server again and again. With solution 1 your pc will not respond to them at all.

Hope this helps.

Check with qk smtp server if you have an option for authorizing the user before sending out emails. If yes, then you can set it to authorize users before sending out emails and those who are not allowed won't be able to relay emails using your mail server.

only allow your local network to send mail.
Sounds to me like you are indeed an open mail relay and orbs etc will block you sooner or later.
Either that or you have in fact been compromised and need to rebuild server.
(You dont HAVE TO rebuild, but you never know what secrets they have left for you so is the prudent thing to do.)