I think you've misunderstood how public cryptography work.
Example (We'll use good ol' Alice and Bob):
Alice has one private and one public key.
Bob got one private and one public key.
Bob import Alice public key into his key chain.
Alice import Bobs public key into your key chain.
Alice encrypts a mail to you Bobs public key.
Bob decrypt the message using his private key.
You see, once the message is encrypted by Alice with Bob's public key she can't decrypt it, even though she was the one doing all the encrypting. For this, only Bobs private key will do.
So, translating this to your problem it would be:
Your web server imports your public key. And you import the web serverís public key. The web server encrypts a message and sends it off to you. You use your private key to decrypt it.
The only danger with having the web serverís private key on the web server is that if someone hacks your computer, they can forge messages. It won't compromise the safety of the encrypted messages though. The same goes for your computer though, if someone "borrows" your computer, they can send encrypted mails from it pretending to be you.
Only way of protecting against this is keeping the servers in a locked room and keep up the general security which you should have during normal operations anyway (update system frequently, use a firewall, implement IDS (or NIDS, or both), have some sort of file system integrity check, review your logs regularly etc.) which I hope you or whomever are in charge of the servers are doing already.
As I read your message, I got the impression you weren't interested in authentication anyway, so this security hazard may not apply to you, in which case you have nothing to worry about at all.
For a basic and easy to follow description on how public and private cryptography work, look here: http://www.geekgirls.com/net_publick.htm