Optimizing Apache/PHP for heavy load -- question for freebsd and experts

I'm trying to optimize Apache for heavy load on a site using PHP 4.0.6 with continuous use of sessions, and many MySQL connections. The server is also using some CGI, and some FrontPage.

Apache was having a lot of trouble handling the sessions, and was frequently failing at writing session files, until I did the following tweaks. It is running OK, now, but still near the limit most of the time.

Here are the tweaks; is there anything else I have missed or done wrong?:

1. recompiled Apache with HARD_SERVER_LIMIT raised to 1024, only the essential modules loaded, and with the following vars set before ./configure:

OPTIM="-O9 -funroll-loops -ffast-math -malign-double -mcpu=i686 -march=i686
-fomit-frame-pointer -fno-exceptions"

2. set the following values for httpd.conf:

MaxKeepAliveRequests 500
KeepAliveTimeout 30

MinSpareServers 20
MaxSpareServers 50
StartServers 35

MaxClients 500

MaxRequestsPerChild 250


3. Linux system changes:

a. Change max filehandles to 16384 or (default was 4096)

b. Change max inodes to 65536 (default was 16384)


4. recompiled PHP with inline optimization and minimal external modules.

---------------

The above is what I have done so far. I am considering two other possibilities:

1. Change /etc/fstab to mount the /usr/ partition with the -noatime option, so that timestamps are not set when files are read-accessed.

2. Mount a RAM partition as the location to save session temp files, (session files need the access timestamp)

The server itself is a Rackspace.com box, with a gigahertz Athlon, 1 GB RAM, 2 18 GB SCSI drives. When I run "top" the CPU is usually at less than 15% usage, but the memory is usually at 85 or 90% used. We can only add another 1/2 GB RAM before we have to upgrade to a larger machine.

Any ideas?

I wouldn't exactly call myself an expert, so I'd take this advice with a grain of salt, but I'll proffer my two bits nonetheless.

Have you considered using more than one build of Apache? If your site is high traffic (it sounds like it), you can save some memory by using multiple builds with only the bare essentials. You need almost nothing installed to serve images, for example. You also don't need to load Front Page Extensions for a PHP page, etc. So, you could probably get a modest performance boost by isolating the needs of specific documents to specific servers.

Also, what OS are you runing? I know that, for instance, Red Hat Linux 7.1 has far more overhead than previous versions as it uses the new kernel.

Try to avoid FrontPage if at all possible. If you can't, set the following is reasonable:

RLimitCPU 60 60
RLimitMEM 60000000 60000000

Increase them:
MinSpareServers 20 -> 30
MaxSpareServers 50 -> 75
StartServers 35 -> 100

MaxClients depends on how many simultaneous requests, setting 500 might be too high.

MaxRequestsPerChild has a very tricky setting and influences memory the most. If it's set too high, the children will die off slowly, thus sucking up memory. Normally you would want to set it as high as 20000 if you serve alot of static files. So try to lower it and adjust it accordingly. If you also use mod_perl, lower it below 80.

>> hange /etc/fstab to mount the /usr/ partition

Disk partitioning could also be a big factor because it affects disk I/O speed. You might want to relocate your docroot partition to a dedicated partition like /www. It's bad if your docroot lives on the same partition as /usr or /home.

>> but the memory is usually at 85 or 90% used

That's normal and optimistic. If it's too low, that's a waste of ram.

Performance tuning of Apache takes time and may vary as it depends on many factors. If you can remove frontpage alone, that should help significantly.

freebsd -

I will definitely be changing the doc_root partition, and testing out your other suggestions in httpd.conf.

Where do I set RLimitCPU and RLimitMEM?

And trust me, I would love to remove FrontPage, but my client insists on using for some stuff :-(. What do you think of Iceman's idea?


Iceman-

It seems an interesting idea, but I'm a little hazy on exactly how I would implement it.

How would I manage to keep each process from trying to serve the wrong files? Would I have to physically separate them, as in build1 serves files from this directory container, and build2 serves files from that directory container?

The OS seems to be some special cooked-up version of RedHat that Rackspace installs, but it's serving kernel 2.2.19, not 2.4.

I personally much prefer running FreeBSD (but I'll live with Slackware), but my client is afraid to change anything right now.

I suppose I could recompile Linux, but since I'm stuck with working remotely on the box, I doubt that is an option.

Thanks for the tips, both of you.

>> Where do I set RLimitCPU and RLimitMEM?

You can define them globally. They impact CGI, SSI and frontpage.

>> What do you think of Iceman's idea?

Great one. You can isolate frontpage alone to another box and NFS exporting or rsync them to your main server. This doesn't help I/O speed but it helps your CPU and Rams just because frontpage eats them like crazy.

Another thing to watch for with Frontpage is the log file (it grows without bound). If users are experiencing connection timeout to frontpage, that's the common cause of the problem.

Thanks so much...

rycamor-

You don't necessarily have to use seperate docroots for the different servers, although it wouldn't hurt.

The Apaches should be bound to different ports/ips as the default server so they don't overlap each other, however. If you can do it, the easiest thing would be give each an IP and set a domain/sub to it.