general - Apache Worker MPM configuration help required

We are currently on Amazon EC2 on Medium Instance. We wanted to know if our Apache

Worker MPM configuration is correct or do we need to tweak this to best suit our

requirement:

Server Configuration:
Intel(R) Xeon(R) CPU E5430 @ 2.66GHz, 2 cores
8 GB Ram
OS: Ubuntu Linux 12.04.1
Kernel & CPU: Linux 3.2.0-31-virtual on x86_64

Apache Worker MPM configuration:



Traffic on the server:
Almost 1000 processes running (High traffic)
600 processes (Normal traffic)
No. of Users online:
150-160 (High traffic)
90-100 (Normal traffic)

MySQL CPU usage is around:
90-100% (high traffic)
40-50% (normal traffic)

Also, MySql Max connections: 150

Could anyone recommend the best possible configuration for Worker MPM. What could be an ideal Max. Clients?

Do we also need to tweak the MySQL settings?

Are you having problems? If not, don't mess with it. If so, tell us what they are.

We generally face issues when the no. of users on the site are more.

We run a social networking site with a chat module enabled for logged in users. The site is database intensive where most of the load is on the MySQL.

Typically there are about 100 users online when the site runs normally with all pages loading fast. As the no. of users increase to 150-160, we see sluggishness on the server - the MySQL CPU usage goes to 100%, shows a memory usage of 3.5%, the no. of processes running shows around 1000-1100. That being a cause of concern to us.

I believe, that if we do the right optimization to the apache server, php settings and mysql settings, the load should come down. Hence, my query.

What does top show during these peak times?

Have you configured Apache's server status handler? What's that look like during peak times?

Are you hitting your MySQL max connections?

Are you hitting your Apache MaxClients?

Are you aware that worker shouldn't be used with MPM? I've seen plenty of people do it, and I've done it, but it's not suppose to be done (reference). Just keep it in mind.

Do you have a MySQL slow query log?

What's the response times for a page on your website under no load (in your development environment)?

Top during peak hours: Top


Apache server status handler: No. We have not configured this.

MySql max. connections reached: No. We have configured max. connections to 150. The figure we reached here is 84 checked via MySql tuner.

Hitting MaxClients: No. We have checked error logs and found nothing on this.

MySql slow query: We have previously run MySql slow queries and optimized / indexed the ones which were logged as slow queries. As of now, there are hardly any queries which show up in slow queries except 1 or 2 in a day.

The avg. response time (production): I am not sure how to test this. But tested this on a response testing site. : gave a result of 0.6 seconds.

I would recommend configuring the Apache server status so you can get a better idea of what Apache is doing. Yes it adds some overhead, but it's well worth it, particularly when troubleshooting. I would also turn ExtendedStatus on so it will tell you what's being requested. This will tell you whether or not you're hitting an Apache connection limit.

top shows that your server has had to use swap space at some point, and that's never a good thing on a web server. You'll either have to increase RAM or decrease use of RAM; the latter can be difficult to pinpoint. You're also pushing your CPUs pretty hard.

PHP sripts are the usual culprit in these situations—and a PHP-based chat script is a good place to start. And often, increasing the RAM and CPUs may allow you to handle more users, but eventually you'll end up in the same situation. That is, these processes will eventually take everything you give them, sometimes without really allowing more users.

One approach, although likely not feasible, would be to disable the chat portion during peak times to see how that affects the server's performance.

Is that a custom built chat or something else?

Thanks Jharnois for pointing out precisely these things.

We will setup the apache server status with extendedstatus and see what it shows up.

Coming to PHP, you are very true. PHP is the main culprit here. Moreso, becuase we have a PHP / MySql based customized chat installed. A few days back, we installed a new chat app which works on Long Polling (Nginx & PushStream based) which has resulted in lowering the usage of PHP. Nonetheless, we are still trying to improve on this front.

I shall update you once we see what Apache Server status tells us.

Thanks again and wishing you a very happy new year!

Hi Jharnois,

Here is the report from the Apache Server Status (part 1):

Apache Status (Part 2):

Is that under load? You have plenty of open slots; I wouldn't expect that under load.

Is /Register/user-img/resize_1285480599.jpg an already resized static image file or is that being rewritten to a dynamic image resizer? If it's the latter, you should consider caching that or doing the resize once and calling a static file instead.

This shows a lot of the same or very similar scripts being called, so you should start with those. Obviously there's your chat script, which is likely the problem, but I'd look at its onlinecount1.php script, too, to see if it could be cleaned up. And usersload.php.

It's difficult to optimize for a PHP-based chat script. There are some basic, always do this on your site things, but I'm sure you've Googled and implemented what you could from those.

Have you used Firebug to profile your pages and requests to see how long each step is taking?

Yes. This report was generated under load.

The /Register/user-img/resize_1285480599.jpg image is just the name of the image. The image could be 400 px x 300 px and when the PHP / HTML page calls is, the image is resized into a 117 px x 147 px box. So, the file is already cached in the browser after the first call.

I will certainly look into the onlinecount1.php & usersload.php. These are some troublesome scripts.

The page loading seems to be fine since most of the images / css / js get cached. The issue seems to be with php files.