Rotatelogs Log Format

Hi,

I was having trouble with extremely large log files which I needed to analyse remotely and thus suffering huge download times.

I added this to my vhosts.conf :

TransferLog "|/usr/local/apache/bin/rotatelogs /home/users/<domain>/logs/access_log 2592000"

This appears to have worked correctly and is indeed rotating the logs. However the newly rotated log is in a different format from the original access.log. How do I keep it in the same format?

Also, it appears the access.log is still being used as well as the newly created log file. Is this supposed to be the case?

Thanks!

>> having trouble with extremely large log files

Rotate them sooner and don't log everything. For example:

>> the newly rotated log is in a different format
>> How do I keep it in the same format?

So what new format?

Ok this is the old format:

xxx.xx.xxx.xx - - [04/Feb/2002:10:38:31 +0000] "GET /support/upgrading/German/framework1.gif HTTP/1.0" 200 8943 "http://www.<domain>.com/support/upgrading/German/whatsnew.htm" "Mozilla/4.0 (compatible; MSIE 5.0; Windows NT; DigExt; MF)"

And this is the new:

xxx.xx.xxx.xx - - [04/Feb/2002:10:38:31 +0000] "GET /support/upgrading/German/framework1.gif HTTP/1.0" 200 8943

This log file is maintained for a client of ours, I want to capture as much detail as possible for analysis, with the latter format it looks like I won't get browser types and the page that the hit relates to.

That have nothing to do with rotatelogs at all because it doesn't trim your log file. Start here to find out more. And TransferLog directive relies on LogFormat so you need to add a proper log format to log Referer and User-Agent.

>> I want to capture as much detail as possible for analysis

I already suggested you not to log everything. Anyway, just don't complain when your access_log grows so large in short period of time.

Ok point taken. There is no real need to have a log of each image hit.

I would appreciate it if you could briefly explain how the below works then so that I can tailor it to my needs:

SetEnvIfNoCase Request_URI "\.(gif|jpe?g|css|js)$" dont_log
LogFormat "%h %a %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" log_this
CustomLog /var/log/apache/access_log log_this env=!dont_log

Thanks for your help.

1) First go to http://httpd.apache.org/docs/mod/mo...ig.html#formats and create your very own log format.
Then assign that LogFormat line a nickname. In my previous example I assigned it with log_this.

2) You then need to decide what you are not interested to log and assign it an env var (nickname). In my SetEnvIfNoCase example, dont_log.
You can define multiple SetEnvIfNoCase or equivalent lines and assign dont_log as its environment variable. For example, you can add:

SetEnv Remote_Addr "^your.ip.addr.here$" dont_log

Or even:

BrowserMatchNoCase ^Googlebot dont_log

Or if CodeRed/Nimda is filling up your access log:

SetEnvIfNoCase Request_URI "^/(scripts|msadc|_vti_bin|_mem_bin)" dont_log

3) Finally, you need to use CustomLog (not TransferLog), the env=!dont_log is to tell Apache not to log those should a log entry matches the values specified in dont_log environment variable.

You just need 3 lines like the ones I posted previously with a little modification ( to work with rotatelogs) by adding a pipe | in the same position as your TransferLog. Check here for the exact syntax.

Thanks for that, I'm starting to get it clearer in my head now!

However, I'm not sure where I should be putting the pipe from custlog to transferlog... This is how I have it currently:

SetEnvIfNoCase Request_URI "\.(gif|jpe?g|css|js|jpg)$" dont_log
LogFormat "%h %a %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" log_this
CustomLog /home/users/intersharedev/logs/access.log log_this env=!dont_log
ErrorLog /home/users/intersharedev/logs/error.log
TransferLog "|/usr/local/apache/bin/rotatelogs /home/users/intersharedev/logs/access_log 2592000"

Should I change this to :

SetEnvIfNoCase Request_URI "\.(gif|jpe?g|css|js|jpg)$" dont_log
LogFormat "%h %a %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" log_this
CustomLog "|/usr/local/apache/bin/rotatelogs /home/users/intersharedev/logs/access_log 2592000 log_this env=!dont_log
ErrorLog /home/users/intersharedev/logs/error.log

Thanks again!

>> Should I change this to :.......

Yes and PERFECT!.

But gif|jpe?g|css|js|jpg

The jpe?g matches jpeg and jpg so you don't need to define your last jpg.

And you need to remove your leading double-quote on your CustomLog line.

Like I said, you can check your old access log and create your own env var or nickname of what you are not interested in seeing in your access log.

Thanks a lot, you've been a great help!

I tried it and got an error when I ran apachectl configtest which was:

Syntax error on line 60 of /usr/local/apache/conf/vhosts.conf:
CustomLog takes two or three arguments, a file name, a custom log format string or format name, and an optional "env=" clause (see docs)

And this is the line:

CustomLog |/usr/local/apache/bin/rotatelogs /home/users/intersharedev/logs/access_log 2592000 log_this env=!dont_log

Any ideas?

Sorry, I don't use rotatelogs so I'm not familiar with the exact syntax. However, from here the format should be:

so you can try double-quoting your rotatelogs portion like so:

CustomLog | "/usr/local/apache/bin/rotatelogs /home/users/intersharedev/logs/access_log 2592000" log_this env=!dont_log

or even:

CustomLog "| /usr/local/apache/bin/rotatelogs /home/users/intersharedev/logs/access_log 2592000" log_this env=!dont_log

Thanks, the double quotes round the whole line version worked. However the log format remains the same. Presumably it will change upon the next log rotation....

Cheers

>> Presumably it will change upon the next log rotation

Most likely. What you can do is to change your 2592000 (1 month) to a lower second like 120, just to see if it works, then switch it back to 1 month, of course, you need to -HUP Apache to do all this.

Ok, did that and the new log file format is looking a bit skewed.

193.109.72.6 193.109.72.6 [04/Feb/2002:13:41:40 +0000] "GET /xxxxxxxx/xxxxxxx.php HTTP/1.0" 200 15439 "http://www.xxxxxxx.com/xxxxxxxx.php?prevpage=http://www.xxxxxx.com/xxxxxx/xxxxx.php?id=22" "Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 4.0; T312461)"

The IP is turning up twice.

Any ideas?

>> The IP is turning up twice

Because not all IPs have an PTR record. Often, an upsteam only leases a netblock to a particular ISP (just leasing, not delegating). When that IP block is not delegated to that ISP, that ISP has no control to the IP and setup the reverse like user-119adhr.biz.mindspring.com.
But some ISPs are even lazy or clueless about setting up the appropriate PTR for their entire netblock. As a result, the reverse DNS of 193.109.72.6 returns nothing. When Apache receives an empty REMOTE_HOST, it will replace that with REMOTE_ADDR.