#1
  1. No Profile Picture
    Contributing User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Apr 2001
    Location
    London, England
    Posts
    262
    Rep Power
    14

    Rotatelogs Log Format


    Hi,

    I was having trouble with extremely large log files which I needed to analyse remotely and thus suffering huge download times.

    I added this to my vhosts.conf :

    TransferLog "|/usr/local/apache/bin/rotatelogs /home/users/<domain>/logs/access_log 2592000"

    This appears to have worked correctly and is indeed rotating the logs. However the newly rotated log is in a different format from the original access.log. How do I keep it in the same format?

    Also, it appears the access.log is still being used as well as the newly created log file. Is this supposed to be the case?

    Thanks!
  2. #2
  3. No Profile Picture
    Contributing User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Jan 2001
    Posts
    4
    Rep Power
    0
    >> having trouble with extremely large log files

    Rotate them sooner and don't log everything. For example:
    Code:
    SetEnvIfNoCase Request_URI "\.(gif|jpe?g|css|js)$" dont_log
    LogFormat "%h %a %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" log_this
    CustomLog /var/log/apache/access_log log_this env=!dont_log
    >> the newly rotated log is in a different format
    >> How do I keep it in the same format?

    So what new format?
  4. #3
  5. No Profile Picture
    Contributing User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Apr 2001
    Location
    London, England
    Posts
    262
    Rep Power
    14
    Ok this is the old format:

    xxx.xx.xxx.xx - - [04/Feb/2002:10:38:31 +0000] "GET /support/upgrading/German/framework1.gif HTTP/1.0" 200 8943 "http://www.<domain>.com/support/upgrading/German/whatsnew.htm" "Mozilla/4.0 (compatible; MSIE 5.0; Windows NT; DigExt; MF)"

    And this is the new:

    xxx.xx.xxx.xx - - [04/Feb/2002:10:38:31 +0000] "GET /support/upgrading/German/framework1.gif HTTP/1.0" 200 8943

    This log file is maintained for a client of ours, I want to capture as much detail as possible for analysis, with the latter format it looks like I won't get browser types and the page that the hit relates to.
  6. #4
  7. No Profile Picture
    Contributing User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Jan 2001
    Posts
    4
    Rep Power
    0
    That have nothing to do with rotatelogs at all because it doesn't trim your log file. Start here to find out more. And TransferLog directive relies on LogFormat so you need to add a proper log format to log Referer and User-Agent.

    >> I want to capture as much detail as possible for analysis

    I already suggested you not to log everything. Anyway, just don't complain when your access_log grows so large in short period of time.
  8. #5
  9. No Profile Picture
    Contributing User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Apr 2001
    Location
    London, England
    Posts
    262
    Rep Power
    14
    Ok point taken. There is no real need to have a log of each image hit.

    I would appreciate it if you could briefly explain how the below works then so that I can tailor it to my needs:

    SetEnvIfNoCase Request_URI "\.(gif|jpe?g|css|js)$" dont_log
    LogFormat "%h %a %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" log_this
    CustomLog /var/log/apache/access_log log_this env=!dont_log

    Thanks for your help.
  10. #6
  11. No Profile Picture
    Contributing User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Jan 2001
    Posts
    4
    Rep Power
    0
    1) First go to http://httpd.apache.org/docs/mod/mod...g.html#formats and create your very own log format.
    Then assign that LogFormat line a nickname. In my previous example I assigned it with log_this.

    2) You then need to decide what you are not interested to log and assign it an env var (nickname). In my SetEnvIfNoCase example, dont_log.
    You can define multiple SetEnvIfNoCase or equivalent lines and assign dont_log as its environment variable. For example, you can add:

    SetEnv Remote_Addr "^your.ip.addr.here$" dont_log

    Or even:

    BrowserMatchNoCase ^Googlebot dont_log

    Or if CodeRed/Nimda is filling up your access log:

    SetEnvIfNoCase Request_URI "^/(scripts|msadc|_vti_bin|_mem_bin)" dont_log

    3) Finally, you need to use CustomLog (not TransferLog), the env=!dont_log is to tell Apache not to log those should a log entry matches the values specified in dont_log environment variable.

    You just need 3 lines like the ones I posted previously with a little modification ( to work with rotatelogs) by adding a pipe | in the same position as your TransferLog. Check here for the exact syntax.
    Last edited by freebsd; February 4th, 2002 at 07:05 AM.
  12. #7
  13. No Profile Picture
    Contributing User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Apr 2001
    Location
    London, England
    Posts
    262
    Rep Power
    14
    Thanks for that, I'm starting to get it clearer in my head now!

    However, I'm not sure where I should be putting the pipe from custlog to transferlog... This is how I have it currently:

    SetEnvIfNoCase Request_URI "\.(gif|jpe?g|css|js|jpg)$" dont_log
    LogFormat "%h %a %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" log_this
    CustomLog /home/users/intersharedev/logs/access.log log_this env=!dont_log
    ErrorLog /home/users/intersharedev/logs/error.log
    TransferLog "|/usr/local/apache/bin/rotatelogs /home/users/intersharedev/logs/access_log 2592000"

    Should I change this to :

    SetEnvIfNoCase Request_URI "\.(gif|jpe?g|css|js|jpg)$" dont_log
    LogFormat "%h %a %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" log_this
    CustomLog "|/usr/local/apache/bin/rotatelogs /home/users/intersharedev/logs/access_log 2592000 log_this env=!dont_log
    ErrorLog /home/users/intersharedev/logs/error.log

    Thanks again!
  14. #8
  15. No Profile Picture
    Contributing User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Jan 2001
    Posts
    4
    Rep Power
    0
    >> Should I change this to :.......

    Yes and PERFECT!.

    But gif|jpe?g|css|js|jpg

    The jpe?g matches jpeg and jpg so you don't need to define your last jpg.

    And you need to remove your leading double-quote on your CustomLog line.

    Like I said, you can check your old access log and create your own env var or nickname of what you are not interested in seeing in your access log.
    Last edited by freebsd; February 4th, 2002 at 07:18 AM.
  16. #9
  17. No Profile Picture
    Contributing User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Apr 2001
    Location
    London, England
    Posts
    262
    Rep Power
    14
    Thanks a lot, you've been a great help!
  18. #10
  19. No Profile Picture
    Contributing User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Apr 2001
    Location
    London, England
    Posts
    262
    Rep Power
    14
    I tried it and got an error when I ran apachectl configtest which was:

    Syntax error on line 60 of /usr/local/apache/conf/vhosts.conf:
    CustomLog takes two or three arguments, a file name, a custom log format string or format name, and an optional "env=" clause (see docs)

    And this is the line:

    CustomLog |/usr/local/apache/bin/rotatelogs /home/users/intersharedev/logs/access_log 2592000 log_this env=!dont_log

    Any ideas?
  20. #11
  21. No Profile Picture
    Contributing User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Jan 2001
    Posts
    4
    Rep Power
    0
    Sorry, I don't use rotatelogs so I'm not familiar with the exact syntax. However, from here the format should be:
    Code:
    CustomLog file|pipe format|nickname  [env=[!]environment-variable]
    so you can try double-quoting your rotatelogs portion like so:

    CustomLog | "/usr/local/apache/bin/rotatelogs /home/users/intersharedev/logs/access_log 2592000" log_this env=!dont_log

    or even:

    CustomLog "| /usr/local/apache/bin/rotatelogs /home/users/intersharedev/logs/access_log 2592000" log_this env=!dont_log
    Last edited by freebsd; February 4th, 2002 at 07:34 AM.
  22. #12
  23. No Profile Picture
    Contributing User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Apr 2001
    Location
    London, England
    Posts
    262
    Rep Power
    14
    Thanks, the double quotes round the whole line version worked. However the log format remains the same. Presumably it will change upon the next log rotation....

    Cheers
  24. #13
  25. No Profile Picture
    Contributing User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Jan 2001
    Posts
    4
    Rep Power
    0
    >> Presumably it will change upon the next log rotation

    Most likely. What you can do is to change your 2592000 (1 month) to a lower second like 120, just to see if it works, then switch it back to 1 month, of course, you need to -HUP Apache to do all this.
  26. #14
  27. No Profile Picture
    Contributing User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Apr 2001
    Location
    London, England
    Posts
    262
    Rep Power
    14
    Ok, did that and the new log file format is looking a bit skewed.

    193.109.72.6 193.109.72.6 [04/Feb/2002:13:41:40 +0000] "GET /xxxxxxxx/xxxxxxx.php HTTP/1.0" 200 15439 "http://www.xxxxxxx.com/xxxxxxxx.php?prevpage=http://www.xxxxxx.com/xxxxxx/xxxxx.php?id=22" "Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 4.0; T312461)"

    The IP is turning up twice.

    Any ideas?
  28. #15
  29. No Profile Picture
    Contributing User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Jan 2001
    Posts
    4
    Rep Power
    0
    >> The IP is turning up twice

    Because not all IPs have an PTR record. Often, an upsteam only leases a netblock to a particular ISP (just leasing, not delegating). When that IP block is not delegated to that ISP, that ISP has no control to the IP and setup the reverse like user-119adhr.biz.mindspring.com.
    But some ISPs are even lazy or clueless about setting up the appropriate PTR for their entire netblock. As a result, the reverse DNS of 193.109.72.6 returns nothing. When Apache receives an empty REMOTE_HOST, it will replace that with REMOTE_ADDR.

IMN logo majestic logo threadwatch logo seochat tools logo