mv, cp, or rm log_files with crontab

Hi there:

Does anyone know how to move or remove a file from a unix server with crontab.

I don't want to delete all *.*
because, i need to keep the files for this month, however, I would like to delete the files for last month.

I have log files named
access_log.20010601.gz
access_log.20010602.gz
...
access_log.`date +%Y%m%d`.gz

I have this script made in bash, but can't seem to get it to work.

---
#!/usr/local/bin/bash

LAST_MONTH=$(( (-1+$(date +%m)) ))
FULL_DATE=$(date +%Y`echo $LAST_MONTH`%d)

# this will move old access_log to logs_expired folder
/bin/mv ~/www/logs2/access_log.`echo $FULL_DATE`.gz ~/www/logs_expired/.
---

Hopefully, someone could assist me with some valuable help.
Thanks,
Ponch

How about moving all rotated logs (the gz files) to ~/www/logs_expired/? I assume your are running as root and HOME dir is /root:

######### rotate_access_log.sh ###########
#!/bin/sh

locate=`/usr/bin/find /root/www/logs2 -type f -name access_log\*gz`
for file in $locate
do
filename=`/bin/echo $file | /usr/bin/awk '{ print substr($1,17) }'`
/bin/mv $file /root/www/logs_expired/$filename
done
##################################

Crontab -e:
# Run at 10PM every Sunday
0 22 * * 0 /path/to/rotate_access_log.sh 2>&1 > /dev/null

Thanks for your reply, however, I need to keep this months log_files in the directory because I am running WebTrends every Sunday and 1st of every Month.

For example, today, July 8th, 2001, I have all of June's log_files in the folder logs2, but I also have July 1-8's log_files, and if i rotate with what you posted, then won't I delete even the July files?

-ponch

>> if i rotate with what you posted, then won't I delete even the July files?

Yes, simply all .gz files.

Here's a revision:

################
#!/bin/sh

# Instead of getting previous month (which would fail on January), let's get the 1st day of current month
firstday=`date '+%Y%m01'`
# Now locate all files named access_log*gz
locate=`/usr/bin/find /root/www/logs2 -type f -name access_log\*gz`
for file in $locate
do
# Remove the path, we just want the file name
filename=`/bin/echo $file | /usr/bin/awk '{ print substr($1,17) }'`
# Remove access_log. from the filename and leaving just the date
filedate=`/bin/echo $filename | /usr/bin/awk ' { print substr($1,12,8) }'`
# Move the files only if the date is less than $firstday
if [ "$filedate" -lt "$firstday" ]
then
/bin/mv $file /root/www/logs_expired/$filename
fi
done

################

Change the substr values if your paths have changed.

GREAT!

However, I left out some vital information...

I'm running a virtual server with multiple domain names.
Those multiple domain names, has their own access_log and error_log.

Therefore, within the logs2 folder, there are virtual_name folders
for example; client1 (i am using the real domain name, but for ease of this posting i call them client1, client2, client3, a.s.o).

I already clear log_files from logs folder every midnight, and move them to logs2 folder.
In the script for clearing the log_files I have a logzapper.conf with client1, client2, client3, a.s.o, to distinguish between virtual_names:

###
# virtual_name listed in logzapper.conf
for VNAME in `cat /usr/home/wide1/usr/local/etc/httpd/conf/logzapper.conf`

do

###

and then the script continues, but I just wanted to show you the $VNAME.

So back to the script that you posted, which worked awesome, however, since I have my log_files named i.e. $VNAME_access_log.20010630.gz
then the ' { print substr($1,12,8) }' won't work.

Can't it be done something like this?

###
#!/usr/local/bin/bash

for VNAME in `cat /usr/home/wide1/usr/local/etc/httpd/conf/logzapper2.conf`

FIRSTDAY=$(date +%Y%m01)
FILENAME=~/www/logs2/`echo $VNAME`/`echo $VNAME`_access_log.`echo $FILEDATE`.gz
FILEDATE=\*

# Move the files only if the date is less than $firstday
if [ $FILEDATE -lt $FIRSTDAY ]
then
/bin/mv $FILENAME ~/www/logs_expired/`echo $VNAME`/$FILENAME
fi
done

###

But somehow, it doesn't like the FILEDATE=\*

It would be to great help if you could assist me with some solution or thoughts.

Thanks a lot,
ponch

>> within the logs2 folder, there are virtual_name folders

So the path to access_log virtual_name is /root/www/logs2/client1/access_log.20010601.gz or /root/www/logs2/client1/client1_access_log.20010601.gz?

>> virtual_name listed in logzapper.conf

I need to know the exact format. How is each vhostname delimited? Any other non-vhostname lines there? BTW, is your vhostname really a domain name like domain.com or even www.domain.com? Or just a username in lowercase and a-z0-9 characters only?

I am now wondering what other files or directories[/b] are in your /root/www/logs2/ directory. Also, do your users have write access to /root/www/logs2 or /root/www/logs/client*/ directory at all?

>> then the ' { print substr($1,12,8) }' won't work

True, your situation is pretty dynamic.

>> I need to know the exact format.

This is how everything is done with the log files.
1. Apache writes the log_files like this for mydomain.com:

# point mydomain.com to subdirectory www/m/mydomain
<VirtualHost mydomain.com www.mydomain.com>
ServerName mydomain.com
ServerAdmin webmaster@mydomain.com
DocumentRoot /usr/local/etc/httpd/htdocs/www/m/mydomain/public_html
TransferLog /usr/local/etc/httpd/logs/mydomain_access_log
ErrorLog /usr/local/etc/httpd/logs/mydomain_error_log
</VirtualHost>

and this for somedomain.com:

# point mydomain.com to subdirectory www/s/somedomain
<VirtualHost somedomain.com www.somedomain.com>
ServerName somedomain.com
ServerAdmin webmaster@somedomain.com
DocumentRoot /usr/local/etc/httpd/htdocs/www/s/somedomain/public_html
TransferLog /usr/local/etc/httpd/logs/somedomain_access_log
ErrorLog /usr/local/etc/httpd/logs/somedomain_error_log
</VirtualHost>

>> virtual_name listed in logzapper.conf

2. Inside the logzapper.conf I have one row for each domain name. So for the domain's mentioned above I would have this:

mydomain
somedomain

3. At midnight everyday, I have a script moving, gzipping, and clearing the old log_files. They get moved to /usr/local/etc/httpd/logs_zapped/$VNAME

Here is the line which gzip the VNAME_access_log:

# this will compress the temporary VNAME_access_log file
/usr/bin/gzip -c ~/www/logs_zapped/`echo $VNAME`/`echo $VNAME`_access_log.tmp > ~/www/logs_zapped/`echo $VNAME`/`echo $VNAME`_access_log.`date +%Y%m%d`.gz


>> So the path to access_log virtual_name
>> is /root/www/logs2/client1/access_log.20010601.gz
>> or /root/www/logs2/client1/client1_access_log.20010601.gz?


4. The path for mydomain.com access_log is as follows:
/usr/local/etc/httpd/logs_zapped/mydomain/mydomain_access_log.20010601.gz

>> I am now wondering what other files or directories[/b] are in your /root/www/logs2/ directory.

5. There are only $VNAME folders for all virtual log_files and a main folder for the main access_log & error_log.

>> Also, do your users have write access to /root/www/logs2 or /root/www/logs/client*/ directory at all?

6. No they don't.

7. I really appreciate your input. Thanks a lot!

-ponch

hi there freebsd:
i tried to work this out, but ran into a wall.
do you have any suggestions? anything would be helpful.
have a great day buddy,
ponch

I tried to figure out all the paths and so (which you have changed its name repeatedly from post to post) so that I lost track and got more confused and therefore didn't continue this thread.

And I found that I had been using the paths (/root) different from yours but you didn't correct me.

Anyhow, writing such a script should not be difficult, it was the paths and the actual directory layouts that made me got confused.

Now that I got what you really wanted to do, if you can repost everything: invent some short and less-confused paths, I am more than willing to continue this thread. You just wanted a sample script, didn't you? As long as you get the idea how to do it, it would be easier for you to convert the paths and fake names back to the actual ones.