Thread: Unix script

    #1
  1. No Profile Picture
    Registered User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Jul 2010
    Posts
    7
    Rep Power
    0

    Unix script


    Has anyone a command I can use to generate a list of files that match the criteria below? I will use the command within a unix shell script which will run weekly.

    I have to 'extract' all file names from a file which are more than 2-weeks old and generate an output file containing the '2-week old' file names.

    The example input file below contains a list of file names. The file names contain a date signifier after the @ symbol.
    i.e. the 1st file-name in the file was created on 11/07/2010

    The file names have two lengths, and I have included two examples of each.

    sd1kappsy15@110710070000
    sd1kappsy16@180710070000
    work_space@110710070000
    work_space@180710070000
  2. #2
  3. No Profile Picture
    Contributing User
    Devshed Beginner (1000 - 1499 posts)

    Join Date
    Jul 2004
    Location
    Middle Europa
    Posts
    1,198
    Rep Power
    17
    hello john

    no chance to do it using that notation: DDMMYYHHMM...
    a better way would be the notation: YYYYMMDD.... that allow you to sort the list
    but, this is still not the way!
    you cannot compare that strings and expect you get 'weeks'

    a script - when not at least perl or c (an awk could help)- is also not appropriate.

    - compute today-date AND transform it in SECONDS, substract 14 days (also in seconds)
    - transform the file-date in SECONDS
    - now you can COMPARE - a stupid 'if' will do the job

    don't forget, years, months and days have the tendency to change,
    so 20110101 and 20101231 are they in the same week??
    in Europe is MO the first day of week, not so elsewhere.
    what about leap-years??
    and and and...

    NOTA: the notation: YYYYWW or YYYYDDD (DDD is day number of year) in the file-name will also help you.
    regards
    working on Solaris[5-9], preferred languages french and C.
  4. #3
  5. No Profile Picture
    Registered User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Jul 2010
    Posts
    7
    Rep Power
    0
    guggach,
    You are correct. This is more difficult than it should be. However, the file names exist in this format already for other purposes. They are snapshots of the system.

    I also understand what you are teling me about leapyear, etc. But I suppose I was looking for an awk/sed command that found (after the @) and 'read' the file-month-number - if it was greater than or equal to todays-month and if the file-day-number was > 15, then the file-name would be more than 2-weeks old.

    I think my logic is correct. But I dont know how to program this logic.
  6. #4
  7. Contributing User
    Devshed Novice (500 - 999 posts)

    Join Date
    Sep 2006
    Posts
    867
    Rep Power
    391

    Cool Gnu date...


    The following works for Linux and GNU date:
    Code:
    #!/bin/bash
    cat - <<EOF >flist.txt
    sd1kappsy15@110710070000
    sd1kappsy16@180710070000
    work_space@110710070000
    work_space@180710070000
    EOF
    
    today=$(date +"%s")
    
    for fn in `cat flist.txt`
    do
     kk=${fn%@*}
     dt=${fn#*@}
     yy="20${dt:4:2}"
     mm=${dt:2:2} 
     dd=${dt:0:2}
     filedt=`date -d "$yy/$mm/$dd" +%s`
     (( diff = (today - filedt)/86400 ))
     [ $diff -gt 15 ] && echo "File $fn is $diff days old."
    done
    Last edited by LKBrwn_DBA; July 28th, 2010 at 11:14 AM.
  8. #5
  9. No Profile Picture
    Registered User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Jul 2010
    Posts
    7
    Rep Power
    0
    Thanks for script, but apparently Solaris doesnt recognise -d or %s. (I get error token is "%s")
  10. #6
  11. Contributing User
    Devshed Novice (500 - 999 posts)

    Join Date
    Sep 2006
    Posts
    867
    Rep Power
    391

    Cool GNU-it


    Originally Posted by john999
    Thanks for script, but apparently Solaris doesnt recognise -d or %s. (I get error token is "%s")
    For Solaris, install GNU date and use the following script:
    Code:
    #!/bin/ksh
    # Solaris with GNU date:
    cat - <<EOF >flist.txt
    sd1kappsy15@110710070000
    sd1kappsy16@180710070000
    work_space@110710070000
    work_space@180710070000done
    EOF
    
    today=$(date +"%s")             # Install GNU date
    echo "Today = $today"
    for fn in `cat flist.txt`
    do
     kk=${fn%@*}
     dt=${fn#*@}
     yy="20`echo $dt|cut -c 5-6`"
     mm=`echo $dt|cut -c 3-4`
     dd=`echo $dt|cut -c 1-2`
     filedt=`date -d "$yy-$mm-$dd" +%s` # Install GNU date
     (( diff = (today - filedt)/86400 )) 
     [ $diff -gt 15 ] && echo "File $fn is $diff days old."
    done

    Now, if you don't care it may be 1 or two days off on leap years, then try this:
    Code:
    #!/bin/ksh
    # Solaris
    cat - <<EOF >flist.txt
    sd1kappsy15@110710070000
    sd1kappsy16@180710070000
    work_space@110710070000
    work_space@180710070000done
    EOF
    # Start script
    xdate()
    {
     (( ss = $1 * 366 + $2 * 31 + $3 ))
     echo $ss
    }
    echo `date '+%y %m %d'`
    today=$(xdate `date '+%Y %m %d'`)
    for fn in `cat flist.txt`
    do
     kk=${fn%@*}
     dt=${fn#*@}
     yy="20`echo $dt|cut -c 5-6`"
     mm=`echo $dt|cut -c 3-4`
     dd=`echo $dt|cut -c 1-2`
     filedt=$(xdate $yy $mm $dd)
     (( diff = today - filedt ))
     [ $diff -gt 15 ] && echo "File $fn is $diff days old."
    done
    Last edited by LKBrwn_DBA; July 29th, 2010 at 12:07 PM.
  12. #7
  13. No Profile Picture
    Registered User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Jul 2010
    Posts
    7
    Rep Power
    0
    LKBrwn_DBA

    Thanks - this works great.
    I used your second suggestion as this will be used to house keep the server snapshots, so the 2-week date will never be critical, so long as we have access to at least a couple of snapshots. Thanks again.
  14. #8
  15. Contributing User
    Devshed Novice (500 - 999 posts)

    Join Date
    Sep 2006
    Posts
    867
    Rep Power
    391

    Cool


    Originally Posted by john999
    LKBrwn_DBA

    Thanks - this works great.
    I used your second suggestion as this will be used to house keep the server snapshots, so the 2-week date will never be critical, so long as we have access to at least a couple of snapshots. Thanks again.
    Here is a better approximation:
    Code:
    #!/bin/ksh
    # Solaris
    #
    cat - <<EOF >flist.txt
    sd1kappsy15@110710070000
    sd1kappsy16@180710070000
    work_space@110710070000
    work_space@180710070000
    EOF
    xdate()
    {
     m=1
     (( ss = $1 * 365 + $3 ))
     while [ $m -lt $2 ]
     do
       if [ $m -eq 2 ]; then
         x=28;
       else
         (( x = ($m - ($m / 8) ) % 2 +30 ))
       fi
       (( ss += $x ))
       (( m += 1 ))
     done
     echo $ss
    }
    today=$(xdate `date +%y\ %m\ %d`)
    #echo "Today = $today"
    for fn in `cat flist.txt`
    do
     kk=${fn%@*}
     dt=${fn#*@}
     yy=`echo $dt|cut -c 5-6`
     mm=`echo $dt|cut -c 3-4`
     dd=`echo $dt|cut -c 1-2`
     echo  $yy $mm $dd
     filedt=$(xdate $yy $mm $dd)
     (( diff = today - filedt ))
     [ $diff -gt 15 ] && echo "File $fn is $diff days old."
    done
    Last edited by LKBrwn_DBA; July 28th, 2010 at 04:14 PM. Reason: Better approximation
  16. #9
  17. No Profile Picture
    Registered User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Jul 2010
    Posts
    7
    Rep Power
    0
    Hi LKBrwn_DBA,

    This was working. But now we are in a new month (?) - it is not finding the correct files. If I change day-number from 15 to a 1 or a 2, I get the following results. I have modified the script slightly so that there is an input and output file. Please see below.

    Also, I have now discovered that the file names are sometimes of a different length than the examples I gave (eg sd1kappsy15_dba@110710070000). I appologise for this, but have only just noticed it while testing on another server. I dont think this will be a problem because I think your script tests for the date after the @ symbol and it is preceeding the @ that the file-name component varies in length.


    flist.txt
    sd1kappsy15@110710070000
    sd1kappsy16@180710070000
    work_space@110710070000
    work_space@180710070000
    delete.out
    sd1kappsy15@110710070000
    work_space@110710070000


    #!/bin/ksh
    #
    # Script to read the snapshot_weekly.dat file, containing
    # list of all snapshots created by the snapshot_weekly.sh cron job.
    # A snapshot.delete file is generated, which contains a list all snapshots
    # older than 14 days. This file is then read and the zones deleted.
    # NOTE: The date calculation is not 100% accurate, so test is set
    # to 16 days to ensure files are at least 2 weeks old (allowing for leapyear, etc).
    #
    xdate()
    {
    (( ss = $1 * 366 + $2 * 12 + $3 ))
    echo $ss
    }
    #echo `date '+%y %m %d'`
    today=$(xdate `date '+%Y %m %d'`)
    for fn in `cat /jd/jd/jd/flist.txt`
    do
    kk=${fn%@*}
    dt=${fn#*@}
    yy="20`echo $dt|cut -c 5-6`"
    mm=`echo $dt|cut -c 3-4`
    dd=`echo $dt|cut -c 1-2`
    filedt=$(xdate $yy $mm $dd)
    (( diff = today - filedt ))
    [ $diff -gt 2 ] && echo "$fn" >> /jd/jd/jd/delete.out
    done
    #
    #for dn in `cat /jd/jd/jd/delete.out`
    # do
    # zfs destroy -r $dn
    # done
  18. #10
  19. Contributing User
    Devshed Novice (500 - 999 posts)

    Join Date
    Sep 2006
    Posts
    867
    Rep Power
    391

    Cool The latest is the greatest...


    Use the last script I posted!!!
    Code:
    #!/bin/ksh
    # Solaris
    #
    cat - <<EOF >flist.txt
    sd1kappsy15@110710070000
    sd1kappsy16@180710070000
    work_space@110710070000
    work_space@180710070000
    sd1kappsy15@110710070000
    sd1kappsy16@180710070000
    work_space1@110710070000
    work_space2@180710070000
    sd1kappsy15@110710070000
    work_space3@110710070000
    EOF
    xdate()
    {
     m=1
     (( ss = $1 * 365 + $3 ))
     while [ $m -lt $2 ]
     do
       if [ $m -eq 2 ]; then
         x=28;
       else
         (( x = ($m - ($m / 8) ) % 2 +30 ))
       fi
       (( ss += $x ))
       (( m += 1 ))
     done
     echo $ss
    }
    today=$(xdate `date +%y\ %m\ %d`)
    #echo "Today = $today"
    for fn in `cat flist.txt`
    do
     kk=${fn%@*}
     dt=${fn#*@}
     yy=`echo $dt|cut -c 5-6`
     mm=`echo $dt|cut -c 3-4`
     dd=`echo $dt|cut -c 1-2`
     echo  $yy $mm $dd
     filedt=$(xdate $yy $mm $dd)
     (( diff = today - filedt ))
     echo "File $fn is $diff days old."
     #[ $diff -gt 15 ] && echo "File $fn is $diff days old."
    done
    "k0" 46 lines, 871 characters
    [mcacst41]oracle@uatlkatp2ors001:/opt/app/oracle/scripts
    $ ./k0
    10 07 11
    File sd1kappsy15@110710070000 is 22 days old.
    10 07 18
    File sd1kappsy16@180710070000 is 15 days old.
    10 07 11
    File work_space@110710070000 is 22 days old.
    10 07 18
    File work_space@180710070000 is 15 days old.
    10 07 11
    File sd1kappsy15@110710070000 is 22 days old.
    10 07 18
    File sd1kappsy16@180710070000 is 15 days old.
    10 07 11
    File work_space1@110710070000 is 22 days old.
    10 07 18
    File work_space2@180710070000 is 15 days old.
    10 07 11
    File sd1kappsy15@110710070000 is 22 days old.
    10 07 11
    File work_space3@110710070000 is 22 days old.
  20. #11
  21. No Profile Picture
    Registered User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Jul 2010
    Posts
    7
    Rep Power
    0
    LKBrwn_DBA,
    Thanks again. Your last script does work much better. I did try it before but got, and still do get, the syntax error below. Weird because cant see what is wrong and it does complete successfully.

    Error message:
    sh -x ./snapshot_housekeping.sh
    + rm /usr/local/etc/delete_snapshot.out
    ./snapshot_housekeping.sh: syntax error at line 17: `(' unexpected
  22. #12
  23. Contributing User
    Devshed Novice (500 - 999 posts)

    Join Date
    Sep 2006
    Posts
    867
    Rep Power
    391

    Cool Post-it


    Originally Posted by john999
    LKBrwn_DBA,
    Thanks again. Your last script does work much better. I did try it before but got, and still do get, the syntax error below. Weird because cant see what is wrong and it does complete successfully.

    Error message:
    sh -x ./snapshot_housekeping.sh
    + rm /usr/local/etc/delete_snapshot.out
    ./snapshot_housekeping.sh: syntax error at line 17: `(' unexpected
    Post the script you are using...
  24. #13
  25. No Profile Picture
    Registered User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Jul 2010
    Posts
    7
    Rep Power
    0
    LKBrwn_DBA,

    Sorry, my fault - should be using ksh -x, not sh -x. Then get no errors.

    I do have one last thing to add to script though. I cant find a 'quiet' mode for diff. If I do 'diff current_file file_deletions > new_file', the resultant new_file contains '> ' and the top line contains stats.

    Is there an easy way strip these out so file only contains normal data?
  26. #14
  27. No Profile Picture
    Contributing User
    Devshed Frequenter (2500 - 2999 posts)

    Join Date
    Mar 2006
    Posts
    2,632
    Rep Power
    1811
    You'd need to pipe the output from diff through something like awk or sed to remove the things you do not want before redirecting output to the final file.
    The moon on the one hand, the dawn on the other:
    The moon is my sister, the dawn is my brother.
    The moon on my left and the dawn on my right.
    My brother, good morning: my sister, good night.
    -- Hilaire Belloc

IMN logo majestic logo threadwatch logo seochat tools logo