#1
  1. No Profile Picture
    Registered User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Apr 2004
    Posts
    28
    Rep Power
    0

    Grepping a date/time range within a log file...


    Hi,

    I've got a new project where I need to identify messages in a log file that occur every morning between 0230 and 0300 and then append this info to another file that I am monitoring using BMC Patrol. I have done this kind of extensive grep before so I am hoping that I can get some help. I am running this on SunOS 5.10 Generic using the standard system grep command. The layout of the file looks like this:

    02-18-2008 02:23:32 INFO com.fred.flintstone.log.Log - some message

    So in this case everything on 2/18 from 0230 to 0300 needs to be pulled from the main log and appended to a different log file elsewhere on the box. If anyone has any suggestions on the best syntax on how to grab everything on that specific range that would be a big help.

    Thanks,

    Jon
  2. #2
  3. No Profile Picture
    Contributing User
    Devshed Regular (2000 - 2499 posts)

    Join Date
    Mar 2006
    Posts
    2,477
    Rep Power
    1752
    I'd create a variable with the date in the right format, then use that and append any needed time component to it for the grep (probably a grep -e or egrep). One question - is it up to or up to and including 03:00?
    so - 02:30-02:59 or 02:30-03:00 that you need?
    The moon on the one hand, the dawn on the other:
    The moon is my sister, the dawn is my brother.
    The moon on my left and the dawn on my right.
    My brother, good morning: my sister, good night.
    -- Hilaire Belloc
  4. #3
  5. No Profile Picture
    Registered User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Apr 2004
    Posts
    28
    Rep Power
    0
    For the sake of making the search the easiest we'll say up to but not including 0300. What I am looking for is the syntax needed to make searching within this range possible. I was thinking the same thing you were regarding the variable because of the difference in the way the OS normally displays the date and time but I am lost on how to search within a range (never done that before )

    Thanks,

    J.
  6. #4
  7. No Profile Picture
    Contributing User
    Devshed Regular (2000 - 2499 posts)

    Join Date
    Mar 2006
    Posts
    2,477
    Rep Power
    1752
    The simple answer is cheat!
    We know the date is 'fixed' in the variable, and we know the hours is always going to be 02:<something> so all we need worry about is the <something> and that, to cover times from 02:30 to 02:59 needs just be 3, 4, or 5.
    Thus, look into a regexp that will find "<date in your format> 02:[345]" as the start of a line, and you are done and dusted.
    The moon on the one hand, the dawn on the other:
    The moon is my sister, the dawn is my brother.
    The moon on my left and the dawn on my right.
    My brother, good morning: my sister, good night.
    -- Hilaire Belloc
  8. #5
  9. No Profile Picture
    Registered User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Apr 2004
    Posts
    28
    Rep Power
    0
    Thank-you sooo much that completely works for what I am doing with it Now just one more question:

    What if I were to want to be able to grep multiple hours so per the original request let's say I needed to also include 0300 as part of the range, how would the search string change? I mean what would the new search look like?

    Thanks man,

    J.
  10. #6
  11. No Profile Picture
    Contributing User
    Devshed Regular (2000 - 2499 posts)

    Join Date
    Mar 2006
    Posts
    2,477
    Rep Power
    1752
    Ok, well the start part, being the date would remain the same - let us hope!, so we are on to a good start here.
    Then we'd need to make the hour component a variable like was done with the date - populating it with "02" in the first instance, along with the minutes part of "[345]" - all as before.

    To roll in a new time to check, just chnage th ehour variable - to in the case - "03" and the minutes to match. If you wanted to capture 03:00-03:09 your just put "0" in the minutes variable, if you only wanted 03:00 your put "00" in.
    The moon on the one hand, the dawn on the other:
    The moon is my sister, the dawn is my brother.
    The moon on my left and the dawn on my right.
    My brother, good morning: my sister, good night.
    -- Hilaire Belloc
  12. #7
  13. No Profile Picture
    Registered User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Jun 2008
    Posts
    1
    Rep Power
    0

    Grep on range of time..


    The client has asked for logs from a particular domain from the exim_mainlog files .. the time was between 10:30 to 12:00 ..

    So this is what we did and got the result .. Basically we did a grep on his domain and then on the time .. after which we had given a range of time..

    cat exim_mainlog |grep domainname.com |grep -E '2008-06-20 (1[0-1]:[0-5][0-9]|12:00)'

    This will grep all the records from exim_mainlog from domainname.com on date 20-6-2008 and time 10:00 - 12:00

  14. #8
  15. No Profile Picture
    Registered User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Mar 2012
    Posts
    6
    Rep Power
    0

    Need Grep info


    Need an help.. want to grep data in a log file from present system time to past 30 mins

    The format of the data will be as follows

    03-07-2012 11:09:58.275 info message
    03-07-2012 11:10:01.575 info message
  16. #9
  17. Contributing User
    Devshed Demi-God (4500 - 4999 posts)

    Join Date
    Aug 2011
    Posts
    4,966
    Rep Power
    481

    Use gawk


    Code:
    gawk -F'[- :.]' 'BEGIN{NOW=systime();THEN=NOW-(30*60+1)}{LOGTIME=mktime($3 " " $1 " " $2 " " $4 " " $5 " " $6);if(THEN<LOGTIME){print}}'
    Last edited by b49P23TIvg; March 8th, 2012 at 11:35 AM. Reason: remove debug junk
    [code]Code tags[/code] are essential for python code and Makefiles!
  18. #10
  19. No Profile Picture
    Registered User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Mar 2012
    Posts
    6
    Rep Power
    0

    Need Grep info


    Originally Posted by b49P23TIvg
    Code:
    gawk -F'[- :.]' 'BEGIN{NOW=systime();THEN=NOW-(30*60+1)}{LOGTIME=mktime($3 " " $1 " " $2 " " $4 " " $5 " " $6);if(THEN<LOGTIME){print}}'
    Thanks for the command but i am not aware of gawk command.. i am with that gawk command

    can you please give me any grep and sed command

    The task is i need to write a script which grep data in a log file from current system time to past 30 minutes.
    i will place this script in crontab so that it will run for every 30 mins.

    Please help me with the script or with an logic to get that

    The format of data in the log file will be as follows
    03-07-2012 11:09:58.275 info message
    03-07-2012 11:10:01.575 info message
    03-0702012 12:05:59.678 info message


    Thanks in Advacne
  20. #11
  21. Contributing User
    Devshed Demi-God (4500 - 4999 posts)

    Join Date
    Aug 2011
    Posts
    4,966
    Rep Power
    481
    Install gawk on your system.
    I think you need it for my program instead of awk or nawk because the field separator is a regular expression.
    If you have linux and an awk program it is assuredly gawk.

    try
    man awk

    Here's a web page.
    http://www.gnu.org/software/gawk/

    Learn to write shell pipes and redirect io.
    [code]Code tags[/code] are essential for python code and Makefiles!
  22. #12
  23. No Profile Picture
    Registered User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Mar 2012
    Posts
    3
    Rep Power
    0
    ls -lrt /usr/local/intranet/areas/prod/output/SRGW_0?/
    O/P of above command.
    drwxr-xr-x 2 mtsadm mts 4096 Mar 13 10:44 153913
    drwxr-xr-x 2 mtsadm mts 4096 Mar 13 10:48 153914
    drwxr-xr-x 2 mtsadm mts 4096 Mar 13 10:53 153915
    drwxr-xr-x 2 mtsadm mts 4096 Mar 13 10:57 153916
    drwxr-xr-x 2 mtsadm mts 4096 Mar 13 11:01 153917
    drwxr-xr-x 2 mtsadm mts 4096 Mar 13 11:05 153918
    drwxr-xr-x 2 mtsadm mts 4096 Mar 13 11:10 153919
    drwxr-xr-x 2 mtsadm mts 4096 Mar 13 11:14 153921
    drwxr-xr-x 2 mtsadm mts 4096 Mar 13 11:14 153920
    Load time of 153913 = 4 minutes.
    I need to in corporate a logic similar to this but need to do it in a loop as there are many directories for load time calculation.
  24. #13
  25. Contributing User
    Devshed Demi-God (4500 - 4999 posts)

    Join Date
    Aug 2011
    Posts
    4,966
    Rep Power
    481
    I hope simon is available. Maybe he would clearly explain to you the importance of writing a specific question.

    What does O/P mean?

    You need to IN CORPORATE logic similar to what?

    What mighty version of ls do you use that displays, with ls -lrt
    Load time of 153913 = 4 minutes.
    ?

    The theme of this thread seems to be handling time strings using bash. Where are your time strings? What do you want done with them? What is your directory structure?
    [code]Code tags[/code] are essential for python code and Makefiles!
  26. #14
  27. No Profile Picture
    Registered User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Mar 2012
    Posts
    3
    Rep Power
    0

    Load time


    A report needs to come some what similar to this
    No of elements Stream Batch No Load time
    A B C D
    A=75,B=SRGW_05,C=153907 im able to get quite easily
    Code:
    wc -l /usr/local/intranet/areas/prod/output/SRGW_0?/*/MESSAGE_T.dat
    Output of above command.
    A B C
    Code:
    75/usr/local/intranet/areas/prod/output/SRGW_05/153907/MESSAGE_T.dat
    26 /usr/local/intranet/areas/prod/output/SRGW_05/153908/MESSAGE_T.dat
    110 /usr/local/intranet/areas/prod/output/SRGW_05/153909/MESSAGE_T.dat
    Code:
    wc -l /usr/local/intranet/areas/prod/output/SRGW_05/*/MESSAGE_T.dat | cut -f1,8,9 -d"/"
    O/P of above command.
    Code:
    159 /SRGW_05/153917
    367 /SRGW_05/153918
    21 /SRGW_05/153919
    12 /SRGW_05/153920
    88 /SRGW_05/153921
    35 /SRGW_05/153922
    36 /SRGW_05/153923
    For D I need to check every 2 Batches and compare so I need to put it in a loop
    Load time needs to be time stamp of folder created of C.
    Code:
    ls -lrt /usr/local/intranet/areas/prod/output/SRGW_0?/
    Output of above command.
    Code:
    drwxr-xr-x 2 mtsadm mts 4096 Mar 13 10:44  153913
    drwxr-xr-x 2 mtsadm mts 4096 Mar 13 10:48  153914
    drwxr-xr-x 2 mtsadm mts 4096 Mar 13 10:53 153915
    drwxr-xr-x 2 mtsadm mts 4096 Mar 13 10:57 153916
    drwxr-xr-x 2 mtsadm mts 4096 Mar 13 11:01 153917
    drwxr-xr-x 2 mtsadm mts 4096 Mar 13 11:05 153918
    drwxr-xr-x 2 mtsadm mts 4096 Mar 13 11:10 153919
    drwxr-xr-x 2 mtsadm mts 4096 Mar 13 11:14 153921
    drwxr-xr-x 2 mtsadm mts 4096 Mar 13 11:14 153920
    This output is required
    Load time of 153913 = 4 minutes.
    I need to in corporate a logic similar to this but need to do it in a loop as there are many directories for load time calculation.
    Store in a variable = ls -
    Code:
    lrt /usr/local/intranet/areas/prod/output/SRGW_05/ | cut -f24 -d"
    "
    Code:
    h1=`echo $T1|cut -d: -f1`
    m1=`echo $T1|cut -d: -f2`
    x1=`echo "$h1*60 + $m1"|bc -l`
    h2=`echo $T2|cut -d: -f1`
    m2=`echo $T2|cut -d: -f2`
    x2=`echo "$h2*60 + $m2"|bc -l`
    if test $x1 -lt $x2
    then
    diff=`echo "$x2 - $x1"|bc -l`
    else
    diff=`echo "$x1 - $x2"|bc -l`
    fi
    echo "Load time is $diff"
    Entire Output should be like this eventually
    No of elements Stream Batch No Load time
    A B C D
    Can some one help me ?

    Appologies if my previous question was not framed properly
    Last edited by peckenson; March 13th, 2012 at 11:40 PM. Reason: Correction
  28. #15
  29. No Profile Picture
    Contributing User
    Devshed Regular (2000 - 2499 posts)

    Join Date
    Mar 2006
    Posts
    2,477
    Rep Power
    1752
    Despite the request having been clarified in this thread I will respond in your other thread ...
    Being as clear as you can helps greatly. For example, in your other thread on this subject it was not wholly obvious if just the time differential for the first two items in the list was needed, or if it was a 'rolling' operation.
    The moon on the one hand, the dawn on the other:
    The moon is my sister, the dawn is my brother.
    The moon on my left and the dawn on my right.
    My brother, good morning: my sister, good night.
    -- Hilaire Belloc

IMN logo majestic logo threadwatch logo seochat tools logo