Grepping a date/time range within a log file...

Dev Shed Forums Sponsor:

February 19th, 2008, 12:59 AM

jtelep

Registered User

Join Date: Apr 2004

Posts: 27

Time spent in forums: 2 h 40 m 3 sec
Reputation Power: 0

Grepping a date/time range within a log file...

Hi,

I've got a new project where I need to identify messages in a log file that occur every morning between 0230 and 0300 and then append this info to another file that I am monitoring using BMC Patrol. I have done this kind of extensive grep before so I am hoping that I can get some help. I am running this on SunOS 5.10 Generic using the standard system grep command. The layout of the file looks like this:

02-18-2008 02:23:32 INFO com.fred.flintstone.log.Log - some message

So in this case everything on 2/18 from 0230 to 0300 needs to be pulled from the main log and appended to a different log file elsewhere on the box. If anyone has any suggestions on the best syntax on how to grab everything on that specific range that would be a big help.

Thanks,

Jon

February 19th, 2008, 03:45 AM

SimonJM

Contributing User

Join Date: Mar 2006

Posts: 2,108

Time spent in forums: 1 Month 1 Week 1 Day 4 h 50 m 50 sec
Reputation Power: 1485

I'd create a variable with the date in the right format, then use that and append any needed time component to it for the grep (probably a grep -e or egrep). One question - is it up to or up to and including 03:00?
so - 02:30-02:59 or 02:30-03:00 that you need?

__________________
The moon on the one hand, the dawn on the other:
The moon is my sister, the dawn is my brother.
The moon on my left and the dawn on my right.
My brother, good morning: my sister, good night.
-- Hilaire Belloc

February 19th, 2008, 08:28 AM

jtelep

Registered User

Join Date: Apr 2004

Posts: 27

Time spent in forums: 2 h 40 m 3 sec
Reputation Power: 0

For the sake of making the search the easiest we'll say up to but not including 0300. What I am looking for is the syntax needed to make searching within this range possible. I was thinking the same thing you were regarding the variable because of the difference in the way the OS normally displays the date and time but I am lost on how to search within a range (never done that before

)

Thanks,

J.

February 20th, 2008, 03:10 AM

SimonJM

Contributing User

Join Date: Mar 2006

Posts: 2,108

Time spent in forums: 1 Month 1 Week 1 Day 4 h 50 m 50 sec
Reputation Power: 1485

The simple answer is cheat!
We know the date is 'fixed' in the variable, and we know the hours is always going to be 02:<something> so all we need worry about is the <something> and that, to cover times from 02:30 to 02:59 needs just be 3, 4, or 5.
Thus, look into a regexp that will find "<date in your format> 02:[345]" as the start of a line, and you are done and dusted.

February 20th, 2008, 10:32 AM

jtelep

Registered User

Join Date: Apr 2004

Posts: 27

Time spent in forums: 2 h 40 m 3 sec
Reputation Power: 0

Thank-you sooo much that completely works for what I am doing with it

Now just one more question:

What if I were to want to be able to grep multiple hours so per the original request let's say I needed to also include 0300 as part of the range, how would the search string change? I mean what would the new search look like?

Thanks man,

J.

February 21st, 2008, 01:56 AM

SimonJM

Contributing User

Join Date: Mar 2006

Posts: 2,108

Time spent in forums: 1 Month 1 Week 1 Day 4 h 50 m 50 sec
Reputation Power: 1485

Ok, well the start part, being the date would remain the same - let us hope!, so we are on to a good start here.
Then we'd need to make the hour component a variable like was done with the date - populating it with "02" in the first instance, along with the minutes part of "[345]" - all as before.

To roll in a new time to check, just chnage th ehour variable - to in the case - "03" and the minutes to match. If you wanted to capture 03:00-03:09 your just put "0" in the minutes variable, if you only wanted 03:00 your put "00" in.

June 20th, 2008, 03:00 AM

kumvinod

Registered User

Join Date: Jun 2008

Posts: 1

Time spent in forums: 18 m 21 sec
Reputation Power: 0

Grep on range of time..

The client has asked for logs from a particular domain from the exim_mainlog files .. the time was between 10:30 to 12:00 ..

So this is what we did and got the result .. Basically we did a grep on his domain and then on the time .. after which we had given a range of time..

cat exim_mainlog |grep domainname.com |grep -E '2008-06-20 (1[0-1]:[0-5][0-9]|12:00)'

This will grep all the records from exim_mainlog from domainname.com on date 20-6-2008 and time 10:00 - 12:00

March 7th, 2012, 11:14 PM

mannan

Registered User

Join Date: Mar 2012

Posts: 6

Time spent in forums: 1 h 2 m
Reputation Power: 0

Need Grep info

Need an help.. want to grep data in a log file from present system time to past 30 mins

The format of the data will be as follows

03-07-2012 11:09:58.275 info message
03-07-2012 11:10:01.575 info message

March 8th, 2012, 10:34 AM

b49P23TIvg

Contributing User

Join Date: Aug 2011

Posts: 3,393

Time spent in forums: 1 Month 2 Weeks 3 Days 15 h 37 m 10 sec
Reputation Power: 383

Use gawk

Code:

gawk -F'[- :.]' 'BEGIN{NOW=systime();THEN=NOW-(30*60+1)}{LOGTIME=mktime($3 " " $1 " " $2 " " $4 " " $5 " " $6);if(THEN<LOGTIME){print}}'

__________________
[code]Code tags[/code] are essential for python code!

Last edited by b49P23TIvg : March 8th, 2012 at 10:35 AM. Reason: remove debug junk

#10

March 8th, 2012, 11:13 PM

mannan

Registered User

Join Date: Mar 2012

Posts: 6

Time spent in forums: 1 h 2 m
Reputation Power: 0

Need Grep info

Quote:

Originally Posted by b49P23TIvg

Code:

gawk -F'[- :.]' 'BEGIN{NOW=systime();THEN=NOW-(30*60+1)}{LOGTIME=mktime($3 " " $1 " " $2 " " $4 " " $5 " " $6);if(THEN<LOGTIME){print}}'

Thanks for the command but i am not aware of gawk command.. i am

with that gawk command

can you please give me any grep and sed command

The task is i need to write a script which grep data in a log file from current system time to past 30 minutes.
i will place this script in crontab so that it will run for every 30 mins.

Please help me with the script or with an logic to get that

The format of data in the log file will be as follows
03-07-2012 11:09:58.275 info message
03-07-2012 11:10:01.575 info message
03-0702012 12:05:59.678 info message

Thanks in Advacne

#11

March 9th, 2012, 06:14 AM

b49P23TIvg

Contributing User

Join Date: Aug 2011

Posts: 3,393

Time spent in forums: 1 Month 2 Weeks 3 Days 15 h 37 m 10 sec
Reputation Power: 383

Install gawk on your system.
I think you need it for my program instead of awk or nawk because the field separator is a regular expression.
If you have linux and an awk program it is assuredly gawk.

try
man awk

Here's a web page.
http://www.gnu.org/software/gawk/

Learn to write shell pipes and redirect io.

#12

March 13th, 2012, 09:33 PM

peckenson

Registered User

Join Date: Mar 2012

Posts: 3

Time spent in forums: 37 m 24 sec
Reputation Power: 0

ls -lrt /usr/local/intranet/areas/prod/output/SRGW_0?/
O/P of above command.
drwxr-xr-x 2 mtsadm mts 4096 Mar 13 10:44 153913
drwxr-xr-x 2 mtsadm mts 4096 Mar 13 10:48 153914
drwxr-xr-x 2 mtsadm mts 4096 Mar 13 10:53 153915
drwxr-xr-x 2 mtsadm mts 4096 Mar 13 10:57 153916
drwxr-xr-x 2 mtsadm mts 4096 Mar 13 11:01 153917
drwxr-xr-x 2 mtsadm mts 4096 Mar 13 11:05 153918
drwxr-xr-x 2 mtsadm mts 4096 Mar 13 11:10 153919
drwxr-xr-x 2 mtsadm mts 4096 Mar 13 11:14 153921
drwxr-xr-x 2 mtsadm mts 4096 Mar 13 11:14 153920
Load time of 153913 = 4 minutes.
I need to in corporate a logic similar to this but need to do it in a loop as there are many directories for load time calculation.

#13

March 13th, 2012, 09:59 PM

b49P23TIvg

Contributing User

Join Date: Aug 2011

Posts: 3,393

Time spent in forums: 1 Month 2 Weeks 3 Days 15 h 37 m 10 sec
Reputation Power: 383

I hope simon is available. Maybe he would clearly explain to you the importance of writing a specific question.

What does O/P mean?

You need to IN CORPORATE logic similar to what?

What mighty version of ls do you use that displays, with ls -lrt
Load time of 153913 = 4 minutes.
?

The theme of this thread seems to be handling time strings using bash. Where are your time strings? What do you want done with them? What is your directory structure?

#14

March 13th, 2012, 10:06 PM

peckenson

Registered User

Join Date: Mar 2012

Posts: 3

Time spent in forums: 37 m 24 sec
Reputation Power: 0

Load time

A report needs to come some what similar to this
No of elements Stream Batch No Load time
A B C D
A=75,B=SRGW_05,C=153907 im able to get quite easily

Code:

wc -l /usr/local/intranet/areas/prod/output/SRGW_0?/*/MESSAGE_T.dat

Output of above command.
A B C

Code:

75/usr/local/intranet/areas/prod/output/SRGW_05/153907/MESSAGE_T.dat
26 /usr/local/intranet/areas/prod/output/SRGW_05/153908/MESSAGE_T.dat
110 /usr/local/intranet/areas/prod/output/SRGW_05/153909/MESSAGE_T.dat

Code:

wc -l /usr/local/intranet/areas/prod/output/SRGW_05/*/MESSAGE_T.dat | cut -f1,8,9 -d"/"

O/P of above command.

Code:

159 /SRGW_05/153917
367 /SRGW_05/153918
21 /SRGW_05/153919
12 /SRGW_05/153920
88 /SRGW_05/153921
35 /SRGW_05/153922
36 /SRGW_05/153923

For D I need to check every 2 Batches and compare so I need to put it in a loop
Load time needs to be time stamp of folder created of C.

Code:

ls -lrt /usr/local/intranet/areas/prod/output/SRGW_0?/

Output of above command.

Code:

drwxr-xr-x 2 mtsadm mts 4096 Mar 13 10:44  153913
drwxr-xr-x 2 mtsadm mts 4096 Mar 13 10:48  153914
drwxr-xr-x 2 mtsadm mts 4096 Mar 13 10:53 153915
drwxr-xr-x 2 mtsadm mts 4096 Mar 13 10:57 153916
drwxr-xr-x 2 mtsadm mts 4096 Mar 13 11:01 153917
drwxr-xr-x 2 mtsadm mts 4096 Mar 13 11:05 153918
drwxr-xr-x 2 mtsadm mts 4096 Mar 13 11:10 153919
drwxr-xr-x 2 mtsadm mts 4096 Mar 13 11:14 153921
drwxr-xr-x 2 mtsadm mts 4096 Mar 13 11:14 153920

This output is required
Load time of 153913 = 4 minutes.
I need to in corporate a logic similar to this but need to do it in a loop as there are many directories for load time calculation.
Store in a variable = ls -

Code:

lrt /usr/local/intranet/areas/prod/output/SRGW_05/ | cut -f24 -d"

Code:

h1=`echo $T1|cut -d: -f1`
m1=`echo $T1|cut -d: -f2`
x1=`echo "$h1*60 + $m1"|bc -l`
h2=`echo $T2|cut -d: -f1`
m2=`echo $T2|cut -d: -f2`
x2=`echo "$h2*60 + $m2"|bc -l`
if test $x1 -lt $x2
then
diff=`echo "$x2 - $x1"|bc -l`
else
diff=`echo "$x1 - $x2"|bc -l`
fi
echo "Load time is $diff"

Entire Output should be like this eventually
No of elements Stream Batch No Load time
A B C D
Can some one help me ?

Appologies if my previous question was not framed properly

Last edited by peckenson : March 13th, 2012 at 10:40 PM. Reason: Correction

#15

March 14th, 2012, 10:46 AM

SimonJM

Contributing User

Join Date: Mar 2006

Posts: 2,108

Time spent in forums: 1 Month 1 Week 1 Day 4 h 50 m 50 sec
Reputation Power: 1485

Despite the request having been clarified in this thread I will respond in your other thread ...
Being as clear as you can helps greatly. For example, in your other thread on this subject it was not wholly obvious if just the time differential for the first two items in the list was needed, or if it was a 'rolling' operation.

Grepping a date/time range within a log file...

Developer Shed Advertisers and Affiliates