The Shed is going Social! Join us on FaceBook and Twitter and chime in on the conversation.
|
 |
|
Dev Shed Forums
> Operating Systems
> UNIX Help
|
Grepping a date/time range within a log file...
Discuss Grepping a date/time range within a log file... in the UNIX Help forum on Dev Shed. Grepping a date/time range within a log file... UNIX Help forum discussing the Unix Operating System and all variants including Irix, Solarix, and AIX. Unix was designed as a true multi-user operating system.
|
|
 |
|
|
|
|

Dev Shed Forums Sponsor:
|
|
|

February 19th, 2008, 12:59 AM
|
|
Registered User
|
|
Join Date: Apr 2004
Posts: 27
Time spent in forums: 2 h 40 m 3 sec
Reputation Power: 0
|
|
|
Grepping a date/time range within a log file...
Hi,
I've got a new project where I need to identify messages in a log file that occur every morning between 0230 and 0300 and then append this info to another file that I am monitoring using BMC Patrol. I have done this kind of extensive grep before so I am hoping that I can get some help. I am running this on SunOS 5.10 Generic using the standard system grep command. The layout of the file looks like this:
02-18-2008 02:23:32 INFO com.fred.flintstone.log.Log - some message
So in this case everything on 2/18 from 0230 to 0300 needs to be pulled from the main log and appended to a different log file elsewhere on the box. If anyone has any suggestions on the best syntax on how to grab everything on that specific range that would be a big help.
Thanks,
Jon
|

February 19th, 2008, 03:45 AM
|
|
|
|
I'd create a variable with the date in the right format, then use that and append any needed time component to it for the grep (probably a grep -e or egrep). One question - is it up to or up to and including 03:00?
so - 02:30-02:59 or 02:30-03:00 that you need?
__________________
The moon on the one hand, the dawn on the other:
The moon is my sister, the dawn is my brother.
The moon on my left and the dawn on my right.
My brother, good morning: my sister, good night.
-- Hilaire Belloc
|

February 19th, 2008, 08:28 AM
|
|
Registered User
|
|
Join Date: Apr 2004
Posts: 27
Time spent in forums: 2 h 40 m 3 sec
Reputation Power: 0
|
|
For the sake of making the search the easiest we'll say up to but not including 0300. What I am looking for is the syntax needed to make searching within this range possible. I was thinking the same thing you were regarding the variable because of the difference in the way the OS normally displays the date and time but I am lost on how to search within a range (never done that before  )
Thanks,
J.
|

February 20th, 2008, 03:10 AM
|
|
|
|
The simple answer is cheat!
We know the date is 'fixed' in the variable, and we know the hours is always going to be 02:<something> so all we need worry about is the <something> and that, to cover times from 02:30 to 02:59 needs just be 3, 4, or 5.
Thus, look into a regexp that will find "<date in your format> 02:[345]" as the start of a line, and you are done and dusted.
|

February 20th, 2008, 10:32 AM
|
|
Registered User
|
|
Join Date: Apr 2004
Posts: 27
Time spent in forums: 2 h 40 m 3 sec
Reputation Power: 0
|
|
Thank-you sooo much that completely works for what I am doing with it  Now just one more question:
What if I were to want to be able to grep multiple hours so per the original request let's say I needed to also include 0300 as part of the range, how would the search string change? I mean what would the new search look like?
Thanks man,
J.
|

February 21st, 2008, 01:56 AM
|
|
|
|
Ok, well the start part, being the date would remain the same - let us hope!, so we are on to a good start here.
Then we'd need to make the hour component a variable like was done with the date - populating it with "02" in the first instance, along with the minutes part of "[345]" - all as before.
To roll in a new time to check, just chnage th ehour variable - to in the case - "03" and the minutes to match. If you wanted to capture 03:00-03:09 your just put "0" in the minutes variable, if you only wanted 03:00 your put "00" in.
|

June 20th, 2008, 03:00 AM
|
|
Registered User
|
|
Join Date: Jun 2008
Posts: 1
Time spent in forums: 18 m 21 sec
Reputation Power: 0
|
|
|
Grep on range of time..
The client has asked for logs from a particular domain from the exim_mainlog files .. the time was between 10:30 to 12:00 ..
So this is what we did and got the result .. Basically we did a grep on his domain and then on the time .. after which we had given a range of time..
cat exim_mainlog |grep domainname.com |grep -E '2008-06-20 (1[0-1]:[0-5][0-9]|12:00)'
This will grep all the records from exim_mainlog from domainname.com on date 20-6-2008 and time 10:00 - 12:00
|

March 7th, 2012, 11:14 PM
|
|
Registered User
|
|
Join Date: Mar 2012
Posts: 6
Time spent in forums: 1 h 2 m
Reputation Power: 0
|
|
|
Need Grep info
Need an help.. want to grep data in a log file from present system time to past 30 mins
The format of the data will be as follows
03-07-2012 11:09:58.275 info message
03-07-2012 11:10:01.575 info message
|

March 8th, 2012, 10:34 AM
|
 |
Contributing User
|
|
|
|
|
Use gawk
Code:
gawk -F'[- :.]' 'BEGIN{NOW=systime();THEN=NOW-(30*60+1)}{LOGTIME=mktime($3 " " $1 " " $2 " " $4 " " $5 " " $6);if(THEN<LOGTIME){print}}'
__________________
[code] Code tags[/code] are essential for python code!
Last edited by b49P23TIvg : March 8th, 2012 at 10:35 AM.
Reason: remove debug junk
|

March 8th, 2012, 11:13 PM
|
|
Registered User
|
|
Join Date: Mar 2012
Posts: 6
Time spent in forums: 1 h 2 m
Reputation Power: 0
|
|
|
Need Grep info
Quote: | Originally Posted by b49P23TIvg
Code:
gawk -F'[- :.]' 'BEGIN{NOW=systime();THEN=NOW-(30*60+1)}{LOGTIME=mktime($3 " " $1 " " $2 " " $4 " " $5 " " $6);if(THEN<LOGTIME){print}}'
|
Thanks for the command but i am not aware of gawk command.. i am  with that gawk command
can you please give me any grep and sed command
The task is i need to write a script which grep data in a log file from current system time to past 30 minutes.
i will place this script in crontab so that it will run for every 30 mins.
Please help me with the script or with an logic to get that
The format of data in the log file will be as follows
03-07-2012 11:09:58.275 info message
03-07-2012 11:10:01.575 info message
03-0702012 12:05:59.678 info message
Thanks in Advacne
|

March 9th, 2012, 06:14 AM
|
 |
Contributing User
|
|
|
|
|
Install gawk on your system.
I think you need it for my program instead of awk or nawk because the field separator is a regular expression.
If you have linux and an awk program it is assuredly gawk.
try
man awk
Here's a web page.
http://www.gnu.org/software/gawk/
Learn to write shell pipes and redirect io.
|

March 13th, 2012, 09:33 PM
|
|
Registered User
|
|
Join Date: Mar 2012
Posts: 3
Time spent in forums: 37 m 24 sec
Reputation Power: 0
|
|
|
ls -lrt /usr/local/intranet/areas/prod/output/SRGW_0?/
O/P of above command.
drwxr-xr-x 2 mtsadm mts 4096 Mar 13 10:44 153913
drwxr-xr-x 2 mtsadm mts 4096 Mar 13 10:48 153914
drwxr-xr-x 2 mtsadm mts 4096 Mar 13 10:53 153915
drwxr-xr-x 2 mtsadm mts 4096 Mar 13 10:57 153916
drwxr-xr-x 2 mtsadm mts 4096 Mar 13 11:01 153917
drwxr-xr-x 2 mtsadm mts 4096 Mar 13 11:05 153918
drwxr-xr-x 2 mtsadm mts 4096 Mar 13 11:10 153919
drwxr-xr-x 2 mtsadm mts 4096 Mar 13 11:14 153921
drwxr-xr-x 2 mtsadm mts 4096 Mar 13 11:14 153920
Load time of 153913 = 4 minutes.
I need to in corporate a logic similar to this but need to do it in a loop as there are many directories for load time calculation.
|

March 13th, 2012, 09:59 PM
|
 |
Contributing User
|
|
|
|
|
I hope simon is available. Maybe he would clearly explain to you the importance of writing a specific question.
What does O/P mean?
You need to IN CORPORATE logic similar to what?
What mighty version of ls do you use that displays, with ls -lrt
Load time of 153913 = 4 minutes.
?
The theme of this thread seems to be handling time strings using bash. Where are your time strings? What do you want done with them? What is your directory structure?
|

March 13th, 2012, 10:06 PM
|
|
Registered User
|
|
Join Date: Mar 2012
Posts: 3
Time spent in forums: 37 m 24 sec
Reputation Power: 0
|
|
|
Load time
A report needs to come some what similar to this
No of elements Stream Batch No Load time
A B C D
A=75,B=SRGW_05,C=153907 im able to get quite easily
Code:
wc -l /usr/local/intranet/areas/prod/output/SRGW_0?/*/MESSAGE_T.dat
Output of above command.
A B C
Code:
75/usr/local/intranet/areas/prod/output/SRGW_05/153907/MESSAGE_T.dat
26 /usr/local/intranet/areas/prod/output/SRGW_05/153908/MESSAGE_T.dat
110 /usr/local/intranet/areas/prod/output/SRGW_05/153909/MESSAGE_T.dat
Code:
wc -l /usr/local/intranet/areas/prod/output/SRGW_05/*/MESSAGE_T.dat | cut -f1,8,9 -d"/"
O/P of above command.
Code:
159 /SRGW_05/153917
367 /SRGW_05/153918
21 /SRGW_05/153919
12 /SRGW_05/153920
88 /SRGW_05/153921
35 /SRGW_05/153922
36 /SRGW_05/153923
For D I need to check every 2 Batches and compare so I need to put it in a loop
Load time needs to be time stamp of folder created of C.
Code:
ls -lrt /usr/local/intranet/areas/prod/output/SRGW_0?/
Output of above command.
Code:
drwxr-xr-x 2 mtsadm mts 4096 Mar 13 10:44 153913
drwxr-xr-x 2 mtsadm mts 4096 Mar 13 10:48 153914
drwxr-xr-x 2 mtsadm mts 4096 Mar 13 10:53 153915
drwxr-xr-x 2 mtsadm mts 4096 Mar 13 10:57 153916
drwxr-xr-x 2 mtsadm mts 4096 Mar 13 11:01 153917
drwxr-xr-x 2 mtsadm mts 4096 Mar 13 11:05 153918
drwxr-xr-x 2 mtsadm mts 4096 Mar 13 11:10 153919
drwxr-xr-x 2 mtsadm mts 4096 Mar 13 11:14 153921
drwxr-xr-x 2 mtsadm mts 4096 Mar 13 11:14 153920
This output is required
Load time of 153913 = 4 minutes.
I need to in corporate a logic similar to this but need to do it in a loop as there are many directories for load time calculation.
Store in a variable = ls -
Code:
lrt /usr/local/intranet/areas/prod/output/SRGW_05/ | cut -f24 -d"
"
Code:
h1=`echo $T1|cut -d: -f1`
m1=`echo $T1|cut -d: -f2`
x1=`echo "$h1*60 + $m1"|bc -l`
h2=`echo $T2|cut -d: -f1`
m2=`echo $T2|cut -d: -f2`
x2=`echo "$h2*60 + $m2"|bc -l`
if test $x1 -lt $x2
then
diff=`echo "$x2 - $x1"|bc -l`
else
diff=`echo "$x1 - $x2"|bc -l`
fi
echo "Load time is $diff"
Entire Output should be like this eventually
No of elements Stream Batch No Load time
A B C D
Can some one help me ?
Appologies if my previous question was not framed properly 
Last edited by peckenson : March 13th, 2012 at 10:40 PM.
Reason: Correction
|

March 14th, 2012, 10:46 AM
|
|
|
|
Despite the request having been clarified in this thread I will respond in your other thread ...
Being as clear as you can helps greatly. For example, in your other thread on this subject it was not wholly obvious if just the time differential for the first two items in the list was needed, or if it was a 'rolling' operation.
|
Developer Shed Advertisers and Affiliates
| Thread Tools |
Search this Thread |
|
|
|
| Display Modes |
Rate This Thread |
Linear Mode
|
|
Posting Rules
|
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts
HTML code is Off
|
|
|
|
|