Grep or sed or awk for file manipulation??

Dev Shed Forums Sponsor:

December 7th, 2010, 02:44 PM

butcher

Contributing User

Join Date: Jul 2001

Location: In a constant state of turmoil

Posts: 858

Time spent in forums: 1 Week 1 Day 20 h 5 m 30 sec
Reputation Power: 81

Grep or sed or awk for file manipulation??

Let me *try* and explain what I'm trying to do, and please keep in mind aside from a little command line stuff I'm a beginner to any of what I'm asking about.

I have a directory structure something like this:

Code:

/usr/local/chatlogs/
    webserver1/
        1.chatlog
        2.chatlog
    webserver2/
        1.chatlog
        2.chatlog
    webserver3/
        1.chatlog
        2.chatlog

The 1 and 2 in the file name (1.chatlog) correspond to a specific room id and there's the possibility they will be different ones each time.

I'm writing a bash script (trying to anyway) that will scoop up the logs in each /webserver(n) directory and put them into a single file but according to the file name. So in this instance I would like the 3 1.chatlog files to end up all togehter in a single 1.chatlog file (and the same with the 2.chatlog files into 1 big 2.chatlog) that I can then parse line by line. I can handle the parsing part, but what I was hoping for help on was getting the files together into 1 big 1.chatlog and 2.chatlog.

The main problem is that I will never know exactly what files are in each /webserver(n) directory and I'm trying to pull off putting them in 1 correspondingly named big file without having to loop through each directory and figure out what's there.

I know I can do something like:

Code:

LOGFILES="/usr/local/chatlogs/*/*.chatlog"
FULLLOGFILE="/path/to/new/filename"
cat $LOGFILES | grep -v "^#" > $FULLLOGFILE

#do what I need to with contents of $FULLLOGFILE

but that doesn't keep them separated as I need them by the file name that I need for a room id reference.
What I'm hoping somebody here can tell me is if there's a 'variable' way that I can get each of the different numbered files into a corresponding large file with the same name. I'm hoping there's a way to do a back reference of some kind that I can catch the number in the start if the file name (1.chatlog) and use it in the same statement for when I jam everything into it. So what I'm thinking is something like:

cat /usr/local/chatlogs/*/(*).chatlog > /new/file/path/\1.chatlog

So that whatever was captured in the () in the first part of the statement would be used in the \1 in the back part of the statement for every n.chatlog that might be in any of the /webserver directories at that time.

Thanks to anybody that stuck around to get to this point, and I hope my explanation was clear enough to understand what I'm hoping to do.

Butcher

__________________
- Butcher -

December 8th, 2010, 11:08 AM

crustymonkey

Contributing User

Join Date: Oct 2003

Location: Minneapolis, MN

Posts: 356

Time spent in forums: 2 Days 13 h 38 m 46 sec
Reputation Power: 11

This short shell script should do the trick:

bash Code:

Original - bash Code

#!/bin/sh
 
OUTDIR=outdir
TOPDIR=/usr/local/chatlogs
logFExt=chatlog
tmpFile=$(tempfile)
trap "rm $tmpFile" 1 2 15
 
# Make sure we have our output directory
mkdir -p $OUTDIR
 
# First get the unique numbers
uniques=$(find $TOPDIR -type f -name "*.$logFExt")
for u in $uniques ; do
    echo $u | sed -e 's#^.*/\([[:digit:]]*\)\.chatlog$#\1#' >> $tmpFile
done
uniqueNums=$(sort $tmpFile | uniq)
 
# Remove our tempfile
rm $tmpFile
 
# Now that we have the unique numbers for the chatlog precursors in all the 
# directories, we will combine them into our outfile
for num in $uniqueNums ; do
    # Get all the relevant files to combine
    toCombine=$(find $TOPDIR -type f -name $num.$logFExt | sort)
    cat $toCombine > $OUTDIR/$num.$logFExt
done

That does assume (based on your example) that there aren't spaces in file or directory names.

__________________
badger badger badger badger
badger badger badger badger
MUSHROOM MUSHROOM

Last edited by crustymonkey : December 8th, 2010 at 11:19 AM.

December 10th, 2010, 06:51 AM

butcher

Contributing User

Join Date: Jul 2001

Location: In a constant state of turmoil

Posts: 858

Time spent in forums: 1 Week 1 Day 20 h 5 m 30 sec
Reputation Power: 81

That's BEAUTIFUL crustymonkey!

Thank you very much for the extra effort.

Grep or sed or awk for file manipulation??

Developer Shed Advertisers and Affiliates