#1
  1. No Profile Picture
    Contributing User
    Devshed Novice (500 - 999 posts)

    Join Date
    Jul 2001
    Location
    In a constant state of turmoil
    Posts
    858
    Rep Power
    83

    Grep or sed or awk for file manipulation??


    Let me *try* and explain what I'm trying to do, and please keep in mind aside from a little command line stuff I'm a beginner to any of what I'm asking about.

    I have a directory structure something like this:
    Code:
    /usr/local/chatlogs/
        webserver1/
            1.chatlog
            2.chatlog
        webserver2/
            1.chatlog
            2.chatlog
        webserver3/
            1.chatlog
            2.chatlog
    The 1 and 2 in the file name (1.chatlog) correspond to a specific room id and there's the possibility they will be different ones each time.

    I'm writing a bash script (trying to anyway) that will scoop up the logs in each /webserver(n) directory and put them into a single file but according to the file name. So in this instance I would like the 3 1.chatlog files to end up all togehter in a single 1.chatlog file (and the same with the 2.chatlog files into 1 big 2.chatlog) that I can then parse line by line. I can handle the parsing part, but what I was hoping for help on was getting the files together into 1 big 1.chatlog and 2.chatlog.

    The main problem is that I will never know exactly what files are in each /webserver(n) directory and I'm trying to pull off putting them in 1 correspondingly named big file without having to loop through each directory and figure out what's there.

    I know I can do something like:
    Code:
    LOGFILES="/usr/local/chatlogs/*/*.chatlog"
    FULLLOGFILE="/path/to/new/filename"
    cat $LOGFILES | grep -v "^#" > $FULLLOGFILE
    
    #do what I need to with contents of $FULLLOGFILE
    but that doesn't keep them separated as I need them by the file name that I need for a room id reference.
    What I'm hoping somebody here can tell me is if there's a 'variable' way that I can get each of the different numbered files into a corresponding large file with the same name. I'm hoping there's a way to do a back reference of some kind that I can catch the number in the start if the file name (1.chatlog) and use it in the same statement for when I jam everything into it. So what I'm thinking is something like:

    cat /usr/local/chatlogs/*/(*).chatlog > /new/file/path/\1.chatlog

    So that whatever was captured in the () in the first part of the statement would be used in the \1 in the back part of the statement for every n.chatlog that might be in any of the /webserver directories at that time.

    Thanks to anybody that stuck around to get to this point, and I hope my explanation was clear enough to understand what I'm hoping to do.

    Butcher
    - Butcher -
  2. #2
  3. Contributing User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Oct 2003
    Location
    Minneapolis, MN
    Posts
    356
    Rep Power
    12
    This short shell script should do the trick:
    bash Code:
    #!/bin/sh
     
    OUTDIR=outdir
    TOPDIR=/usr/local/chatlogs
    logFExt=chatlog
    tmpFile=$(tempfile)
    trap "rm $tmpFile" 1 2 15
     
    # Make sure we have our output directory
    mkdir -p $OUTDIR
     
    # First get the unique numbers
    uniques=$(find $TOPDIR -type f -name "*.$logFExt")
    for u in $uniques ; do
        echo $u | sed -e 's#^.*/\([[:digit:]]*\)\.chatlog$#\1#' >> $tmpFile
    done
    uniqueNums=$(sort $tmpFile | uniq)
     
    # Remove our tempfile
    rm $tmpFile
     
    # Now that we have the unique numbers for the chatlog precursors in all the 
    # directories, we will combine them into our outfile
    for num in $uniqueNums ; do
        # Get all the relevant files to combine
        toCombine=$(find $TOPDIR -type f -name $num.$logFExt | sort)
        cat $toCombine > $OUTDIR/$num.$logFExt
    done

    That does assume (based on your example) that there aren't spaces in file or directory names.
    Last edited by crustymonkey; December 8th, 2010 at 11:19 AM.
    badger badger badger badger
    badger badger badger badger
    MUSHROOM MUSHROOM
  4. #3
  5. No Profile Picture
    Contributing User
    Devshed Novice (500 - 999 posts)

    Join Date
    Jul 2001
    Location
    In a constant state of turmoil
    Posts
    858
    Rep Power
    83
    That's BEAUTIFUL crustymonkey!

    Thank you very much for the extra effort.
    - Butcher -

IMN logo majestic logo threadwatch logo seochat tools logo