#1
  1. No Profile Picture
    Contributing User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Sep 2003
    Location
    sydney
    Posts
    47
    Rep Power
    12

    Smile getline with if condition


    Hi
    I am new with shell script programming could some one please help.
    How can I get the hostname with his infoline-1 to infoline-n if the total error for the hostname is >0 and then the next hostname-n and infoline-n (Skip the xxxx data and read and print the hostname No of errors and his information

    Data file is:
    Xxxxxxxxxx
    Xxxxxxxxx
    Hostname1
    Infoline-1
    Error=1
    Infoline-n
    Error=0
    xxxxx
    Total Error for hostname1 =1

    Xxxxxxxxxxx
    Xxxxxxxxxxx
    Xxxxxxxxxx
    Hostname2
    Infoline-1
    Error=0
    Infoline-n
    Error=0
    Xxxxxxx
    Total Error for hostname2 =0
    Xxxxxxx
    Hostname 3
    Infoline-1
    error=3
    infoline-n
    error=1
    Total Error for hostname3 =4
  2. #2
  3. No Profile Picture
    Contributing User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Sep 2003
    Posts
    137
    Rep Power
    0
    It's not the most elegant script I wrote, but it does what you want (I think):
    Code:
    #!/bin/bash
    #
    # infile must be given
    # outfile is optional (if none given: host-info.<yymmdd>)
    
    INFILE="$1"
    OUTFILE="$2"
    
    # Are input/output file provided:
    [[ -z ${INFILE} ]] && echo "No input file given. Quiting" && exit 1
    [[ -z ${OUTFILE} ]] && OUTFILE="host-info.`date '+%y%m%d'`"
    
    cat ${INFILE} | \
    sed -n '/^Hostname/,/^Total Error/{
    s/Total Error for hostname\(.*\)/Total Error for hostname\1\
    TAB/
    p
    }' | \
    awk '
    BEGIN { FS = "\n" ; RS = "\t\n" ; OFS = "%" ; ORS = "\n" }
    $(NF-1) !~ /Total Error for hostname.*=0/ {
      for ( x = 1; x <= NF; ++x ) {
        print $x
      }
    }' | \
    egrep "^Hostname|^Infoline|^Total|^$" > ${OUTFILE}
    IMPORTANT: You will probably copy and paste this, so I have to warn you
    for one thing: TAB (line 16) should be replaced by a real tab!!

    Using your inputfile example, this will be in the output file:
    Code:
    Hostname1 
    Infoline-1 
    Infoline-n 
    Total Error for hostname1 =1 
    
    Hostname 3 
    Infoline-1 
    Total Error for hostname3 =4
    Explaining the code:

    I used sed to create 'paragraphs' that are more uniform. These paragraphs
    start with Hostname.... and end with Total Error....... and I added a line with
    a tab in it (which I use with awk later on)

    After sed has done it's job, you are left with the following (substitute <TAB> with a real tab):
    Code:
    Hostname1 
    Infoline-1 
    Error=1 
    Infoline-n 
    Error=0 
    xxxxx 
    Total Error for hostname1 =1 
    <TAB>
    Hostname2 
    Infoline-1 
    Error=0 
    Infoline-n 
    Error=0 
    Xxxxxxx 
    Total Error for hostname2 =0 
    <TAB>
    Hostname 3 
    Infoline-1 
    error=3 
    infoline-n 
    error=1 
    Total Error for hostname3 =4
    <TAB>
    Although we have an irregular amount of fields, we do have TAB
    seperated records and can continue using awk. After setting FS,RS,OFS,
    ORS (input/ouput field and record seperators) we check if this record has
    errors. If it doesn't: don't show it. The for loop is needed because of the
    irregular amount of fields.

    After awk has done its job we are left with:
    Code:
    Hostname1 
    Infoline-1 
    Error=1 
    Infoline-n 
    Error=0 
    xxxxx 
    Total Error for hostname1 =1 
    
    Hostname 3 
    Infoline-1 
    error=3 
    infoline-n 
    error=1 
    Total Error for hostname3 =4
    What's left is to discard the unwanted lines and only show what we want to
    see. This is what the egrep line does.
    Last edited by druuna; September 20th, 2003 at 10:08 AM.
  4. #3
  5. No Profile Picture
    Contributing User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Sep 2003
    Location
    sydney
    Posts
    47
    Rep Power
    12

    Thumbs up getline and if condition


    Excellent and thank you,
    But the data output display all infolines (Contains Errors and non Errors) for the hostnameX.
    What if we want only display the infolines CONTAINS ERRORS only for the hostnameX with Total errors >0 with the same pattern of output or
    In Column format
    (INFOLINES are text lines Ex. UNIX Programming Environment)

    Example

    Hostname1
    Infonline-2
    Error=4
    Infoline-3
    Error=7
    UNIX Programming Environment = 2
    OR

    HOSTNAME INFOLINE ERRORS

    Hostname1 infoline2 4
    Infoline3 7
    UNIX…… 2

    Hostname3 Infoline1 3
    Infoline-n 1

    Column formatted


    Your data output currently producing all Infolines

    Hostname1
    Infoline-1
    Error=1
    Infoline-n
    Error=0
    xxxxx
    Total Error for hostname1 =1

    Hostname 3
    Infoline-1
    error=3
    infoline-n
    error=1
    Total Error for hostname3 =4
  6. #4
  7. No Profile Picture
    Contributing User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Sep 2003
    Posts
    137
    Rep Power
    0
    I'm not entirely clear what it is you want. If I understand correctly you want the error amount/number printed after the Infoline.

    One other thing, you say that my script produces this:

    Hostname1
    Infoline-1
    Error=1
    Infoline-n
    Error=0
    xxxxx
    Total Error for hostname1 =1

    I do believe the output is:

    Hostname1
    Infoline-1
    Infoline-n
    Total Error for hostname1 =1

    Could you post some real examples from your inputfile. I need to know if the xxxxx in your previous posts can be anything at all or are actually messagelines that start with certain keywords.
    I.e. Does xxxx always starts with 'UNIX Programming Environment'.

    If that is the case:

    Code:
    #!/bin/bash
    #
    # infile must be given
    # outfile is optional (if none given: host-info.<yymmdd>)
    
    INFILE="$1"
    OUTFILE="$2"
    
    # Are input/output file provided:
    [[ -z ${INFILE} ]] && echo "No input file given. Quiting" && exit 1
    [[ -z ${OUTFILE} ]] && OUTFILE="host-info.`date '+%y%m%d'`"
    
    cat ${INFILE} | \
    sed -n '/^Hostname/,/^Total Error/{
    s/Total Error for hostname\(.*\)/Total Error for hostname\1\
    TAB/
    p
    }' | \
    awk '
    BEGIN { FS = "\n" ; RS = "\t\n" ; OFS = "%" ; ORS = "\n" }
    $(NF-1) !~ /Total Error for hostname.*=0/ {
      for ( x = 1; x <= NF; ++x ) {
        #print $x
        printf("%s", $x )
      }
    printf("\n")
    }' | \
    sed -e 's/^Hostname/\
    Hostname/' -e 's/[Ee]rror=//g' -e 's/line-/line /g' -e 's/ [Ii]nfoline/\
    - Infoline/g' -e 's/ UNIX Programming Environment =/\
    - UNIX Programming Environment/g' -e 's/Total Error/\
    Total Error/' -e 's/=\([0-9]\)/\1/'
    If I test this with the following input:

    Xxxxxxxxxx
    Xxxxxxxxx
    Hostname1
    Infoline-1
    Error=1
    Infoline-7
    Error=0
    UNIX Programming Environment = 5
    Total Error for hostname1 =1

    Hostname1
    Infonline-2
    Error=4
    Infoline-3
    Error=7
    UNIX Programming Environment = 2
    Total Error for hostname1 =7

    Xxxxxxxxxxx
    Xxxxxxxxxxx
    Xxxxxxxxxx
    Hostname2
    Infoline-1
    Error=0
    Infoline-4
    Error=0
    UNIX Programming Environment = 1
    Total Error for hostname2 =0
    Xxxxxxx
    Hostname 3
    Infoline-1
    error=3
    infoline-8
    error=1
    Total Error for hostname3 =4

    This is the result:

    Hostname1
    - Infoline 1 1
    - Infoline 7 0
    - UNIX Programming Environment 5
    Total Error for hostname1 1

    Hostname1 Infonline 2 4
    - Infoline 3 7
    - UNIX Programming Environment 2
    Total Error for hostname1 7

    Hostname 3
    - Infoline 1 3
    - Infoline 8 1
    Total Error for hostname3 4
  8. #5
  9. No Profile Picture
    Contributing User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Sep 2003
    Location
    sydney
    Posts
    47
    Rep Power
    12

    Thumbs up limits on size of input file


    I have run the script successfully (your first script) on small size of input file only let say less than 10000 bytes.
    But when I run it with larger file I get the following errors could you please let me know how can I rectify the problem what I need to modify in your first script to read larger input files, is there any limit size of reading files

    The error is
    awk: 0602-534 Input line 1) Hostname: Hostname1 cannot be longer than 10,239 bytes
    .
    The source line number is 2.

    Thank you.
  10. #6
  11. No Profile Picture
    Contributing User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Sep 2003
    Location
    sydney
    Posts
    47
    Rep Power
    12

    Thumbs up limits on size of input file


    I have run the script successfully (your first script) on small size of input file only let say less than 10000 bytes.
    But when I run it with larger file I get the following errors could you please let me know how can I rectify the problem what I need to modify in your first script to read larger input files, is there any limit size of reading files

    The error is
    awk: 0602-534 Input line 1) Hostname: Hostname1 cannot be longer than 10,239 bytes
    .
    The source line number is 2.

    Thank you.
  12. #7
  13. No Profile Picture
    Contributing User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Sep 2003
    Posts
    137
    Rep Power
    0
    I cannot reproduce the error you are describing.

    Made an inputfile (1.5 Mb large, 113541 lines long) and fed this to the script. It runs perfectly. No errors whatsoever. I also made one of the Hostname lines long (166 chars), also no errors.

    3 things come to mind:

    1) Your inputfile is different from the example you posted before (that's why I asked for a real example in a previous post). The other reason I asked for (part of) a real file: The example you provide is very irregular, that's one of the reasons why I had to account for things that probably are not present, or more consistent in the real inputfile.

    2) Your input file has dos line terminators (do you see ^M at the end of the lines if you open it with vi??) or other unrecognized chars (again, I need a real example to check some of this).

    3) Your computer has limited memory. The test with the 1.5 Mb input file was done on a machine with 'only' 64Mb of physical memory. I have encounter sed/awk problems that had to do with machine limits (memory/swap/diskspace). But I'm talking about an input file that was over 2Gb on a machine with 512 Mb physical mem.

    I did find one thing (but it is sed related and doesn't produce errors) that could be done better:

    In the following two lines (near bottom of script):

    => Hostname/' -e 's/[Ee]rror=//g' -e 's/line-/line /g' -e 's/ [Ii]nfoline/\
    Remove the space before [Ii]nfoline.

    => - Infoline/g' -e 's/ UNIX Programming Environment =/\
    Remove the space before UNIX.

    In the end it should look like this:

    Hostname/' -e 's/[Ee]rror=//g' -e 's/line-/line /g' -e 's/[Ii]nfoline/\
    - Infoline/g' -e 's/UNIX Programming Environment =/\
  14. #8
  15. No Profile Picture
    Contributing User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Sep 2003
    Location
    sydney
    Posts
    47
    Rep Power
    12

    error condition


    Thanks for your help and follow-up

    I have modify your script as bellow and the data file is attached I am running your script on AIX 4.3.3 and the error message is as follow

    /tmp> ./your_script.sh datafile
    awk: 0602-534 Input line 1) Hostname: bhccms
    cannot be longer than 10,239 bytes.
    The source line number is 2.


    #!/bin/ksh
    #
    # infile must be given
    # outfile is optional (if none given: host-info.<yymmdd> )

    INFILE="$1"
    OUTFILE="$2"

    # Are input/output file provided:
    [[ -z ${INFILE} ]] && echo "No input file given. Quiting" && exit 1
    [[ -z ${OUTFILE} ]] && OUTFILE="host-info.`date '+%y%m%d'`"

    cat ${INFILE} | \
    sed -n '/Hostname/,/Total Violations/{
    s/Total Violations\(.*\)/Total Violations\1\
    TAB/
    p
    }' | \
    awk '
    BEGIN { FS = "\n" ; RS = "\t\n" ; OFS = "%" ; ORS = "\n" }
    $(NF-1) !~ /"Total Violations = 0"/ {
    for ( x = 1; x <= NF; ++x ) {
    print $x }
    }' | \
    egrep "Hostname|Total|^$" > ${OUTFILE}


    Thanks
    Attached Files
  16. #9
  17. No Profile Picture
    Contributing User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Sep 2003
    Posts
    137
    Rep Power
    0
    Glad to know what os and version you are using.

    There's probably nothing wrong with the script(s), its awk that's behaving odd when using aix 4.3.3 (found this after a google search). I've had problems with awk before (on Stratus FTX, but the problem was different then yours).

    I also tested this script on my hp box and it worked fine.

    Possible solutions:
    - use nawk instead if awk (should be present in /usr/bin),
    - ask sysadmin about limits.

    There is an awk testscript that you could run to see if awk is working the way it should:

    http://www.shelldorado.com/articles/testawk.sh

    I haven't used AIX before, so I cannot help you with problems related to this os.
  18. #10
  19. No Profile Picture
    Contributing User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Sep 2003
    Location
    sydney
    Posts
    47
    Rep Power
    12

    getline


    Thank you so much you have been very help full
  20. #11
  21. Moderator Emeritus
    Devshed Supreme Being (6500+ posts)

    Join Date
    Feb 2002
    Location
    Austin, TX
    Posts
    7,188
    Rep Power
    2265
    Moved to the UNIX forum
    DrGroove, Devshed Moderator | New to Devshed? Read the User Guide | Connect with me on LinkedIn

IMN logo majestic logo threadwatch logo seochat tools logo