Page 1 of 2 12 Last
  • Jump to page:
    #1
  1. PHP & Java Error Master
    Devshed Beginner (1000 - 1499 posts)

    Join Date
    Jun 2003
    Location
    My Computer
    Posts
    1,219
    Rep Power
    0

    Shell scripting > Looping through the contents of a file


    I'm trying to loop through a file that'll containt machine names to connect to through SSH and it doesn't seem to perform all the actions in the loop before it tries to go to the next line.
    I'll get a ksh error message stating that the name of the server cannont be found (the error you get when you try to execute a command that's not found etc) before it does anything else.

    Here's what the output says:
    Reading in node names from file...
    Pseudo-terminal will not be allocated because stdin is not a terminal.
    user@box_name's password:
    stty: tcgetattr: A specified file does not support the ioctl system call.
    ksh: box_name2: not found.
    Connected to box_name!
    Node name : box_name
    OS Name :
    OS Version :
    IP Address :
    Memory : MB
    CPU Count :
    Waiting 2 before reconnect...
    Here's my code:
    Code:
    while read linevar
    do
            ssh -l username $linevar
            echo "Connected to $linevar!"
    
            ### Obtain system information ###
    
            node=$(uname -n | tr '[A-Z]' '[a-z]')
            os=$(uname)
    
            if test $os = AIX ; then
               osv=`oslevel`
               ip=$(ping -a -c1 $linevar | head -1 | awk '{print $3}' | tr -d ":")
               mem=$(lsattr -E -l mem0 | tail -1 | awk '{print $2}')
               cpus=$(lscfg | grep proc | wc -l | awk '{print $1}')
            fi
            if test $os = SunOS ; then
               osv=`uname -r`
               ip=$(ping -a -c1 dingo | head -1 | awk '{print $2}')
               mem=$(/usr/sbin/prtconf | grep 'Memory size' | awk '{print $3}')
               cpus=$(/usr/sbin/psrinfo | wc -l | awk '{print $1}')
            fi
            if test $os = Linux ; then
               osv=`uname -r`
               ip=$(ping -c1 $linevar | head -1 | awk '{print $3}' | tr -d ":")
               mem=$(grep MemTotal /proc/meminfo | tail -1 | awk '{print $2}')
               cpus=$(cat /proc/cpuinfo | grep ^processor| wc -l | awk '{print $1}')
               let "mem = $mem / 1024"
            fi
    
            if test -r ../data/node_details.txt ; then
                    rm ../data/node_details.txt
            else
                    touch ../data/node_details.txt
            fi
    
            echo "Node name   :  $node" #>> ../data/node_details.txt
            echo "OS Name     :  $os" #>> ../data/node_details.txt
            echo "OS Version  :  $osv" #>> ../data/node_details.txt
            echo "IP Address  :  $ip" #>> ../data/node_details.txt
            echo "Memory      :  $mem MB" #>> ../data/node_details.txt
            echo "CPU Count   :  $cpus" #>> ../data/node_details.txt
    
            sleep 2
            echo "Waiting 2 before reconnect..."
    done < node_list.txt
    Thanks.
    Last edited by StevenC; September 23rd, 2004 at 04:19 PM.
  2. #2
  3. No Profile Picture
    Contributing User
    Devshed Beginner (1000 - 1499 posts)

    Join Date
    Jul 2004
    Location
    Middle Europa
    Posts
    1,198
    Rep Power
    17
    may i kindly ask you:
    please take a couple of shell scripting lessons ?
    just an example (i mean a s.....y):
    cpus=$(cat /proc/cpuinfo | grep ^processor| wc -l | awk '{print $1}')
    cat is useless, you did not understand: grep does not need 'cat'
    wc is useless, for what ??
    awk is more then useles, but the moderator will not like my commentar
    why not a simply:
    grep -c '^processor' /proc/cpuinfo
    show us: you know *nix, try it!
    Last edited by guggach; September 24th, 2004 at 02:14 AM. Reason: typo
  4. #3
  5. No Profile Picture
    Contributing User
    Devshed Beginner (1000 - 1499 posts)

    Join Date
    Jul 2004
    Location
    Middle Europa
    Posts
    1,198
    Rep Power
    17
    OK, you are in the middle
    a lot of people does: cat|grep|awk
    they(you also) don't realized:
    'grep' does not need 'cat'
    and 'awk' does not need 'grep'

    but it's modern so we do the s..t
    i am an older unix-guy, maybe too old
    i would not accept that script in my (test)team.
    Last edited by guggach; September 24th, 2004 at 02:29 AM. Reason: typo
  6. #4
  7. No Profile Picture
    Contributing User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Sep 2004
    Posts
    60
    Rep Power
    14
    Hi StevenC,

    You'll find it a lot easier to have a list of nodes that you want to loop round and a script that reads this list to execute the command you want.

    I've not got a copy of my "allsh" script but it used to work as follows:

    Have an "all.nodes" file containing lines such as:
    Code:
    #Nodename:Status:Application:Other stuff...
    grumpy:Production:Payroll
    lazy:Development:Payroll
    sleepy:Production:Sales
    happy:Development:Sales
    funny:Production:Manufacturing
    gloopy:Development:Manufacturing
    Execute commands against selected nodes using the allsh command (which egrep'd out the requested nodes from the all.nodes file). Examples being:

    Code:
    #1) run the hostname command against all nodes:
    allsh hostname   
    
    #2) run the df -k command against all Production nodes:
    allsh Production "df -k"
    
    #3) run script /var/tmp/somescript.ksh against all 
    # Manufacturing and Sales nodes:
    # Note that the script /var/tmp/somescript.ksh would 
    # have to reside on the remote node
    allsh "Manufacturing|Sales" /var/tmp/somescript.ksh
    As you can see from the 3rd example, sometimes the command I was wanting to run was too complicated to put on the command line. So I wrote a script, then used an "allftp" script to put it to my selected nodes, then I'd run the command (after giving it execute permission of course).

    I do have an allftp script I'm currently using. You can probably make up the allsh script from this quite easily. Once we have ssh installed on our environments I'll create an allssh and forward it onto you if you want.

    Please find script allftp below:

    Code:
    #!/usr/bin/ksh
    
    #------------------------------------------------------------------------------------------------
    usage()
    #------------------------------------------------------------------------------------------------
    {
    cat <<END
    
    Filename    : allftp
    Version     : 1.0
    Summary     : Executes the given ftpscript against given nodes in ~oradba/dat/all.nodes
    
    Parameters  : The last parameter is the name of the file containing the FTP commands
                : to be executed.
                : Any parameters in between are egrep search strings that will be applied to
                : file all.nodes to determine which nodes to execute the commands against.
    
    Calls       : No other scripts
    
    Date       Author   Version
    04-Feb-04  ASBlack  1.0      Initial version.
    
    EXAMPLES:
    1) allftp /tmp/put_files.ftp
    Run FTP commands in file /tmp/put_files.ftp against all nodes.
    
    2) allftp "lims|mcas" /tmp/create_dirs.ftp
    Run FTP commands in file /tmp/create_dirs.ftp against LIMS and MCAS nodes.
    
    END
    }
    
    
    #------------------------------------------------------------------------------------------------
    mk_ftpscript()
    #------------------------------------------------------------------------------------------------
    {
    echo "ftp ${NODE} <<END_OF_FTP"
    cat ${FTPSCRIPT}
    echo "bye"
    echo "END_OF_FTP"
    }
    
    
    #------------------------------------------------------------------------------------------------
    # MAIN PROGRAM STARTS HERE
    #------------------------------------------------------------------------------------------------
    
    #-- Display usage if requested
    if [[ ${1} = "help" ]]; then usage; exit 0; fi
    
    #-- Check that parameters have been passed
    NUM_PARAMS=$#
    if [[ ${NUM_PARAMS} -lt 1 ]]; then
      usage
      echo "ERROR: at least the file containing the FTP commands needs to be passed"
      exit 10
    fi
    
    #-- Set variables
    PRG=`basename $0`
    TMP=/var/tmp/${PRG}.$$
    ALLNODES=~oradba/dat/all.nodes
    FAILEDLIST=""  # List of NODES that the FTP has failed to run against.
    
    #-- Determine the command to select the records from all.nodes
    CMD="egrep -iv '^#|^$' ${ALLNODES}"
    COUNTER=1
    while [ ${COUNTER} -lt ${NUM_PARAMS} ]
    do
      CMD="${CMD} |egrep -i '$1'" 
      shift
      COUNTER=`echo ${COUNTER} + 1 | bc` 
    done
    
    FTPSCRIPT=$1
    echo CMD=${CMD}
    echo FTPSCRIPT=${FTPSCRIPT}
    
    #-- Check that the file exists
    if [ ! -f ${FTPSCRIPT} ]; then
      echo "ERROR: File ${FTPSCRIPT} does not exist"
      exit 20
    fi
    
    #-- Execute the ftpscript against all nodes in all.nodes
    for NODE in `echo ${CMD} |ksh |awk -F":" '{print $1}'`
    do
    
      echo "#-----------------------------------------------------------------"
      grep "^${NODE}:" ${ALLNODES}  |sed -e 's/^/# /'
    
      #-- Check if we can ping the host
      ping ${NODE} 5 >/dev/null 2>&1
      if [ $? -ne 0 ]; then
        echo "${PRG}: ERROR: Cannot ping ${NODE}"
        FAILEDLIST=`echo "${FAILEDLIST}|${NODE}" |sed -e 's/^|//'`
        echo "FAILEDLIST=${FAILEDLIST}"
        continue
      fi
    
    
      #-- Create the ftpscript for this node
      mk_ftpscript > ${TMP}.ftpscript
    
      #-- Execute the ftpscript for this node
      ksh ${TMP}.ftpscript
    
    done
    
    #-- Display command to run the FTP on the failed NODES
    echo
    if [ ! -z "${FAILEDLIST}" ]; then
      echo "Command to rerun against the FAILEDLIST:"
      echo "allftp \"${FAILEDLIST}\" ${FTPSCRIPT}"
    else
      echo "Command appeared to execute against all nodes."
    fi
    
    #-- Tidy up
    rm ${TMP}.*
    And it being run:
    Code:
    $cat /var/tmp/asb.ftp
    put /var/tmp/asb.ksh
    dir /var/tmp/asb.ksh
    
    $ allftp MCAS /var/tmp/asb.ftp
    CMD=egrep -iv '^#|^$' /export/home/oradba/dat/all.nodes |egrep -i 'MCAS'
    FTPSCRIPT=/var/tmp/asb.ftp
    #-----------------------------------------------------------------
    # dssmcdb01:MCAS
    -rw-r--r--   1 oradba   dba          814 Sep 24 15:53 /var/tmp/asb.ksh
    #-----------------------------------------------------------------
    # dssmcdb02:MCAS
    -rw-r--r--   1 oradba   dba          814 Sep 24 15:53 /var/tmp/asb.ksh
    
    Command appeared to execute against all nodes.

    Hope this helps,
    Andy
    Last edited by andyb1ack; October 15th, 2004 at 08:39 AM. Reason: Removed UNIX prompt from sample output
  8. #5
  9. PHP & Java Error Master
    Devshed Beginner (1000 - 1499 posts)

    Join Date
    Jun 2003
    Location
    My Computer
    Posts
    1,219
    Rep Power
    0
    OK, I think I've been misunderstood.

    guggach: cpus=$(cat /proc/cpuinfo | grep ^processor| wc -l | awk '{print $1}') works exactly as I want it to. It gets the # of cpu's for the box.
    Also, cristo showed me that command. (I added awk part to get rid of extra whitespace.)

    Andy: that pretty much went way over my head.
    All I really need is to know why my loop seems to mess up per my description above.

    I'm really pretty new to shell scripting, so I'm not sure what your showing me.

    The point of my script is to connect to each node listed in my node list file, get a bunch of system info, store it in a file, then go to the next one in the list.
    That's all.

    Thanks.
  10. #6
  11. No Profile Picture
    Contributing User
    Devshed Beginner (1000 - 1499 posts)

    Join Date
    Jul 2004
    Location
    Middle Europa
    Posts
    1,198
    Rep Power
    17
    StevenC: did you try 'grep -c' ?
    or: man grep ? it's also very interesting.
    btw:
    Code:
    for file in `ls|cat|grep -v " "|sed -n '/./p`
    do ls -lisag $file
    done
    works perfectly too.
    Last edited by guggach; September 24th, 2004 at 02:24 PM. Reason: typo
  12. #7
  13. No Profile Picture
    Contributing User
    Devshed Beginner (1000 - 1499 posts)

    Join Date
    Jul 2004
    Location
    Middle Europa
    Posts
    1,198
    Rep Power
    17
    sorry i forgot: awk
  14. #8
  15. No Profile Picture
    Contributing User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Sep 2004
    Posts
    60
    Rep Power
    14
    Oops. Sorry SteveC.
    Once you've sorted this particular problem and find you're regularly writing code to loop round nodes then you'll end up writing some script that will read in the list of nodes and do whatever command you give it... Which is what I was waffling on about
  16. #9
  17. PHP & Java Error Master
    Devshed Beginner (1000 - 1499 posts)

    Join Date
    Jun 2003
    Location
    My Computer
    Posts
    1,219
    Rep Power
    0
    Well, what I want to know is is the loop above written correctly?

    And by correctly I mean is will the loop above actually read in the file specified, and perform the actions after the do and before the done once for each line in the input file?

    Thanks.
  18. #10
  19. No Profile Picture
    Contributing User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Sep 2004
    Posts
    60
    Rep Power
    14
    Can you let me know what happens if you do the following:
    Code:
    cat node_list.txt |while read NODENAME
    do
      echo
      echo Running the hostname command against ${NODENAME} as username
      ssh username@${NODENAME} hostname
      echo Running the hostname command against ${NODENAME} as ${username}
      ssh ${username}@${NODENAME} hostname
    done
  20. #11
  21. PHP & Java Error Master
    Devshed Beginner (1000 - 1499 posts)

    Join Date
    Jun 2003
    Location
    My Computer
    Posts
    1,219
    Rep Power
    0
    I of course changed username to my own username.
    There's 2 lines in the file right now:
    LIGHTNING2
    RANGER
    Code:
    $ ./loop_test
    
    Running the hostname command against LIGHTNING2 as username
    username@lightning2's password:
    lightning2
    Running the hostname command against LIGHTNING2 as
    Usage: ssh [options] host [command]
    Options:
      -l user     Log in using this user name.
      -n          Redirect input from /dev/null.
      -F config   Config file (default: ~/.ssh/config).
      -A          Enable authentication agent forwarding.
      -a          Disable authentication agent forwarding (default).
      -X          Enable X11 connection forwarding.
      -x          Disable X11 connection forwarding (default).
      -i file     Identity for public key authentication (default: ~/.ssh/identity)
      -t          Tty; allocate a tty even if command is given.
      -T          Do not allocate a tty.
      -v          Verbose; display verbose debugging messages.
                  Multiple -v increases verbosity.
      -V          Display version number only.
      -P          Don't allocate a privileged port.
      -q          Quiet; don't display any warning messages.
      -f          Fork into background after authentication.
      -e char     Set escape character; ``none'' = disable (default: ~).
      -c cipher   Select encryption algorithm
      -m macs     Specify MAC algorithms for protocol version 2.
      -p port     Connect to this port.  Server must be on the same port.
      -L listen-port:host:port   Forward local port to remote address
      -R listen-port:host:port   Forward remote port to local address
                  These cause ssh to listen for connections on a port, and
                  forward them to the other side by connecting to host:port.
      -D port     Enable dynamic application-level port forwarding.
      -C          Enable compression.
      -N          Do not execute a shell or command.
      -g          Allow remote hosts to connect to forwarded ports.
      -1          Force protocol version 1.
      -2          Force protocol version 2.
      -4          Use IPv4 only.
      -6          Use IPv6 only.
      -o 'option' Process the option as if it was read from a configuration file.
      -s          Invoke command (mandatory) as SSH2 subsystem.
      -b addr     Local IP address.
  22. #12
  23. No Profile Picture
    Contributing User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Sep 2004
    Posts
    60
    Rep Power
    14
    Sorry this is taking so long to-ing and fro-ing but we don't have ssh at work so I can't try this myself... (if anyone else uses ssh a lot feel free to put their two pennies worth in! )
    .
    .
    I wasn't sure what was going on with username hence that bit of code. It's interesting what it did though...

    Are LIGHTNING2 and RANGER the names of two servers you're wanting to loop round?

    LIGHTNING2 was read successfully into the NODENAME variable, but I don't see how it appeared again below (and in lower case too!):
    Running the hostname command against LIGHTNING2 as username
    username@lightning2's password:
    lightning2
    Is the ssh command wanting you to input the password for the user, or have you set it up so that a password is not required?
    If not or you're not able/allowed to set it up I think I had a script in the past that passed the password to ssh via the method below (it might work!):
    Code:
    echo password |ssh username@hostname
    Given you are changing username to your correct one, I guess the next stage is see what the following does on it's own:
    Code:
    cat node_list.txt |while read NODENAME
    do
      echo
      echo Running the hostname command against ${NODENAME} as username
      ssh username@${NODENAME} hostname
    done
  24. #13
  25. PHP & Java Error Master
    Devshed Beginner (1000 - 1499 posts)

    Join Date
    Jun 2003
    Location
    My Computer
    Posts
    1,219
    Rep Power
    0
    Yes, they're hostnames of boxes on our intranet here.

    The lowercase part is probably just the real name (probably case insensivity on the server end; it sees the uppercase name and it doesn't hurt, but when it spits back into it's in lowercase form).

    No, there's no keys setup to bypass passwords.

    And that last bit of code returns:
    Code:
    $ ./loop_test
    
    Running the hostname command against LIGHTNING2 as username
    username@lightning2's password:
    lightning2
  26. #14
  27. No Profile Picture
    Contributing User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Sep 2004
    Posts
    60
    Rep Power
    14
    Try ssh -n

    Code:
    echo LIGHTNING2 RANGER |while read NODENAME
    do
      echo
      echo Running the hostname command against ${NODENAME}
      ssh -n username@${NODENAME} hostname
    done
  28. #15
  29. PHP & Java Error Master
    Devshed Beginner (1000 - 1499 posts)

    Join Date
    Jun 2003
    Location
    My Computer
    Posts
    1,219
    Rep Power
    0
    Code:
    $ . ./loop_test
    
    Running the hostname command against LIGHTNING2 RANGER
    username@lightning2's password:
    ksh: RANGER:  not found.
Page 1 of 2 12 Last
  • Jump to page:

IMN logo majestic logo threadwatch logo seochat tools logo