#1
  1. No Profile Picture
    Contributing User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Feb 2003
    Posts
    156
    Rep Power
    17

    Deployment Script


    Hi,

    I have been developing a deployment script utilising expect (written in tcl) which scp's to copy artifacts onto each box and then ssh's onto each box and executes a puppet command.

    The main deployment script is as follows:

    Code:
    #!/bin/bash
    
    environmenthosts=$1
    puppetfacts=$2
    user=$3
    userpassword=$4
    releaseartifact=puppetmodules.tar.gz
    
    for i in $environmenthosts;do echo $i; done | xargs -I{} bash -c "
    {
      /bin/scppass $userpassword $releaseartifact $user@{}:/tmp;
    }"
    
    for i in $environmenthosts;do echo $i; done | xargs -I{} bash -c "
    {
      ./sshpasscmd $userpassword $user@{} sudo /bin/bash -c \"rm -rf /tmp/puppet; mkdir -p /tmp/puppet; cd /tmp/puppet; tar -xvf /tmp/$releaseartifact; export PATH=/usr/bin:/usr/local/bin:/opt/vagrant_ruby/bin; $puppetfact puppet apply --modulepath=. \"
    }"
    The sshpasscmd is

    Code:
    #!/usr/bin/expect -f
    
    set password [lindex $argv 0]
    set host [lindex $argv 1]
    set args1 [lindex $argv 2]
    set args2 [lindex $argv 3]
    set args3 [lindex $argv 4]
    set args4 [lindex $argv 5]
    set args5 [lindex $argv 6]
    set args6 [lindex $argv 7]
    set args7 [lindex $argv 8]
    set args8 [lindex $argv 9]
    set args9 [lindex $argv 10]
    set args10 [lindex $argv 11]
    set args11 [lindex $argv 12]
    set args12 [lindex $argv 13]
    set args13 [lindex $argv 14]
    set args14 [lindex $argv 15]
    set args15 [lindex $argv 16]
    set args16 [lindex $argv 17]
    set args17 [lindex $argv 18]
    set args18 [lindex $argv 19]
    set args19 [lindex $argv 20]
    set args20 [lindex $argv 21]
    set args21 [lindex $argv 22]
    set args22 [lindex $argv 23]
    set args23 [lindex $argv 24]
    set args24 [lindex $argv 25]
    set args25 [lindex $argv 26]
    
    spawn ssh -oStrictHostKeyChecking=no -oCheckHostIP=no $host $args1 $args2 $args3 $args4 $args5 $args6 $args7 $args8 $args9 $args10 $args11 $args12 $args13 $args14 $args15 $args16 $args17 $args18 $args19 $args20 $args21 $args22 $args23 $args24 $args25
    expect "password"
    send "$password\r"
    expect eof

    When I execute this script, this does partially work, although I do receive the error: 'rm: missing operand'


    If I then add in an ls at the beginning of the set of commands as follows:

    Code:
    ./sshpasscmd $userpassword $user@{} sudo /bin/bash -c \"ls; rm -rf /tmp/puppet; mkdir -p /tmp/puppet; cd /tmp/puppet; tar -xvf /tmp/$releaseartifact; export PATH=/usr/bin:/usr/local/bin:/opt/vagrant_ruby/bin; $puppetfact puppet apply --modulepath=. \"
    I then receive the following error:

    ssh: Could not resolve hostname {}: nodename nor servname provided, or not known
    send: spawn id exp7 not open
    while executing
    "send "$password\r" "
    (file "./sshpasscmd" line 37)


    Appreciate any insights into where the issue might lie in these scripts and why they seem to be rather fragile as they currently stand.

    Many Thanks
  2. #2
  3. No Profile Picture
    Contributing User
    Devshed Frequenter (2500 - 2999 posts)

    Join Date
    Mar 2006
    Posts
    2,632
    Rep Power
    1811
    It's been a while since i have butted heads against scripts, but ... I am confused! You set variable environmenthosts from the 1st parameter passed to the command, but then it is used in a loop which suggests it's a list of name - I presume it looks something like: ./YourCommand "host1 host2 host3" PuppetFacts User UserPwd?
    In addition I'm not a fan of the way you are processing that list of hosts and not really seeing the need to involve xargs? Would not something like
    Code:
    for i in $environmenthosts
    do
      /bin/scpass $userpassword $releaseartifact $user@$i:/tmp
    done
    do the job?
    For the second script, seeing as it's tcl, could you not drop the arg=[lindex ..] list of commands and just specify [lrange $argv 2 end]?
    Last edited by SimonJM; July 26th, 2015 at 05:59 PM.
    The moon on the one hand, the dawn on the other:
    The moon is my sister, the dawn is my brother.
    The moon on my left and the dawn on my right.
    My brother, good morning: my sister, good night.
    -- Hilaire Belloc
  4. #3
  5. No Profile Picture
    Contributing User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Feb 2003
    Posts
    156
    Rep Power
    17
    Many thanks for your feedback... you are indeed correct that the 1st arg is in fact a single string with whitespace separated hostnames.

    The reason for using xargs is that this command needs to run in parallel across all hosts/boxes. I don't want this to be achieved through using background processing as I want the stdout of all concurrent executions of the for loop across all boxes to be piped to the current shell's stdout. I have omitted how this is achieved from the above example, but basically, I am using tee to output to both stdout and to a file. The reason I need all concurrent executions to be running in the foreground and outputting to stdout is that this script is to be run from a continuous integration app (jenkins or bamboo) and I want to be able to see the output during execution from within the UI of the CI server.

    Interesting point about the tcl script. Could you expand on how you might incorporate that range expansion into the above script?

    Appreciate your thoughts on how I might be able to achieve parallel execution of sshpasscmd across multiple boxes while having interlaced output coming from the stdout of all running executions across all boxes (environmenthosts)?

    Thanks.
  6. #4
  7. No Profile Picture
    Contributing User
    Devshed Frequenter (2500 - 2999 posts)

    Join Date
    Mar 2006
    Posts
    2,632
    Rep Power
    1811
    I always had/have to sit and think about logging when lobbing commands across the network, so I feel your pain! I presume the output of the scppass is not a problem as it's actually being driven by the server running the actual base command (if that's not the case my memory is worse than I thought! ) So, to the second loop, the one that runs stuff on the target/remote server(s)!
    In theory (i.e., to the best of my knowledge!) you could wrap the whole thing in {} and deal with the output en masse that way?
    Code:
     for i in $environmenthosts
    do
       { ./sshpasscmd $userpassword $user@$i sudo /bin/bash -c \"rm -rf /tmp/puppet; mkdir -p /tmp/puppet; cd /tmp/puppet; tar -xvf /tmp/$releaseartifact; export PATH=/usr/bin:/usr/local/bin:/opt/vagrant_ruby/bin; $puppetfact puppet apply --modulepath=. \" } | tee YourLog.log
    done
    So far as the tcl bit goes, as 'all' you seem to be doing is 'un-listing' and 're-listing' the arguments you should be able to achieve the same thing with [lrange $argv 2 end] (You may need to wrap that in [string ...], so your spawn command would become:
    Code:
    spawn ssh -oStrictHostKeyChecking=no -oCheckHostIP=no $host [lrange $argv 2 end]
    The moon on the one hand, the dawn on the other:
    The moon is my sister, the dawn is my brother.
    The moon on my left and the dawn on my right.
    My brother, good morning: my sister, good night.
    -- Hilaire Belloc

IMN logo majestic logo threadwatch logo seochat tools logo